Skip to content

1.11.22

⚙️ Patch 1.11.22 — Parfümerie Godel catalog scrape, GTIN enrich, Price Alerts tab

🚀 Summary

Adds a Parfümerie Godel (Shopware 6) catalog scraper: paginate category listings, enrich each product from the PDP for GTIN, upsert into godel_products, optional wholesale EAN mirror, CSV export, and a cron. Active HTTP proxies are loaded automatically from the shared proxies table. UI lives under Price Alerts → Godel. Cron status now respects is_active after stop (same fix on Nexalia).

✨ Details

Area Behavior
Listing /{category}?order=price-desc&p={n} until empty / duplicate IDs / declared max page
GTIN PDP microdata (itemprop=gtin13 + fallbacks); concurrent detail fetches
Proxies Active rows from shared proxies table (rotated; retry on 403/429) — no frontend/env proxy config
Storage godel_credentials + godel_products; optional wholesale_shopping_list.godel_preis / godel_bestand
Defaults neuheiten, parfum, pflege, make-up, premium, herren, x-mas/adventskalender
API Config CRUD/connectivity test, POST /catalog/scrape, GET /products, GET /products/export.csv, cron start/stop/status
Frontend Price Alerts tab: categories, concurrency, cron builder, Import Products, product table, CSV download
Cron UI /cron/status returns active from DB is_active (stop no longer looks stuck Active); same for Nexalia

📌 Notes

  • Redeploy service-omni and frontend. Apply docker/postgres/updates/add_godel_tables.sql (or restart omni for auto-create / wholesale column patch).
  • Full catalog scrape is PDP-heavy (~8k products); first verified run: 7,892 upserted, 7,872 with GTIN (~99.7%), ~48 min with 100 DB proxies.
  • Stock on listings is limited; in-stock qty comes from config in_stock_qty when the PDP looks sellable.
  • /godel redirects to /price-alerts.

✅ Verify

  1. Price Alerts → Godel → save config + Test connectivity (logs show proxies loaded from DB).
  2. Import Products → table fills with name/url/sku/gtin/price/currency.
  3. Export CSV downloads those columns (+ brand/category).
  4. Start cron → status Active with next run; Stop cron → Inactive / paused (not stuck Active).
  5. Tests: pytest backend/service_omni/app/tests/test_godel_client.py

📁 Files

  • backend/service_omni/app/services/godel/ — client + scraper
  • backend/service_omni/app/api/v1/endpoints/godel.py — API + CSV + cron
  • backend/service_omni/app/crud/godel.py, schemas/godel.py
  • backend/service_omni/app/crud/proxy.py, schemas/proxy.py — shared DB proxies
  • backend/service_omni/app/core/schema_patches.py, main.py, api/v1/api.py
  • backend/service_omni/app/services/qogita/wholesale_offers.py — Godel supplier
  • docker/postgres/tables/godel_*.sql, updates/add_godel_tables.sql
  • frontend/src/pages/PriceAlerts/Tabs/GodelTab.jsx
  • frontend/src/services/api/omni/godel.js
  • docs/docs/services/service_omni/services/godel.md
  • docs/docs/patchnotes/1.11/1.11.22.md — this file