Skip to content

1.8.1

⚙️ Patch 1.8.1 — Qogita catalog webhook, resumable Keepa pipeline, push guards

🚀 Summary

  • Qogita catalog pipeline via Public API: register webhook → request full catalog CSV → import GTINs into existing wholesale_shopping_list (qogita_preis / qogita_bestand) — no separate GTIN index table.
  • Resumable Keepa pipeline (keepa_pipeline_tick cron): token-aware sync/fetch with progress persisted in cron_jobs.arguments.pipeline; frontend progress panel with ETA and stop.
  • Push to Qogita only for GTINs present in wholesale_shopping_list; leading-zero EAN normalization; blocks push when catalog is empty.
  • Unit tests for Keepa pricing, pipeline, mocked HTTP client, scanner, and catalog CSV parsing — no Keepa tokens required.
  • DB sync preset qogita for pulling wholesale_shopping_list from production.

✨ Improvements

Qogita catalog (webhook + wholesale_shopping_list)

  • qogita_webhook_settings singleton — webhook registration + latest download state (catalog_request_id, catalog_status, catalog_gtin_count, …).
  • Public webhook callback — POST /api/v1/omni/qogita/webhooks/incoming (no JWT; HMAC verify when signing secret stored).
  • Catalog API:
  • GET /qogita/catalog/status
  • POST /qogita/catalog/setup-webhook
  • POST /qogita/catalog/request-download
  • On catalog_download.completed: download CSV, parse GTIN, € Lowest Price inc. shipping, Lowest Priced Offer Inventory → upsert wholesale_shopping_list by EAN; clear qogita_preis / qogita_bestand for EANs no longer in catalog.
  • in_qogita_catalog on qogita_watchlist_items — synced after catalog import and during Analyze.
  • PUBLIC_API_BASE_URL env (default https://dashboard.lage-beauty.de/api/v1/omni) — HTTPS base for Qogita webhook URL.

Frontend — Qogita catalog panel

  • Qogita catalog (GTIN index) section: setup webhook → request catalog → poll status.
  • Filter chips: In Qogita catalog / Not in Qogita catalog.
  • Row chips for catalog membership; analyze/push messages include catalog stats.

Keepa — resumable pipeline

  • Pipeline state in cron_jobs.arguments.pipeline (no keepa_pipeline_jobs table).
  • Cron keepa_pipeline_tick every 30 minutes (*/30 * * * *); burns tokens until low, then waiting_tokens until bucket refills (~280 max, 5/min).
  • Mutex — only one pipeline job (sync_asins or fetch_products) at a time.
  • API: GET/POST /keepa/pipeline/status, start/sync-asins, start/fetch-products, stop.
  • Manual /sync-asins, /fetch-products, /scan return 409 while pipeline is active.
  • Frontend: progress bar (processed/remaining/ETA/runs/tokens), polling, Sync/Fetch disabled during active run.

Push & analyze guards

  • Analyze sets in_qogita_catalog when wholesale_shopping_list has Qogita data; returns in_catalog / not_in_catalog counts.
  • Push checks wholesale_shopping_list directly (is_gtin_in_catalog); skips unknown GTINs (skipped_not_in_catalog); errors if catalog never loaded.
  • EAN normalization — leading-zero tolerant lookup (0792971283512 ↔ 792971283512).
  • Candidates / awaiting_push filtered by catalog when wholesale data exists.

Rate limiting & logging

  • Keepa client — token bucket tracking, pre-request throttle, 429 retry, per-page/batch logs.
  • Qogita client — request spacing, 429 backoff, per-request logs.
  • Discord webhooks — rate-limited delivery; POST /price-alerts/cron/discord/test.

Tests (no Keepa tokens)

  • test_keepa_pricing.py — EK/margin, metrics, EAN keys.
  • test_keepa_pipeline.py — state serialization, ETA helpers, cron **kwargs.
  • test_keepa_client_mocked.py — mocked httpx (query, products, throttle, 429).
  • test_keepa_scanner_mocked.py — full workflow with mocked clients (@pytest.mark.db).
  • test_qogita_catalog_unit.py — CSV parse + wholesale upsert/stale-clear (@pytest.mark.db).
  • Existing test_keepa_workflow_integration.py remains opt-in (RUN_KEEPA_INTEGRATION=1).

DB sync

  • Preset qogita — syncs wholesale_shopping_list from production.
  • schema-patches.sql — ensures wholesale_shopping_list exists locally before restore.
.\scripts\db-sync\sync-from-prod.ps1 -Preset qogita

🐛 Bug Fixes

  • Keepa pipeline cron crash on startup — run_keepa_pipeline_tick(**_kwargs) accepts pipeline from cron_jobs.arguments (APScheduler was passing unsupported kwarg).
  • Push ignored catalog — previously only filtered when webhook catalog_status === completed; now checks wholesale_shopping_list whenever Qogita price/stock data exists.
  • 400 spam on push — GTINs not in Qogita assortment are skipped before API call instead of failing one-by-one.

📁 Files changed

Backend (service_omni)

  • app/api/v1/endpoints/qogita.py, qogita_webhooks.py, keepa.py
  • app/services/qogita/catalog.py, catalog_setup.py, ean_keys.py, qogita_client.py
  • app/services/keepa/scanner.py, pipeline_runner.py, pipeline_state.py, constants.py, keepa_client.py
  • app/crud/qogita_catalog.py, qogita.py, keepa_pipeline.py
  • app/schemas/qogita_catalog.py, wholesale_shopping_list.py, qogita.py
  • app/models/qogita.py, keepa.py
  • app/core/config.py, app/main.py, app/api/v1/api.py
  • app/tests/ — fixtures_keepa.py, test_keepa_*.py, test_qogita_catalog_unit.py, conftest.py
  • pytest.ini

Frontend

  • src/pages/Statistics/Tabs/QogitaWatchlistTab.jsx, qogitaWatchlistPrefs.js
  • src/services/api/omni/qogita.js, keepa.js

Postgres

  • docker/postgres/migrations/add_qogita_catalog.sql
  • docker/postgres/migrations/add_keepa_pipeline_job.sql

Scripts

  • scripts/db-sync/tables.json, sync-from-prod.ps1, sync-from-prod.sh, schema-patches.sql

🔧 Database & SQL

Run on server before deploy (if not already applied):

\i docker/postgres/migrations/add_qogita_catalog.sql
\i docker/postgres/migrations/add_keepa_pipeline_job.sql

If keepa_pipeline_tick already exists with * * * * *:

UPDATE cron_jobs SET expression = '*/30 * * * *' WHERE name = 'keepa_pipeline_tick';

If legacy catalog tables were created by an earlier draft:

DROP TABLE IF EXISTS qogita_catalog_gtins;
DROP TABLE IF EXISTS qogita_catalog_snapshots;

Set PUBLIC_API_BASE_URL on service-omni to your public HTTPS API base (Qogita rejects loopback/private webhook URLs).

Restart service-omni. Rebuild frontend.

Usage notes

Qogita catalog (first time)

  1. Run migrations above.
  2. Set PUBLIC_API_BASE_URL (e.g. https://dashboard.lage-beauty.de/api/v1/omni).
  3. In Price Alerts → Qogita Watchlist: Setup webhook → Request catalog (CSV arrives via webhook; may take several minutes).
  4. Analyze watchlist rows → Push to Qogita (only in-catalog GTINs).

Local catalog data from prod

.\scripts\db-sync\sync-from-prod.ps1 -Preset qogita

Keepa pipeline (long runs)

  • Use Sync ASINs or Fetch Products in the UI — starts background pipeline; cron resumes after token wait.
  • GET /keepa/pipeline/status for progress; Stop to cancel.

Tests (no tokens)

cd backend/service_omni
pytest app/tests/test_keepa_pricing.py app/tests/test_keepa_pipeline.py app/tests/test_keepa_client_mocked.py -v

With local Postgres for DB-marked tests:

pytest app/tests/test_keepa_scanner_mocked.py app/tests/test_qogita_catalog_unit.py -v -m db