Skip to content

1.8.2

⚙️ Patch 1.8.2 — Keepa token wait fix, Qogita webhook 401, catalog pending timeout

🚀 Summary

  • Keepa pipeline no longer stalls in waiting_tokens when the token bucket is negative — Sync ASINs and Fetch Products wait only for the next step’s token cost and refresh tokens_left from Keepa after resume.
  • Qogita catalog webhook POST /qogita/webhooks/incoming accepts signed payloads again — request body is preserved for HMAC verification; signature parsing supports common v1= / hex / base64 formats.
  • Qogita catalog download no longer stays pending forever if the webhook never arrives — auto-fails after 30 minutes; UI allows retry after ~5 minutes (Qogita does not resend missed webhooks).

🐛 Bug Fixes

Keepa pipeline — stuck at waiting_tokens

Symptom: Pipeline frozen (e.g. Sync ASINs at 2800/28999), tokens_left=-8, logs every 30 min:

resuming after token wait → immediately waiting for tokens (left=-8, resume in 3456s) with no API calls and no progress.

Cause:

  1. Wait time targeted a full 280-token bucket (~58 min) instead of tokens needed for the next page (1) or batch (100).
  2. On resume, stale tokens_left from DB was reused without calling Keepa — pre-check re-scheduled wait every cron tick.

Fix:

  • _seconds_until_tokens_available(tokens_left, needed) — wait only until needed tokens exist (1 for sync, fetch_batch_size for fetch).
  • On resume after wait_until, set tokens_left = None so the next tick calls Keepa and reads fresh tokensLeft.
  • Logs now include need= e.g. waiting for tokens (left=-8, need=1, resume in 108s).

Unblock a stuck job after deploy (optional):

UPDATE cron_jobs
SET arguments = jsonb_set(arguments, '{pipeline,tokens_left}', 'null'::jsonb)
WHERE name = 'keepa_pipeline_tick';

Restart service-omni — next keepa_pipeline_tick should process pages again.

Qogita webhook — 401 Invalid signature

Symptom: POST /api/v1/omni/qogita/webhooks/incoming returned 401 even with correct Qogita signing secret; catalog download never imported; UI stuck on Latest download: pending.

Cause: log_requests middleware called await request.body() before the handler — body consumed, HMAC verified against empty bytes. Qogita does not retry failed webhooks.

Fix:

  • Re-inject cached body via request._receive in log_requests so downstream handlers see raw bytes.
  • Webhook path on PUBLIC_ENDPOINTS (no JWT).
  • verify_webhook_signature — Qogita uses X-Qogita-Signature: t={unix},v={hmac} over {timestamp}.{webhook_url}.{raw_body} (not body alone); falls back to raw-body HMAC for older formats.
  • Setup webhook deletes any existing endpoint at the callback URL and re-registers to obtain a fresh signingSecret (secret is only returned once on create).
  • Clearer failure logging in qogita_webhooks.py (body_len, signature presence).

After deploy: restart service-omni, then Request catalog again (or Retry catalog request in the UI).

Qogita catalog — pending never cleared

Symptom: Waiting for Qogita webhook… indefinitely after a failed/missed callback; Request catalog button disabled.

Fix:

  • expire_stale_pending_catalog() — on GET /catalog/status (and before a new request), mark pending as failed after 30 minutes with a clear error message.
  • API adds pending_timeout_minutes and pending_expires_at on catalog status.
  • Frontend: shows auto-fail time, failed error text, and enables Retry catalog request after ~5 minutes when webhook is overdue.

Reset a stuck pending download immediately (optional):

UPDATE qogita_webhook_settings
SET catalog_status = 'failed',
    catalog_error_message = 'Webhook not received — reset manually',
    catalog_completed_at = NOW()
WHERE catalog_status = 'pending';

Env override: QOGITA_CATALOG_PENDING_TIMEOUT_MINUTES (default 30).

✨ Tests

  • test_keepa_pipeline.py — fetch batch wait math (-8 + batch 100 → 1296s, not 3456s); resume after token wait calls fetch (no stale re-wait).
  • test_qogita_catalog_unit.py — webhook signature (v1=, invalid body); pending timeout expiry (@pytest.mark.db).
cd backend/service_omni
pytest app/tests/test_keepa_pipeline.py app/tests/test_qogita_catalog_unit.py -v

📁 Files changed

Backend (service_omni)

  • app/services/keepa/pipeline_runner.py
  • app/main.py
  • app/services/qogita/catalog.py, catalog_setup.py, constants.py
  • app/crud/qogita_catalog.py
  • app/api/v1/endpoints/qogita_webhooks.py
  • app/models/qogita.py
  • app/tests/test_keepa_pipeline.py, test_qogita_catalog_unit.py

Frontend

  • src/pages/Statistics/Tabs/QogitaWatchlistTab.jsx

🔧 Deploy

No new migrations. Restart service-omni after deploy. Rebuild frontend for catalog pending/retry UI.

If a Keepa pipeline job was stuck overnight, clear stale tokens_left (SQL above) or wait for the next cron tick after restart.

Verify Qogita catalog

  1. Setup webhook → Request catalog.
  2. CSV usually arrives in 2–3 minutes via webhook; status becomes completed and wholesale_shopping_list gains rows.
  3. If still pending after ~5 min, use Retry catalog request (only after 1.8.2 is deployed).
  4. Check logs for Qogita webhook received: type=catalog_download.completed.

Verify Keepa pipeline: Start Sync ASINs or Fetch Products → when tokens run low, wait ETA should be minutes (per step), not ~58 min loop; processed count should increase after each resume.