Skip to content

1.8.12

⚙️ Patch 1.8.12 — Rituals invoice PDF download & Otto offers-frame scraping

🚀 Summary

  • Rituals invoices (EUW / DACH) — reliable PDF download from Gmail: correct click.c.rituals.com link (invoice aria-label), HTML parsing, DACH two-step QR flow, and a public test endpoint.
  • Otto site prices — scrape via /dcs/offers-page-frame/ instead of blocked moin PDPs; resolve variationId from moin HEAD redirect; sync stores offers-frame URLs.
  • n8n invoice storage — shared invoices volume on n8n + runners (/home/node/.n8n-files/invoices), same data as service-invoice and file-manager.

✨ Improvements

Invoice — Rituals special case

Before After
Any click.c.rituals.com near “ZUR RECHNUNG” Only anchors with aria-label … Rechnung im PDF-Format …
Plain-text email body (no aria-label) HTML body for link extraction
EUW search used full EUW234… or wrong link → promotions page Strip EUW for Gmail search; PDF button link in shipping/invoice mail
DACH single-email assumption Shipping mail → extract OR237018980 from QR → re-search 237018980 → invoice mail
Corrupt “PDF” (HTML promotions saved as file) Reject non-invoice redirects; verify %PDF magic bytes
No local test hook POST /api/v1/invoice/gmail/test/rituals/{invoice_number} (public, step log)

Verified: EUW234505767, DACH06758065 — valid PDFs downloaded in Docker.

Multi-repricer — Otto scraping

Before After
Scrape https://www.otto.de/p/?moin=… → HTTP 429 from server IP HEAD moin → canonical URL + variationId; offers-frame GET → 200
product_url built from moin or empty productDescription.productUrl Sync resolves /dcs/offers-page-frame/{item}?variationId=…
marketplace-status shop link only GET /api/v1/multi/otto/products/{sku}/shop-context for debugging
Offers-frame 404 / empty / blocked → failed scrape Solo buybox → price toward max
Full PDP scrape as fallback Offers page only — no moin PDP HTML load
Seller / buybox unknown after offers scrape → blind undercut Parse buybox seller from .of_offers-row → span.of_offers-seller__retailer-name; is_in_buybox when seller is eCommerce GmbH

Offers-page logic

  1. Resolve offers_frame_url via marketplace-status + moin HEAD redirect.
  2. Scrape /dcs/offers-page-frame/{item}?variationId=… only.
  3. Each .js_of_offers-row.of_offers-row → price + seller; first row = buybox.
  4. No offers page (404 / no rows / unreachable) → assume solo buybox → max price.
  5. Competing offers but buybox seller unknown → skip price change (no blind undercut).

Note: productDescription.productUrl in Otto Partner API is the seller’s own URL (usually empty), not otto.de.

🔧 Deploy

  1. Rebuild / restart service-invoice and service-multi-repricer.
  2. Rebuild docs if you publish patchnotes from the repo.
  3. No database migration.
  4. Rituals: re-run Otto/Rituals sync or wait for next Gmail download batch to pick up new logic.
  5. Otto: optional re-sync articles to refresh stored product_url to offers-frame URLs.

✅ Verify

  1. Rituals EUW:
    POST /api/v1/invoice/gmail/test/rituals/EUW234505767 → success: true, extracted_link contains invoice tracking URL, download_link file opens as PDF.

  2. Rituals DACH:
    POST /api/v1/invoice/gmail/test/rituals/DACH06758065 → steps show QR 237018980 and second email; PDF valid.

  3. Otto shop context:
    GET /api/v1/multi/otto/products/{sku}/shop-context (auth) → offers_frame_url populated.

  4. Otto scrape logs: offers-frame HTTP OK with buybox_seller=…, in_buybox=True/False (not moin PDP 429).

⚠️ Notes

  • Rituals test endpoint is public (PUBLIC_ENDPOINT_PREFIXES in service-invoice main.py) — use only on trusted networks or remove prefix after testing.
  • Otto full PDP may still 429 from datacenter IPs; offers-frame is the supported server-side path.
  • Playwright + proxy still used for Rituals click.c.rituals.com redirects when direct download fails.

Files touched

service-invoice

  • app/services/gmail.py — Rituals EUW/DACH flow, HTML extraction, aria-label link, PDF validation, test_rituals_invoice_flow
  • app/api/v1/endpoints/gmail.py — POST /gmail/test/rituals/{invoice_number}
  • app/main.py — public prefix for Rituals test route

service-multi-repricer

  • app/services/otto/otto_service.py — offers-frame-only scrape, seller/buybox parsing, solo buybox fallback
  • app/api/v1/endpoints/fbm_articles.py — FBM error flush on sync/recalc/export

infrastructure

  • docker-compose.yml, docker-compose-portainer.yml — shared invoices volume mounted on n8n + runners for invoice downloads

docs

  • docs/docs/patchnotes/1.8/1.8.12.md — this file