Skip to content

1.8.9

⚙️ Patch 1.8.9 — Otto scrape resilience, CDP on Hetzner & db-sync proxies

🚀 Summary

  • Otto site scraping — HTTP 429 / blocked responses detected early; 5 retries with shuffled proxy rotation, exponential backoff, and clear proxy logging.
  • Hetzner / Docker — optional Chrome CDP on the host so the repricer container uses real Chrome instead of headless Patchright (fixes persistent 429 on server).
  • Otto price monitor — rate-limit scrape failures no longer set status to ERROR.
  • db-sync — repricer preset includes proxies table.

✨ Improvements

Otto — scrape retry, proxies & logging

Before After
3 retries, ~1s, same session 5 retries, backoff, new proxy per attempt
No proxy visibility in logs Logs via proxy host:port (user=…) or warns when pool empty
429 page parsed as “no price” OttoScrapeError on blocked HTTP codes immediately
google_search=False (no referer) Google referer, de-DE locale, rotated User-Agent
HTTP + browser double-hit in Docker HTTP skipped in Docker by default (OTTO_SCRAPE_HTTP_FIRST)

Blocked codes retried: 401, 403, 407, 429, 444, 500, 502, 503, 504.

Proxies loaded from DB at Otto_Service init; monitor passes active pool explicitly.

Otto — Chrome CDP (Hetzner / Portainer)

Headless Patchright inside Docker is often blocked by Otto even with datacenter proxies. Option B: real Google Chrome on the host with remote debugging; repricer connects via CDP.

Host (one-time, SSH):

cd /tmp/otto-chrome   # or repo: hetzner-deployment/otto-chrome
sudo bash install.sh
curl -s http://127.0.0.1:9222/json/version   # expect Chrome JSON

Container env (compose or Portainer UI):

OTTO_SCRAPE_CDP_URL=ws://172.17.0.1:9222
OTTO_SCRAPE_HTTP_FIRST=false

Compose also needs:

extra_hosts:
  - "host.docker.internal:host-gateway"
shm_size: "2gb"

Docs: hetzner-deployment/otto-chrome/README.md (includes Portainer notes when no git checkout on server).

Test inside container:

python scripts/test_otto_cdp.py

Otto price monitor — rate limit ≠ ERROR

If validation_errors mentions rate limit / HTTP 429 → status unchanged, error_message set; other scrape failures → ERROR.

db-sync — repricer preset

scripts/db-sync/tables.json — repricer preset now includes proxies.

cd scripts/db-sync
.\sync-from-prod.ps1 -Preset repricer

🔧 Environment variables

Variable Default (Docker) Purpose
OTTO_SCRAPE_CDP_URL unset / compose: ws://host.docker.internal:9222 Connect to host Chrome CDP
OTTO_SCRAPE_HTTP_FIRST false in Docker Skip curl attempt before browser
OTTO_SCRAPE_REAL_CHROME false Use installed Chrome (non-CDP)
OTTO_SCRAPE_HEADLESS true Browser headless mode

📁 Files changed

Multi-repricer

  • app/services/otto/otto_service.py — OttoScrapeError, proxy pool, CDP env, StealthyFetcher, HTTP-first toggle
  • app/api/v1/endpoints/otto_articles.py — proxy logging; rate-limit ≠ ERROR
  • scripts/test_otto_cdp.py — CDP smoke test
  • docker/service_multi-repricer/Dockerfile — copy scripts/

Deploy / Hetzner

  • hetzner-deployment/otto-chrome/ — install.sh, systemd unit, README
  • hetzner-deployment/docker-compose.https.yml — CDP env, extra_hosts, shm_size
  • docker-compose-portainer.yml — same for Portainer stacks
  • docker-compose.yml — shm_size, extra_hosts (local)

Scripts

  • scripts/db-sync/tables.json — proxies in repricer preset

🔧 Deploy

1. Hetzner host (SSH, once)

# install Chrome CDP — see hetzner-deployment/otto-chrome/README.md
sudo bash install.sh
ufw allow from 172.17.0.0/16 to any port 9222 proto tcp   # if using ufw

2. Portainer / stack

  • Rebuild service-multi-repricer image (Dockerfile + app code changed).
  • Add env + extra_hosts on service-multi-repricer (see above).
  • Recreate container.

3. Local dev

  • Restart service-multi-repricer; optional db-sync repricer preset for proxies.

No database migration.

✅ Verify

CDP (server)

  1. Host: curl -s http://127.0.0.1:9222/json/version
  2. Container: python scripts/test_otto_cdp.py → status 200 + price
  3. Monitor logs: Otto scraping via CDP browser at http://host.docker.internal:9222

Scrape (any env)

  1. Logs show scrape attempt N/5 via proxy …
  2. Persistent 429: SKU status not flipped to ERROR (rate-limit path)

db-sync

  1. .\sync-from-prod.ps1 -Preset repricer -DryRun — includes proxies

⚠️ If still HTTP 429 with CDP

CDP is working but Otto may block datacenter Webshare IPs — switch to residential proxies while keeping CDP enabled.