Skip to content

Proxy Endpoints

Base path: /api/v1/multi + router prefix /proxy → full base: /api/v1/multi/proxy

Summary: Manage local proxy records and Webshare (third-party proxy provider) credentials; test proxies via Selenium, fetch proxies from Webshare, and manage active/disabled flags. These endpoints are used by Selenium-based scrapers and monitor flows to obtain working proxies.

Auth: Bearer JWT required for all endpoints.

Primary models: Proxy, ProxyCreate, ProxyUpdate, ProxyTestRequest, ProxyTestResponse, WebshareCredentials, WebshareCredentialsCreate, WebshareCredentialsUpdate (see schemas.proxy and models.proxy).

External services used: services.webshare.get_proxies, services.webshare.test_key, selenium-wire for runtime proxy testing.

Endpoints (summary)

  • POST / — Create a proxy (ProxyCreate) and persist to DB.
  • GET / — List proxies (pagination: skip, limit).
  • GET /{proxy_id} — Get proxy by ID.
  • PUT /{proxy_id} — Update proxy record (ProxyUpdate).
  • DELETE /clear — Remove all proxies from DB.
  • DELETE /{proxy_id} — Remove a single proxy.
  • POST /{proxy_id}/toggle-active/ — Toggle active flag on a proxy (returns new status).

Webshare credential management

  • POST /webshare/credentials/ — Create Webshare credential (stored encrypted in DB).
  • GET /webshare/credentials/ — List stored Webshare credentials.
  • GET /webshare/credentials/{id} — Get a specific credential.
  • PUT /webshare/credentials/{id} — Update a credential.
  • DELETE /webshare/credentials/{id} — Delete a credential.
  • POST /webshare/credentials/{id}/toggle-active — Toggle active on a credential (activating one deactivates others).
  • POST /webshare/test/ — Test a raw Webshare API key supplied in the request body.
  • POST /webshare/credentials/{id}/test — Test a stored Webshare credential (DB key is decrypted before testing).
  • POST /webshare/fetch/ — Fetch proxies from Webshare using the active credential and store any new proxies in DB.

Proxy runtime test

  • POST /test — Test an existing proxy via a headless Selenium browser and selenium-wire. The request body (ProxyTestRequest) includes proxy_id and url to visit. Returns ProxyTestResponse with success, message, detected proxy_ip and a short page_content_preview.

Implementation details & behavior

  • fetch_webshare_proxies:
  • Retrieves the active Webshare credential from DB, decrypts the api_key via core.security.decrypt_string, and calls services.webshare.get_proxies(api_key).
  • get_proxies paginates the Webshare API (/proxy/list/), parses each entry into Proxy schemas and returns a list.

  • The endpoint upserts any non-existing proxies into the proxy table using proxy_crud.get_by_host_port_user_pass to deduplicate.

  • test_webshare_credential / test_stored_webshare_credential:

  • Use services.webshare.test_key(api_key) which calls https://proxy.webshare.io/api/v2/profile/ to validate the token. Returns { "success": true } on 200.

  • test_proxy (Selenium-based):

  • Uses selenium-wire to create a local proxy endpoint that forwards traffic through the target proxy. The code:
    • Starts selenium-wire with seleniumwire_options (proxy config, addr='0.0.0.0', port=8087, auto_config=False).
    • Starts a remote Selenium WebDriver connected to http://selenium:<SELENIUM_PORT>/wd/hub (env SELENIUM_PORT, default 4444).
    • Visits https://api.ipify.org?format=json to detect the public IP seen by the browser and optionally visits the provided url and returns a page preview.
    • Updates proxy status in DB with proxy_crud.update_status(db, id=proxy.id, success=True|False).
  • Note: The application monkey-patches selenium.webdriver.Remote.__init__ to remove desired_capabilities for compatibility with selenium-wire and Selenium 4.10+.

Examples

  • Test a raw Webshare API key:
curl -X POST "http://localhost:8000/api/v1/multi/proxy/webshare/test/" \
    -H "Content-Type: application/json" \
    -d '{"api_key": "WEBSHARE_API_KEY"}'
  • Fetch proxies from Webshare (requires active Webshare credential stored in DB):
curl -X POST "http://localhost:8000/api/v1/multi/proxy/webshare/fetch/" \
    -H "Authorization: Bearer <TOKEN>"
  • Test a proxy via Selenium (proxy_id must exist):
curl -X POST "http://localhost:8000/api/v1/multi/proxy/test" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer <TOKEN>" \
    -d '{"proxy_id": 1, "url": "https://example.com"}'

Operational notes

  • Ensure a Selenium hub/container is reachable at hostname selenium and port set by SELENIUM_PORT (default 4444). The test flow expects to connect to http://selenium:<SELENIUM_PORT>/wd/hub.
  • selenium-wire requires binding a local port (default 8087). If running multiple concurrent proxy tests, ensure unique ports or serialize tests.
  • Stored Webshare api_key values are encrypted in the DB; webshare_crud handles encryption/decryption for create/update, but test endpoints explicitly decrypt via core.security.decrypt_string when reading.
  • Proxy tests perform real network requests and may reveal real IPs — avoid running tests with sensitive targets in production environments.
  • Endpoint implementation: backend/service_multi-repricer/app/api/v1/endpoints/proxy.py
  • Webshare service: backend/service_multi-repricer/app/services/webshare.py
  • Proxy CRUD: backend/service_multi-repricer/app/crud/proxy.py

Generated: detailed reference for proxy endpoints used by multi-repricer.