Scraping manager

Generator for real-estate listing scrapers. You hand a URL to the Hermes CLI agent (web browsing + Bright Data MCP); it writes a Python scraper into backend/scrapers/ itself. A React frontend drives the whole flow through a FastAPI backend, both served from a single container on port 8000.

1. New Scraper — paste a listing URL, generate a detail scraper, preview it against the live page, give feedback to regenerate as many times as needed, then lock it and repeat the same loop for a search-results URL collector.
2. Saved Portals — save the pair (detail scraper + URL collector) as a portal, download the generated .py files.
3. Runs — pick a portal, give it a search URL, launch a run that collects every listing URL and scrapes each one; progress is polled live. Runs can also be scheduled.
4. Logs — structured event log of generations, portal saves, and run outcomes.

Requirements

• Docker + Docker Compose
• A Bright Data account: an MCP API token + browser zone (for the agent) and a browser WebSocket endpoint (for the scrapers it writes)

Quick start

cp .env.example .env          # container environment + Hermes config secrets
cp .env.example backend/.env  # same values, read when scrapers are executed
# fill both in (see "Environment" below)

docker compose build
docker compose up

Open http://localhost:8000. The MinIO console is on http://localhost:9001 (minioadmin / minioadmin).

Two notes on the setup:

• Create backend/.env as a file before the first docker compose up. Compose bind-mounts it, so if it doesn't exist Docker silently creates a directory with that name and the scrapers run without their environment.
• There is no hot reload. This is a production-style single-container build (the frontend is compiled into static files at image build time), so any code change needs docker compose build again.

Useful commands:

docker compose exec app bash                 # shell inside the running container
docker compose run --rm app which hermes     # verify the Hermes CLI is on PATH

Environment

Both .env files use the same keys — copy .env.example and fill in:

Variable	Purpose
BRIGHTDATA_WS_ENDPOINT	Browser WebSocket endpoint the generated scrapers connect to. Generation works without it; running a scraper does not.
BRIGHTDATA_API_TOKEN	Bright Data MCP token used by the Hermes agent to browse.
BRIGHTDATA_MCP_BROWSER_ZONE	Bright Data browser zone name for that MCP server.
OLLAMA_API_KEY	Ollama Cloud key for the model Hermes runs on.
HERMES_MODEL, HERMES_PROVIDER	Optional per-invocation override, to benchmark another codegen model without editing config.yaml.
AI_FALLBACK_MODEL, AI_FALLBACK_MAX_CALLS_PER_RUN, AI_FALLBACK_COST_PER_1K_TOKENS	Runtime AI fallback (hermes -z) when a scraper misses a field, plus its per-run budget.
STORAGE_BACKEND, S3_*	local writes to backend/storage_local/, s3 to MinIO/AWS. docker-compose.yml already overrides these to point at the MinIO service.
AUTO_HEAL	true lets a drifting portal trigger an automatic healing regeneration at the end of a run (candidates still await human review).
OPENROUTER_API_KEY	Only needed if you keep the fallback_model block in config.yaml.

The root .env is loaded by Compose into the container environment (and is what fills the ${...} placeholders in config.yaml). backend/.env is mounted at /app/.env and read when the backend shells out to a scraper, so editing it on the host takes effect on the next run — no restart needed.

Never commit a real .env. If one ever was, revoke and re-issue the Bright Data token and the Ollama key.

Hermes agent config

config.yaml at the repo root is the Hermes configuration template. On every container start, docker-entrypoint.sh renders it through envsubst into /root/.hermes/config.yaml, substituting secrets from the environment. You do not need to run hermes setup or hermes config edit in the container — and an in-container hermes config set is overwritten on the next restart, so make changes here and rebuild.

What you're most likely to change:

• model.default / model.provider — which model plans and writes the code.
• model.context_length — set it to the model's useful ceiling, not its physical window. It's required: Hermes only auto-compacts when it knows the window, and without it long sessions fail with "context too large".
• compression.threshold — fraction of context_length at which a session is compacted. Lower is safer, higher keeps more raw history.
• agent.max_turns / agent.reasoning_effort — how long and how hard the agent works on one generation.
• mcp_servers.brightdata — the Bright Data MCP server; its values come from the env vars above.
• fallback_model — provider failover on rate limits or outages.

Any new ${VAR} you add to this file must also exist in the root .env, or envsubst will render it empty.

CLI usage (bypassing the UI)

docker compose run --rm app python step1_single_scraper_generation.py <url> [--generate-only]
docker compose run --rm app python step2_urls_scraper.py <search_url> [--generate-only]

Local backend development (no Docker, no Hermes)

The FastAPI app runs standalone for the parts that don't need Hermes (portals registry, run lifecycle, logs):

cd backend
pip install -r requirements.txt
uvicorn main:app --reload

Without a backend/static/ directory (only produced by the Docker frontend build), the API serves no static files — hit it directly at http://localhost:8000/api/.... For frontend iteration, run npm run dev in frontend/ (port 5173) against it; CORS for that origin is already enabled.

Project layout

.
├── Dockerfile                              # Builds the UI, then serves it with the API
├── docker-compose.yml                      # app (:8000) + MinIO (:9000 API, :9001 console)
├── docker-entrypoint.sh                    # Renders config.yaml into /root/.hermes/
├── config.yaml                             # Hermes agent config template
├── .env.example                            # Template for both .env files
├── backend/                                # FastAPI app
│   ├── main.py                             # Entrypoint: routers, CORS, startup hooks
│   ├── step1_single_scraper_generation.py  # CLI for step 1 (listing scraper)
│   ├── step2_urls_scraper.py               # CLI for step 2 (links scraper)
│   ├── requirements.txt
│   ├── app/
│   │   ├── api/                            # Thin HTTP layer over the packages below
│   │   │   ├── step1.py                    # Listing scraper: generate/preview/chat
│   │   │   ├── step2.py                    # Same for the links (search) scraper
│   │   │   ├── portals.py                  # Portal CRUD, heal, promote, rollback
│   │   │   ├── runs.py                     # Launch and monitor runs
│   │   │   ├── schedules.py                # Cron schedule CRUD + "run now"
│   │   │   ├── scrapers.py                 # Hand-edit a generated file, no LLM
│   │   │   ├── jobs.py                     # Re-attach to a running generation
│   │   │   ├── logs.py                     # Query the event log
│   │   │   └── schemas.py                  # Pydantic request/response models
│   │   ├── generation/                     # URL -> working scraper
│   │   │   ├── hermes_client.py            # CLI subprocess: stream, timeout, resume
│   │   │   ├── pipeline.py                 # generate -> validate -> critique -> retry
│   │   │   ├── prompt_templates.py         # Prompts sent to the agent
│   │   │   ├── harness.py                  # Multi-URL validation + critique
│   │   │   ├── code_checks.py              # AST lint before a human sees the code
│   │   │   ├── recon.py                    # Turns the page probe into prompt input
│   │   │   └── jobs.py                     # Runs the pipeline off the HTTP thread
│   │   ├── runtime/                        # Executing scrapers at scale
│   │   │   ├── runs_manager.py             # Worker pool, retries, rate limit, drift
│   │   │   ├── run_store.py                # SQLite run state (WAL), stdlib only
│   │   │   ├── script_runner.py            # Runs one scraper, parses its JSON
│   │   │   ├── scheduler.py                # Cron ticking for scheduled runs
│   │   │   ├── ai_fallback.py              # `hermes -z` for missing core fields
│   │   │   ├── ai_completion.py            # Applies that fallback to a finished run
│   │   │   ├── drift_detector.py           # Run fill-rate vs. portal baseline
│   │   │   └── healing_manager.py          # Regenerates a drifting scraper
│   │   ├── portals/                        # Saved portal registry
│   │   │   ├── portal_store.py             # Pairs, golden URLs, baseline, history
│   │   │   ├── scraper_origins.py          # Which URL produced each file on disk
│   │   │   ├── scraper_sessions.py         # Last Hermes session id per file
│   │   │   └── favicon.py                  # Portal logos for the UI cards
│   │   ├── core/
│   │   │   ├── config.py                   # Paths, .env loading, shared timeouts
│   │   │   ├── storage.py                  # Object storage: local dir or S3/MinIO
│   │   │   └── logs_store.py               # Append-only JSONL event log
│   │   ├── schema/
│   │   │   └── listing_schema.py           # Canonical schema + field validators
│   │   └── tools/                          # Standalone probes run as subprocesses
│   │       ├── page_recon.py               # Structured-data sources a page exposes
│   │       └── page_dump.py                # Page text as JSON, for the fallback
│   ├── tests/                              # pytest: schema, lint, harness, sessions
│   ├── scrapers/                           # Generated scraper files (exec cache)
│   ├── portals.json                        # Saved portals
│   ├── runs/runs.db                        # Run state
│   ├── logs/                               # Event log
│   └── storage_local/                      # Storage when STORAGE_BACKEND=local
└── frontend/                               # React + Vite + TypeScript (Figma export)
    └── src/
        ├── main.tsx                        # Vite entry
        ├── app/
        │   ├── App.tsx                     # Routing and shell
        │   ├── api/client.ts               # Single typed client for every API call
        │   ├── types.ts                    # Types mirroring the backend schemas
        │   ├── pages/                      # NewScraper, Portals, Runs, Edit, Logs
        │   ├── components/                 # Sidebar, controls, cards, progress
        │   │   └── ui/                     # Generated shadcn/ui primitives
        │   └── hooks/                      # useWakeLock, for long generations
        └── styles/                         # Tailwind + theme

What persists where

backend/scrapers/, backend/runs/, backend/portals.json and backend/logs/ are bind-mounted, so generated scrapers, saved portals, run history and logs survive on the host across rebuilds. Durable artifacts (run exports, page dumps, portal manifests, scraper copies) go to object storage under users/<user_id>/... — MinIO by default, or backend/storage_local/ with STORAGE_BACKEND=local.