avis-en-cours

Query a public OpenDataSoft (ODS) procurement portal for currently-open SERVICES-type avis relevant to a mobility-planning consultancy, and classify each one as relevant, travaux (works), or excluded (out-of-domain).

Built with Bun + TypeScript. Hits the portal's REST API directly — no browser, no scraping. Optional semantic classification through OpenRouter (any model, with a fallback chain and price-sorted providers).

Two ways to use it:

• Local CLI (bun run src/index.ts) — writes timestamped markdown reports, as before.
• Deployed service (Clever Cloud) — daily cron run stored in PostgreSQL, a French web dashboard with passwordless email-code login (src/server.ts + src/auth.ts), and a daily email digest via MailPace (src/run.ts). See Deployment.

What it does

1. Builds a single ODS v2.1 where= clause combining a French keyword OR (mobilité OR vélo OR stationnement OR "schéma directeur" …) with a date filter (datelimitereponse >= today or datefindiffusion >= today when no deadline is set).
2. Pages through every matching record (limit 100/req, sorted by deadline ASC), extracts structured fields (id, objet, acheteur, département, date limite, type d'avis, procédure) and caches the lot to .cache/scrape.json.
3. Reads a second source, the achatpublic.com buyer-profile platform (where many sub-threshold MAPA studies are published without ever reaching BOAMP): every open SERVICES and FOURNITURES consultation is listed (~120 form POSTs, 10 cards each, ~2–3 minutes) and matched locally against per-family keyword lists (seeded from src/defaults.ts, mirroring the consultancy's manual watch procedure, then edited on /configuration): mobilité (services, classified like BOAMP avis), kiomda (fournitures — counters/sensors for bike and pedestrian counting) and inondations (services — AMC / PAPI / GEMAPI flood-prevention studies). The two non-mobility families are classified by their own rule in src/familles.ts, never by the LLM. Only retained consultations have their public detail page fetched (description, CPV, opening date). The source is non-blocking: if the site is down, the run keeps the BOAMP results and records a warning. ACHATPUBLIC=0 disables it.
4. Reads a third source, the AFD notices on dgMarket (afd.dgmarket.com, category « Transports », open notices only): list, category filter and full notice text are public (only document downloads need a subscription); the first request goes through an anonymous auto-login redirect chain that sets a session cookie, handled by a small per-domain cookie jar in src/afd.ts. Notices are abroad and often in English; they go through the normal mobility classifiers. About 10 requests per run. AFD=0 disables it.
5. Reads the atexo MPE platforms listed in MPE_SITES (src/mpe.ts): Maximilien (marches.maximilien.fr, Île-de-France, ~450 open consultations) and AMPA (demat-ampa.fr, Marchés publics d'Aquitaine, ~300). Same software, same HTML: the full list of open consultations (20 per page, each next page a PRADO postback replaying the page state with the session cookie) is filtered locally on SERVICES + the client's keyword list (seeded from MPE_SEARCHES in src/defaults.ts, edited on /configuration; families mobilité and inondations); detail pages (public) are fetched only for retained consultations (CPV, type d'annonce). MAXIMILIEN=0 / AMPA=0 disable a site; adding another MPE site (marches-publics.gouv.fr is one) is one entry in MPE_SITES.
6. Reads Marchés Online (marchesonline.com, Le Moniteur's aggregator, "en accès libre"): no account for what we use — the public keyword pages (/appels-offres/top-recherches/<mot>, 20 per page) are walked for every keyword of the client's five search profiles (seeded from MARCHESONLINE_SEARCHES in src/defaults.ts, edited on /configuration: mobilité, kiomda counters, inondations; ~250 pages per run) and filtered locally on the card's activity label (services / études vs fournitures) and on the keywords at start of word (the site's search is lexical). Public detail pages are read only for retained notices the title alone does not already exclude (descriptif, CPV). The site rate-limits (Cloudflare), so requests are paced and wide keywords are capped to their newest pages (~3–4 minutes per run once known notices are reused; ~15 minutes the first time). MARCHESONLINE=0 disables it.
7. Classifies each avis into relevant / travaux / excluded. Writes two timestamped markdown reports per run (avis-en-cours-YYYY-MM-DD-HH-MM-SS.md and the travaux equivalent). Excluded avis are logged to stderr with their reason.

Deployed, the same pipeline runs daily into PostgreSQL behind a French dashboard (passwordless email-code login). Each avis has a detail page with a live comment thread and a collaborative status (« à évaluer », « répondu », « gagné »… — list editable by admins on /admin, every change kept as history in the thread). The dashboard's primary click opens the internal detail page (the official notice stays one click away), lets you change an avis' status from the list, filters by text, status, source (with counts) and family, sorts by deadline or recency, marks what is new since your previous visit, applies filters as you change them, and can hide rejected avis; the same appel d'offres published on several platforms (or republished / rectified on BOAMP) is grouped into one row listing every publication, with one status and one comment thread; /configuration lists the sources (what each reads, active/disabled/failed, current avis per source) and, while an update runs, shows its milestones (one step per source, then classification, then the email) with counters and a progress bar, updated in place through the Skip engine (the job writes runs.progress, the adapter watches runs, the page refetches a server-rendered fragment on each event); the dashboard banner carries a one-line summary the same way. /configuration (admins, or users delegated the right from /admin) edits the pipeline live: search keywords (one list per search engine: BOAMP query, achatpublic, the atexo MPE sites, Marchés Online, with a family per term), « toujours garder / toujours exclure » rules (each one shows how many current avis it touches, and a candidate rule can be tried on the current avis before being added; keywords and rules can be paused instead of deleted and carry an optional note; a rule can be limited to one source, to the title only, or to the whole word), classifier mode, LLM model chain (with live OpenRouter prices), digest window, départements, a switch per secondary source (the server-side <SOURCE>=0 variables still win), the reading limits (BOAMP pages, Marchés Online pages per keyword) and the OpenRouter price cap, plus a « Relancer maintenant » button; every change is listed in a « Journal des modifications » card with its author. Threads update in real time through the Skip reactive engine running in-process (WASM) on Postgres LISTEN/NOTIFY; set LIVE_COMMENTS=0 to disable it (threads then show as unavailable). /sante reports live (engine up) and livePg (its Postgres connection + number of watched tables).

Install

bun install
cp .env.example .env       # then edit — see below

That's it — no browser binaries needed.

Environment

Everything goes in .env (auto-loaded by Bun):

Variable	Required	Default	Notes
PORTAL_API_URL	yes	—	ODS v2.1 datasets base URL, no trailing slash. e.g. https://<portal>/api/explore/v2.1/catalog/datasets.
PORTAL_DATASET	yes	—	Dataset id. Run will fail at startup if either is unset.
OPENROUTER_API_KEY	no	—	Enables semantic (LLM) and hybrid classifier modes.
LLM_MODELS	no	mistralai/mistral-nemo,… (see src/defaults.ts)	Comma-separated OpenRouter model ids, fallback order (max 5). Overridden by the /configuration setting.
LLM_MAX_PRICE_PER_M	no	1	Price cap, USD per million tokens, applied to every model of the chain. Overridden by the /configuration setting.
OPENROUTER_BASE_URL	no	https://openrouter.ai/api/v1	Override for tests / a proxy.
CLASSIFIER	no	hybrid if key set, else regex	Override: regex | llm | hybrid.
MAX_PAGES	no	100	Pagination cap (100 items/page). Stops earlier when results are exhausted. Overridden by the /configuration setting (server only).
USE_CACHE	no	unset	1 skips the API and re-runs classification on .cache/scrape.json.

Run

# fresh fetch + classify + write reports
bun run src/index.ts

# pass a custom OR-query (overrides the default keyword list)
bun run src/index.ts "mobilité OR cyclable OR stationnement"

# re-classify using the cached scrape (fast, ~seconds)
USE_CACHE=1 bun run src/index.ts

# force a specific classifier
CLASSIFIER=regex USE_CACHE=1 bun run src/index.ts

All sources are read in parallel, so a run lasts as long as its slowest source (Marchés Online, a few minutes). A full run hits the portal API ~30 times (the first page alone, then four pages at a time with retry; ~3 000 records over a 14-keyword query), re-reads a notice's detail page only when its card changed (URL/version, title or deadline — the rest is reused from the previous run), and only asks the LLM about avis it has not already classified with the same text, prompt and model chain (the verdict is stored with a content key in announcements.llm_key; LLM_MEMO=0 forces a full reclassification). The first run sends ~2 000 avis to the LLM, about 2.4 M prompt tokens — roughly 5 cents of OpenRouter credit with the default chain (mistral-nemo, 39/40 on bun run eval and the cheapest JSON-capable model in the catalog; the previous llama-3.3-70b head cost ~27 cents per run) when the hybrid classifier is on (the exact cost, token counts and which models answered are recorded on each run and shown on /configuration and in the digest footer). Each call carries the whole model chain: OpenRouter switches to the next model on outage, rate limit, moderation or context overflow, and the client retries timeouts / 5xx and re-asks the rest of the chain once when a model returns unusable JSON. A dead key, an empty credit or five consecutive chain failures open a circuit breaker: the rest of the run is classified by regex, the run is flagged with a warning, ALERT_RECIPIENT gets an email, and the digest is held until the next healthy run.

Checks

bun run typecheck && bun run lint && bun test && bun run build   # unit tests only
TEST_DATABASE_URL=postgresql://<user>@localhost:5432/avis_test bun test   # + integration tests — what CI runs

Unit tests are pure logic (no DB, no network). With TEST_DATABASE_URL pointing at a throwaway local Postgres (createdb avis_test) the three integration files also run, and CI runs them against a postgres:16 service container before every deploy:

• db.integration — the real schema: the jsonb shape stored for run statistics and its migration, the run advisory lock seen from a second connection, orphaned « running » rows closed at start, dashboard filters by source and family, counts per source.
• server.integration — spawns the real bun src/server.ts on a test port, logs in with the dev login code printed on stderr, and checks /sante, the dashboard (badges, ?source=, ?famille=), /configuration (sources card, LLM statistics, an orphaned run being closed) and a notice page.
• live.integration — boots the Skip engine, subscribes to a thread over SSE, writes comments and status changes, kills the engine's Postgres connection with pg_terminate_backend and checks the thread keeps updating. Run it before touching the Skip packages, their patch, or src/live.ts.

Classifier modes

• regex — deterministic rules in src/classify.ts. Free, instant, reproducible. Good at hard rules (télécom, contrôle de travaux, assurances…). Bad at nuance.
• llm — one OpenRouter call per avis with a few-shot French prompt (src/classify-llm.ts, prompt tuned on mistral-small). Good at semantic distinctions (e.g. "restructuration des services" vs. physical restructuring). Cheap but adds some non-determinism. Occasionally misses hard rules.
• hybrid (default when OPENROUTER_API_KEY is set) — regex first. If the regex confidently labels an avis excluded or travaux, that stands. Otherwise the LLM makes the final call. Best of both: hard rules are enforced deterministically, borderline cases get semantic judgement. Only the items that pass regex go to the LLM.

Customising the scope

The scope is hard-coded for mobility-planning consultancy work. Three places to edit:

• Keyword list — src/defaults.ts: the KEYWORDS array gets OR'd into a single where= clause.
• Regex rules — src/classify.ts: hard exclusions (télécom, contrôle de travaux, operator procurement, etc.) and travaux patterns (MOE, rénovation, fouilles, reconnaissance géotechnique).
• LLM prompt — src/classify-llm.ts: the system prompt describing in-scope vs. out-of-scope work, plus a few-shot of labelled examples.

All three files are intentionally small and self-contained. Fork and tweak.

File layout

src/
  defaults.ts         KEYWORDS list + default search params (type_marche=SERVICES, deadline>=today)
  params.ts           SearchParams type + ODS v2.1 URL builder (where= with search() + date filter)
  scraper.ts          Paginated fetch loop + ODS record → Announcement mapping (BOAMP)
  achatpublic.ts      Second source: achatpublic.com form scraper (session cookie, 0-based pages, card + fiche parsers)
  afd.ts              Third source: AFD notices on dgMarket, « Transports » category (cookie jar, redirect chain, list + notice parsers)
  progress.ts         Run progress: ProgressTracker (steps, throttled writes), summary/percent helpers
  sources.ts          Registry of sources (label, URL, description, env kill switch) for the pipeline, dashboard filter and configuration page
  mpe.ts              atexo MPE platforms (Maximilien, AMPA…): PRADO postback pagination, row + fiche parsers, MPE_SITES registry
  marchesonline.ts    Marchés Online (Le Moniteur): public keyword pages walked per keyword, card + fiche parsers
  mois.ts             Month name (FR/EN, short/long) → « 01 »…« 12 », shared by the sources with dates in words
  familles.ts         Watch families (mobilité / kiomda / inondations): keyword matcher + per-family classification rule
  departements.ts     Département name → INSEE code (achatpublic only exposes the label)
  classify.ts         Regex classifier (normalise, hard exclusions, travaux patterns)
  classify-llm.ts     LLM classifier (system prompt + few-shot, per-run context: chain, stats, breaker)
  llm.ts              OpenRouter client: fallback chain, price-sorted providers, retries, stats, circuit breaker
  http.ts             postWithRetry shared by llm.ts and email.ts (retries statuses, timeouts and network errors)
  openrouter-catalog.ts  Public model catalog (prices, JSON support) + key credit, for /configuration
  eval-llm.ts         `bun run eval`: labelled set × candidate models → accuracy, cost, latency
  classify-hybrid.ts  Regex-then-LLM composition
  pipeline.ts         Shared core: fetch → dedupe → classify (used by CLI and server)
  report.ts           Markdown renderer (sorted by deadline)
  index.ts            Local CLI: pipeline → write markdown reports
  rules.ts            « toujours garder / exclure » substring rules applied before the classifier
  db.ts               PostgreSQL (Bun.sql): schema, run tracking, upserts, dashboard queries
  auth.ts             Passwordless auth: email OTP codes, opaque session cookies, users, rate limits
  run.ts              Daily job: pipeline → upsert DB → email digest (alerts on failure)
  email.ts            MailPace sender + French digest/alert/login-code HTML
  live.ts             Live threads: Skip reactive graph over Postgres LISTEN/NOTIFY + SSE relay
  router.ts           Tiny path matcher for routes with :param segments
  server.ts           Dashboard + login + admin/configuration pages (Bun.serve) + /sante health endpoint
  *.test.ts           Unit tests; live.integration.test.ts needs TEST_DATABASE_URL (see Checks)
patches/              Local patch to @skip-adapter/postgres (reconnects after a dropped LISTEN connection)
clevercloud/
  cron.json           Daily schedule (04:30 UTC)
  daily-run.sh        Cron entry: env-loading wrapper around src/run.ts
.cache/               Gitignored scrape cache (raw API output)

Per-run outputs (avis-en-cours-*.md, avis-travaux-*.md) are gitignored.

Deployment (Clever Cloud)

One Node.js app (Bun is auto-detected from bun.lock; scripts.start serves the dashboard on port 8080) plus a PostgreSQL add-on. The Clever Cloud filesystem is ephemeral — PostgreSQL is the source of truth.

1. Create a Node.js application (1 instance, smallest size, disable autoscaling) and a PostgreSQL add-on (DEV plan), and link them — this injects POSTGRESQL_ADDON_URI.
2. Email first: MailPace token with a DKIM-verified sending domain (the add-on injects no env var — copy the token manually). Login is by emailed code, so a deployed app without working email is unreachable. Smoke-test the token with a curl to https://app.mailpace.com/api/v1/send before deploying; a 403 means the domain is not verified.
3. Set the env vars from .env.example: portal vars, OPENROUTER_API_KEY (set it before the push: the cron and the « Relancer maintenant » button inherit the app env), MAILPACE_API_TOKEN, EMAIL_FROM, OTP_PEPPER, ADMIN_EMAILS (your email — seeded as admin at startup), ALERT_RECIPIENT, DASHBOARD_URL.
4. clever deploy. Tables are created automatically at startup; ADMIN_EMAILS accounts are upserted as admins.
5. Log in at /connexion with your email code, then add teammates from /admin.
6. The cron (clevercloud/cron.json) fires every day at 04:30 UTC: scrape → classify → store → digest email. On failure, ALERT_RECIPIENT gets an alert email and the dashboard banner turns red.

The daily digest goes to users who ticked the opt-in on /profil; it is sent every day even when empty — if it stops arriving, something is wrong. It is skipped (with a log line) when MAILPACE_API_TOKEN is unset or nobody opted in.

License

No license specified yet. Add one before publishing if you want others to reuse the code.