feat(host-store): antseed file state + call-ledger route identity off the filesystem#38
Conversation
Move the antseed marketplace book from market.json (a file on a shared volume, unioned by hand in merge-market.js) into the Postgres host store — the next slice of the JSON/in-process migration after #36. Form delta: - Definition: a new `peer_offers` table holds one RAW row per (peer, service) — the seller's announced prices/cap/reputation as columns, not interpreted. The antseed sidecar is the sole writer (it runs `antseed network browse`); sources/antseed._load_market is the sole reader. The 15-min sliding window that merge-market.js unioned by hand is now a read-time filter on observed_at (WHERE observed_at >= now - window); the sidecar prunes rows past the window. - Invariants: store raw, derive by query — no scoring host-side; the negative / cached>input / reputation gates stay in offers_sync. Fail-soft: a DB error degrades to "no antseed candidates" exactly as a missing dump did. Behaviour preserved: offers_sync / market_book unchanged. - Irreversible: peer_offers is new DB state; market.json is retired. Changes: - host_store.py: peer_offers schema (PK (peer_id, service) + observed_at index) and a window-filtered peer_offers() reader; truncate hook updated. - antseed/write-market.js: replaces merge-market.js — flattens the browse dump to (peer, service) rows, UPSERTs into peer_offers (type-cleaning at the write, mirroring the old Python coercion), prunes past the window. - sources/antseed.py: _load_market reads host_store.peer_offers(); the file / staleness / flatten code and the now-dead coercion helpers are removed. - Dockerfile.antseed: pin pg@8.16.3 + NODE_PATH so the writer can require it. - compose.yml: DATABASE_URL + postgres dependency for the antseed service (it already shares the llm-router-internal network with postgres). - tests: seed peer_offers (shared conftest helper) instead of market.json; new host_store peer_offers round-trip + window tests. Sovereignty (Axis 4): pg is the boring standard Postgres client, pinned, and lives only in the sidecar; no new Python dependency (psycopg is from #36). Verification: full suite 409 passed, 2 skipped, 0 failed against the compose Postgres; the real Node writer -> Postgres -> Python reader round-trip, non-dump validation and window prune checked; the full stack boots healthy and /x/market surfaces a seeded antseed peer end to end.
|
Warning Review limit reached
Next review available in: 12 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR moves AntSeed marketplace offers and buyer status from shared JSON files to Postgres-backed tables and APIs. It adds DB writers, rewires the source and compose setup to use ChangesAntSeed market and status storage migration
Response metadata recording
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@antseed/entrypoint.sh`:
- Around line 66-68: The Postgres write path in entrypoint.sh can hang
indefinitely during connect/query, so wrap the node "$LIB/write-market.js"
invocation in a timeout and treat timeout as a soft failure that preserves the
last good window. Update the write step inside the existing refresh loop guard
so stalled pg access exits non-zero without blocking subsequent iterations, and
keep the current failure handling behavior for the write-market.js path.
In `@antseed/write-market.js`:
- Around line 72-83: The refresh logic in write-market.js is committing each
UPSERT and the prune independently, so a failure can leave peer_offers partially
updated. Update the main async flow that uses Client.connect(), the rows loop,
and the DELETE from peer_offers to run inside a single database transaction, and
make sure any error triggers a rollback before the existing stderr/exitCode
handling.
In `@sources/antseed.py`:
- Around line 112-113: The browse freshness check in antseed’s host snapshot
logic is incorrectly deriving staleness from whether peer_offers() returns any
rows, so an empty but successful browse is marked stale. Update the browse flow
around peer_offers() and the _stats["stale"] assignment to use a separate
heartbeat/freshness signal (or freshness metadata returned with the rows)
instead of not rows, and make snapshot_stats() reflect actual data freshness
rather than row count.
- Around line 112-113: The discover path is still vulnerable to a slow Postgres
read because `host_store.peer_offers()` can block while lazily opening the pool
and initializing schema before `antseed` reaches its fail-soft handling. Update
the `peer_offers()` flow in `host_store` to use explicit connection and
statement timeouts so the call fails fast and returns an empty result quickly,
and keep the `antseed` discovery path (`rows = host_store.peer_offers(...)`)
relying on that bounded behavior.
In `@tests/test_antseed_offers.py`:
- Around line 24-28: The autouse _clean fixture currently only relies on
host_store_clean, but it also needs to reset the in-memory route cache state so
tests do not leak global mutations between cases. Update _clean in
tests/test_antseed_offers.py to clear the route_tool_capability cache/state used
by test_offers_sync_drops_supports_tools_for_learned_incapable_route before
yielding, so test_offers_sync_defaults_tool_capability_for_meets_req always
starts from a clean route state regardless of file order.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 47519e2b-4ea3-4ca8-8d71-d728286a663c
📒 Files selected for processing (11)
Dockerfile.antseedantseed/entrypoint.shantseed/merge-market.jsantseed/write-market.jscompose.ymlhost_store.pysources/antseed.pytests/conftest.pytests/test_antseed_offers.pytests/test_host_store.pytests/test_sources.py
💤 Files with no reviewable changes (1)
- antseed/merge-market.js
| (async () => { | ||
| const client = new Client({ connectionString: process.env.DATABASE_URL }); | ||
| try { | ||
| await client.connect(); | ||
| for (const r of rows) await client.query(UPSERT, r); | ||
| await client.query("DELETE FROM peer_offers WHERE observed_at < $1", | ||
| [now - WINDOW_MS]); | ||
| } catch (e) { | ||
| process.stderr.write(`write-market: ${e.message}\n`); | ||
| process.exitCode = 4; // DB error -> keep window | ||
| } finally { | ||
| await client.end().catch(() => {}); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== antseed/write-market.js ==\n'
wc -l antseed/write-market.js
sed -n '1,140p' antseed/write-market.js
printf '\n== antseed/entrypoint.sh ==\n'
wc -l antseed/entrypoint.sh
sed -n '1,220p' antseed/entrypoint.sh
printf '\n== transaction usage search ==\n'
rg -n --hidden --glob '!node_modules' --glob '!dist' --glob '!build' '\b(BEGIN|COMMIT|ROLLBACK)\b|client\.query\("BEGIN"|client\.query\("COMMIT"|client\.query\("ROLLBACK"' antseed . || trueRepository: genlayerlabs/unhardcoded
Length of output: 8600
Wrap the refresh in a transaction.
write-market.js autocommits each UPSERT and the prune separately. If a later row or the DELETE fails, earlier rows stay committed even though entrypoint.sh treats the non-zero exit as “keep the last good window,” leaving a partially refreshed peer_offers.
🧰 Tools
🪛 ast-grep (0.44.0)
[error] 76-77: Avoid SQL injection
Context: client.query("DELETE FROM peer_offers WHERE observed_at < $1",
[now - WINDOW_MS])
Note: [CWE-89] Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection').
(sql-injection-javascript)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@antseed/write-market.js` around lines 72 - 83, The refresh logic in
write-market.js is committing each UPSERT and the prune independently, so a
failure can leave peer_offers partially updated. Update the main async flow that
uses Client.connect(), the rows loop, and the DELETE from peer_offers to run
inside a single database transaction, and make sure any error triggers a
rollback before the existing stderr/exitCode handling.
| rows = host_store.peer_offers(STALE_AFTER_S * 1000) | ||
| self._stats["stale"] = not rows |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Fresh empty browses now look stale.
peer_offers() only returns rows that are still inside the sliding window, and Line 113 turns [] into stale=True. That means a successful browse that finds zero peers/services is indistinguishable from a stale snapshot or a DB read failure, so snapshot_stats()/dashboard freshness regresses even when the data is current. This needs a separate browse heartbeat (or freshness metadata returned alongside the rows) instead of deriving staleness from row presence alone.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@sources/antseed.py` around lines 112 - 113, The browse freshness check in
antseed’s host snapshot logic is incorrectly deriving staleness from whether
peer_offers() returns any rows, so an empty but successful browse is marked
stale. Update the browse flow around peer_offers() and the _stats["stale"]
assignment to use a separate heartbeat/freshness signal (or freshness metadata
returned with the rows) instead of not rows, and make snapshot_stats() reflect
actual data freshness rather than row count.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Files:\n'
git ls-files | rg '(^|/)(antseed\.py|.*host_store.*|.*store.*|.*peer.*)$' || true
printf '\nSearch for symbols:\n'
rg -n "def (_get_pool|peer_offers|offers_sync)|ConnectionPool\(|statement_timeout|connect_timeout|timeout" sources . || trueRepository: genlayerlabs/unhardcoded
Length of output: 15890
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'host_store.py relevant sections:\n'
sed -n '120,190p;380,460p' host_store.py | cat -n
printf '\nantseed.py relevant section:\n'
sed -n '90,150p' sources/antseed.py | cat -n
printf '\nTests around host_store timeouts/behavior:\n'
sed -n '1,220p' tests/test_host_store.py | cat -nRepository: genlayerlabs/unhardcoded
Length of output: 22850
Bound the Postgres read on the discover path. sources/antseed.py:23 calls host_store.peer_offers(), and host_store.py lazily opens the pool and runs schema init without connect/statement timeouts. A slow or unreachable Postgres can block discovery before the fail-soft except runs; add explicit DB timeouts so this degrades to [] quickly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@sources/antseed.py` around lines 112 - 113, The discover path is still
vulnerable to a slow Postgres read because `host_store.peer_offers()` can block
while lazily opening the pool and initializing schema before `antseed` reaches
its fail-soft handling. Update the `peer_offers()` flow in `host_store` to use
explicit connection and statement timeouts so the call fails fast and returns an
empty result quickly, and keep the `antseed` discovery path (`rows =
host_store.peer_offers(...)`) relying on that bounded behavior.
| @pytest.fixture(autouse=True) | ||
| def _clean(host_store_clean): | ||
| # The market book now lives in the host store (peer_offers); every test seeds | ||
| # it and needs the per-test truncation (skips if Postgres is unavailable). | ||
| yield |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Reset the in-memory route caches in the autouse fixture.
This fixture now only truncates Postgres, but test_offers_sync_drops_supports_tools_for_learned_incapable_route mutates global route_tool_capability state for antseed|qwen3-235b-a22b|peerA, while test_offers_sync_defaults_tool_capability_for_meets_req expects that same route to start clean. That makes the file order-dependent.
Suggested fix
import route_reliability as rr # noqa: E402
import route_latency as rl # noqa: E402
+import route_tool_capability as tc # noqa: E402
from sources.antseed import AntSeedSource # noqa: E402
from conftest import seed_peer_offers as _seed_market # noqa: E402
`@pytest.fixture`(autouse=True)
def _clean(host_store_clean):
# The market book now lives in the host store (peer_offers); every test seeds
# it and needs the per-test truncation (skips if Postgres is unavailable).
+ rr.reset()
+ rl.reset()
+ tc.reset()
yield📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| @pytest.fixture(autouse=True) | |
| def _clean(host_store_clean): | |
| # The market book now lives in the host store (peer_offers); every test seeds | |
| # it and needs the per-test truncation (skips if Postgres is unavailable). | |
| yield | |
| `@pytest.fixture`(autouse=True) | |
| def _clean(host_store_clean): | |
| # The market book now lives in the host store (peer_offers); every test seeds | |
| # it and needs the per-test truncation (skips if Postgres is unavailable). | |
| rr.reset() | |
| rl.reset() | |
| tc.reset() | |
| yield |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_antseed_offers.py` around lines 24 - 28, The autouse _clean
fixture currently only relies on host_store_clean, but it also needs to reset
the in-memory route cache state so tests do not leak global mutations between
cases. Update _clean in tests/test_antseed_offers.py to clear the
route_tool_capability cache/state used by
test_offers_sync_drops_supports_tools_for_learned_incapable_route before
yielding, so test_offers_sync_defaults_tool_capability_for_meets_req always
starts from a clean route state regardless of file order.
Twin of the peer_offers move: the antseed buyer's status (session pin + escrow + wallet) goes from status-<id>.json on the shared volume to the Postgres host store. With both off the filesystem, sources/antseed.py no longer touches disk and the antseed-market volume is removed entirely. Form delta: - Definition: a new `buyer_status` table holds one row per buyer pid — the raw buyer-reported fields (pinned_peer_id, deposits_available/_reserved, wallet_address, connection_state) as columns. The antseed sidecar writes it (write-status.js on the poll loop + control.js after a wallet op); sources/antseed reads it (_pinned_peer + balances). - Invariants: store raw — deposits stay the strings the buyer reports and are coerced on read, exactly as the JSON status was. Fail-soft: a missing row / store error degrades to "no pin, no balance" as a missing status file did. Behaviour preserved: _pinned_peer / balances unchanged but for the source. - Irreversible: buyer_status is new DB state; status-<id>.json is retired and the antseed-market volume (+ both mounts) is dropped. Changes: - host_store.py: buyer_status schema + a buyer_status(pid) reader; truncate hook updated. - antseed/store.js: shared buyer_status row shape + UPSERT, used by both writers so they can't drift. - antseed/write-status.js: replaces the inline node -e + atomic_write; reads `buyer status --json`, UPSERTs buyer_status, validates (non-status -> no write). - antseed/control.js: refreshStatus UPSERTs buyer_status via a pg pool instead of writing the file; still returns the fresh status for the HTTP response. - antseed/entrypoint.sh: write_status calls write-status.js; the now-dead atomic_write helper is removed; comments updated. - sources/antseed.py: _pinned_peer + balances read host_store.buyer_status; the file / json / Path / market_dir machinery is removed (no disk access). - Dockerfile.antseed: COPY store.js + write-status.js. - compose.yml: drop the antseed-market volume and its router/antseed mounts. - tests: seed buyer_status (shared conftest helper) instead of status files; new host_store buyer_status round-trip/absent test. Verification: full suite 410 passed, 2 skipped, 0 failed against the compose Postgres; the real write-status.js -> Postgres -> Python reader round-trip and non-status validation checked; all four sidecar JS files pass node --check; the full stack boots healthy and creates buyer_status on boot.
jmlago
changed the title
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/test_sources.py (1)
493-497: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winDon't hide schema/setup regressions behind
pytest.skip().This skips on any exception, not just an unavailable Postgres. If the new
peer_offers/buyer_statussetup is broken (schema init,TRUNCATE, SQL shape, etc.), the AntSeed source tests will be reported as skipped instead of failing, which can mask the migration break this helper is meant to catch. Limit the skip to connectivity/bootstrap failures and let other exceptions fail the test.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_sources.py` around lines 493 - 497, The test helper in host_store reset/truncate handling is catching every Exception and turning all failures into pytest.skip(), which can hide schema or SQL regressions. Update the try/except around host_store.reset() and host_store.truncate_all_for_tests() so only expected Postgres connectivity/bootstrap failures are skipped, while other exceptions are re-raised and fail the AntSeed source tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@antseed/control.js`:
- Around line 55-61: The refreshStatus() flow currently parses stdout from
run(['buyer', 'status', '--json'], STATUS_TIMEOUT_MS) without checking the
returned exit code, so non-zero CLI failures can still be treated as valid
status data. Update refreshStatus() to inspect the run() result code before JSON
parsing/upsert, and immediately return a failed/null status when the CLI exits
non-zero; keep the existing JSON parse and pool.query/UPSERT_BUYER_STATUS path
only for successful runs.
- Line 26: The shared Postgres pool created in control.js currently has no
timeout, so pool.query() calls in refreshStatus() can hang indefinitely when the
database is wedged. Update the Pool initialization for the shared pool to
include an appropriate timeout setting, and make sure the existing
refreshStatus(), /deposit, /withdraw, and /status flows use that same pool
instance so requests fail fast instead of waiting forever.
In `@antseed/entrypoint.sh`:
- Line 74: The status writer call in the entrypoint can still hang during pg
connect/query because the current fallback only ignores exit failures. Update
the `node "$LIB/write-status.js"` invocation in `entrypoint.sh` to run under a
timeout, or add an explicit database timeout inside `write-status.js` so the
`writeStatus` flow cannot block indefinitely.
In `@antseed/store.js`:
- Around line 21-24: The UPSERT row builder in buyerStatusRow currently accepts
any object and turns missing fields into nulls, which can overwrite a valid
buyer_status record. Add a shared shape validator for status objects and use it
before calling buyerStatusRow and before any connect/upsert path so only
properly shaped status data is accepted. Keep the check centralized near
buyerStatusRow and update callers to reject invalid CLI error objects before row
creation.
In `@antseed/write-status.js`:
- Around line 16-22: The current object-only guard in write-status.js lets
invalid status objects like {} or {"error": ...} proceed into the UPSERT path
and open Postgres unnecessarily. Reuse the shared status validator from store.js
in the write-status entrypoint before creating the Client, and exit early for
any invalid shape so only a valid status reaches buyerStatusRow and
UPSERT_BUYER_STATUS.
In `@tests/conftest.py`:
- Around line 81-100: `seed_buyer_status` is overwriting previously seeded buyer
fields with NULL when callers omit them. Update the UPSERT in this helper so
omitted values preserve the existing `buyer_status` columns instead of replacing
`pinned_peer_id` and `connection_state`, or change the helper contract to
require a complete row and update the `_antseed_source()`/balance test call
sites accordingly. Use the `seed_buyer_status` helper and its `ON CONFLICT (pid)
DO UPDATE` clause as the fix point.
---
Outside diff comments:
In `@tests/test_sources.py`:
- Around line 493-497: The test helper in host_store reset/truncate handling is
catching every Exception and turning all failures into pytest.skip(), which can
hide schema or SQL regressions. Update the try/except around host_store.reset()
and host_store.truncate_all_for_tests() so only expected Postgres
connectivity/bootstrap failures are skipped, while other exceptions are
re-raised and fail the AntSeed source tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 49ba1da5-8470-4d8a-91d0-9d22473b47ba
📒 Files selected for processing (12)
Dockerfile.antseedantseed/control.jsantseed/entrypoint.shantseed/store.jsantseed/write-status.jscompose.ymlhost_store.pysources/antseed.pytests/conftest.pytests/test_antseed_offers.pytests/test_host_store.pytests/test_sources.py
🚧 Files skipped from review as they are similar to previous changes (2)
- Dockerfile.antseed
- tests/test_antseed_offers.py
|
|
||
| // One pool for the long-lived control server (write-status.js, the poll-loop | ||
| // twin, is one-shot and uses a plain Client instead). | ||
| const pool = new Pool({ connectionString: process.env.DATABASE_URL }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- antseed/control.js ---'
nl -ba antseed/control.js | sed -n '1,240p'
echo
echo '--- searches ---'
rg -n "statement_timeout|query_timeout|connectionTimeoutMillis|Pool\\(|DATABASE_URL|refreshStatus|/deposit|/withdraw|/status" antseed . -g '!node_modules' -g '!dist' -g '!build'Repository: genlayerlabs/unhardcoded
Length of output: 229
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect package metadata to identify pg version and any timeout-related docs/config.
echo '--- package files mentioning pg ---'
rg -n '"pg"|node-postgres|postgres' package.json antseed package.json . -g '!node_modules' -g '!dist' -g '!build'
echo
echo '--- antseed/control.js nearby context (if file is large) ---'
wc -l antseed/control.jsRepository: genlayerlabs/unhardcoded
Length of output: 1648
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- antseed/control.js ---'
nl -ba antseed/control.js | sed -n '1,240p'
echo
echo '--- timeout-related search ---'
rg -n "statement_timeout|query_timeout|connectionTimeoutMillis|Pool\\(|DATABASE_URL|refreshStatus|/deposit|/withdraw|/status" . -g '!node_modules' -g '!dist' -g '!build'
echo
echo '--- package metadata ---'
if [ -f package.json ]; then
sed -n '1,220p' package.json
fi
if [ -f antseed/package.json ]; then
sed -n '1,220p' antseed/package.json
fiRepository: genlayerlabs/unhardcoded
Length of output: 229
🌐 Web query:
node-postgres Pool connectionTimeoutMillis query_timeout statement_timeout documentation
💡 Result:
In node-postgres, the Pool configuration supports several timeout parameters that govern different stages of the database interaction lifecycle [1][2]. connectionTimeoutMillis This parameter controls the maximum time, in milliseconds, that the client will wait to establish a connection to the database [3][4]. If a connection cannot be acquired from the pool within this time, it returns an error [3][4]. The default is no timeout [1][2]. query_timeout This is a client-side timeout managed by the node-postgres library [4][5]. It specifies the maximum time, in milliseconds, that the library will wait for a response from the database after a query has been sent [1][4][5]. If the time is exceeded, the driver stops listening for a response, but the database may continue executing the query on the server side [5]. The default is no timeout [1][2]. statement_timeout This parameter is passed directly to the PostgreSQL server [5]. It defines the maximum time, in milliseconds, that an individual SQL statement is allowed to run on the server [1][4]. If the limit is reached, PostgreSQL cancels the statement and returns an error to the client [4][5]. This is generally considered the more robust way to handle long-running queries, as it ensures the server stops processing the request [5]. The default is no timeout [1][2]. Recommended Relationship Best practice often involves using both, setting the query_timeout slightly longer than the statement_timeout [6][5]. This allows the database to gracefully cancel and report the error (via statement_timeout), while providing a client-side safety net (via query_timeout) in the event of network issues or other failures where the database fails to send a response [5]. Configuration These options can be passed directly to the Pool constructor [3][7]: const pool = new Pool({ connectionTimeoutMillis: 3000, statement_timeout: 20000, query_timeout: 25000 }); [8], [3], [1], [4], [7], [2], [5], [9]
Citations:
- 1: https://github.com/brianc/node-postgres/blob/master/docs/pages/apis/client.mdx
- 2: https://node-postgres.com/apis/client
- 3: https://github.com/brianc/node-postgres/blob/master/packages/pg-pool/README.md
- 4: Documented lock_timeout in client.mdx brianc/node-postgres#3199
- 5: https://stackoverflow.com/questions/59155572/how-to-set-query-timeout-in-relation-to-statement-timeout
- 6: Add configurable query timeout brianc/node-postgres#1760
- 7: https://github.com/brianc/node-postgres/tree/master/packages/pg-pool
- 8: https://node-postgres.com/apis/pool
- 9: https://www.nazarboyko.com/articles/database-connection-pooling-in-nodejs
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- antseed/control.js (first 260 lines with line numbers via awk) ---'
awk '{printf "%4d %s\n", NR, $0}' antseed/control.js | sed -n '1,260p'
echo
echo '--- pool/timeout usage in antseed ---'
grep -RInE 'Pool\\(|connectionTimeoutMillis|query_timeout|statement_timeout|refreshStatus|/deposit|/withdraw|/status' antseed || true
echo
echo '--- antseed/control.js line count ---'
wc -l antseed/control.jsRepository: genlayerlabs/unhardcoded
Length of output: 6055
Add a timeout to the shared Postgres pool. refreshStatus() awaits pool.query(), so a wedged database can leave successful /deposit, /withdraw, and /status responses hanging indefinitely. antseed/control.js:26
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@antseed/control.js` at line 26, The shared Postgres pool created in
control.js currently has no timeout, so pool.query() calls in refreshStatus()
can hang indefinitely when the database is wedged. Update the Pool
initialization for the shared pool to include an appropriate timeout setting,
and make sure the existing refreshStatus(), /deposit, /withdraw, and /status
flows use that same pool instance so requests fail fast instead of waiting
forever.
| const r = await run(['buyer', 'status', '--json'], STATUS_TIMEOUT_MS); | ||
| let data; | ||
| try { data = JSON.parse(r.stdout); } catch (_) { return null; } | ||
| if (data === null || typeof data !== 'object') return null; | ||
| data.fetched_at_ms = Date.now(); | ||
| const tmp = STATUS_FILE + '.' + process.pid + '.tmp'; | ||
| try { | ||
| fs.writeFileSync(tmp, JSON.stringify(data)); | ||
| fs.renameSync(tmp, STATUS_FILE); | ||
| } catch (_) { try { fs.unlinkSync(tmp); } catch (__) {} } | ||
| await pool.query(UPSERT_BUYER_STATUS, buyerStatusRow(data, PID)); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not trust non-zero status CLI output.
run() returns code, but refreshStatus() ignores it. If the CLI exits non-zero while emitting JSON, this path can still return ok: true and upsert bad status data.
Suggested guard
const r = await run(['buyer', 'status', '--json'], STATUS_TIMEOUT_MS);
+ if (r.code !== 0) return null;
let data;
try { data = JSON.parse(r.stdout); } catch (_) { return null; }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const r = await run(['buyer', 'status', '--json'], STATUS_TIMEOUT_MS); | |
| let data; | |
| try { data = JSON.parse(r.stdout); } catch (_) { return null; } | |
| if (data === null || typeof data !== 'object') return null; | |
| data.fetched_at_ms = Date.now(); | |
| const tmp = STATUS_FILE + '.' + process.pid + '.tmp'; | |
| try { | |
| fs.writeFileSync(tmp, JSON.stringify(data)); | |
| fs.renameSync(tmp, STATUS_FILE); | |
| } catch (_) { try { fs.unlinkSync(tmp); } catch (__) {} } | |
| await pool.query(UPSERT_BUYER_STATUS, buyerStatusRow(data, PID)); | |
| const r = await run(['buyer', 'status', '--json'], STATUS_TIMEOUT_MS); | |
| if (r.code !== 0) return null; | |
| let data; | |
| try { data = JSON.parse(r.stdout); } catch (_) { return null; } | |
| if (data === null || typeof data !== 'object') return null; | |
| data.fetched_at_ms = Date.now(); | |
| try { | |
| await pool.query(UPSERT_BUYER_STATUS, buyerStatusRow(data, PID)); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@antseed/control.js` around lines 55 - 61, The refreshStatus() flow currently
parses stdout from run(['buyer', 'status', '--json'], STATUS_TIMEOUT_MS) without
checking the returned exit code, so non-zero CLI failures can still be treated
as valid status data. Update refreshStatus() to inspect the run() result code
before JSON parsing/upsert, and immediately return a failed/null status when the
CLI exits non-zero; keep the existing JSON parse and
pool.query/UPSERT_BUYER_STATUS path only for successful runs.
| < "$raw" | atomic_write "$MARKET_DIR/status-antseed.json" || true | ||
| # upsert the buyer status into the host store's buyer_status (validate + DB | ||
| # error -> exit non-zero, keep the last good row) | ||
| node "$LIB/write-status.js" < "$raw" || true |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate relevant files
git ls-files 'antseed/*' | sed -n '1,200p'
# Map the entrypoint and status writer structure
ast-grep outline antseed/entrypoint.sh --view expanded || true
ast-grep outline antseed/write-status.js --view expanded || true
ast-grep outline antseed/lib/write-status.js --view expanded || true
# Read the relevant script sections with line numbers
for f in antseed/entrypoint.sh antseed/write-status.js antseed/lib/write-status.js; do
if [ -f "$f" ]; then
echo "===== $f ====="
wc -l "$f"
cat -n "$f" | sed -n '1,220p'
fi
done
# Find PostgreSQL usage and any timeout handling in the antseed directory
rg -n --hidden --no-messages 'timeout|pg\.|new Client|Client\(|pool|connect\(|query\(' antseedRepository: genlayerlabs/unhardcoded
Length of output: 8136
Bound the status writer’s DB call. node "$LIB/write-status.js" can still block in pg connect/query because write-status.js has no timeout, so || true only masks exited failures. Wrap this invocation in timeout or add a DB timeout in antseed/write-status.js.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@antseed/entrypoint.sh` at line 74, The status writer call in the entrypoint
can still hang during pg connect/query because the current fallback only ignores
exit failures. Update the `node "$LIB/write-status.js"` invocation in
`entrypoint.sh` to run under a timeout, or add an explicit database timeout
inside `write-status.js` so the `writeStatus` flow cannot block indefinitely.
| function buyerStatusRow(d, pid) { | ||
| return [pid, str(d.pinnedPeerId), str(d.depositsAvailable), | ||
| str(d.depositsReserved), str(d.walletAddress), | ||
| str(d.connectionState), Date.now()]; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Reject non-status objects before shaping the UPSERT row.
buyerStatusRow({}) currently produces a row of null status fields, so object-shaped CLI errors can clobber the last good buyer_status row. Centralize a shape check here and have callers use it before connecting/upserting.
Suggested shared validator
const str = (v) => (v === null || v === undefined) ? null : String(v);
+const STATUS_KEYS = [
+ "pinnedPeerId", "depositsAvailable", "depositsReserved",
+ "walletAddress", "connectionState",
+];
+
+function isBuyerStatus(d) {
+ return d !== null && typeof d === "object" &&
+ STATUS_KEYS.some((k) => Object.prototype.hasOwnProperty.call(d, k));
+}
function buyerStatusRow(d, pid) {
+ if (!isBuyerStatus(d)) throw new TypeError("not an antseed buyer status");
return [pid, str(d.pinnedPeerId), str(d.depositsAvailable),
str(d.depositsReserved), str(d.walletAddress),
str(d.connectionState), Date.now()];
}
-module.exports = { UPSERT_BUYER_STATUS, buyerStatusRow };
+module.exports = { UPSERT_BUYER_STATUS, buyerStatusRow, isBuyerStatus };📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| function buyerStatusRow(d, pid) { | |
| return [pid, str(d.pinnedPeerId), str(d.depositsAvailable), | |
| str(d.depositsReserved), str(d.walletAddress), | |
| str(d.connectionState), Date.now()]; | |
| const str = (v) => (v === null || v === undefined) ? null : String(v); | |
| const STATUS_KEYS = [ | |
| "pinnedPeerId", "depositsAvailable", "depositsReserved", | |
| "walletAddress", "connectionState", | |
| ]; | |
| function isBuyerStatus(d) { | |
| return d !== null && typeof d === "object" && | |
| STATUS_KEYS.some((k) => Object.prototype.hasOwnProperty.call(d, k)); | |
| } | |
| function buyerStatusRow(d, pid) { | |
| if (!isBuyerStatus(d)) throw new TypeError("not an antseed buyer status"); | |
| return [pid, str(d.pinnedPeerId), str(d.depositsAvailable), | |
| str(d.depositsReserved), str(d.walletAddress), | |
| str(d.connectionState), Date.now()]; | |
| } | |
| module.exports = { UPSERT_BUYER_STATUS, buyerStatusRow, isBuyerStatus }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@antseed/store.js` around lines 21 - 24, The UPSERT row builder in
buyerStatusRow currently accepts any object and turns missing fields into nulls,
which can overwrite a valid buyer_status record. Add a shared shape validator
for status objects and use it before calling buyerStatusRow and before any
connect/upsert path so only properly shaped status data is accepted. Keep the
check centralized near buyerStatusRow and update callers to reject invalid CLI
error objects before row creation.
| if (d === null || typeof d !== "object") process.exit(3); // JSON but not a status | ||
|
|
||
| (async () => { | ||
| const client = new Client({ connectionString: process.env.DATABASE_URL }); | ||
| try { | ||
| await client.connect(); | ||
| await client.query(UPSERT_BUYER_STATUS, buyerStatusRow(d, PID)); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Validate the status shape before opening Postgres.
With the current object-only check, {} or {"error": ...} reaches the UPSERT path. After adding the shared validator in store.js, use it here so invalid status exits before any DB work.
Suggested caller update
-const { UPSERT_BUYER_STATUS, buyerStatusRow } = require("./store.js");
+const { UPSERT_BUYER_STATUS, buyerStatusRow, isBuyerStatus } = require("./store.js");
...
-if (d === null || typeof d !== "object") process.exit(3); // JSON but not a status
+if (!isBuyerStatus(d)) process.exit(3); // JSON but not a status🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@antseed/write-status.js` around lines 16 - 22, The current object-only guard
in write-status.js lets invalid status objects like {} or {"error": ...} proceed
into the UPSERT path and open Postgres unnecessarily. Reuse the shared status
validator from store.js in the write-status entrypoint before creating the
Client, and exit early for any invalid shape so only a valid status reaches
buyerStatusRow and UPSERT_BUYER_STATUS.
| def seed_buyer_status(pid, pinned_peer_id=None, deposits_available=None, | ||
| deposits_reserved=None, wallet_address=None, | ||
| connection_state=None): | ||
| """Seed the antseed buyer status (pin + escrow + wallet) as the sidecar's | ||
| write-status.js / control.js do — one row per buyer pid. Deposits are the raw | ||
| buyer-reported strings.""" | ||
| import time | ||
| with host_store._get_pool().connection() as conn: | ||
| conn.execute( | ||
| "INSERT INTO buyer_status (pid, pinned_peer_id, deposits_available," | ||
| " deposits_reserved, wallet_address, connection_state, fetched_at)" | ||
| " VALUES (%s,%s,%s,%s,%s,%s,%s)" | ||
| " ON CONFLICT (pid) DO UPDATE SET" | ||
| " pinned_peer_id=EXCLUDED.pinned_peer_id," | ||
| " deposits_available=EXCLUDED.deposits_available," | ||
| " deposits_reserved=EXCLUDED.deposits_reserved," | ||
| " wallet_address=EXCLUDED.wallet_address," | ||
| " connection_state=EXCLUDED.connection_state, fetched_at=EXCLUDED.fetched_at", | ||
| (pid, pinned_peer_id, deposits_available, deposits_reserved, | ||
| wallet_address, connection_state, int(time.time() * 1000))) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Preserve existing buyer fields when reseeding a status row.
This helper always UPSERTs the full buyer_status row, so callers that only pass deposits/wallet values overwrite the preseeded pinned_peer_id and connection_state with NULL. _antseed_source() seeds default pins first, and the later balance tests already reuse this helper without pinned_peer_id, so the stored row no longer mirrors the sidecar state these fixtures are supposed to represent. Either preserve existing columns when an argument is omitted, or require callers to pass a complete row explicitly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/conftest.py` around lines 81 - 100, `seed_buyer_status` is overwriting
previously seeded buyer fields with NULL when callers omit them. Update the
UPSERT in this helper so omitted values preserve the existing `buyer_status`
columns instead of replacing `pinned_peer_id` and `connection_state`, or change
the helper contract to require a complete row and update the
`_antseed_source()`/balance test call sites accordingly. Use the
`seed_buyer_status` helper and its `ON CONFLICT (pid) DO UPDATE` clause as the
fix point.
…bump) #3 of the operational-store migration: enrich the `calls` fact table with the two raw per-call facts the #4 route/analytics views will derive from — the executed route identity and the cache-token breakdown. Prerequisite for keying per-route stats off the ledger. - Submodule bump core 97d0333 -> 537e204 (unhardcoded-engine #23): the engine's `chosen` now carries `served_by` — the marketplace peer that served the call, or the provider itself for a direct route (never nil). Host suite green on it. - host_store.py: `calls` gains `served_by TEXT` + `tokens_cached BIGINT`, applied to existing tables via idempotent `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` (CREATE TABLE IF NOT EXISTS never alters an existing table — the store gains its first in-place migration). insert_call maps both. route_key is left unchanged: deriving a peer-granular route key from served_by is #4's job; this commit only captures the raw fact. - shim.py: `_build_x_router` surfaces `served_by` from `chosen` (tokens_cached was already there). - auth_proxy.py: the ingress threads served_by + tokens_cached off x_router into the recorded call (both stream and unary paths) -> insert_call. ttft was intentionally NOT added: nothing measures it yet, so the column would be idle (Axis 3). error_type was already a column. Verification: full suite 411 passed, 2 skipped, 0 failed against the compose Postgres; the ALTER migration applies in place on boot; a live chat records served_by + tokens_cached in `calls` end to end against engine #23.
jmlago
changed the title
…olumn contract peer_offers and buyer_status are CREATEd by the Python host store but WRITTEN by the Node antseed sidecar (write-market.js, antseed/store.js) and seeded by Python test mimics (conftest). Three places must agree on the column set and nothing at runtime makes them: the readers are fail-soft, so a renamed/added/dropped column degrades antseed to "no candidates" silently -- and the unit suite can't see it, because it seeds via the Python mimic, not the real Node writer (green proves the reader works, not that Node and Python agree). Add a static contract test that parses the column list out of all three sources and asserts it matches per table. Pure text parsing: no DB, no node runtime, runs in the ordinary unit suite; red on any drift (verified by injecting a rename). The live behave e2e stays the only thing exercising the real Node writer; this guards the part that drifts.
Prod runs the sidecar as the image now (not the inline node command), so the entrypoint must keep the inline's safety: a CHANGE_ME / unset-secret placeholder is not a valid identity and the CLI would reject it. Unset it when it isn't a 64-hex string so the buyer falls back to a generated key on the data volume (matching the previous inline behaviour); the prod secret is a real hot-wallet.
1 participant
What this does (and why)
The host's operational state — the antseed marketplace book, the buyer's
wallet/escrow status, and the per-call ledger — lived in JSON/JSONL files
on a shared volume and in in-process dicts. That state is per-pod and can't be
shared between the router and the ingress (they run as separate containers), nor
survive a restart. #36 began moving it into a single Postgres store both
containers reach; this PR continues that migration.
Three atomic commits. 1 and 2 move the antseed market and status files
into the store (after which
sources/antseed.pyno longer touches disk at all,and the shared market volume is deleted). 3 enriches the call ledger with the
identity of the route that actually served each call — the raw fact the next
step (#4) will derive per-route stats from. Commit 3 depends on engine #23
(
chosen.served_by), already merged and pinned here.Commit 1 —
peer_offers(antseed market book)One RAW row per
(peer, service)— the seller's announced prices/cap/reputationas columns. The antseed sidecar is the sole writer (
antseed network browse),_load_marketthe sole reader. The 15-min sliding window thatmerge-market.jsunioned by hand becomes a read-time filter on
observed_at; the sidecar prunespast the window.
Commit 2 —
buyer_status(pin + escrow + wallet)One row per buyer pid — raw buyer-reported fields. Written by
write-status.js(poll loop) and
control.js(after a wallet op); read by_pinned_peer+balances. With both off the filesystem,sources/antseed.pyno longer touchesdisk and the
antseed-marketvolume (+ both mounts) is removed.Commit 3 —
calls.served_by+calls.tokens_cachedThe fact table gains the executed route identity (
served_by, from theengine's
chosen— the marketplace peer that served the call, or the provideritself for a direct route) and the cache-token breakdown. These are the raw
per-call facts #4 derives per-route stats from. Threaded engine
chosen→x_router→ ingress →calls. Bumps the core submodule to engine #23 (merged).route_keyis left unchanged — deriving a peer-granular route key fromserved_byis #4's job; this commit only captures the raw fact.ttftomitted(nothing measures it yet → an idle column).
Invariants
scoring host-side.
served_byis captured raw; combining it into a route keyis deferred to test: BDD user-flow suite (behave) + AntSeed dev-wallet setup #4.
callsevolves via idempotentALTER TABLE ... ADD COLUMN IF NOT EXISTS— the store's first table evolution (CREATE TABLE IF NOTEXISTS never alters an existing table).
offers_sync/market_book/_pinned_peer/balances/ OpenAI-compat unchanged but for their source).peer_offers,buyer_status, twocallscolumns);
market.json/status-<id>.jsonretired.Dependency sovereignty (Axis 4)
pg@8.16.3only in the sidecar (Node), pinned; the buyer_status UPSERT shared inantseed/store.jsso the two writers can't drift. No new Python dep (psycopgfrom #36). The sidecar→Postgres edge is the same shape as router→PG / ingress→PG:
all three depend on the shared store, not on each other.
Verification
populates
peer_offers(5 live peers) +buyer_status(the real wallet) inPostgres;
/x/marketsurfaces them; the router routes off them.served_by+tokens_cached: the ALTER migration applies in place on boot; alive chat records both in
callsend to end (against engine fix(core): bump engine to ee31e81 — preserve marketplace offer prices #23).54 scenarios passed. The remaining failures are pre-existing environment
limits of this dev stack, NOT this change: codex isn't configured as a
subscription provider here, so the codex-seeded flows fail (
05_providersapi_kind,
09_flow1no_candidates, the08_dashboard_uirow that needs theseeded activity); and one antseed peer returned
insufficient_deposits. Theantseed scenario that failed still routed via
peer_offers— i.e. thischange worked; the peer rejected the payment.
Remaining migration (#36's plan)
route_*views derived fromcalls(keyed byserved_by) · 5._stats/usage-history → queries overcalls· 6. cleanup (drop legacy JSON +migrate_legacy_json).