Skip to content

feat(api): add admin backup endpoint - #162

Merged
jack-arturo merged 4 commits into
mainfrom
feat/admin-backup-endpoint
Jun 6, 2026
Merged

feat(api): add admin backup endpoint#162
jack-arturo merged 4 commits into
mainfrom
feat/admin-backup-endpoint

Conversation

@jack-arturo

@jack-arturo jack-arturo commented May 1, 2026

Copy link
Copy Markdown
Member

Summary

  • add admin-only GET /backup that streams a restore-compatible tar.gz export of FalkorDB and/or Qdrant
  • refactor the existing backup script onto shared backup helpers
  • let restore_from_backup.py --backup-dir accept downloaded .tar.gz archives
  • document the API backup export workflow

API / config

  • GET /backup requires X-Admin-Token or X-Admin-Api-Key using ADMIN_API_TOKEN
  • regular AUTOMEM_API_TOKEN auth is intentionally bypassed for this endpoint
  • optional include=falkordb,qdrant defaults to both stores

Breaking Changes

None.

Testing

  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest -q
  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest tests/test_backup_endpoint.py tests/contracts/test_routes_contract.py -q
  • pre-commit hooks on commit

Railway

Ready for branch deployment smoke testing on Railway.

@railway-app
railway-app Bot temporarily deployed to automem / production May 1, 2026 01:43 Inactive
@jack-arturo
jack-arturo marked this pull request as ready for review June 6, 2026 05:19
Copilot AI review requested due to automatic review settings June 6, 2026 05:19
Copilot AI reviewed Jun 6, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an admin-only API endpoint to export restore-compatible backups (FalkorDB and/or Qdrant) as a streamed .tar.gz, refactors backup script logic into shared helpers, and updates restore tooling + docs to support backups downloaded from the API.

Changes:

  • Introduce GET /backup (admin-token only) that streams a restore-compatible tar.gz archive.
  • Centralize backup creation into automem/backup.py and refactor scripts/backup_automem.py to use it.
  • Extend scripts/restore_from_backup.py to accept a downloaded .tar.gz archive via --backup-dir, plus add tests and documentation.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/test_backup_endpoint.py Adds end-to-end tests for admin-only /backup tarball export and restore-from-tar behavior.
tests/support/fake_graph.py Extends FakeGraph to support backup-export query pagination semantics.
tests/contracts/test_routes_contract.py Freezes the route contract to include GET /backup.
scripts/restore_from_backup.py Adds .tar.gz detection + safe extraction wrapper for restore flows.
scripts/backup_automem.py Refactors the script to use shared backup helpers and cleanup utilities.
docs/MONITORING_AND_BACKUPS.md Documents the API backup export workflow and restore command.
docs/API.md Documents /backup behavior, auth requirements, and payload type.
automem/backup.py New shared backup helpers for exporting/writing/streaming FalkorDB + Qdrant backups.
automem/api/runtime_bootstrap.py Registers the new backup blueprint.
automem/api/backup.py Implements the /backup endpoint and request auditing.
app.py Bypasses regular API-token auth for /backup so admin-only auth applies.

Comment thread scripts/restore_from_backup.py Outdated
Comment thread app.py Outdated
Comment thread automem/backup.py Outdated
@railway-app
railway-app Bot temporarily deployed to automem / production June 6, 2026 05:28 Inactive
@railway-app
railway-app Bot temporarily deployed to automem / production June 6, 2026 05:30 Inactive
@jack-arturo
jack-arturo merged commit 8b1f264 into main Jun 6, 2026
8 checks passed
@jack-arturo
jack-arturo deleted the feat/admin-backup-endpoint branch June 6, 2026 05:43
This was referenced Jun 6, 2026
Merged
Merged
jack-arturo added a commit that referenced this pull request Jun 6, 2026
@jack-arturo
feat(lab): isolate production clone restores (#171)
aef90c0 
## Summary
- allow Docker Compose host ports to be overridden for isolated lab
stacks
- make clone_production restore-only workflows accept saved snapshot
paths or names
- add compose project and local credential overrides so destructive
experiments can run without touching the default dev stack

## Testing
- bash -n scripts/lab/clone_production.sh
- docker compose config
- pre-commit run --files docker-compose.yml
scripts/lab/clone_production.sh

## Release note
Follow-up for the local commit that was intended to accompany PR #162
before the v0.16.0 release.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
@zackkatz zackkatz mentioned this pull request Jun 7, 2026
Merged
jack-arturo pushed a commit that referenced this pull request Jun 8, 2026
@zackkatz
fix(backup): make backup_automem.py runnable as `python scripts/backu…
edd9742 
…p_automem.py` (#175)

## Summary

`scripts/backup_automem.py` can no longer be run as a script — `python
scripts/backup_automem.py` crashes immediately on import:

```
Traceback (most recent call last):
  File ".../scripts/backup_automem.py", line 31, in <module>
    from automem.backup import (
ModuleNotFoundError: No module named 'automem'
```

## Root cause

#162 (`8b1f2649`) moved the backup helpers into the `automem.backup`
package, and the script now does `from automem.backup import ...`. But
when a file is run as `python scripts/backup_automem.py`, Python puts
the **script's own directory** (`scripts/`) on `sys.path[0]` — not the
repo root — so the `automem` package one level up isn't importable.
Nothing installs the package either (`requirements.txt` has no `-e .`).

## Impact

The scheduled **AutoMem Backup** workflow
(`.github/workflows/backup.yml`) runs exactly this command and has been
**failing on every run since the refactor landed (2026-06-06)** — no
automated FalkorDB/Qdrant backup has succeeded for ~2 days. The failure
is identical across all runs (`ModuleNotFoundError: No module named
'automem'`).

## Fix

Prepend the repo root to `sys.path` before the package import, using the
**same idiom already in `scripts/backfill_tag_prefixes.py`**:

```python
PROJECT_ROOT = Path(__file__).resolve().parents[1]
if str(PROJECT_ROOT) not in sys.path:
    sys.path.insert(0, str(PROJECT_ROOT))
```

This fixes the documented usage (`python scripts/backup_automem.py`),
the CI workflow, Railway cron, and manual/local runs in one place — no
workflow change required.

## Verification

- `python -m py_compile scripts/backup_automem.py` ✅
- `PROJECT_ROOT` resolves to the repo root containing
`automem/__init__.py` and `automem/backup.py` ✅
- `black --check` and `isort --check-only --profile black` both pass ✅
(and E402 is not in `.flake8`'s `select` set)
- The equivalent fix (adding the repo root to the import path) was
applied to a downstream wrapper of this workflow and a manual run
completed end-to-end: `✅ Backup completed successfully` →
`"s3_uploaded": true`.
jack-arturo added a commit that referenced this pull request Jun 26, 2026
@jack-arturo
chore(main): release 0.16.0 (#154)
8c67c16 
🤖 I have created a release *beep* *boop*
---


##
[0.16.0](v0.15.2...v0.16.0)
(2026-06-26)


### Features

* **api:** add admin backup endpoint
([#162](#162))
([8b1f264](8b1f264))
* **api:** support bulk memory associations
([1221e36](1221e36))
* **api:** support bulk memory associations
([#198](#198))
([28eb916](28eb916))
* **benchmarks:** LongMemEval failure-mode diagnosis harness + judge
quota preflight
([#183](#183))
([f99bece](f99bece))
* **consolidation:** expose cluster threshold and min size as env vars
([#163](#163))
([7e731f3](7e731f3))
* **enrichment:** expose classification fallback-rate metrics in
/enrichment/status
([#188](#188))
([0b522a9](0b522a9))
* **entity:** harden identity cleanup and repair tooling
([#176](#176))
([827dfbc](827dfbc))
* **eval:** recall-quality optimization harness — lab foundation +
design ([#197](#197))
([431433e](431433e))
* **graph:** support unbounded visualizer snapshots
([#141](#141))
([c730128](c730128))
* **lab:** add aged labelled distractor injection
([cc5d546](cc5d546))
* **lab:** add config_complexity simplicity metric
([dfb10d9](dfb10d9))
* **lab:** add distractor_rate_at_k precision guardrail metric
([872eab2](872eab2))
* **lab:** add lab_corpus with parameterized recall
([5e1e071](5e1e071))
* **lab:** add pick_winner scorecard decision rule
([3187eac](3187eac))
* **lab:** add real consolidation pass helper
([48a7d4a](48a7d4a))
* **lab:** isolate production clone restores
([#171](#171))
([aef90c0](aef90c0))
* **lab:** wire scorecard, distractors, recall params, consolidation
into runner
([589ec30](589ec30))
* **recall:** add metadata sidecar search
([#177](#177))
([4e7956e](4e7956e))
* **recall:** add state_mode=current|history recall alias
([#173](#173))
([b1df86c](b1df86c))
* **recall:** cap tag-score denominator to fix query-length bias
([#193](#193))
([cefa516](cefa516))
* **recall:** date-aware ranking + latest-fact selection
([#158](#158),
[#159](#159))
([#187](#187))
([a6ed945](a6ed945))
* **recall:** make recency decay window and curve configurable
([#182](#182))
([dbb933f](dbb933f))
* **recall:** ranking release — recency config, tag-score cap, relevance
gate, date-aware ranking
([#182](#182),
[#193](#193),
[#186](#186),
[#187](#187),
[#183](#183),
[#184](#184),
[#188](#188))
([#194](#194))
([337fe98](337fe98))
* **scripts:** safer reclassify_with_llm.py with provider flags +
tighter prompt
([#164](#164))
([a742602](a742602))


### Bug Fixes

* **api:** address copilot review on PR
[#198](#198)
([0466a1e](0466a1e))
* **api:** handle grouped association write failures
([cd93df9](cd93df9))
* **backup:** make backup_automem.py runnable as `python
scripts/backup_automem.py`
([#175](#175))
([edd9742](edd9742))
* **benchmarks:** add publication verification bundle
([#166](#166))
([420d721](420d721))
* **consolidation:** skip eager first tick at startup to avoid FalkorDB
load race
([#165](#165))
([1b812cf](1b812cf))
* **docs:** keep dispatch payload arrays stable
([df6e9e8](df6e9e8))
* **embedding:** fall back to per-item real embeddings before
placeholders in batch path
([#189](#189))
([6e9c62c](6e9c62c))
* **entity:** restore person-shape exemption on the slug validation path
([#179](#179))
([5e29960](5e29960))
* **entity:** stop validator over-rejecting real people, code tools, and
event categories
([#178](#178))
([193b730](193b730))
* **lab:** address copilot review on PR
[#197](#197)
([45f80d6](45f80d6))
* **lab:** align scorecard key contract (build_scorecard -&gt;
pick_winner)
([7d91530](7d91530))
* **mcp-sse:** decouple /health liveness from upstream readiness
([#151](#151))
([5bcfb8b](5bcfb8b))
* **mcp:** cap association failure summary
([ea4e08f](ea4e08f))
* **mcp:** surface stored metadata and updated_at in detailed recall
format ([#184](#184))
([230416e](230416e))
* **recall:** address copilot review on PR
[#194](#194)
([50b1647](50b1647))
* **recall:** canonicalize / and : separators in context_tag matching
([3afd9d3](3afd9d3))
* **recall:** canonicalize / and : separators in context_tag matching
([#203](#203))
([ba5e9ff](ba5e9ff))
* **recall:** gate query-independent scoring on topical evidence within
tag scope
([#130](#130))
([#186](#186))
([c11b594](c11b594))
* **recall:** hydrate semantic recall summaries
([#192](#192))
([76e845d](76e845d))
* **recall:** normalize graph keyword scores into the 0-1 component
range ([#191](#191))
([3653ddf](3653ddf))
* **recall:** respect current memory state
([#170](#170))
([ed36b98](ed36b98)),
closes [#169](#169)
[#158](#158)
[#159](#159)
* **scripts:** add sys.path guard to reembed_embeddings.py
([d333cf0](d333cf0))


### Documentation

* add scripts catalog, recall-quality-lab guide, and 0.16.0 migrations
([f20c664](f20c664))
* **bench:** log full judged 500q LongMemEval ship-config run with churn
attribution
([41bf8d0](41bf8d0))
* **eval:** Plan A — lab metric foundation (TDD, 9 tasks)
([0087dda](0087dda))
* **eval:** Plan B — parallel matrix harness (TDD, 9 tasks)
([c8ddfb2](c8ddfb2))
* **evals:** mark Memora/FAMA/WRIT lifecycle diagnostics as
diagnostic-only
([#174](#174))
([e8a3285](e8a3285))
* **eval:** spec for recall-quality optimization harness
([b1a1995](b1a1995))
* fix stale claims and document gated flags for 0.16.0
([b152d64](b152d64))
* note develop-branch contribution policy in README
([ccf02dd](ccf02dd))
* **positioning:** add scout reference
([#168](#168))
([922d23b](922d23b))
* refresh benchmark currency for the neutral AMB run and prune stale
archive docs
([3ff95bd](3ff95bd))
* refresh benchmark currency for the neutral AMB run and prune stale
archive docs
([#204](#204))
([89c30e0](89c30e0))
* refresh README and benchmark guidance
([#157](#157))
([bba31cc](bba31cc))
* **runtime:** align Docker viewer paths and setup guidance
([#155](#155))
([bbda79b](bbda79b))
* scripts catalog, recall quality lab guide, and 0.16.0 migration
runbook ([#199](#199))
([f190ae5](f190ae5))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jack-arturo