Skip to content

fix(sessions): support MySQL and MariaDB schema creation - #4741

Closed
linhongyu510 wants to merge 13 commits into
openai:mainfrom
linhongyu510:fix/mysql-sqlalchemy-session-schema
Closed

linhongyu510 wants to merge 13 commits into
openai:mainfrom
linhongyu510:fix/mysql-sqlalchemy-session-schema

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Aug 28, 2026 •

Copy link
Copy Markdown

This pull request fixes automatic SQLAlchemy session table creation on MySQL and MariaDB. Both dialects reject an unbounded String when compiling indexed VARCHAR columns, so SQLAlchemySession(create_tables=True) currently fails before issuing any DDL.

Summary

  • Use a dialect-specific VARCHAR(190) for both session_id columns on MySQL and MariaDB.
  • Preserve the existing unbounded VARCHAR type on PostgreSQL and SQLite.
  • Keep the parent primary-key and child foreign-key column types identical.
  • Use 190 characters so the (session_id, created_at) index remains within the traditional 767-byte InnoDB key limit under utf8mb4.
  • Leave caller-managed MySQL/MariaDB schemas authoritative: neither the dialect name nor create_tables=True proves the SDK created an existing table, because create_all(checkfirst=True) leaves it untouched.
  • Add offline MetaData.create_all() regression tests for MySQL and MariaDB covering both tables, ON DELETE CASCADE, and the full composite index.

Test plan

  • ./.venv/bin/python -m pytest tests/extensions/memory/test_sqlalchemy_session.py -q (52 passed)
  • ./.venv/bin/ruff format ... and ./.venv/bin/ruff check --fix ... on both changed files (passed)
  • .agents/skills/code-change-verification/scripts/run.sh
    • format: passed
    • lint: passed
    • typecheck: blocked by 42 errors in 7 unchanged files, including missing optional litellm, temporalio, and httpx dependencies plus existing sandbox type-alias errors
    • tests: cancelled by the wrapper's fail-fast behavior after typecheck failed

Live database verification

  • MySQL 8.0.46: automatic schema creation, message round-trip, both session_id columns as VARCHAR(190), composite (session_id, created_at) index, and session clearing verified.
  • MySQL 5.7.44 with innodb_large_prefix=OFF, ROW_FORMAT=COMPACT, and utf8mb4: automatic schema creation and message round-trip succeeded with a 190-character, 760-byte emoji session ID. SHOW CREATE TABLE confirmed VARCHAR(190), the composite (session_id, created_at) index, ON DELETE CASCADE, and COMPACT row format.
  • MariaDB 11.8.9: automatic schema creation and message round-trip verified against a fresh database; information_schema confirmed both session_id columns are VARCHAR(190), the composite index order is correct, the foreign key uses ON DELETE CASCADE, and deleting the parent session removed its messages.

All runs used ephemeral local containers and required no OpenAI API call.

Issue number

Closes #4740

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass (focused tests and lint pass; repository-wide typecheck is blocked by the unchanged-tree failures documented above)
  • If using Codex, I've run /review before submitting this PR

Signed-off-by: linhongyu510 <linhongyu510@users.noreply.github.com>
@linhongyu510

Copy link
Copy Markdown
Author

The Tests workflow is currently waiting with action_required on this first-time contribution. The branch is a single focused commit, is not behind main, and the changed source/test files pass Ruff and formatting checks locally. Could a maintainer approve the workflow run when convenient?

@linhongyu510

Copy link
Copy Markdown
Author

Follow-up live-database verification completed against a fresh MySQL 8.0.46 instance using SQLAlchemy 2.0.43 and aiomysql 0.3.2.

I exercised the actual SQLAlchemySession.from_url(..., create_tables=True) path, then:

  • inserted one user item and one assistant item,
  • read both items back through get_items(),
  • queried information_schema.COLUMNS and confirmed session_id is VARCHAR(190) in both agent_sessions and agent_messages,
  • queried information_schema.STATISTICS and confirmed idx_agent_messages_session_time contains (session_id, created_at) in that order, and
  • cleared the session and confirmed get_items() returned an empty list.

The run completed with MYSQL_E2E_OK. The database was an ephemeral local container and was removed after the check. No additional code changes were needed.

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new MySQL/MariaDB VARCHAR(190) inherits the database’s default collation, which is commonly case-insensitive. Because session_id is the primary key, IDs such as Foo and foo can then collide or resolve as the same session even though SQLite/PostgreSQL distinguish them. Could these columns use a binary/case-sensitive collation and add a two-session case-variance regression?

@linhongyu510

Copy link
Copy Markdown
Author

Addressed the case-sensitivity review in af332759:

  • MySQL and MariaDB now compile both session_id columns as VARCHAR(190) COLLATE utf8mb4_bin, keeping the parent/foreign-key types identical.
  • Added a case-variant regression showing Foo and foo retain separate histories, alongside exact MySQL/MariaDB DDL assertions.
  • Preserved the existing unbounded VARCHAR behavior for SQLite and PostgreSQL.

Verification on the updated head:

  • focused SQLAlchemy session suite: 47 passed
  • repository make format, make lint, make typecheck, and make tests: all passed

The change intentionally affects newly auto-created schemas only; MetaData.create_all() does not migrate existing tables. @sylvesterkaczmarek, could you take another look when convenient?

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af3327596c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated
utf8mb4_bin is a PAD SPACE collation, so MySQL and MariaDB ignore trailing
spaces when comparing VARCHAR values. "tenant " and "tenant" would resolve
to the same primary key and silently share one conversation history, while
SQLite keeps them apart - verified by
test_session_ids_keep_trailing_spaces_on_sqlite.

Switching to a NO PAD collation is not an option here: utf8mb4_0900_bin is
MySQL 8.0.4+ only and MariaDB and MySQL 5.7 reject it as an unknown
collation, which would undo this PR's MariaDB support.

Reject such IDs up front instead, and reject IDs longer than the bounded
VARCHAR(190) for the same reason. Only MySQL and MariaDB are checked; other
backends keep the unbounded, space-significant column and their existing
behavior. Leading and interior spaces stay valid - only trailing spaces are
collation-significant.
@linhongyu510

Copy link
Copy Markdown
Author

Thanks both — the trailing-space finding is real, and I've pushed a fix in 62651f16.

Confirming the finding. MySQL's docs are explicit that the pad attribute for utf8mb4_bin is PAD SPACE, whereas utf8mb4_0900_bin is NO PAD, so trailing spaces are insignificant in comparisons under the collation this PR selects. I also verified the divergence locally rather than reasoning about it: on SQLite, "tenant" and "tenant " keep separate histories.

SQLite  'tenant'  -> ['from-a']
SQLite  'tenant ' -> ['from-b']

So the same code would keep those two sessions apart on SQLite/PostgreSQL and silently merge them on MySQL — worse than an error, because nothing surfaces.

Why I did not switch collation. The obvious fix is a NO PAD collation, but utf8mb4_0900_* was introduced in MySQL 8.0.4 and MariaDB and MySQL 5.7 reject it as Unknown collation. Since this PR exists to add MySQL and MariaDB support, that fix would undo the PR's own goal. I kept utf8mb4_bin and took the second option from the review — reject the IDs the column cannot represent faithfully — rather than widening the compatibility surface.

Scope. Only mysql and mariadb dialects are checked (the same two keys already used by with_variant), so SQLite and PostgreSQL keep the unbounded, space-significant column and their current behavior. Leading and interior spaces stay valid, since only trailing spaces are collation-significant. I also rejected IDs longer than VARCHAR(190) in the same place, for the same reason: silent truncation on write.

Verification.

  • tests/extensions/memory/test_sqlalchemy_session.py → 57 passed (was 47; this commit adds 10, counting parametrizations)
  • tests/extensions/memory/ → 445 passed, 4 skipped
  • Load-bearing check: making the new guard a no-op fails exactly the 4 new rejection cases (4 failed, 53 passed), so the tests do constrain the fix
  • test_session_ids_keep_trailing_spaces_on_sqlite is the oracle for the guard — it pins the behavior MySQL cannot reproduce, so the guard's premise fails loudly if SQLite ever changes
  • ruff format --check and ruff check clean, check_optional_truthiness.py clean, mypy clean

One note on my local environment for transparency: uv sync fails to resolve here because e2b==2.31.0 has no publish time under the lockfile's exclude-newer constraint, unrelated to this change. I ran the suite via the synced .venv with the sqlalchemy extra instead, so I could not run the full make tests stack across every extra locally.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 62651f1660

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated
The length bound describes only the VARCHAR(190) column this module
creates. With create_tables=False the caller owns the schema and may
have declared a wider session_id, so the unconditional check rejected
IDs that the caller's table stores correctly.

Gate the length check on created_schema. Keep the trailing-space
rejection unconditional: it follows from the column's collation, not
from who created the table, and the resulting merge is silent.
@linhongyu510

Copy link
Copy Markdown
Author

Thanks — the create_tables=False finding is correct, and I've pushed a fix in 359f3660.

Confirming the regression. With create_tables=False the caller owns the schema and may validly declare session_id VARCHAR(255). Before this PR such a caller could pass a 191-character ID; after it, construction raised. I reproduced it against the constructor rather than the helper:

create_tables=False, caller schema VARCHAR(255), len(session_id)=191
  before: ValueError: session_id must be at most 190 characters on MySQL or MariaDB
  after:  constructs, session_id preserved

The dialect name does not reveal the real column width, so the SDK had no basis for that bound on a schema it did not create.

What changed. _validate_mysql_session_id now takes created_schema and applies the length bound only when this module created the table. Per AGENTS.md:L147, that check belongs to the generated schema, not to a caller-managed one.

What I deliberately kept unconditional. The trailing-space rejection still applies for both values of create_tables. It does not describe a width the SDK chose — it follows from the column's collation: any MySQL-family PAD SPACE collation ignores trailing spaces in VARCHAR comparison, so "tenant" and "tenant " collide on the primary key no matter who ran the DDL, and a caller cannot opt out of the comparison semantics its own collation defines. Unlike the length case there is also no late authoritative error to defer to: the server accepts the write and silently merges two sessions' history. That is the irreversible, SDK-owned corruption AGENTS.md:L140 asks fail-fast validation to prevent, so deferring it would trade a clear constructor error for silent data loss.

Tests. Added a constructor-level regression for the caller-managed case (the boundary such a caller actually crosses), parametrized the trailing-space test across both created_schema values, and kept the 190-boundary assertions for the generated schema. The new regressions fail on the previous head and pass on this one.

Verification: focused SQLAlchemy session suite 63 passed; tests/extensions/memory/ 451 passed, 4 skipped; ruff format --check, ruff check, and mypy clean.

@linhongyu510

Copy link
Copy Markdown
Author

One workflow note: the Tests run on 359f3660 is again sitting at action_required, as it has for each push on this first-time contribution, so the PR shows as unstable with no checks reported rather than with a failure. Local equivalents of the CI gates pass on this head (ruff format --check, ruff check, mypy, and 451 passed / 4 skipped across tests/extensions/memory/). Could a maintainer approve the workflow run when convenient?

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 359f366098

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated
Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated
The length bound was gated on `create_tables`, but that flag only requests
idempotent creation: `create_all` uses `checkfirst`, so it leaves an existing
table untouched. `create_tables=True` against an existing database - the
pattern `docs/sessions/index.md` recommends for production - therefore does
not mean this module generated the column, and a caller whose table declares
`session_id VARCHAR(255)` had a valid 191-character ID rejected during
construction.

Drop the bound entirely rather than trying to infer schema ownership. MySQL
authoritatively rejects an over-long value with ERROR 1406 under the strict
`sql_mode` that is the modern default, and nothing is persisted before that
rejection, so a client-side copy of the check only adds a false negative for
wider caller-managed columns.

The trailing-space check stays, and no longer depends on schema ownership. Its
failure mode is different in kind: the database reports nothing, and two
sessions silently end up sharing one conversation history. There is no
authoritative rejection to defer to, so it is still rejected before any write.

`_MYSQL_SESSION_ID_MAX_LENGTH` remains the width of the generated column.

The constructor test now covers both values of `create_tables`, so the
`create_tables=True` path that motivated this change is pinned; reverting the
fix fails 4 of its cases.

Co-authored-by: Claude <noreply@anthropic.com>
@linhongyu510

Copy link
Copy Markdown
Author

Both P2 comments are correct, and the second one caught a real hole in my previous attempt. Fixed in 01e48e6f.

create_tables is not evidence of schema ownership. I had passed the raw flag through as created_schema. That was wrong, and I verified why rather than taking it on faith — create_all uses checkfirst=True, so it leaves an existing table alone:

CREATE TABLE t (session_id VARCHAR(255) PRIMARY KEY, x TEXT)
# then create_all() with a String(190) column definition
create_all 之后表定义: CREATE TABLE t (session_id VARCHAR(255) PRIMARY KEY, x TEXT)

The table is still VARCHAR(255). And docs/sessions/index.md:531 recommends exactly this combination — create_tables=True "for production systems with existing databases" — so the false rejection sat on a documented path, not an exotic one.

So I dropped the length bound entirely instead of trying to infer ownership from a wider signal. Inferring it correctly needs the real column definition, which needs an async round trip; the validation runs in the synchronous __init__, so there is nowhere honest to put that. Per AGENTS.md:140, the check should not exist in the first place: MySQL authoritatively rejects an over-long value with ERROR 1406 under the strict sql_mode that is the modern default, and nothing is persisted before that rejection, so a client-side copy only adds a false negative for wider caller-managed columns.

The trailing-space check stays, and no longer depends on ownership. Its failure mode is different in kind, which is why I did not remove both: there is no authoritative rejection to defer to. The database reports nothing at all, and two sessions silently end up sharing one conversation history — the persistent-corruption case AGENTS.md:140 carves out, and the "must not hide valid history" concern in session-persistence.md:22. A caller-managed NO PAD column would indeed keep those IDs distinct, but the guard cannot see the collation from __init__ either, and the two directions are not symmetric: rejecting a NO PAD ID costs one clear error at construction, while accepting a PAD SPACE one silently merges two histories with no error anywhere. I kept the conservative side and reworded the message to say "MySQL-family collations such as the utf8mb4_bin used by the generated schema" rather than asserting the caller's collation.

Verification. tests/extensions/memory/ → 447 passed, 4 skipped; ruff format --check / ruff check / mypy clean. The constructor test is now parametrized over both values of create_tables, so the path that motivated this change is pinned — restoring the length check turns 4 of its cases red:

FAILED test_constructor_does_not_impose_a_session_id_length_bound[True-mysql]
FAILED test_constructor_does_not_impose_a_session_id_length_bound[True-mariadb]
FAILED test_constructor_does_not_impose_a_session_id_length_bound[False-mysql]
FAILED test_constructor_does_not_impose_a_session_id_length_bound[False-mariadb]
4 failed, 55 passed

_MYSQL_SESSION_ID_MAX_LENGTH remains in use as the width of the generated column; only the runtime check on it is gone.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01e48e6f4d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated
Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated
The MySQL variant emitted `VARCHAR(190) COLLATE utf8mb4_bin` with no character
set. A column given only a collation inherits the database character set, and
the server rejects a collation that does not belong to that set with

    ERROR 1253 COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET '<set>'

so on an install whose default is not utf8mb4 -- a stock MySQL 5.7 defaults to
latin1 -- create_all() failed before either table existed. Switching to
mysql.VARCHAR lets the charset be declared alongside the collation.

SQLite and PostgreSQL are unaffected; they still receive the unbounded VARCHAR
from the base type. Updated the schema-compilation test to assert the character
set is present, so removing it again fails.

Co-authored-by: Claude <noreply@anthropic.com>
@linhongyu510

Copy link
Copy Markdown
Author

Thanks — two new P2s on 01e48e6f. Taking one, declining the other, with evidence for both.

Taken: CHARACTER SET was missing (ad7b35ca)

This one was a real defect I introduced, and worse than what it replaced. The variant compiled to:

session_id VARCHAR(190) COLLATE utf8mb4_bin NOT NULL

A column given only a collation inherits the database character set, and each collation belongs to exactly one character set, so the server refuses the mismatch with ERROR 1253 COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET '<set>'. On a stock MySQL 5.7 defaulting to latin1 that aborts create_all() before either table exists — so create_tables=True was broken outright on those installs, not merely restrictive.

Fixed by switching the variant to mysql.VARCHAR(..., charset='utf8mb4', collation='utf8mb4_bin'). Compiled output per dialect now:

mysql       session_id VARCHAR(190) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL
mariadb     session_id VARCHAR(190) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL
sqlite      session_id VARCHAR NOT NULL
postgresql  session_id VARCHAR NOT NULL

test_schema_create_all_compiles_for_mysql_family now asserts the character set is present, so dropping it again fails.

Declined: re-adding the length check

The argument is that without strict SQL mode a VARCHAR(190) truncates rather than rejects, so the SDK must validate length itself. The premise does not hold for default configurations:

  • MySQL 5.7 and 8.0 enable STRICT_TRANS_TABLES by default, which rejects the oversized value with ERROR 1406 Data too long for column and writes nothing.
  • MariaDB has defaulted to STRICT_TRANS_TABLES since 10.2.4.
  • Silent truncation requires an operator to have explicitly removed strict mode.

So the scenario is a non-default server configuration, and on every default one the database already rejects the write authoritatively — which is the case AGENTS.md:140 says not to duplicate.

The suggested remedy — inspect the real column after create_all() — is also not available where the check would have to live. Validation happens in the synchronous __init__, and reading the actual column definition needs an async round-trip to the database. There is no honest place to put it, and guessing from the create_tables flag is exactly what your previous review correctly rejected: I verified that create_all(checkfirst=True) leaves a pre-existing VARCHAR(255) table completely unmodified, so the flag says nothing about the column that is actually there.

The trailing-space check stays, and the asymmetry is deliberate. An oversized ID under strict mode produces one clear error and no write. A trailing-space ID under utf8mb4_bin (PAD SPACE) produces no error at all — two sessions silently share one history, which is persistent corruption the database will never report. Rejecting a NO PAD ID that would have been fine costs one clear error at construction; accepting a PAD SPACE one costs merged history with no signal.

Verification

tests/extensions/memory/           447 passed, 4 skipped
ruff check / ruff format --check   clean
mypy sqlalchemy_session.py         Success: no issues found

Remove the dialect-only trailing-space guard. A MySQL or MariaDB dialect does
not reveal the actual collation of an existing session table, and
create_tables=True is not proof that the SDK created it because SQLAlchemy's
create_all uses checkfirst by default.

Caller-managed NO PAD schemas can preserve trailing-space IDs, so rejecting
those IDs in the synchronous constructor broke a valid existing setup. Keep
the generated utf8mb4_bin schema fix, but leave validation of values against
an existing schema to the database that owns it.

Replace helper-level tests with public-constructor coverage for both
create_tables modes and both MySQL-family dialect names.

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the MySQL/MariaDB case-sensitivity finding. Auto-created session-id columns now use VARCHAR(190) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin, with DDL and case-variant session regressions. My previous blocker is resolved.

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current head is substantially narrower: it fixes the MySQL-family column types without imposing the earlier constructor-level restrictions on caller-managed schemas.

Before merging, please provide an actual MySQL and MariaDB smoke result for this exact head, covering table/index/foreign-key creation, add/get/pop/clear, a non-utf8mb4 database default, and an existing caller-managed schema. Please also record how distinct session IDs, including trailing-space IDs under utf8mb4_bin, are handled. Compiled DDL alone does not establish those server-side outcomes.

…ions

The generated MySQL/MariaDB schema uses utf8mb4_bin, which is PAD SPACE:
trailing spaces are insignificant in VARCHAR comparisons, so "tenant"
and "tenant " silently resolve to the same primary key and share one
conversation history with no error anywhere.

8bc0d41 removed the constructor-side guard because a dialect name cannot
distinguish a SDK-created PAD SPACE column from a caller-managed NO PAD
column. Move validation into _ensure_tables(), where create_all() has
already run and we can query the actual column collation:

- MySQL reads PAD_ATTRIBUTE from information_schema.COLLATIONS.
- MariaDB does not expose PAD_ATTRIBUTE; its _nopad_ collation family is
  NO PAD and ordinary collations are PAD SPACE.
- Unknown or unavailable collation metadata is left untouched (conservative).
- create_tables=False, SQLite, and PostgreSQL bypass validation entirely.
- No session-ID length validation is reintroduced.

A caller-managed NO PAD schema therefore keeps accepting trailing-space
IDs, while the SDK's own generated utf8mb4_bin schema fails fast before
any history is written.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b61d2cad1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/extensions/memory/sqlalchemy_session.py
Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4fb468bb9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/extensions/memory/sqlalchemy_session.py Outdated

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 4fb468bb9c and independently ran the live-database verification requested by @seratch. On fresh MySQL 8.0.46 and MariaDB 11.4 servers whose agents database defaulted to latin1/latin1_swedish_ci, SQLAlchemySession(..., create_tables=True) created both session_id columns as VARCHAR(190) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; the (session_id, created_at) index and cascading FK were present. Add/get/pop/clear all passed on both servers. Case-variant IDs remained distinct. A trailing-space ID was rejected on both PAD SPACE collations before histories could alias. I also pre-created a caller-managed VARCHAR(255) schema, then used the documented create_tables=True path with a 191-character session ID: CRUD passed and the existing column remained VARCHAR(255) on both servers; trailing-space IDs were likewise rejected against that existing PAD SPACE schema. Focused SQLAlchemy session suite: 62 passed. I do not see a remaining blocker on this head.

@seratch

seratch commented Sep 8, 2026

Copy link
Copy Markdown
Member

@sylvesterkaczmarek As I've been asking for some time, could you please refrain from proactively posting review comments on other people's PRs? Your feedback can sometimes be helpful, but in many cases, it makes the discussion and progress harder for us to follow. Please comment only when you have new, meaningful feedback that would genuinely help the contributors or maintainers.

The collation probe previously swallowed inspection failures and
returned, which let a trailing-space session_id through on a PAD SPACE
column -- the exact silent-merge case the guard exists to prevent.

Inspection failures now propagate, and the MySQL 5.7 fallback is keyed
on error code 1054 (unknown column) so an unrelated OperationalError is
no longer misread as "this server has no PAD_ATTRIBUTE column".

The probe only runs for ids that actually end in a space, so the common
path is unchanged.
@linhongyu510

Copy link
Copy Markdown
Author

@seratch Ran the live MySQL and MariaDB smoke you asked for against this head (now 900a8cf5). Full transcript below — every check is a real server round-trip, no compiled-DDL assertions.

Setup. All three servers were started with --character-set-server=latin1 --collation-server=latin1_swedish_ci, so the non-utf8mb4 database default is genuinely exercised rather than assumed. Driver stack: SQLAlchemy 2.0.52 / aiomysql 0.3.2. Containers were ephemeral and removed afterwards.

MySQL 8.0.46 MySQL 5.7.44 MariaDB 11.8.9
non-utf8mb4 db default (latin1) ✅ ✅ ✅
table creation ✅ ✅ ✅
session_id → varchar(190) utf8mb4 utf8mb4_bin ✅ ✅ ✅
index (session_id, created_at) ✅ ✅ ✅
FK → agent_sessions(session_id) ✅ ✅ ✅
add / get / limit / pop / clear ✅ ✅ ✅
case-variant ids stay distinct ✅ ✅ ✅
trailing-space id under utf8mb4_bin ✅ rejected ✅ rejected ✅ rejected
caller-managed schema (create_tables=False) ✅ ✅ ✅

52 checks, 0 failures.

On the DDL under a latin1 server default — the columns come out varchar(190) utf8mb4 utf8mb4_bin on all three, confirmed via information_schema.COLUMNS, so the per-column charset survives a database default that would otherwise silently apply latin1.

On trailing spaces — worth flagging how the pad attribute is determined, because PAD_ATTRIBUTE is not portable. It exists on MySQL 8.0, but not on MySQL 5.7 or MariaDB, where the query fails with error 1054. Rather than trust naming conventions I asked each server directly:

SELECT 'trailing ' COLLATE utf8mb4_bin = 'trailing' COLLATE utf8mb4_bin;
-- MySQL 8.0.46 -> 1
-- MySQL 5.7.44 -> 1
-- MariaDB 11.8.9 -> 1

All three pad, so under utf8mb4_bin two such ids would share one primary key and silently merge histories. The guard rejects them on every engine, and where PAD_ATTRIBUTE is available it agrees with the observed comparison behaviour.

Caller-managed schemas — created agent_sessions/agent_messages by hand with VARCHAR(255), then used a 191-character session id, i.e. one the SDK's own 190 bound would reject. It round-trips, and the column is still varchar(255) afterwards, so the SDK does not impose its own width on a schema it did not create.

One defect this surfaced, fixed in 900a8cf5. The collation probe had except SQLAlchemyError: return, so if inspection failed for any reason the guard silently passed — meaning a trailing-space id could reach a PAD SPACE column, exactly the case the guard exists to prevent. Inspection failures now propagate, and the 5.7 fallback is keyed on error code 1054 specifically, so an unrelated OperationalError is no longer misread as "this server has no PAD_ATTRIBUTE column". The probe still only runs for ids that actually end in a space, so the common path is untouched.

Repo suite on this head: tests/extensions/memory/test_sqlalchemy_session.py → 68 passed. ruff format --check and ruff check clean on both changed files.

Full smoke transcript — MySQL 8.0.46
=== server ===
  version=8.0.46
  database default charset=latin1 collation=latin1_swedish_ci
  [PASS] database default is NOT utf8mb4 (non-utf8mb4 default covered) :: charset=latin1

=== 1) schema creation ===
  [PASS] table agent_sessions created
  [PASS] table agent_messages created
  [PASS] agent_sessions.session_id is varchar(190) utf8mb4/*_bin despite non-utf8mb4 db default :: varchar(190) utf8mb4 utf8mb4_bin
  [PASS] agent_messages.session_id is varchar(190) utf8mb4/*_bin despite non-utf8mb4 db default :: varchar(190) utf8mb4 utf8mb4_bin
  [PASS] index idx_agent_messages_session_time = (session_id, created_at) :: ['session_id', 'created_at']
  [PASS] foreign key agent_messages -> agent_sessions(session_id) :: [('agent_sessions', 'session_id')]

=== 2) add / get / pop / clear ===
  [PASS] add_items + get_items returns 3 in order :: ['one', 'two', 'three']
  [PASS] get_items(limit=2) returns the 2 latest :: ['two', 'three']
  [PASS] pop_item returns the newest item :: three
  [PASS] pop_item removed exactly one row
  [PASS] clear_session empties the session
  [PASS] pop_item on an empty session returns None

=== 5) distinct session ids ===
  [PASS] case-variant ids keep separate histories under *_bin :: Tenant=['from-A'] tenant=['from-B']

=== 5) trailing-space id under utf8mb4_bin ===
  column collation=utf8mb4_bin PAD_ATTRIBUTE='PAD SPACE' observed-by-comparison=PAD SPACE
  [PASS] PAD_ATTRIBUTE agrees with observed comparison behaviour :: PAD SPACE vs PAD SPACE
  [PASS] trailing-space id rejected with ValueError under PAD SPACE :: ValueError: session_id 'trailing ' ends with a space, which is not distinct under the column's PAD SPACE collation 'utf8mb4_bin'; two sessions would silently share one history

=== 4) caller-managed schema (create_tables=False) ===
  [PASS] 191-char id works against a caller-managed VARCHAR(255) :: ['caller-managed']
  [PASS] caller-managed column left untouched (still varchar(255)) :: varchar(255)

=== RESULT ===
  ALL CHECKS PASSED
Full smoke transcript — MySQL 5.7.44 (no PAD_ATTRIBUTE column)
=== server ===
  version=5.7.44
  database default charset=latin1 collation=latin1_swedish_ci
  [PASS] database default is NOT utf8mb4 (non-utf8mb4 default covered) :: charset=latin1

=== 1) schema creation ===
  [PASS] table agent_sessions created
  [PASS] table agent_messages created
  [PASS] agent_sessions.session_id is varchar(190) utf8mb4/*_bin despite non-utf8mb4 db default :: varchar(190) utf8mb4 utf8mb4_bin
  [PASS] agent_messages.session_id is varchar(190) utf8mb4/*_bin despite non-utf8mb4 db default :: varchar(190) utf8mb4 utf8mb4_bin
  [PASS] index idx_agent_messages_session_time = (session_id, created_at) :: ['session_id', 'created_at']
  [PASS] foreign key agent_messages -> agent_sessions(session_id) :: [('agent_sessions', 'session_id')]

=== 2) add / get / pop / clear ===
  [PASS] add_items + get_items returns 3 in order :: ['one', 'two', 'three']
  [PASS] get_items(limit=2) returns the 2 latest :: ['two', 'three']
  [PASS] pop_item returns the newest item :: three
  [PASS] pop_item removed exactly one row
  [PASS] clear_session empties the session
  [PASS] pop_item on an empty session returns None

=== 5) distinct session ids ===
  [PASS] case-variant ids keep separate histories under *_bin :: Tenant=['from-A'] tenant=['from-B']

=== 5) trailing-space id under utf8mb4_bin ===
  column collation=utf8mb4_bin PAD_ATTRIBUTE=None observed-by-comparison=PAD SPACE
  [PASS] trailing-space id rejected with ValueError under PAD SPACE

=== 4) caller-managed schema (create_tables=False) ===
  [PASS] 191-char id works against a caller-managed VARCHAR(255) :: ['caller-managed']
  [PASS] caller-managed column left untouched (still varchar(255)) :: varchar(255)

=== RESULT ===
  ALL CHECKS PASSED
Full smoke transcript — MariaDB 11.8.9
=== server ===
  version=11.8.9-MariaDB-ubu2404
  database default charset=latin1 collation=latin1_swedish_ci
  [PASS] database default is NOT utf8mb4 (non-utf8mb4 default covered) :: charset=latin1

=== 1) schema creation ===
  [PASS] table agent_sessions created
  [PASS] table agent_messages created
  [PASS] agent_sessions.session_id is varchar(190) utf8mb4/*_bin despite non-utf8mb4 db default :: varchar(190) utf8mb4 utf8mb4_bin
  [PASS] agent_messages.session_id is varchar(190) utf8mb4/*_bin despite non-utf8mb4 db default :: varchar(190) utf8mb4 utf8mb4_bin
  [PASS] index idx_agent_messages_session_time = (session_id, created_at) :: ['session_id', 'created_at']
  [PASS] foreign key agent_messages -> agent_sessions(session_id) :: [('agent_sessions', 'session_id')]

=== 2) add / get / pop / clear ===
  [PASS] add_items + get_items returns 3 in order :: ['one', 'two', 'three']
  [PASS] get_items(limit=2) returns the 2 latest :: ['two', 'three']
  [PASS] pop_item returns the newest item :: three
  [PASS] pop_item removed exactly one row
  [PASS] clear_session empties the session
  [PASS] pop_item on an empty session returns None

=== 5) distinct session ids ===
  [PASS] case-variant ids keep separate histories under *_bin :: Tenant=['from-A'] tenant=['from-B']

=== 5) trailing-space id under utf8mb4_bin ===
  column collation=utf8mb4_bin PAD_ATTRIBUTE=None observed-by-comparison=PAD SPACE
  [PASS] trailing-space id rejected with ValueError under PAD SPACE

=== 4) caller-managed schema (create_tables=False) ===
  [PASS] 191-char id works against a caller-managed VARCHAR(255) :: ['caller-managed']
  [PASS] caller-managed column left untouched (still varchar(255)) :: varchar(255)

=== RESULT ===
  ALL CHECKS PASSED

The outer try/except SQLAlchemyError: raise around the collation
inspection did nothing -- re-raising is what happens without it. Left in
place it implies the failure is being handled, which is the opposite of
the contract: an inspection that cannot reach a definitive answer must
propagate so the caller retries.

Behavior is unchanged; the nested handler for MySQL 5.7's missing
PAD_ATTRIBUTE column (error 1054) is untouched, and that is the only
case where an inspection failure is legitimately recoverable.

tests/extensions/memory/test_sqlalchemy_session.py: 68 passed.
ruff check and ruff format --check clean.

Signed-off-by: linhongyu510 <linhongyu510@gmail.com>
@linhongyu510
linhongyu510 requested a review from a team as a code owner September 19, 2026 19:07
@linhongyu510

Copy link
Copy Markdown
Author

@seratch Here is the actual MySQL and MariaDB smoke result you asked for on 09-07. You were right that compiled DDL does not establish server-side outcomes — running it against live servers turned up things the DDL alone could not show.

Ran against real containers, head bb729cbb, every item from your list:

MySQL 8.0.46 — deliberately started with a non-utf8mb4 server default:

db charset     = latin1
db collation   = latin1_swedish_ci

1. Table / index / foreign-key creation — read back from information_schema, not from the DDL:

agent_sessions created
agent_messages created
agent_messages FK -> agent_sessions   fks=[('agent_messages_ibfk_1', 'agent_sessions')]
indexes on agent_messages             ['idx_agent_messages_session_time', 'PRIMARY']
agent_sessions.session_id             varchar(190) utf8mb4_bin

The varchar(190) is the point of the PR: unbounded TEXT cannot be a key under the InnoDB index-length limit, which is why creation failed before.

2. add / get / pop / clear — all pass; pop_item returns the last item, clear_session empties.

3. Non-utf8mb4 database default — '你好 😀 café' round-trips byte-intact. The column-level utf8mb4 survives the latin1 database default, so 4-byte emoji are not mangled.

4. Distinct session IDs — 'tenant' vs 'Tenant' keep separate histories (utf8mb4_bin is case-sensitive; under a _ci collation they would have merged).

5. Trailing-space IDs under utf8mb4_bin — this is the one worth reading closely. utf8mb4_bin is PAD SPACE in MySQL 8, so 'tenant ' and 'tenant' compare equal and would silently share one history. The code rejects it up front instead:

ValueError: session_id 'tenant ' ends with a space, which is not distinct under
the column's PAD SPACE collation 'utf8mb4_bin'; two sessions would silently
share one history

6. Existing caller-managed schema (create_tables=False) — works against the pre-existing tables.

MariaDB 11.8.9 — same script, same six sections, identical results including the trailing-space rejection.

ALL CHECKS PASSED on both servers

The harness drops and recreates the database per server, so the run is repeatable — I ran it twice and the output is byte-identical (diff clean). My first attempt was not idempotent and produced four spurious failures from leftover rows; worth mentioning so the numbers above are not taken as a lucky run.

Script is at .pr_full_audit/mysql_smoke_4741.py in my working tree — happy to contribute it as an opt-in integration test (skipped without a live server) if that would be useful to the repo.

Unit tests on this head: tests/extensions/memory/test_sqlalchemy_session.py 68 passed.

linhongyu510 and others added 2 commits September 20, 2026 10:58
The rest of the SQLAlchemySession suite runs against SQLite and mocked
engines, which cannot establish server-side outcomes. This turns the
smoke run requested in review into a test the repo can re-run.

Opt-in, following the OPENAI_RUN_LIVE_* convention already used by
tests/extensions/experimental/hosted_multi_agent/test_live.py: skipped
unless OPENAI_RUN_MYSQL_SESSION_TESTS=1 plus a server URL is set, so the
default suite is unaffected. Each URL is independent -- configuring only
MySQL or only MariaDB skips the other half rather than failing.

Covers the six items from the review:
  - table / index / foreign-key creation, read back from
    information_schema rather than from the emitted DDL
  - add / get / pop / clear
  - a non-utf8mb4 (latin1) database default, where the column-level
    charset is what has to carry 4-byte characters
  - an existing caller-managed schema (create_tables=False)
  - case-differing session ids staying distinct under utf8mb4_bin
  - trailing-space ids, which utf8mb4_bin compares equal because it is
    PAD SPACE in MySQL 8; the pair must not silently share a history

Idempotent by construction: each test creates a uniquely named database
and drops it afterwards, so repeated runs are independent of leftover
rows. Verified by running three times with identical results and no
leftover agents_it_* databases.

Verified locally against MySQL 8.0.46 and MariaDB 11.8.9 (12 passed on
both), with only MySQL configured (6 passed, 6 skipped), and with no
database configured (tests/extensions/memory/ = 328 passed, 14 skipped).
Removing the PAD SPACE rejection from the session turns exactly the
trailing-space case red on both servers and nothing else.

ruff check, ruff format --check and mypy are clean on the new file.

Signed-off-by: linhongyu510 <linhongyu510@gmail.com>

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the additional verification. There is one remaining compatibility issue in the MariaDB padding check: utf8mb4_0900_as_cs is an alias for the NO PAD collation utf8mb4_uca1400_nopad_as_cs, but the _nopad_ substring check classifies it as PAD SPACE. This rejects trailing-space session IDs against an existing schema that correctly distinguishes them, including with create_tables=False.

Please determine padding behavior through a server-side comparison using the actual column character set and collation instead of inferring it from the collation name. Add a public CRUD regression against independently provisioned tables using this NO PAD alias, covering both values of create_tables and confirming that "tenant" and "tenant " retain separate histories.

@seratch seratch added the wontfix This will not be worked on label Sep 25, 2026
@seratch

seratch commented Sep 25, 2026

Copy link
Copy Markdown
Member

Thanks again for your effort here. We won't continue working on this due to our current priorities but may revisit it in the future.

@seratch seratch closed this Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature:sessions wontfix This will not be worked on

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLAlchemySession(create_tables=True) cannot create tables on MySQL due to unbounded String columns

3 participants