Skip to content

Break actor on errors that rollback transactions - #3728

Merged
shrima-cf merged 4 commits into
cloudflare:shrima/STOR-3841from
shrima-cf:shrima/STOR-3841
Mar 28, 2025
Merged

shrima-cf merged 4 commits into
cloudflare:shrima/STOR-3841from
shrima-cf:shrima/STOR-3841

Conversation

@shrima-cf

@shrima-cf shrima-cf commented Mar 14, 2025 •

Copy link
Copy Markdown
Contributor

We should break output gates and reset actors in the event that we catch the following exceptions because they implicitly roll back any transaction:

SQLITE_FULL: database or disk full
SQLITE_IOERR: disk I/O error
SQLITE_BUSY: database in use by another process
SQLITE_NOMEM: out of memory
https://www.sqlite.org/lang_transaction.html#response_to_errors_within_a_transaction

@shrima-cf
shrima-cf requested review from a team as code owners March 14, 2025 18:12
@shrima-cf

Copy link
Copy Markdown
Contributor Author

@kentonv For some reason, I don't have permissions to edit the reviewers list to add you explicitly, would be great if you could review this PR

@kentonv
kentonv self-requested a review March 14, 2025 22:52
@shrima-cf shrima-cf changed the title STOR-3841 Break actor on errors that rollback transactions Break actor on errors that rollback transactions Mar 14, 2025
Comment thread src/workerd/util/sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite.c++ Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite-test.c++ Resolved
Comment thread src/workerd/util/sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite-test.c++ Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite-test.c++ Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
Comment thread src/workerd/util/sqlite.c++ Outdated
code == SQLITE_NOMEM || code == SQLITE_INTERRUPT) {

sqlite3* db = &KJ_ASSERT_NONNULL(maybeDb, "previous reset() failed");
// The transaction was rolledback, re-enabling the auto commit mode, so we should fail

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we might not have been in a transaction to start with, in which case there's no need to treat this as a critical error.

You can check if we were in a transaction before the error by checking inTransaction || !savepoints.empty().

We should probably then do inTransaction = false; savepoints.clear(); here since we need these to track what sqlite thinks the state is... although this feels precarious. I kind of wonder whether SqliteDatabase itself should refuse to execute any more statements after a critical error... not sure.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the check for inTransaction || !savepoints.empty() , but got rid of it as it felt checking if the db is in auto_commit mode was sufficient.
I was also not able to get into the code path of if(inTransaction || !savepoints.empty()) through my test - any idea why this could be the case with the test I have?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inTransaction || !savepoints.empty() determines if we were in a transaction before the error.

sqlite3_get_autocommit() tells us if we are in a transaction now.

This is how we tell if the critical error caused a rollback -- we have to compare these two things.

If we were not in a transaction before, then we don't need to consider the error critical, because it only rolled back the immediate statement that caused it, which is what we expect from any error. The critical problem is if past statements were rolled back since then the database state may no longer be consistent with the expectations of the application.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed the check here - 9b726ce

But two tests are failing:
KJ_TEST("SQLite critical error handling for SQLITE_BUSY")
KJ_TEST("SQLite critical error handling for SQLITE_NOMEM")

Both are failing because the commit is not auto-rolledback when run inside a transaction. Appreciate help to debug this

Comment thread src/workerd/util/sqlite.h Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
Comment thread src/workerd/io/actor-sqlite.c++ Resolved
@shrima-cf
shrima-cf changed the base branch from main to shrima/STOR-3841 March 28, 2025 22:51
@shrima-cf

Copy link
Copy Markdown
Contributor Author

I am going to merge this into my personal branch in cf repo (shrima/STOR-3841) and will open a new PR from there.
This is the branch from CF repo - https://github.com/cloudflare/workerd/tree/shrima/STOR-3841

Will keep this PR open as well since the comments need to be present.

I had to do this update since something changed in the the main cf repo's build generation logic, that I am not able to reproduce here. There is no way to sync those two repos without discarding these commits either.

@shrima-cf
shrima-cf merged commit 714c2bb into cloudflare:shrima/STOR-3841 Mar 28, 2025
@shrima-cf

Copy link
Copy Markdown
Contributor Author

Pull request moved - #3832

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants