You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the moment, all database changes and uploads are gone once the page is refreshed. Preserving them would be useful for courses, technical demos, even sharing a link to your changes.
changed the title [-][Browser] Persist the database after page refresh[/-][+][Browser] Persist filesystem changes after page refresh[/+]on Oct 17, 2022
James Long's aptly-named absurd-sql demonstrates persistent browser-side sqlite3 by storing the databases in IndexedDB storage. We experimented with that approach but it's... well, absurd 😉. It's too slow for anything beyond trivial databases and it's easy to get a database in an inconsistent state by visiting the page from two tabs.
About persisting changes to the database, I recently learned about SQLite's Session extension.
The session extension provide a mechanism for recording changes to some or all of the rowid tables in an SQLite database, and packaging those changes into a "changeset" or "patchset" file that can later be used to apply the same set of changes to another database with the same schema and compatible starting data. A "changeset" may also be inverted and used to "undo" a session.
I read about it in an article about someone developing an iOS app with a local SQLite database (full copy of the user data in the app itself).
The session extension lets you start a session on an SQLite connection. All changes made to the database through that connection are bundled into a patchset blob. The extension also provides method for applying the generated patchset to a table.
This can be used to build a very simple client-sync system. Collect the changes made in a client, periodically bundle them up into a changeset and upload it to the server where it is applied to a backup copy of the database.
Perhaps this could be a lighter-weight approach to saving SQLite state in browser local storage, by storing (and restoring) the session history instead of a snapshot of the whole database.
IndexedDB is way too slow for persistence backend.
Emscripten supports storing files in IndexedDb via IDBFS. I got a demo to work. If you run it and refresh the page, the initial console.log() will show the test.txt file:
// In loadPHPRuntime():PHPRuntime.FS.mkdir('/offline');PHPRuntime.FS.mount(PHPRuntime.FS.filesystems.IDBFS,{},'/offline');console.log(PHPRuntime.FS.readdir('/offline'));constsyncFs=(populate:boolean)=>newPromise((resolve,reject)=>{console.log('Syncing FS...');PHPRuntime.FS.syncfs(populate,function(err: Error){console.log('done',err);if(err){reject(err);}else{resolve(true);}});});// Populate /offline from IndexedDb:awaitsyncFs(true);PHPRuntime.FS.writeFile('/offline/test.txt',newTextEncoder().encode('Hello world!'));// Write files from /offline to IndexedDb:awaitsyncFs(false);
However, if you do that with WordPress files, each sync will take minutes:
The easiest way to implement this is through explicit buttons like:
Save - exports the site, writes the zip to localStorage
Restore - imports the last zip written to localStorage
These could be integrated with the version switchers and the beforeunload event to give an illusion of a continuous synchronisation. On top of it, we could have an autosave that runs this sync once a minute or so, if anything in the filesystem changed.
In fact, we could retain the zip file in memory, keep track of all VFS writes, and update only specific files inside the archive.
A few more ideas:
Write the VFS object directly to localStorage without going through zip. Synchronize all the writes as diffs and flatten everything every X writes.
Observe writes and perform them again in an Origin Private Filesystem. It is pretty well supported. After page refresh, initialize from OPFS. Perhaps we could just switch to OPFS as a backend?
@jimmywarting both are on the roadmap! JSPI will need to stabilize first - right now it’s chrome-only and limited to arm64 and x64. Also, we’ll need to build PHP with pthreads support as I think Emscripten can only support OPFS via threaded WASMFS (although I’d love to be wrong here).
Activity
[-][Browser] Persist the database after page refresh[/-][+][Browser] Persist filesystem changes after page refresh[/+][-][Browser] Persist filesystem changes after page refresh[/-][+]Persist filesystem changes after page refresh[/+]adamziel commentedon Nov 23, 2022
Relevant: https://sqlite.org/wasm/doc/tip/about.md
Specifically:
eliot-akira commentedon Jan 12, 2023
About persisting changes to the database, I recently learned about SQLite's Session extension.
I read about it in an article about someone developing an iOS app with a local SQLite database (full copy of the user data in the app itself).
Perhaps this could be a lighter-weight approach to saving SQLite state in browser local storage, by storing (and restoring) the session history instead of a snapshot of the whole database.
swissspidy commentedon Feb 2, 2023
Related: https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system/
adamziel commentedon May 14, 2023
IndexedDB is way too slow for persistence backend.
Emscripten supports storing files in IndexedDb via IDBFS. I got a demo to work. If you run it and refresh the page, the initial
console.log()will show thetest.txtfile:However, if you do that with WordPress files, each sync will take minutes:
Perhaps storing just diffs would work better, but it sounds like a lot of work.
Here's another idea
Exporting and importing a zipped WordPress site works in a blink of an eye. Let's reuse it for persistence.
adamziel commentedon May 21, 2023
More thoughts:
The easiest way to implement this is through explicit buttons like:
These could be integrated with the version switchers and the
beforeunloadevent to give an illusion of a continuous synchronisation. On top of it, we could have an autosave that runs this sync once a minute or so, if anything in the filesystem changed.In fact, we could retain the zip file in memory, keep track of all VFS writes, and update only specific files inside the archive.
A few more ideas:
jimmywarting commentedon May 21, 2023
OPFS is a grate choice.
together with asyncify -2 and JSPI the WebAssembly runtime can be suspended while the js glue awaits a read/write operation.
you should look into switching to OPFS and JSPI
adamziel commentedon May 22, 2023
@jimmywarting both are on the roadmap! JSPI will need to stabilize first - right now it’s chrome-only and limited to arm64 and x64. Also, we’ll need to build PHP with pthreads support as I think Emscripten can only support OPFS via threaded WASMFS (although I’d love to be wrong here).
adamziel commentedon Nov 22, 2023
Done in #196