Skip to content

Persist filesystem changes after page refresh #19

Closed

Description

@adamziel

What problem is this issue trying to solve?

At the moment, all database changes and uploads are gone once the page is refreshed. Preserving them would be useful for courses, technical demos, even sharing a link to your changes.

What solution does this issue propose?

A few ideas:

Activity

  1. changed the title [-][Browser] Persist the database after page refresh[/-] [+][Browser] Persist filesystem changes after page refresh[/+] on Oct 17, 2022
  2. changed the title [-][Browser] Persist filesystem changes after page refresh[/-] [+]Persist filesystem changes after page refresh[/+] on Oct 22, 2022
  3. adamziel commented on Nov 23, 2022

    @adamziel
    CollaboratorAuthor

    Relevant: https://sqlite.org/wasm/doc/tip/about.md

    Specifically:

    James Long's aptly-named absurd-sql demonstrates persistent browser-side sqlite3 by storing the databases in IndexedDB storage. We experimented with that approach but it's... well, absurd 😉. It's too slow for anything beyond trivial databases and it's easy to get a database in an inconsistent state by visiting the page from two tabs.

  4. eliot-akira commented on Jan 12, 2023

    @eliot-akira
    Collaborator

    About persisting changes to the database, I recently learned about SQLite's Session extension.

    The session extension provide a mechanism for recording changes to some or all of the rowid tables in an SQLite database, and packaging those changes into a "changeset" or "patchset" file that can later be used to apply the same set of changes to another database with the same schema and compatible starting data. A "changeset" may also be inverted and used to "undo" a session.

    I read about it in an article about someone developing an iOS app with a local SQLite database (full copy of the user data in the app itself).

    The session extension lets you start a session on an SQLite connection. All changes made to the database through that connection are bundled into a patchset blob. The extension also provides method for applying the generated patchset to a table.

    This can be used to build a very simple client-sync system. Collect the changes made in a client, periodically bundle them up into a changeset and upload it to the server where it is applied to a backup copy of the database.

    Perhaps this could be a lighter-weight approach to saving SQLite state in browser local storage, by storing (and restoring) the session history instead of a snapshot of the whole database.

  5. swissspidy commented on Feb 2, 2023

    @swissspidy
    Member
  6. adamziel commented on May 14, 2023

    @adamziel
    CollaboratorAuthor

    IndexedDB is way too slow for persistence backend.

    Emscripten supports storing files in IndexedDb via IDBFS. I got a demo to work. If you run it and refresh the page, the initial console.log() will show the test.txt file:

    	// In loadPHPRuntime():
     
    	PHPRuntime.FS.mkdir('/offline');
    	PHPRuntime.FS.mount(PHPRuntime.FS.filesystems.IDBFS, {}, '/offline');
    	console.log(PHPRuntime.FS.readdir('/offline'));
    
    	const syncFs = (populate:boolean) => new Promise((resolve, reject) => {
    		console.log('Syncing FS...');
    		PHPRuntime.FS.syncfs(populate, function (err: Error) {
    			console.log('done', err);
    			if (err) {
    				reject(err);
    			} else {
    				resolve(true);
    			}
    		});
    	});
    
    	// Populate /offline from IndexedDb:
    	await syncFs(true);
    
    	PHPRuntime.FS.writeFile(
    		'/offline/test.txt',
    		new TextEncoder().encode('Hello world!')
    	);
    
    	// Write files from /offline to IndexedDb:
    	await syncFs(false);

    However, if you do that with WordPress files, each sync will take minutes:

    	PHPRuntime.FS.mkdir('/wordpress');
    	PHPRuntime.FS.mount(PHPRuntime.FS.filesystems.IDBFS, {}, '/wordpress');
    	//await syncFs(true);
    	console.log(PHPRuntime.FS.readdir('/wordpress'));
    	for (const { default: loadDataModule } of dataDependenciesModules) {
    		loadDataModule(PHPRuntime);
    	}
    	if (!dataDependenciesModules.length) {
    		resolveDepsReady();
    	}
    
    	await depsReady;
    	await phpReady;
    	console.log(PHPRuntime.FS.readdir('/wordpress'));
    	await syncFs(false);
    	console.log(PHPRuntime.FS.readdir('/wordpress'));

    Perhaps storing just diffs would work better, but it sounds like a lot of work.

    Here's another idea

    Exporting and importing a zipped WordPress site works in a blink of an eye. Let's reuse it for persistence.

  7. adamziel commented on May 21, 2023

    @adamziel
    CollaboratorAuthor

    More thoughts:

    The easiest way to implement this is through explicit buttons like:

    • Save - exports the site, writes the zip to localStorage
    • Restore - imports the last zip written to localStorage

    These could be integrated with the version switchers and the beforeunload event to give an illusion of a continuous synchronisation. On top of it, we could have an autosave that runs this sync once a minute or so, if anything in the filesystem changed.

    In fact, we could retain the zip file in memory, keep track of all VFS writes, and update only specific files inside the archive.

    A few more ideas:

    • Write the VFS object directly to localStorage without going through zip. Synchronize all the writes as diffs and flatten everything every X writes.
    • Observe writes and perform them again in an Origin Private Filesystem. It is pretty well supported. After page refresh, initialize from OPFS. Perhaps we could just switch to OPFS as a backend?
  8. jimmywarting commented on May 21, 2023

    @jimmywarting

    OPFS is a grate choice.
    together with asyncify -2 and JSPI the WebAssembly runtime can be suspended while the js glue awaits a read/write operation.

    you should look into switching to OPFS and JSPI

  9. adamziel commented on May 22, 2023

    @adamziel
    CollaboratorAuthor

    @jimmywarting both are on the roadmap! JSPI will need to stabilize first - right now it’s chrome-only and limited to arm64 and x64. Also, we’ll need to build PHP with pthreads support as I think Emscripten can only support OPFS via threaded WASMFS (although I’d love to be wrong here).

  10. adamziel commented on Nov 22, 2023

    @adamziel
    CollaboratorAuthor

    Done in #196

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions