You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Explore interplay between Blueprints and Assembler #117
This is an issue to publicly capture the exploration how Blueprints and Assembler can align and interact. Blueprints are files used to create WordPress sites with specific settings, themes, plugins, content, and anything else that can be configured in WordPress. Assembler is a way for users to declare pages and settings for their site by selecting patterns, fonts, and styles.
Note: The Assembler discussed here is incidentally an Automattic-internal prototype for a simple way to create an outline of site structure and styles, but we can still conceptually consider how something like Assembler can inform the design of Blueprints v2.
We began this discussion with considering how Assembler could speak in terms of a Blueprint. I imagined using Assembler as a utility to create a Blueprint or input for a Blueprint. One can imagine that conceptually, but in practice, it turned out to be a bit more difficult.
The Assembler is implemented using block editor primitives and saves changes to WordPress as it progresses through its flow. Because Assembler makes changes to WordPress, we can collect those changes at the end of its flow and incorporate those in a Blueprint, but when we asked whether Assembler could reasonably be made to function without saving changes to WordPress, the answer was No due to the way block editor primitives work. In addition, some folks see Blueprints as a way to create different initial sites before Assembler further configures them.
We started by thinking Assembler would create steps for Blueprints. Others imagined Blueprints simply as a way to preset a site before it is modified by Assembler. Both are practical possibilities.
What should we focus on to get the most out of this exploration?
If Blueprints are just a way to prep a site for Assembler, then AFAICT, there is not very much to learn from their interaction because Assembler simply continues a site where a Blueprint left off. There is no direct interaction.
If we use Assembler to create content for a Blueprint, there is more of a relationship between the two, and we can consider how the design of Blueprints v2 might incorporate that kind of input.
As for using Assembler without persisting changes to WordPress, I wonder whether we could use something like Sandbox Site powered by Playground, though sandboxing with an entire site's data may be heavy-handed. (I'm not familiar with the sandbox plugin's implementation and do not know whether this concern is based on reality or not.)
We started by thinking Assembler would create steps for Blueprints. Others imagined Blueprints simply as a way to preset a site before it is modified by Assembler. Both are practical possibilities.
What about having both? We discussed allowing users to run Blueprints multiple times and it's technically possible.
I wonder whether we could use something like Sandbox Site powered by Playground, though sandboxing with an entire site's data may be heavy-handed.
The goal of that plugin is to extract a live site into Playground, using SQL and file exports.
There is no way to move the changes you made in a Sandbox back to a live site.
The current implementation is experimental and broken, but even when it works it's not scalable to large sites.
When we finish our Data Liberation work, it should be possible to Sandbox large sites.
I wonder whether we could use something like Sandbox Site powered by Playground, though sandboxing with an entire site's data may be heavy-handed.
The goal of that plugin is to extract a live site into Playground, using SQL and file exports. There is no way to move the changes you made in a Sandbox back to a live site.
@bgrgicak Thanks for your feedback on this. My main thought here was the possibility of using an approach like the sandbox plugin to have Assembler work on a "virtual WP" instead of the live site. Then it might be used as a utility that produces output rather than something that always makes changes to the current site.
My main thought here was the possibility of using an approach like the sandbox plugin to have Assembler work on a "virtual WP" instead of the live site.
That sounds like a great idea. If this is a new site with a small amount of content, we could even use the Sandbox plugin for it.
What should we focus on to get the most out of this exploration?
If Blueprints are just a way to prep a site for Assembler, then AFAICT, there is not very much to learn from their interaction because Assembler simply continues a site where a Blueprint left off. There is no direct interaction.
If we use Assembler to create content for a Blueprint, there is more of a relationship between the two, and we can consider how the design of Blueprints v2 might incorporate that kind of input.
With the understand that I'm commenting as an outsider to this project: like Bero, I think the two ultimately belong together. As an analogy, the Create Block Theme plugin leverages the Site Editor to create new themes, while, in turn, themes are fed into the Site Editor machine. But if the question is which task to prioritise, I would say that, as far as I understood, the mandate was "Assembler → Blueprint" first.
But if the question is which task to prioritise, I would say that, as far as I understood, the mandate was "Assembler → Blueprint" first.
@bgrgicak and @mcsf Thank you for your feedback here. This was my original understanding as well.
It's been a bit quiet here because I've been AFK, but I'm scheduled to be back on Thursday and plan to check in again then.
Offhand, I think some good initial steps are:
Demonstrate an ability to collect the outcome of the Assembler flow.
Make sure that outcome can be represented and duplicated using a Blueprint
Bonus: If we want to be able to use the Assembler without ultimately changing a site, explore ways to sandbox it. Perhaps one of the following would be good:
Use an approach like the Sandbox plugin to capture Assembler output without mutating underlying WP
See if there are adequate WP hooks to capture Assembler changes onto some kind of DB overlay.
Use an approach like the Sandbox plugin to capture Assembler output without mutating underlying WP
This will only work for small sites today. If that's all it's needed for, that's perfect. For larger sites, we'll need to support partial sync to avoid downloading gigabytes of data just to add a contact page.
See if there are adequate WP hooks to capture Assembler changes onto some kind of DB overlay.
This sounds interesting. I'm not convinced about DB overlay as, at that level, we're starting to reason about raw SQL queries. Security-wise that seems fine, as templates are PHP files anyway so we're already in the code execution world. But I worry about IDs and data conflicts. However, it would be lovely if this general direction of explorations would yield a declarative data bundle that can be imported in a conflict-less way.
This sounds interesting. I'm not convinced about DB overlay as, at that level, we're starting to reason about raw SQL queries.
I have a crazy See-If-It-Can-Be-Done urge related to this and the thorough MySQL parser we're working on. 😄 But really, this is the kind of thing that is better implemented as a DBMS feature. AFAICT, MySQL doesn't have a first-class feature like this, but it reminds me of the Playground Explore Dolt issue and the kinds of things Dolt could be used to do.
I think the discussions of comprehensive export formats and the idea of snapshots are important. But since I'm not so familiar with WP exports today, I'm just going to see how far I can get with a naive approach like: blueprint-as-from-blueprint-builder + WXR import + uploads zip. It would be good to get something working as a prototype and hopefully demonstrate how an approach like this is insufficient.
This sounds interesting. I'm not convinced about DB overlay as, at that level, we're starting to reason about raw SQL queries.
I have a crazy See-If-It-Can-Be-Done urge related to this and the thorough MySQL parser we're working on. 😄 But really, this is the kind of thing that is better implemented as a DBMS feature. AFAICT, MySQL doesn't have a first-class feature like this, but it reminds me of the Playground Explore Dolt issue and the kinds of things Dolt could be used to do.
You can also parse the MySQL bin log if it is enabled and readable, but we cannot count on this for WP because it requires editing MySQL config.
I'm just going to see how far I can get with a naive approach like: blueprint-as-from-blueprint-builder + WXR import + uploads zip. It would be good to get something working as a prototype and hopefully demonstrate how an approach like this is insufficient.
Actually... Will check out the Sandbox plugin approach first, which should be a great place to start for new sites, which is what Assembler targets at the moment.
After more thought, I'm exploring using a zip containing a blueprint at a well-known location and allowing that Blueprint to reference assets within the zip. That should be extremely flexible and useful for Blueprints outside of the export use case.
For a basic WP export, such a package would:
Include wp-content/uploads
Include a full export to WXR
Copy the uploads into place
Import the WXR while skipping attachment downloads
There is unfortunately a catch with the current WordPress importer -- attachments are only created during WXR processing if fetching attachments is enabled. We can probably adjust this behavior, but if pressed for a short-term workaround, we could:
Copy the zipped wp-content/uploads to a non-standard location
Preprocess the WXR attachment URLs to refer to the non-standard location within the local Playground instance
Rely on the import process to fetch the media from Playground and create the attachment posts
Cleanup: Delete the original media from the non-standard location
@zaerl -- I wanted to loop you in here. Please chime in if you have any feedback on the export-related comment above (or anything else :). We started this issue to consider how work on Assembler** and Blueprints can align in core. At first I thought of this as Assembler generating a Blueprint, but due to way Assembler works, it looks like all we need is a full, self-contained export of the site at the end of the Assembler flow.
I'm not sure there's a lot of interesting work to do related to Assembler and exports, but at least it looks like a place to try the Blueprint + assets idea.
One question I had for you is:
Would it be reasonable to update the WordPress Importer so attachments can be created without fetching the files? If we include uploads within a Blueprint package, we can copy them into place, but currently the WP importer will not create attachments unless fetch_attachments is enabled.
** An Automattic theme that allows users to declare pages and settings for their site by selecting patterns, fonts, and styles in a wizard-like flow.
I'm sorry, but I don't understand correctly what we want to do here. Do we want to import media copies from the FS instead of downloading them from somewhere? Or, in other words, create rows in the wp_posts table with post_type "attachment" for populating the media library?
fetch_attachments is disabled by default because it is a heavy operation, so it should be discouraged from starting if a user inadvertently starts it. WP-CLI, for example, is enabled by default when you run wp import (see here). Adjusting that variable before running import() is enough.
Changing the way the wp-importer works is doable. The wp_insert_attachment function is perfectly fine for adding a post, regardless of whether the file has been downloaded. It is okay if it is in the wp_upload_dir().
There are multiple ways to do this. One, for example, is to get the path protocol of the attachment and run wp_safe_remote_get only if needed. This will not be a backward compatibility problem, but it will require some security measures to be in place, such as only accepting the local path from inside the WP folder.
I'm sorry, but I don't understand correctly what we want to do here. Do we want to import media copies from the FS instead of downloading them from somewhere? Or, in other words, create rows in the wp_posts table with post_type "attachment" for populating the media library?
I'm sorry, @zaerl. I pinged you but didn't follow up on your reply.
Yes, the idea is that such an export package would include media files. It wouldn't necessarily have to be on disk in all cases, but it would have to be accessible and addressable. For example, we might either:
completely unpack an export zip that includes media
Activity
brandonpayton commentedon Nov 7, 2024
I would love to hear others' thoughts on this.
For my part, I think it probably makes sense to see how we can extract a narrowly-scoped site export at the end of an Assembler flow.
brandonpayton commentedon Nov 7, 2024
As for using Assembler without persisting changes to WordPress, I wonder whether we could use something like Sandbox Site powered by Playground, though sandboxing with an entire site's data may be heavy-handed. (I'm not familiar with the sandbox plugin's implementation and do not know whether this concern is based on reality or not.)
bgrgicak commentedon Nov 7, 2024
What about having both? We discussed allowing users to run Blueprints multiple times and it's technically possible.
The goal of that plugin is to extract a live site into Playground, using SQL and file exports.
There is no way to move the changes you made in a Sandbox back to a live site.
The current implementation is experimental and broken, but even when it works it's not scalable to large sites.
When we finish our Data Liberation work, it should be possible to Sandbox large sites.
brandonpayton commentedon Nov 7, 2024
@bgrgicak Thanks for your feedback on this. My main thought here was the possibility of using an approach like the sandbox plugin to have Assembler work on a "virtual WP" instead of the live site. Then it might be used as a utility that produces output rather than something that always makes changes to the current site.
This is super hand-wavy, but that's the gist.
bgrgicak commentedon Nov 8, 2024
That sounds like a great idea. If this is a new site with a small amount of content, we could even use the Sandbox plugin for it.
mcsf commentedon Nov 8, 2024
With the understand that I'm commenting as an outsider to this project: like Bero, I think the two ultimately belong together. As an analogy, the Create Block Theme plugin leverages the Site Editor to create new themes, while, in turn, themes are fed into the Site Editor machine. But if the question is which task to prioritise, I would say that, as far as I understood, the mandate was "Assembler → Blueprint" first.
brandonpayton commentedon Nov 12, 2024
@bgrgicak and @mcsf Thank you for your feedback here. This was my original understanding as well.
It's been a bit quiet here because I've been AFK, but I'm scheduled to be back on Thursday and plan to check in again then.
Offhand, I think some good initial steps are:
adamziel commentedon Nov 13, 2024
Looping in @mtias about Assembler -> Blueprint vs Blueprint -> Assembler.
One idea here: https://github.com/Emilia-Capital/blueprint-builder
URL rewriting, assets export/import, and the rest of the data liberation machinery seems highly relevant here.
Highly relevant issue that we may need to solve as a part of this exploration:
I just updated it with a few relevant writeups.
This will only work for small sites today. If that's all it's needed for, that's perfect. For larger sites, we'll need to support partial sync to avoid downloading gigabytes of data just to add a contact page.
This sounds interesting. I'm not convinced about DB overlay as, at that level, we're starting to reason about raw SQL queries. Security-wise that seems fine, as templates are PHP files anyway so we're already in the code execution world. But I worry about IDs and data conflicts. However, it would be lovely if this general direction of explorations would yield a declarative data bundle that can be imported in a conflict-less way.
brandonpayton commentedon Nov 16, 2024
I have a crazy See-If-It-Can-Be-Done urge related to this and the thorough MySQL parser we're working on. 😄 But really, this is the kind of thing that is better implemented as a DBMS feature. AFAICT, MySQL doesn't have a first-class feature like this, but it reminds me of the Playground Explore Dolt issue and the kinds of things Dolt could be used to do.
brandonpayton commentedon Nov 16, 2024
I think the discussions of comprehensive export formats and the idea of snapshots are important. But since I'm not so familiar with WP exports today, I'm just going to see how far I can get with a naive approach like:
blueprint-as-from-blueprint-builder + WXR import + uploads zip. It would be good to get something working as a prototype and hopefully demonstrate how an approach like this is insufficient.brandonpayton commentedon Nov 16, 2024
You can also parse the MySQL bin log if it is enabled and readable, but we cannot count on this for WP because it requires editing MySQL config.
brandonpayton commentedon Nov 16, 2024
Actually... Will check out the Sandbox plugin approach first, which should be a great place to start for new sites, which is what Assembler targets at the moment.
brandonpayton commentedon Nov 19, 2024
After more thought, I'm exploring using a zip containing a blueprint at a well-known location and allowing that Blueprint to reference assets within the zip. That should be extremely flexible and useful for Blueprints outside of the export use case.
For a basic WP export, such a package would:
There is unfortunately a catch with the current WordPress importer -- attachments are only created during WXR processing if fetching attachments is enabled. We can probably adjust this behavior, but if pressed for a short-term workaround, we could:
brandonpayton commentedon Nov 21, 2024
note: The blueprints package idea has its own issue now.
@zaerl -- I wanted to loop you in here. Please chime in if you have any feedback on the export-related comment above (or anything else :). We started this issue to consider how work on Assembler** and Blueprints can align in core. At first I thought of this as Assembler generating a Blueprint, but due to way Assembler works, it looks like all we need is a full, self-contained export of the site at the end of the Assembler flow.
I'm not sure there's a lot of interesting work to do related to Assembler and exports, but at least it looks like a place to try the Blueprint + assets idea.
One question I had for you is:
Would it be reasonable to update the WordPress Importer so attachments can be created without fetching the files? If we include uploads within a Blueprint package, we can copy them into place, but currently the WP importer will not create attachments unless fetch_attachments is enabled.
** An Automattic theme that allows users to declare pages and settings for their site by selecting patterns, fonts, and styles in a wizard-like flow.
zaerl commentedon Nov 21, 2024
I'm sorry, but I don't understand correctly what we want to do here. Do we want to import media copies from the FS instead of downloading them from somewhere? Or, in other words, create rows in the
wp_poststable withpost_type"attachment" for populating the media library?fetch_attachmentsis disabled by default because it is a heavy operation, so it should be discouraged from starting if a user inadvertently starts it. WP-CLI, for example, is enabled by default when you runwp import(see here). Adjusting that variable before runningimport()is enough.Changing the way the wp-importer works is doable. The
wp_insert_attachmentfunction is perfectly fine for adding a post, regardless of whether the file has been downloaded. It is okay if it is in thewp_upload_dir().There are multiple ways to do this. One, for example, is to get the path protocol of the attachment and run
wp_safe_remote_getonly if needed. This will not be a backward compatibility problem, but it will require some security measures to be in place, such as only accepting the local path from inside the WP folder.brandonpayton commentedon Jan 23, 2025
I'm sorry, @zaerl. I pinged you but didn't follow up on your reply.
Yes, the idea is that such an export package would include media files. It wouldn't necessarily have to be on disk in all cases, but it would have to be accessible and addressable. For example, we might either:
Does that make sense?