Skip to content

Add an "embeddings" mode to Content Classification - #1252

Open
dkotter wants to merge 21 commits into
WordPress:developfrom
dkotter:feature/content-classification-embeddings
Open

dkotter wants to merge 21 commits into
WordPress:developfrom
dkotter:feature/content-classification-embeddings

Conversation

@dkotter

@dkotter dkotter commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What?

Adds an embedding-based "Similarity matching" mode to the Content Classification experiment. It's the first real feature built on the embeddings stack.

Why?

The embeddings work so far (#976, #1088, #1135) added storage, math and background sync, but nothing user-facing used it. This PR proves that stack works in a real feature (and surfaced a few areas that needed improvements/changes). It also gives Content Classification an option that suggests existing terms by meaning, with no post content sent to a language model.

How?

  • New settings: a "Suggestion method" (AI model or Similarity matching) and an "Embedding model" picker that only lists embedding-capable models. Settings fields can now be hidden or disabled based on other settings. Embedding mode disables the taxonomy strategy and the LLM model selection.
  • Background term embeddings: once embedding mode is on, Content Classification registers as a sync consumer so categories and tags get embedded. An "Embeddings" panel on the settings screen shows coverage and lets an admin start or cancel the backfill, with a confirmation that it may cost money. It's backed by new REST endpoints under ai/v1/embeddings/consumers/{id}. Changing the model clears the old model's vectors.
  • Suggestions: the new Embedding_Suggester embeds the post (title and content) on click, reusing stored vectors if the text hasn't changed. It then ranks term vectors by cosine similarity, applying a minimum score (0.3) and a minimum term usage count (2).
  • Similarity search: the new reusable Similarity_Search does a quick first pass on the one-bit coarse codes, then reranks a shortlist by exact cosine similarity, so large taxonomies never get loaded fully into memory.

Notes: I ran this on a site that had ~26,000 terms and using Ollama, took about 7 minutes to generate embeddings for all of them using the admin UI (CLI command was slightly faster). Testing with OpenAI, took about 17 minutes. Actually running comparisons only took about 1 second

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Fable 5.1, Opus 5.5
Used for: Initial design spec and implementation plan; review of plan by me; execution of plan by AI with final review and testing by me.

Testing Instructions

  1. Ensure you have some content: a handful of posts, tags and categories. We only consider terms if they've been used at least twice so assign each two or more times
  2. Connect a provider that has an embedding model (e.g. OpenAI).
  3. Go to Settings > AI, open Developer Tools, and turn on Advanced settings.
  4. Under Content Classification, set Suggestion method to Similarity matching (embeddings), pick an embedding model (e.g. text-embedding-3-small), and save.
  5. Before generating embeddings, open a post in the editor. The Suggest Tags/Categories buttons should be disabled with a notice.
  6. Back in settings, click Generate embeddings in the Embeddings panel and confirm. Wait for "All N terms are embedded." (or run wp ai embeddings sync run).
  7. Reload the editor and click Suggest Tags and Suggest Categories. The suggestions should be existing terms related to the post.
  8. Change the embedding model and save. The panel should show 0 terms embedded.
  9. Switch Suggestion method back to AI model and confirm the LLM suggestions work as before.

Screenshots or screencast

New settings Confirmation to generate embeddings
Screenshot 2026-10-28 at 2 12 24 PM Screenshot 2026-10-28 at 2 12 37 PM
Embeddings generating Switching models
Screenshot 2026-10-28 at 2 13 05 PM Screenshot 2026-10-28 at 3 00 19 PM

Changelog Entry

Added - Similarity matching (embeddings) mode for Content Classification, which suggests existing tags and categories by comparing the post against pre-generated term embeddings.

Open WordPress Playground Preview

… Add in necessary methods to more easily utilize the coarse values
…hen a more in-depth rank of those results. Built to support this particular feature but also work for other use cases
…d or visible based on other settings. Add the ability to set a different set of capabilities than the model chosen
… and then compares against terms, ranking from closest to farthest match. Update our ability to use this when embedding mode is on
…ly for things like hidden and disabled fields, model picker and status of the sync
…cs; better copy; skip processing if we have no terms; skip bad vectors
@dkotter dkotter added this to the 1.6.0 milestone Oct 30, 2026
@dkotter dkotter self-assigned this Oct 30, 2026
@dkotter
dkotter requested a review from a team October 30, 2026 14:58
@dkotter
dkotter requested a review from jeffpaul as a code owner October 30, 2026 14:58
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: itsgajendraSingh <gajendrasingh@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

… in the method and model they want, so they can use the ability without having to use the experiment or it's UI
@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.21683% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.39%. Comparing base (9f03daf) to head (63dc033).

Files with missing lines Patch % Lines
.../Content_Classification/Content_Classification.php 83.70% 29 Missing ⚠️
...ies/Content_Classification/Embedding_Suggester.php 86.61% 17 Missing ⚠️
includes/Embeddings/Similarity_Search.php 85.00% 9 Missing ⚠️
includes/REST/Embeddings_Controller.php 85.45% 8 Missing ⚠️
.../Content_Classification/Content_Classification.php 93.67% 5 Missing ⚠️
includes/Embeddings/Embedding_Repository.php 92.30% 4 Missing ⚠️
includes/REST/Models_Controller.php 25.00% 3 Missing ⚠️
includes/Settings/Settings_Page.php 85.71% 3 Missing ⚠️
includes/Embeddings/Sync/Embedding_Sync.php 95.00% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #1252      +/-   ##
=============================================
+ Coverage      85.14%   85.39%   +0.25%     
- Complexity      4826     4996     +170     
=============================================
  Files            189      192       +3     
  Lines          17910    18501     +591     
=============================================
+ Hits           15249    15799     +550     
- Misses          2661     2702      +41     
Flag Coverage Δ
unit 85.39% <87.21%> (+0.25%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dkotter dkotter mentioned this pull request Nov 2, 2026
9 of 10 tasks
@itsgajendraSingh

Copy link
Copy Markdown

Hi @dkotter

Tested this locally (LocalWP + Ollama with nomic-embed-text) and the similarity suggestions are working nicely. A few things I noticed while testing:

  1. When I click Generate embeddings, there's nothing that shows it's actually working. A loading icon or spinner while it runs would help, otherwise it feels like nothing is happening.

  2. After generating embeddings, I disabled Content Classification and then enabled it again. The Generate embeddings button became disabled and showed:

    Embeddings aren't active yet. Make sure Content Classification is enabled and saved, then reload this page.

    But in the editor, the Suggest Categories and Suggest Tags buttons still looked active. When I clicked one, it showed:

    No Category suggestions were generated.

    and then the buttons became disabled. Once I reloaded the settings page, it showed all terms as embedded again, so it looks like the state only updates after a reload.

  3. When I get suggested categories or tags, each one has a × icon. I assumed this would remove the suggestion permanently, but when I click Suggest again the same term shows up again. Not sure if that's intended, but it was a bit confusing.

Screenshot 2026-11-05 at 6 16 57 PM

@dkotter

dkotter commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for testing @itsgajendraSingh!

  1. When I click Generate embeddings, there's nothing that shows it's actually working. A loading icon or spinner while it runs would help, otherwise it feels like nothing is happening.

I'm assuming this is referring to the button on the settings screen? If so, you should see a progress message (screenshot). Do you not see that? I have added a spinner next to it now to make it more obvious loading is in progress.

  1. After generating embeddings, I disabled Content Classification and then enabled it again. The Generate embeddings button became disabled and showed:

    Embeddings aren't active yet. Make sure Content Classification is enabled and saved, then reload this page.

    But in the editor, the Suggest Categories and Suggest Tags buttons still looked active. When I clicked one, it showed:

    No Category suggestions were generated.

    and then the buttons became disabled. Once I reloaded the settings page, it showed all terms as embedded again, so it looks like the state only updates after a reload.

Thanks, I was able to reproduce this. Should be addressed in 5daaf99

  1. When I get suggested categories or tags, each one has a × icon. I assumed this would remove the suggestion permanently, but when I click Suggest again the same term shows up again. Not sure if that's intended, but it was a bit confusing.

This is expected as it's just the standard term UI that WordPress uses. Clicking the icon just removes the term but we don't store that anywhere.

That's a potential idea we can think about though, saving that dismissal and assuming that means someone doesn't want to ever see that term suggested again on that post. Not sure if that is what people would expect to happen or not though.

@itsgajendraSingh

Copy link
Copy Markdown

Thanks @dkotter! Pulled the latest changes and tested again, both work as expected now:

  1. The spinner shows while embeddings are generating, much clearer now 👍
  2. After disabling and re-enabling Content Classification, the settings panel and editor buttons show the correct state without needing a reload.
  1. When I get suggested categories or tags, each one has a × icon. I assumed this would remove the suggestion permanently, but when I click Suggest again the same term shows up again. Not sure if that's intended, but it was a bit confusing.

For point 3, a couple of ideas:

  • Store dismissed terms, with a way to bring them back. If we save the dismissal so a term isn't suggested again on that post, there should also be an easy way to undo it, like a "Show dismissed suggestions" link or a reset option, so users don't lose a term without knowing why.
  • Replace instead of remove. When a user dismisses a suggested term, we could fill that spot with the next best match instead of just removing it. That way nothing needs to be stored permanently, and the user still gets a full set of suggestions to choose from.

I think the second option might be simpler to start with since it doesn't need any extra storage. What do you think? Curious to hear your thoughts on whether something like this would make sense here.

@dkotter

dkotter commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

I think the second option might be simpler to start with since it doesn't need any extra storage. What do you think? Curious to hear your thoughts on whether something like this would make sense here.

I'm not opposed to trying this out though I think this should be applied to the result regardless of whether embeddings mode is on or not, meaning it's likely something best to add in a separate PR instead of having that be tied to this one.

There's a few things I'd call out though. First, users can choose the maximum number of terms to show (default is 5). So if we show additional ones as existing ones are removed, technically we aren't respecting that maximum anymore. That setting instead becomes a maximum that we show at once, not a maximum we show ever. That may be fine but don't want to confuse people that have purposely set that number.

We also don't show terms that are below our threshold and I think we'd want to keep that. So likely scenarios where we don't have more than 5 to show once we filter out those low quality results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants