diff --git a/.agents/security.md b/.agents/security.md index ed77fe8e63ea..33e6450517e1 100644 --- a/.agents/security.md +++ b/.agents/security.md @@ -219,33 +219,51 @@ repository does not. When the copy is there, start with these paths: `posthog/security/crosslang/oob_domains.json` lists the services that capture out-of-band interactions, such as Interactsh and Burp Collaborator, and the DNS rebinding services. -Today the guards only log a match. Nothing is refused yet. +A guard refuses a listed name, on the name alone, before any resolve. The matcher reports the +match and the guard decides the refusal, so each surface can report it the way its caller expects. -The warehouse guards emit the `oob_domain` stage of `data_imports.host_check`, which carries -`team_id`, and the `oob_domain_resolved` stage when the caller resolved the host first. The HTTP -adapter emits `security.oob_domain_seen` and takes its team from the activity's bound log context. -`posthog/security/url_validation.py` emits the same event with no team at all, because no team is -in scope there, so those lines are attributable to a host and not to a project. +The warehouse guards emit the `oob_domain` stage of `data_imports.host_check` with the `block` +decision, which carries `team_id`, and the `oob_domain_resolved` stage when the caller resolved +the host first. The HTTP adapter emits `security.oob_domain_seen` and raises +`OutOfBandDomainError`. `posthog/security/url_validation.py` emits the same event and returns a +`BlockedName`, so every caller of `is_url_allowed`, `validate_external_url` and +`validate_external_host` refuses the host. + +Three DNS helpers refuse the name and report nothing for it: `posthog/dns_hosts.py` finds no DNS +host, `email_domain_dns.find_published_records` publishes no record so the domain never verifies, +and `existing_email_tools.lookup_existing_email_tools` reports no tool. + +A refusal must be final, because the host sits in the customer's own config and every retry +replays it. The warehouse refusals carry a prefix `Any_Source_Errors` matches, so the schema +pauses, and they subclass `NonReportableError`, so the retries stay out of error tracking. +`OutOfBandDomainError` is deliberately not a `RequestException`: the sources catch that broadly +and several retry on it. + +`TeamAndOrgViewSetMixin` binds `team_id` into the log context of every team-scoped view, so a line +from a guard reached through the API names the project that drove it. In Node, `validateHostnameIPLiteral` (`nodejs/src/common/utils/request.ts`) and `EmailValidationService.getSkipReason` emit `security.oob_domain_seen` too, under the guard `node_request` or `node_email_validation`. Both read the hand-written mirror in `nodejs/src/common/utils/oob-domains.ts`, which a parity test holds against the JSON. Both log `team_id`, the field the Python guards bind. The email guard always has a team. The request guard has one only when the fetch runs under `fetchAttribution`, as a CDP destination fetch and a recording export's stylesheet fetch do, and logs `team_id: null` for any other caller. -`noteOobDomain` logs one line per host, team and guard every 15 minutes, because a retrying destination would otherwise log per attempt. +`checkOobDomain` logs one line per host, team and guard every 15 minutes, because a retrying destination would otherwise log per attempt. +The window gates the log line and nothing else, so every match is refused. -The Rust guards call `common_dns::oob_domains::note_oob_domain` and emit `security.oob_domain_seen` under three guard names. +The Rust guards call `common_dns::oob_domains::check_oob_domain` and emit `security.oob_domain_seen` under three guard names. They read the hand-written mirror in `rust/common/dns/src/oob_domains.rs`, which a parity test holds against the JSON. -`note_oob_domain` takes the team as an `Option`, logs one line per host, team and guard every 15 minutes, and omits the `team_id` field rather than printing an empty one. +`check_oob_domain` takes the team as an `Option`, logs one line per host, team and guard every 15 minutes, and omits the `team_id` field rather than printing an empty one. `cymbal_sourcemap` names the team on the source and sourcemap fetches, and omits it on a redirect hop, because one HTTP client serves every team. `batch_import_s3_endpoint` always names the team, which the job model carries. `common_dns_resolver` never names one: it fires inside the shared reqwest and Smithy resolvers, which no caller passes a team to. -Every check runs on cloud only, and each runs before its guard's own DNS resolution, because the lookup alone confirms the path to whoever runs the collaborator. +Every check runs on cloud only, so a self-hosted deployment reaches these names as it always did. Each runs before its guard's own DNS resolution, because the lookup alone confirms the path to whoever runs the collaborator. It stops the default scanner configuration. It does not stop a determined attacker, who points a CNAME at the collaborator or runs their own server. The IP checks and the address pinning remain -the real controls. +the real controls. Refusing the name stops the public rebinding services on the list, `rbndr.us` and `1u.ms`, +on the paths that do not pin the address. A rebinding record on any other name still passes +the name check, so address pinning and the egress proxy stay the controls there. ## Content Security Policy diff --git a/.agents/skills/adding-api-scopes/SKILL.md b/.agents/skills/adding-api-scopes/SKILL.md index 1e23fcc46071..ae5f4e4868ae 100644 --- a/.agents/skills/adding-api-scopes/SKILL.md +++ b/.agents/skills/adding-api-scopes/SKILL.md @@ -9,8 +9,9 @@ A scope has two parts: an object and an action, as in `feature_flag:read`. Adding a scope means adding a scope object, such as `feature_flag`. It gives tokens `feature_flag:read` and `feature_flag:write`. `posthog/scopes.py` is the source of the object list. The frontend type and the MCP OAuth list are generated from it. -The picker rows and the groups are kept by hand in `frontend/src/lib/scopes.tsx`, and a test checks them. -They stay in the frontend on purpose: labels, plurals, groups and picker omissions are UI decisions, while `posthog/scopes.py` decides what exists and what it grants. +Each object a picker offers also has a `ScopeDoc`: its name, plural, one-sentence description, group, docs link, the levels a picker offers, whether it reaches the whole organization, and any scope its writes also need. +The owning product declares it in `products//backend/scopes.py`, and core objects are in `CORE_SCOPE_DOCS` in `posthog/scope_docs.py`. +The web app's `API_SCOPES` and `API_SCOPE_GROUPS` are built from those declarations, so the frontend keeps no copy. ## Decide if you need a new scope @@ -23,7 +24,7 @@ Name its object with a `snake_case` singular noun, such as `feature_flag`. - **Public:** OAuth lists it, MCP can request it, and the key picker offers it. This is the default. - **Internal:** only the server creates tokens with it. Use `INTERNAL_API_SCOPE_OBJECTS`. - **OAuth-hidden:** a person can paste it into a personal API key, but OAuth clients do not see it. Use this for staff-only or unreleased surfaces. Use `OAUTH_HIDDEN_SCOPE_OBJECTS`. -- **Privileged:** only PostHog staff can give it to an OAuth app, through Django admin or a data migration. An app that registers itself cannot get it, and the CLI login and the key picker presets never include it. Use this for a scope that a partner app must not grant to itself, such as `llm_gateway`. Use `PRIVILEGED_SCOPES`, and set `unprivilegedExcluded: true` on the picker row. +- **Privileged:** only PostHog staff can give it to an OAuth app, through Django admin or a data migration. An app that registers itself cannot get it, and the CLI login and the key picker presets never include it. Use this for a scope that a partner app must not grant to itself, such as `llm_gateway`. Use `PRIVILEGED_SCOPES`. The picker row reads it from the generated scope docs. Then decide two more things, separately from the kind: @@ -39,24 +40,24 @@ Then decide two more things, separately from the kind: - An endpoint set to `scope_object = "INTERNAL"` accepts only logged-in sessions. Change it to the new object to open it to tokens. - Then call each custom action with a personal API key that has only the new scope. No test checks this. 3. **Regenerate.** Run `hogli build:openapi`, which carries the object to the frontend type, and `hogli build:projections`, which updates the OAuth lists of the MCP server and the web app. **DO NOT EDIT GENERATED FILES BY HAND.** -4. **Show it in the pickers.** Skip this step for an OAuth-hidden object: no picker shows it, and the tests do not ask for a row or a group. Otherwise, in `frontend/src/lib/scopes.tsx`: - - Add a row to `API_SCOPES` with a sentence-case label and plural (`/writing-user-facing-copy`). Disable `write` if no endpoint writes. If the key modal should not offer the object, add a reason to `API_SCOPES_OMITTED_FROM_MODAL` instead. - - Add the object to one group in `API_SCOPE_GROUPS`. See "Choose a group". +4. **Explain it.** Skip this step for an OAuth-hidden object: no picker shows it, and the tests do not ask for a doc or a group. + - Add a `ScopeDoc` to `SCOPE_DOCS` in `products//backend/scopes.py`, or to `CORE_SCOPE_DOCS` for a core object. Give it a sentence-case name, a plural that is lowercase except for proper nouns and acronyms, and one sentence under 120 characters on what the scope covers, without "read" or "write" (`/writing-user-facing-copy`). Link the product's page under `https://posthog.com/docs/` when one exists. Pick its `group` (see "Choose a group"). Set `actions=("read",)` if no endpoint writes, and `organization_wide=True` if it reaches past one project. If writing it also writes another object, as a survey writes its targeting flag, set `write_requires` to that scope and the reason. + - If no picker should offer the object, add a reason to `SCOPE_OBJECTS_OMITTED_FROM_PICKERS` in `posthog/scopes.py` instead. + - Run `hogli build:projections` again to regenerate `frontend/src/lib/scopeDocs.generated.ts`. 5. **Let MCP tools request it.** For each MCP tool that calls the endpoint, add the scope under `scopes:` in `products//mcp/tools.yaml`. ## Choose a group -The OAuth consent screen and the scope pickers show objects in groups, from `API_SCOPE_GROUPS` in `frontend/src/lib/scopes.tsx`. +The OAuth consent screen and the scope pickers show objects in groups, from the `group` of each `ScopeDoc`. The groups and their order are the `ScopeGroup` literal in `posthog/scope_docs.py`. The groups make a long list readable, so a person can find a product quickly. - A group is a product area that a person recognizes, such as "Session replay" or "Feature flags, experiments & surveys". It is not a code module or a team. - Put the object in the existing group that a person would look in first. -- Add a new group only when it gets two or more objects. A group with one object makes the list longer, not easier to read. +- Add a new group to `ScopeGroup` only when it gets two or more objects. A group with one object makes the list longer, not easier to read. ## If a test fails -- **`frontend/src/lib/scopes.test.ts`, the coverage test:** a grantable object has no picker row and no omission reason. Add a row to `API_SCOPES`, or a reason to `API_SCOPES_OMITTED_FROM_MODAL`. If the object is missing from `APIScopeObject`, run `hogli build:openapi` first. If the object is OAuth-hidden, run `hogli build:projections` instead, so the test learns to skip it. -- **`frontend/src/lib/scopes.test.ts`, the group test:** an object is in no group, or in two groups. Put it in exactly one group of `API_SCOPE_GROUPS`. It also fails when an OAuth-hidden object has a row or a group. Remove them, because no picker shows a hidden object. +- **`posthog/test/repo_invariants/test_scope_docs.py`:** an exposed object has no `ScopeDoc`, or a `ScopeDoc` names an object that is hidden or omitted from pickers. Add the doc, or the omission reason, as in step 4. A `ValueError` at import names a `ScopeDoc` with a missing field, a long description, a dash, a docs link outside posthog.com/docs, an unknown group, or a `write_requires` that names an unknown scope. - **`posthog/test/test_scopes.py`:** an internal, OAuth-hidden or privileged scope leaks into a list it must stay out of, or the project secret API key list in `posthog/scopes.py` differs from the copy in `frontend/src/lib/scopes.tsx`. Fix the set, or make the two lists equal. - **`services/mcp/tests/unit/tool-filtering.test.ts`, the completeness test:** an MCP tool requires a scope that OAuth does not advertise. If the scope is new, run `hogli build:projections`. If it is internal, add it to the test's server-only list. Otherwise fix the scope name in `tools.yaml`. @@ -65,8 +66,8 @@ The groups make a long list readable, so a person can find a product quickly. - A custom `@action` without `required_scopes`. Tokens get a 403. - A read scope on an action that changes data. - An MCP tool scope that differs from its endpoint's `scope_object`. -- A group that makes no sense for the object, or a new group that should not exist. The group test only checks that each object has exactly one group. Check that it sits where a person would look for it, and that a new group has at least two objects. See "Choose a group". -- A wrong or unclear label. +- A group that makes no sense for the object, or a new group that should not exist. `ScopeDoc` only checks that the group exists. Check that the object sits where a person would look for it, and that a new group has at least two objects. See "Choose a group". +- A wrong or unclear name or description. ## Where to read scopes in code @@ -79,5 +80,6 @@ Use these instead of writing your own list of scopes. - `get_oauth_scopes_supported()`: what the OAuth server advertises. - **Frontend type:** `APIScopeObject` from `~/types`. - **Frontend list at runtime:** `Object.values(ScopeObjectEnumApi)`, from `products/access_control/frontend/generated/api.schemas`. +- **Python names and descriptions:** `get_scope_docs()` from `posthog.scope_docs`. - **Frontend labels and groups:** `API_SCOPES`, `API_SCOPE_GROUPS` and `getScopeDescription` from `lib/scopes`. - **OAuth lists:** `OAUTH_SCOPES_SUPPORTED` and `OAUTH_SCOPES_HIDDEN`, from `lib/oauthScopes.generated` in the web app and `services/mcp/src/lib/oauth-scopes.generated.ts` in the MCP server. The two files are the same. diff --git a/.agents/skills/building-the-toolbar/SKILL.md b/.agents/skills/building-the-toolbar/SKILL.md index 768fe58b8f26..7ca3ab50b7db 100644 --- a/.agents/skills/building-the-toolbar/SKILL.md +++ b/.agents/skills/building-the-toolbar/SKILL.md @@ -1,6 +1,6 @@ --- name: building-the-toolbar -description: Builds, watches, checks and verifies the PostHog toolbar in both of its forms, the floating toolbar that posthog-js loads on customer pages and the browser extension that shows the toolbar in a Chrome side panel. Use when changing anything under frontend/src/toolbar/ (floating toolbar or extension), frontend/toolbar-config.mjs, or the toolbar build and guard scripts, or when asked to build, watch, load unpacked, or test the toolbar or the toolbar extension in a browser. Covers `pnpm build:toolbar-all` and `pnpm watch:toolbar-all`, the check-toolbar-size, check-toolbar-graph and check-toolbar-csp-eval guards, why both forms must be built and kept in sync, and how to check each one in a real browser. +description: Builds, watches, checks and verifies the PostHog toolbar in both of its forms, the floating toolbar that posthog-js loads on customer pages and the browser extension that shows the toolbar in a Chrome side panel. Use when changing anything under frontend/src/toolbar/ (floating toolbar or extension), frontend/toolbar-config.mjs, or the toolbar build and guard scripts, or when asked to build, watch, load unpacked, or test the toolbar or the toolbar extension in a browser. Covers `pnpm build:toolbar-all` and `pnpm watch:toolbar-all`, the check-toolbar-size, check-toolbar-graph and check-toolbar-csp-eval guards, why both forms must be built and kept in sync, how to check each one in a real browser, and when an extension change needs a Sampo changeset to reach the Chrome Web Store. --- # Building the toolbar @@ -100,3 +100,10 @@ Tests and builds don't show a broken overlay, a missing style, or a CSP refusal. Check the forms you didn't mean to change as well. A change to a shared logic shows up in both. + +## Ship an extension change + +The extension reaches users only through a Chrome Web Store release, which starts from a Sampo changeset. +When a change alters what the extension does, add a changeset to the same pull request, as [the extension README](../../../frontend/src/toolbar/extension/README.md#releases) shows. +A change that only touches the floating toolbar, tests or stories needs none. +Never edit the version in `release/package.json` or the changelog by hand. diff --git a/.agents/skills/clickhouse-migrations/SKILL.md b/.agents/skills/clickhouse-migrations/SKILL.md index cae5c87997aa..f1fc3eccf4c4 100644 --- a/.agents/skills/clickhouse-migrations/SKILL.md +++ b/.agents/skills/clickhouse-migrations/SKILL.md @@ -25,8 +25,8 @@ operations = [ ### Node roles (choose by the cluster that owns the object) Only the members of `NodeRole` in `posthog/clickhouse/client/connection.py` are valid: `ALL`, `DATA`, -`INGESTION_EVENTS`, `INGESTION_SMALL`, `INGESTION_MEDIUM`, `ENDPOINTS`, `LOGS`, `APM`, `AI_EVENTS`, -`AUX`, `BATCH_EXPORTS`, `OPS`, `SESSIONS`. A name that is not in that enum fails at import, which aborts +`INGESTION_EVENTS`, `INGESTION_SMALL`, `INGESTION_MEDIUM`, `INGESTION_METRICS`, `ENDPOINTS`, `LOGS`, `METRICS`, `APM`, +`AI_EVENTS`, `AUX`, `BATCH_EXPORTS`, `OPS`, `SESSIONS`. A name that is not in that enum fails at import, which aborts migration discovery and takes every job that runs migrations down with it. Pick the role by **which cluster owns the object**, not by its type alone: @@ -42,6 +42,9 @@ Pick the role by **which cluster owns the object**, not by its type alone: - `[NodeRole.APM]`: the `ingestion-apm` nodes, the metrics ingestion layer. Kafka tables, `Null` input tables, materialized views and writable `Distributed` tables that write into storage on `LOGS`. Stateless like the other ingestion nodes, so storage tables never go here. +- `[NodeRole.METRICS]`, `[NodeRole.INGESTION_METRICS]`: the `metrics` cluster and the `ingestion-metrics` + nodes that write to it. The metrics4 tables run on both these and `LOGS`/`APM` until reads cut over, so a + metrics4 change has to target both pairs. - `[NodeRole.OPS]`, `[NodeRole.LOGS]`, `[NodeRole.AUX]`, `[NodeRole.AI_EVENTS]`, `[NodeRole.SESSIONS]`, `[NodeRole.BATCH_EXPORTS]`: objects that live on a satellite cluster. A table for one of those on `DATA` lands on the wrong nodes and leaves the intended cluster without it, so check where the diff --git a/.agents/skills/debugging-mcp-analytics/SKILL.md b/.agents/skills/debugging-mcp-analytics/SKILL.md index cd85e5763590..045c6d75220c 100644 --- a/.agents/skills/debugging-mcp-analytics/SKILL.md +++ b/.agents/skills/debugging-mcp-analytics/SKILL.md @@ -72,9 +72,11 @@ These are the failure modes that produce a plausible-looking answer rather than `coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name))`. It exists because a single-exec server can report the tool two different ways, and the two eras of data coexist. Today `services/mcp` resolves the inner tool itself and passes it - straight in as the tool name (`execToolName()` in `src/hono/tool-executor.ts`, which falls - back to the literal `exec` when the inner command isn't recognized), so - `$mcp_tool_name` usually already holds the real tool. `$mcp_exec_tool_call_name` is + straight in as the tool name (`analyticsToolName()` in `src/hono/tool-executor.ts`), so + `$mcp_tool_name` usually already holds the real tool. It reads the literal `exec` for the + discovery verbs (`info`, `schema`, `search`, `learn`, `help`, `tools`), which are most + `exec` rows, and for calls naming an unknown tool. Split that bucket by `$mcp_exec_verb` + and `$mcp_exec_target_tool` before reading it as unresolved calls. `$mcp_exec_tool_call_name` is registered in `posthog/taxonomy/taxonomy.py` and coalesced defensively here, but **nothing on master emits it** — treat it as historical rows plus in-flight work, not current producer behaviour. Either way, aggregate through the coalesce: hand-rolling diff --git a/.agents/skills/debugging-mcp-analytics/references/event-vocabulary.md b/.agents/skills/debugging-mcp-analytics/references/event-vocabulary.md index fccf340a96ef..cdcdc16ffc70 100644 --- a/.agents/skills/debugging-mcp-analytics/references/event-vocabulary.md +++ b/.agents/skills/debugging-mcp-analytics/references/event-vocabulary.md @@ -34,7 +34,7 @@ On `$mcp_tool_call` unless noted. These are the properties any instrumented serv | Property | Notes | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `$mcp_tool_name` | The tool the agent called. In single-exec mode this holds the resolved inner tool, or the literal `exec` when unrecognized — see exec-mode below. | +| `$mcp_tool_name` | The tool the agent called. In single-exec mode this holds the inner tool a `call` named, or the literal `exec` for discovery verbs and unknown tools — see exec-mode below. | | `$mcp_tool_description` | The description the agent saw. In single-exec mode this is the dispatcher's static text on every call. | | `$mcp_tool_category` | Server-assigned grouping. Used by the Signals scout to group problem tools. | | `$mcp_intent` | Why the agent made the call — the product's differentiator. Populated from the injected `context` argument, or a configured fallback. | @@ -120,8 +120,8 @@ recovered from the exec command's `call ...` form. **Read this before writing an exec-mode query.** These properties are registered in `posthog/taxonomy/taxonomy.py` and coalesced by the read path, but **no producer on master emits them**. `services/mcp`'s `ToolExecutor.callExecTool()` resolves the inner tool itself and passes -it to `trackToolCall()` as the tool name, falling back to the literal `exec` when the command -isn't recognized — so in current data the inner tool arrives in `$mcp_tool_name`, and these +it to `trackToolCall()` as the tool name, falling back to the literal `exec` for discovery verbs +and unknown tools — so in current data the inner tool arrives in `$mcp_tool_name`, and these dedicated fields are empty. A query built only on them returns nothing. | Property | Notes | diff --git a/.agents/skills/extending-the-admin-console/SKILL.md b/.agents/skills/extending-the-admin-console/SKILL.md index 460cd7665c1a..6c669564e44c 100644 --- a/.agents/skills/extending-the-admin-console/SKILL.md +++ b/.agents/skills/extending-the-admin-console/SKILL.md @@ -12,10 +12,10 @@ Read [`posthog/admin/console/AGENTS.md`](../../../posthog/admin/console/AGENTS.m ## 1. Decide where it goes - **Plain create, read, update or delete on one model** stays in Django admin. The console is not a second CRUD layer. -- **Facts and actions about one team** go in a panel. One panel per product concern, owned by the team that owns the data. -- **A report across many teams** is a tool. Tools only read. An action from a tool row links to the team page. +- **Facts and actions about one team or one organization** go in a panel on that record's page. One panel per product concern, owned by the team that owns the data. +- **A report across many teams, or an action that is not about one record** (a backfill, a re-save across teams) is a tool. An action on one team from a tool row links to that team's page instead of becoming a tool operation. - **A staff page your product already has in the app** (a DRF staff endpoint and scene) is legacy. Do not extend it and do not bridge to it. Ask the user whether to move it into the console as its own change. -- **Anything else** (a new record type such as an organization or user page, a new block kind, a new tool group, a template change) is a shell change. Stop and ask the user: those need a review by the console owners (team-devex), not a product team alone. +- **Anything else** (a new record type such as a user or project page, a new block kind, a new tool group, a template change) is a shell change. Stop and ask the user: those need a review by the console owners (team-devex), not a product team alone. ## 2. Write the panel or tool as data @@ -39,6 +39,7 @@ Invoke `/writing-user-facing-copy`. In the console that means: - **Internal and reversible** (rebuild, clear or purge a cache that refills itself) is a plain operation. - **Gives the customer something back** (enable access, clear a revoke, re-enable sending, add credit) is a plain operation. Give it a form only when it needs input, as Add credit does for the amount and the reason. - **Takes something away from the customer, or cannot be undone** (suspend, revoke, replace a token) sets `danger=True`, takes a form with a `reason` field, and sets `undo_hint` to how staff undo it, or that they cannot. The console writes the reason to the audit log and shows the undo hint on the confirm page. +- **Only the owning team's staff may run it** (approve a deletion, send usage reports): set `gate=Gate.owner()`, and set `staff_group` on the panel or tool's `Owner` to that team's Django group. Use `Gate.group(name)` only when the operation belongs to another team's group. - A handler that moves money or sends messages builds its idempotency key from the `form_nonce` the console posts, so a double submit runs once. - Offer an operation only in the state where it makes sense. The console runs only operations the panel offers right now. - A form that edits current state sets `form_initial` from the record, so an unchanged submit changes nothing. @@ -47,13 +48,13 @@ Invoke `/writing-user-facing-copy`. In the console that means: ## 5. Move a Django admin action without losing anyone 1. Point the operation at the old view with `AdminViewBridge(Model, SomeAdmin.view_method)` while the logic still lives in the `ModelAdmin`. -2. Replace the old button in Django admin with `MovedToConsole.team_action(team, "", "")`, so staff who look in the usual place find the new one. +2. Replace the old button in Django admin with `MovedToConsole.action("", obj, "", "")`, so staff who look in the usual place find the new one. 3. When a whole page moves, keep its old URL name and route it to `MovedToConsole.page_redirect`, which tells staff the page moved. -4. Every old URL keeps working, query parameters included. Support staff keep bookmarks to old admin pages and actions. When you delete a team action's view, add its old path to `MOVED_TEAM_ACTIONS` in `posthog/admin/console/urls.py`, so it redirects to the console form or panel. +4. Every old URL keeps working, query parameters included. Support staff keep bookmarks to old admin pages and actions. When you delete a record action's view, add its old path to the `MOVED_*_ACTIONS` map of its record type in `posthog/admin/console/urls.py`, so it redirects to the console form or panel. ## 6. Before you call it done -- Extend `posthog/admin/test_admin_console.py`: add the panel title to the team page test, and a parameterized case for each new operation's success and refusal. Do not add a new test file. +- Extend `posthog/admin/test_admin_console.py`: add the panel title to the record page test, and a parameterized case for each new operation's success and refusal. Do not add a new test file. - Run `pnpm --filter=@posthog/frontend build:admin-console` when you add an owner crest. Commit the output. -- Open `/admin/console/teams//` on the local stack (log in as `test@posthog.com`) and look at the page at a narrow and a wide window. A panel that reads wrong in the browser is not done, whatever the tests say. +- Open the record page (`/admin/console/teams//` or `/admin/console/organizations//`) on the local stack (log in as `test@posthog.com`) and look at the page at a narrow and a wide window. A panel that reads wrong in the browser is not done, whatever the tests say. - Put a screenshot of the panel in the PR description. diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md index a6da89278b05..838f27f4678a 100644 --- a/.agents/skills/implementing-warehouse-sources/SKILL.md +++ b/.agents/skills/implementing-warehouse-sources/SKILL.md @@ -511,7 +511,22 @@ A source's OAuth field (`SourceFieldOauthConfig`, a `*_integration_id` input) ca A source opts in with two hooks on its class: `has_legacy_oauth_tokens(job_inputs)` (no network) and `adopt_legacy_oauth(job_inputs)` (can call the provider). HubSpot is the only source that uses them. - `hubspot_legacy_adopted`: the source has `hubspot_secret_key` and `hubspot_refresh_token` and no `hubspot_integration_id`. With `--execute`, the command exchanges the refresh token, reads the hub id from HubSpot, and reuses the `hubspot` row of the team for that hub id or creates one. It then sets `hubspot_integration_id` and the column, and removes the two legacy keys. Preview only counts. `reconcile_warehouse_source_credentials --verify` reports it as `legacy_oauth_tokens_in_job_inputs`. -- `skipped_legacy_token_unusable`: HubSpot rejected the token, returned no hub id, failed after the retries, or the inputs changed during the run. The command writes nothing. A later run tries again. +- `skipped_legacy_token_unusable`: HubSpot rejected the token, returned no hub id, or the inputs changed during the run. The command writes nothing. A later run tries again. Only a confirmed rejection of the refresh token (`invalid_grant`, `BAD_REFRESH_TOKEN` or `BAD_HUB`) makes the grant unusable. Any other failure, such as a network error, a 408, a 429, a 5xx, a malformed body or a refusal of PostHog's app credentials (`invalid_client`), is a retryable error: the backfill counts it as `failed`, and the cleanup command counts it as `skipped_transient`. + +#### Cleanup of dead credentials + +`cleanup_dead_warehouse_source_credentials` clears credentials that no sync can use. Some sources hold legacy OAuth tokens that HubSpot no longer accepts, or an integration pointer whose row is gone. These sources block a clean `reconcile_warehouse_source_credentials --verify`. + +A source is dead only when both rules hold: + +- Its credentials are dead. Either it holds legacy OAuth tokens (`has_legacy_oauth_tokens`) and `adopt_legacy_oauth` raises `LegacyOAuthGrantUnusable`, or its column or its in-use OAuth pointer names no integration row of the team. A source that still has a live reference is left alone. +- No schema with `should_sync` synced in the last `--idle-days` days (default 30), and no schema is running. + +Preview is the default and makes no network call. With `--execute`, the command takes the source lock, writes a backup with the model of the strip command, removes the legacy token keys (`legacy_oauth_keys()` of the source class) and the dangling pointers at their real paths, clears a dangling column, and sets the marker `dead_credentials_cleared_at`. It never deletes a source, a schema or a row. Output is counts only. Stripped sources are skipped. + +- The backfill skips a marked source (`skipped_cleaned`), and `reconcile --verify` treats it as clean. +- `restore_warehouse_source_connection_fields` puts the backed-up `job_inputs` back and removes the marker. +- Do not write the marker key from source code. `SourceInputsStore` hides it from readers and keeps it on save. ## Implementing `source_for_pipeline` diff --git a/.agents/skills/setting-up-devbox/SKILL.md b/.agents/skills/setting-up-devbox/SKILL.md index 4d87e2de762f..c6937a374922 100644 --- a/.agents/skills/setting-up-devbox/SKILL.md +++ b/.agents/skills/setting-up-devbox/SKILL.md @@ -218,6 +218,7 @@ The script runs the main checkout's hogli, so pull the main checkout after a cha - Superset, Orca and worktrunk run `bin/devbox-worktree-hook removing` before they delete a worktree. It runs `hogli devbox:worktree:teardown --background`, which stops the mirror while the files still exist and then stops the box in the background (log: `~/.config/posthog/hogli-devbox-teardown.log`). The box's record is then dropped, so the box no longer belongs to any worktree. - Emdash and Conductor run `bin/devbox-worktree-hook archiving` instead, because they keep an archived worktree on disk. It adds `--keep-record`, so when the app restores the worktree, its create hook starts the same box again. - herdr reports a removal only after the folder is gone, so its plugin runs `devbox:worktree:prune --path `. t3code has no removal hook, so run `hogli devbox:worktree:prune` by hand. +- herdr has two ways to use devboxes. With `--auto herdr`, agents run on the laptop and each herdr worktree gets its own box. With `hogli devbox:open --herdr`, the box is a herdr machine, and agents and worktrees run on it with no mirror. `devbox:open --herdr