Repository navigation
[Docs][Backports] Update developer documentation to reflect new automated backport workflow (#19284) - #67
Closed
mrodm wants to merge 41 commits into
Closed
[Docs][Backports] Update developer documentation to reflect new automated backport workflow (#19284)#67mrodm wants to merge 41 commits into
mrodm wants to merge 41 commits into
Conversation
- Add grok processors to better handle different formattings of the user fields in checkpoint logs. Added support for full name with email, email, and name only.
…tic#20031) Set event.kind, event.category and event.type for every SNS log family through a table-driven Painless script keyed on logtype, with allowed/denied event.type modifiers derived case-insensitively from the rule action. The periodic statistics families (monitor, filterstat, count, routerstat, authstat, ipsecstat) are tagged event.kind: metric so health telemetry can be excluded from detection scopes with a single filter. Unknown families fall back to event.kind: event without asserting a category or type, covered by a dedicated test sample. The script only runs on documents without an error, so the pipeline_error kind set by sub-pipeline failure handlers is preserved. Add the ECS network mappings tracked in elastic#10114: rcvd to destination.bytes (mirroring the existing sent mapping), network.bytes as the sum of both, srcif/srcifname/dstif/dstifname copied to observer.ingress and observer.egress interface fields, and network.direction computed with private ranges as internal networks. Promote alarm fields (alarmid to event.code, risk to event.risk_score, pri to event.severity on the alarm scale where 1 is major and 4 is minor), HTTP proxy fields (arg to url.original, url.domain from destination.domain), and populate related.hosts. All mappings are copies; no stormshield.* field moves or changes location. Guard the ASN geoip processors with null checks, matching the geo processors. Fix the count sub-pipeline, which failed on every count log since creation: the script called splitByToken, which does not exist in Painless, and collapsed all rules into a single map. Rule counters are now a sorted array under stormshield.metadata.rule_stats. Extend the metadata move list with the documented alarm, proxy, vpn and auth fields so they land under stormshield.metadata instead of being left unmapped. Add pipeline test samples for the alarm, auth, web, vpn, xvpn, count and filter families, derived from the log format examples in the official Stormshield documentation with sanitized addresses, including uppercase and empty action values. Add an Alarms (IPS) dashboard keyed on the new categorization fields. Document the categorization table in the README.
…ix CI for snapshot (elastic#20097) Set the mock service hostname to svc-<service> for atlassian_confluence, atlassian_jira, cisco_secure_endpoint, and okta so stream-generated absolute Link/next URLs match the origin injected into system test configs, fixing flaky pagination in system tests.
…ert CEL programs (elastic#20190) [crowdstrike]: use inclusive >= cursor boundary in vulnerability and alert CEL programs Strict > on updated_timestamp permanently skips records that share a boundary timestamp when a page or error boundary falls mid-group. Switch to >= so same-timestamp records are always re-queried. Re-fetched boundary records are absorbed without creating duplicates: both pipelines run fingerprint processors that include updated_timestamp so events that were already ingested are dropped.
Add a new entry in `.backports.yml` for the `aws` package backport branch `backport-aws-6.x`, based on version 6.21.0 (commit 913b779). This branch is active (`archived: false`) and configured to strip all other packages (`remove_other_packages: true`).
…stic#20233) Bumps [updatecli/updatecli-action](https://github.com/updatecli/updatecli-action) from 3.3.0 to 3.4.0. - [Release notes](https://github.com/updatecli/updatecli-action/releases) - [Commits](updatecli/updatecli-action@5bda7da...5dfc616) --- updated-dependencies: - dependency-name: updatecli/updatecli-action dependency-version: 3.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lastic#20236) The `check-backports-inventory` step had no `if:` condition, causing it to run on backport branches that only carry a subset of packages, which made the validation fail unnecessarily. Add an `if:` guard so the step only runs on PRs targeting `main` or direct pushes/merges to `main`, mirroring the guards already on the dependent `trigger-backport-dryrun` and `trigger-backport-create` steps. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6 to 7. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ld mappings, add pipeline tests (elastic#20132)
…cation_risk (elastic#20211) Add a time watermark (updatedAt__gte) to the threat_event CEL program so each interval only fetches threats updated since the last completed walk, rather than re-walking the entire catalog. The high-water mark is tracked in cursor.last_timestamp and seeded from a new initial_interval manifest variable on the first run. The default poll interval drops from 24h to 5m since the watermark eliminates full re-walks. Add fingerprint processors to both the threat_event and application_risk ingest pipelines so that re-emitted documents with the same id are rejected within a backing index rather than accumulating as duplicates.
* [osquery_manager] Refine query profiles dashboard Apply the UX review while refreshing saved object metadata so package installation does not replay incompatible legacy Lens migrations. * [osquery_manager] Link dashboard changelog to PR * [osquery_manager] Clarify query profiling dashboard copy Describe profiling as enabled by default, document how to disable it, and clarify that the executions metric counts profiled events. Assisted-By: Cursor
…layout (elastic#20048) * [IIS] Add grok pattern for s-ip + cookie + referer + split-bytes log layout The documented IIS access log field layout at packages/iis/docs/README.md:69 (s-ip, cs(cookie), cs(Referer), sc-bytes, cs-bytes as separate fields) has no matching grok pattern in the ingest pipeline, so any log using this exact layout fails ingestion with a grok/number-conversion error instead of parsing cleanly. Adds the missing pattern (SDH-beats#7343). * [IIS] Add generated expected output for cookie/bytes-split pipeline test Adds the missing test-iis-access-cookie-bytes-split.log-expected.json fixture generated via elastic-package test pipeline --generate, which resolves the failing IIS pipeline test. --------- Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
…tic#20251) Add the `changelog-link-check:skip` label when creating sync-changelog PRs so Buildkite does not validate changelog PR links — those links always point to the original backport PR, not the sync PR number. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…API 400 errors (elastic#20254) ti_google_threat_intelligence: recover ioc_stream from persistent API 400 errors The ioc_stream CEL program could never make progress past a collection window that the API persistently rejects with HTTP 400 (for example, a window containing an IOC whose URL identifier exceeds the API's internal 500-byte key limit). On error the program left the stale next-page cursor in state, so every subsequent poll replayed the same failing request with the dead pagination cursor attached, and the window start was never advanced. The input stayed degraded until the Filebeat registry was deleted by hand. On errors the program now discards the in-flight page cursor so retries start from a clean first page. HTTP 400 responses additionally maintain a consecutive-failure counter persisted in the cursor; after three consecutive 400s for the same window the program advances the cursor past the window and resumes collection, surfacing the retries and the skip as error events in the data stream. Other error classes keep the existing retry-forever behaviour. Add a system test that drives the input through the failing-window cycle against the mock API and asserts recovery. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…tic#20248) * [Security Rules] Update security rules package to v9.5.1-beta.1
…on upgrade (elastic#20191) elastic#20080 configured lookup index mode but did not bump the destination index suffix, so existing upgrades kept the old index without the mode. Bump dest_asset-2 → dest_asset-3 (and fleet_transform_version) so a new destination index is created with lookup mode.
Add a self-contained Docker/cgroup harness under packages/o365/_dev/scripts/memcap-agent/ to measure the o365 audit CEL input's single-blob peak memory for the agentless ORR. It runs the real elastic-agent image the way agentless-controller deploys it (single container, self-monitoring off, no GOMEMLIMIT, GODEBUG=madvdontneed=1) against the elastic/stream mock serving one large content blob under a Docker --memory cap, and reads the cgroup memory.peak high-water mark. - run.sh: one capped run (sweep via TOTAL_EVENTS / MEM_LIMIT). - sweep.sh: several blob sizes at a non-OOM cap; fits memory.peak = base + k*blob and derives the OOM boundary for 1Gi / 512Mi. - elastic-agent.yml: input config whose CEL program is a verbatim copy of the shipping data_stream/audit/agent/stream/cel.yml.hbs program block (kept in sync). - corpus/ + mock-config.yml: generator inputs and mock config (self-contained). Measures memory capacity only (events per blob at the OOM ceiling), not throughput/EPS.
… streams (elastic#20192) [tenable_io] Add configurable max_executions to asset, audit, plugin, and scan data streams The max_executions parameter was only exposed in the Fleet UI for the vulnerability data stream. The asset, audit, plugin, and scan streams all use CEL inputs with paginated API calls that can exceed the default limit of executions under sufficient data volume, causing them to go DEGRADED. This change brings those streams to parity with vulnerability by exposing the same optional parameter (default: 2000) in both the manifest and the CEL template for each affected data stream.
…lastic#20182) Bump version for each packages affected by Agentless renaming Packages had documentation changes to reflect the change in naming from "Agentless" to "Elastic Managed" in elastic#19756. This change bumps the version to publish those changes.
* [openai]: remove alerting rule template * add pr number * remove alert from readme
…lastic#20277) Add @elastic/sit-crest-contractors alongside @elastic/security-service-integrations for axonius, beyondtrust_epm, doppel, doppler, workday, and xm_cyber. Relates: elastic#19089
* [Security Rules] Update security rules package to v9.5.1 * Add changelog entry for 9.5.1
WHAT: Map the enriched auditd username/group fields (UID, GID, EUID, EGID, FSUID, FSGID, SUID, SGID, AUID, AGID) that appear as key=value suffixes on kernel audit records (e.g. UID="root" AUID="unset") to their corresponding ECS user.* name fields. The mapping uses set processors with override: false so pre-existing values are never clobbered, and skips placeholder values ("unset", "?") that carry no meaningful identity. All mapped names are also appended to related.user for cross-referencing.
WHY: The auditd enriched format already maps the numeric ID fields (e.g. user.id, user.audit.id) from type=SYSCALL records, but the human-readable name counterparts were left unmapped. This meant searches and alerts built on user.name or related.user would miss events where only the enriched format was available, making the integration inconsistent compared to the go-libaudit preparsed path which does populate these fields.
…pAddress (elastic#20282) island_browser: fix device pipeline parsing of multi-NIC internalIpAddress The device ingest pipeline used a single dissect on the first "/" to extract an IP from internalIpAddress. When the Island API returned a "/"-separated list of "<ip> | <adapter>" pairs for hosts with 3+ NICs, the leftover separators failed the subsequent convert-to-ip and produced event.kind: pipeline_error. Strip the adapter labels with gsub, split the cleaned list into individual addresses, and append each one to related.ip. The single-IP path is unchanged. Adds multi-NIC, two-NIC, and single-NIC-with-adapter cases to the device pipeline test. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
… asset types (elastic#19795) * cloud_asset_inventory: add extra fields * changelog: update PR link * fix: move CreatedAt from lifecycle to attributes * cloud_asset_inventory: add EKS, route53 fields; updates * update version requirements * include elastic#20024
…stic#20269) (elastic#20271) * changelog: security_detection_engine 9.4.8 (backport sync from PR elastic#20269) * Add entry from 20249 * Fix alignment Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Format chnagelog --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com> Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Co-authored-by: Shashank K S <Shashank.Suryanarayana@elastic.co>
* add unix input custom integration * add CODEOWNERS * Fix PR in changelog * fix owner in manifest * trap the SIGHUP * switch to stream for testing * Add change to set permissions for socket dir * respond to feedback * update readme
…ic#20304) Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…lastic#20194) Expand FDR ingest categorization for additional event types, including agentic/AI telemetry, Public Cloud asset inventory, on-demand scan (ODS), sensor channel downloads, Custom IOA/AMS/CSA detections, data protection egress, Open Directory IAM, WSL distribution lifecycle, and accessory device events. Map Public Cloud FDR events to cloud.* and host.ip, DataEgress fields to file.*. Derive FDR event.outcome from crowdstrike.Status using NTSTATUS severity bits. The anonymized samples are derived from CrowdStrike documentation.
Add a new "Backport checklist comment" section to developer-workflow-support-old-package.md explaining when the post-backport-checklist workflow posts a comment, what it shows, that it is currently informational only (no checkboxes yet), and how to suppress a branch via `archived` or `maintained_until` in .backports.yml. Remove the orphaned backport_input_step.png screenshot, which was no longer referenced by any documentation page. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a new "Package owner synchronization" section to developer-workflow-support-old-package.md covering: - Automatic sync during apply: what is synced (manifest.yml owner.github, CODEOWNERS package line and sub-path entries), the separate "Sync <pkg> package owners from main" commit, no-op behavior when already in sync, and warn-and-continue when main is unreachable or the package has been removed. - check-backport-owners CI step: when it runs (PRs targeting backport-* branches on packages/** or CODEOWNERS changes), the three PR comment outcomes (in-sync / mismatch with team list / check-failed), required author action on mismatch, and that soft_fail: true means a mismatch does not block merge. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
) Expand the maintained_until and archived field descriptions in the .backports.yml schema table to document their full semantics: maintained_until excludes the branch automatically once the date passes (strictly before today UTC), and is preferred when the end-of-life date is known; archived: true excludes the branch immediately and does not delete it. Also add the previously undocumented remove_other_packages field, explaining its default true behavior and why it exists. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… step Add a note in ci_pipelines.md documenting the check-changelog-pr-links Buildkite step: what it validates (changelog entry links match the PR being merged), that it is soft_fail, and how to skip it via the changelog-link-check:skip label. Explain that the sync-backport-changelog workflow adds this label automatically on sync PRs since their entries intentionally reference the original backport PR. Also clarify that check-backports-inventory runs on PRs targeting main and direct pushes to main only — it is skipped on backport-branch PRs, which carry only a subset of packages. Relates to elastic#19284. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…check (elastic#19214) Rewrite step 3 in developer-workflow-support-old-package.md to document backport_apply.sh as the recommended tool: syntax, required/optional arguments, and a numbered description of what it does end-to-end (cherry-pick, version bump, changelog entry with placeholder link, owner sync commit, PR creation, and link fix). Keep the manual cherry-pick as an alternative and preserve the note about subsequent fixes not needing a new branch. Add a forward-looking note about the auto-backport workflow (PR elastic#20197). Add a note to ci_pipelines.md about the coming check-changelog-versions-in-main step, which catches sync collisions on backport-branch PRs before merge. Relates to elastic#19284. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…le date Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ckages Update docs to reflect the now-merged auto-backport automation (PR elastic#20197) and upcoming skip_checklist_packages field (PR elastic#20384): - Replace "Coming soon" note with description of auto-backport.yml: on merge, the workflow reads checked branches from the checklist and creates backport PRs automatically; post-merge checklist edits also trigger it. - Update checklist comment section: describe checkboxes and auto-backport trigger, update example to show [ ] checkbox format and new intro text, and note that the comment is now recreated (delete+POST) on each push. - Add "Suppressing a package from the checklist" block documenting the new top-level skip_checklist_packages field (elastic#20384). - ci_pipelines.md: remove stale "coming soon" qualifier from the check-changelog-versions-in-main step — it shipped with elastic#20197. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backport branchesImportant Only branches for packages touched by this PR's current diff are shown. Tick the branches you want to backport to. PRs will be created automatically on merge, or when you update this checklist after merge. cloud_asset_inventory
cloud_security_posture
crowdstrike
kubernetes
tenable_io
Tip If a branch above is no longer required, set |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
WHAT:
docs/extend/developer-workflow-support-old-package.md:
backport_apply.shas the recommended tool: syntax, required/optional argument tables, and a
numbered description of the 6 steps it performs (cherry-pick, version bump,
changelog entry, owner sync, PR creation, sentinel link fix). Keeps manual
cherry-pick as an alternative path.
a description of the automatic
sync-backport-changelog.ymlflow — fires onbackport PR merge, opens a sync PR against
mainlabelledbackport:sync-changelogandchangelog-link-check:skip, posts a commenton the merged backport PR.
performed by
backport_apply.sh(what is synced, commit message, no-op andwarn-and-continue behaviours) and the
check-backport-ownersCI step (threepossible comment outcomes, soft_fail behaviour).
content with
[ ]checkboxes, howauto-backport.ymltriggers on checkedbranches at merge (and on post-merge edits), how to suppress a branch via
archived/maintained_until, and how to suppress a package via the newtop-level
skip_checklist_packagesfield (Skip security_detection_engine from backport checklist comment elastic/integrations#20384)..backports.ymlfield descriptions: fullmaintained_untilandarchivedsemantics (auto-exclusion, archiving does not delete the branch,excluded from checklist and branch creation — not schema validation), added
the previously undocumented required
remove_other_packagesfield (includingit in both manual entry examples).
docs/extend/images/backport_input_step.png(no longerreferenced in any doc).
docs/ci_pipelines.md:
check-changelog-pr-linksdocumentation: validates changelog entry linksmatch the PR being merged, soft_fail, changelog-link-check:skip label, and why
sync-changelog PRs use it automatically.
check-backports-inventoryruns on PRs targetingmainonly (skippedon backport-branch PRs).
integrations-backport-dispatchpipeline as the seventhpipeline, with description of the two-pipeline architecture and why the trigger
was split from the public pipeline.
check-changelog-versions-in-mainstep documentation: verifies that nochangelog version introduced by a backport PR already exists on
main.WHY:
The backport workflow has been significantly automated across several merged PRs
(elastic#19997, elastic#20062, elastic#20119, elastic#20236, elastic#19930, elastic#20251, elastic#20197, elastic#20284) with further
automation in progress (elastic#20384). The developer documentation still described the
old fully-manual 4-step procedure. This update brings the docs in line with the
current automated workflow so contributors have accurate guidance on how to
release hotfixes for older package versions.
Author's Checklist
How to test this PR locally
Review the rendered output of the two changed files:
docs/extend/developer-workflow-support-old-package.mddocs/ci_pipelines.mdCross-check key claims against the codebase:
dev/scripts/backport_apply.sh— script CLI matches documented syntaxdev/backports/checklist/checklist.go— checklist comment matches example.buildkite/pipeline.yml— check-backports-inventory guard, check-changelog-pr-links stepdev/backports/changelog/sync.go— sync PR labelsRelated issues