[nginx_otel_integ] Add composable Nginx OpenTelemetry integration

[ishleenk17]

Summary

• Add a composable Nginx (OpenTelemetry) integration that collects stub_status metrics via nginx_otel_input and access/error logs via filelog_otel.
• Parse combined access logs and multiline error logs with transform processors owned by this package (processors.yml.hbs), so filelog_otel 0.2.0 is reused without changes.
• Depend on nginx_otel content assets for dashboards and keep native OTel datasets (nginxreceiver.otel, nginx.access.otel, nginx.error.otel).

Test plan

• elastic-package test policy --data-streams access,error,metrics against local nginx_input_otel and filelog_otel sources
• elastic-package test system --data-streams access,error,metrics on a 9.4+ stack
• Install the package on a local stack, point the agent at a live Nginx stub_status URL and access/error files, and confirm grokked fields such as http.response.status_code, url.original, log.level, and process.pid

Made with Cursor

[github-actions]

Elastic Docs Style Checker (Vale)

Summary: 1 warning, 6 suggestions found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.

File	Line	Rule	Message
packages/nginx_otel_integ/_dev/build/docs/README.md	3	Elastic.EndPuntuaction	Don't end headings with punctuation.

💡 Suggestions (6): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/nginx_otel_integ/_dev/build/docs/README.md	43	Elastic.Semicolons	Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.
packages/nginx_otel_integ/_dev/build/docs/README.md	44	Elastic.Semicolons	Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.
packages/nginx_otel_integ/_dev/build/docs/README.md	46	Elastic.Semicolons	Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.
packages/nginx_otel_integ/_dev/build/docs/README.md	47	Elastic.Semicolons	Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.
packages/nginx_otel_integ/_dev/build/docs/README.md	48	Elastic.Semicolons	Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.
packages/nginx_otel_integ/changelog.yml	1	Elastic.Versions	Use 'later versions' instead of 'newer versions' when referring to versions.

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[Copilot]

🟡 Changes recommended

Unresolved manifest, storage, dependency-test, and log parsing issues block approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a composable Nginx OpenTelemetry integration for metrics, access logs, and error logs using reusable OTel input packages.

Changes:

• Adds metrics, access, and error data streams.
• Adds access/error parsing processors and test fixtures.
• Adds documentation, policy/system tests, Docker setup, and changelog entries.

File summaries

File	Reviewed changes
packages/nginx_otel_integ/manifest.yml	Integration metadata and dependencies. Moderate (1 vote): add persistent file offset storage. Critical (1 vote): bump format version to 3.6.5.
packages/nginx_otel_integ/img/logo_nginx.svg	Adds the integration icon.
packages/nginx_otel_integ/docs/README.md	Documents setup and collected data.
packages/nginx_otel_integ/data_stream/metrics/sample_event.json	Metrics sample event.
packages/nginx_otel_integ/data_stream/metrics/manifest.yml	Metrics stream definition.
packages/nginx_otel_integ/data_stream/metrics/fields/base-fields.yml	Metrics fields.
packages/nginx_otel_integ/data_stream/metrics/_dev/test/system/test-default-config.yml	Metrics system test configuration.
packages/nginx_otel_integ/data_stream/metrics/_dev/test/policy/test-default.yml	Metrics policy test input.
packages/nginx_otel_integ/data_stream/metrics/_dev/test/policy/test-default.expected	Expected metrics policy output.
packages/nginx_otel_integ/data_stream/error/sample_event.json	Error-log sample event.
packages/nginx_otel_integ/data_stream/error/manifest.yml	Error-log stream definition.
packages/nginx_otel_integ/data_stream/error/fields/base-fields.yml	Error-log fields.
packages/nginx_otel_integ/data_stream/error/agent/stream/processors.yml.hbs	Error-log parsing. Moderate (2 votes): assign the extracted error time to time so @timestamp reflects event time.
packages/nginx_otel_integ/data_stream/error/_dev/test/system/test-default-config.yml	Error system test. Nit (1 vote each): exercise multiline continuation handling and assert representative parsed fields.
packages/nginx_otel_integ/data_stream/error/_dev/test/policy/test-default.yml	Error policy test input.
packages/nginx_otel_integ/data_stream/error/_dev/test/policy/test-default.expected	Expected error policy output.
packages/nginx_otel_integ/data_stream/access/sample_event.json	Access-log sample event.
packages/nginx_otel_integ/data_stream/access/manifest.yml	Access-log stream definition. Nit (1 vote): add override coverage for include and start_at, including analogous streams.
packages/nginx_otel_integ/data_stream/access/fields/base-fields.yml	Access-log fields.
packages/nginx_otel_integ/data_stream/access/agent/stream/processors.yml.hbs	Access-log parsing. Moderate (3 votes): support IPv4, IPv6, and host-name remote addresses.
packages/nginx_otel_integ/data_stream/access/_dev/test/system/test-default-config.yml	Access system test. Nit (1 vote): assert representative parsed access fields.
packages/nginx_otel_integ/data_stream/access/_dev/test/policy/test-default.yml	Access policy test input.
packages/nginx_otel_integ/data_stream/access/_dev/test/policy/test-default.expected	Expected access policy output.
packages/nginx_otel_integ/changelog.yml	Release notes.
packages/nginx_otel_integ/_dev/test/config.yml	Local dependency test configuration. Moderate (2 votes): add the local dependency mapping under system as well as policy.
packages/nginx_otel_integ/_dev/deploy/docker/sample_logs/error.log	Error-log fixture. Nit (2 votes): add a continuation line while retaining three expected events.
packages/nginx_otel_integ/_dev/deploy/docker/sample_logs/access.log	Access-log fixture.
packages/nginx_otel_integ/_dev/deploy/docker/nginx.conf	Nginx test configuration.
packages/nginx_otel_integ/_dev/deploy/docker/nginx-continuous.conf	Continuous-log configuration.
packages/nginx_otel_integ/_dev/deploy/docker/docker-compose.yml	Docker test services.
packages/nginx_otel_integ/_dev/build/docs/README.md	Built documentation.

Review details

Suppressed comments (5)

packages/nginx_otel_integ/data_stream/access/_dev/test/system/test-default-config.yml:10

• This system test only checks that five records arrive; it would still pass if the access transform or its grok pattern stopped populating the fields promised by the package. Add field assertions for representative parsed values such as http.request.method, http.response.status_code, url.original, and user_agent.name.

  hit_count: 5

packages/nginx_otel_integ/data_stream/access/manifest.yml:15

• These input variables are exposed in Fleet, but the only policy test for this stream supplies their default values; there is no override expectation verifying that user-selected values flow through the merged filelog_otel template. Add an overrides test (and generated expectation) for at least include and start_at, and cover the analogous error and metrics streams.

      - name: include
        type: text
        title: Access log paths
        required: true
        multi: true

packages/nginx_otel_integ/data_stream/error/_dev/test/system/test-default-config.yml:11

• This fixture has exactly one physical line per error entry, so hit_count: 3 passes even if multiline aggregation is removed or broken. Add a continuation line to one sample entry while keeping the expected record count at three, so the system test verifies the multiline behavior advertised by this stream.

  hit_count: 3

packages/nginx_otel_integ/data_stream/error/_dev/test/system/test-default-config.yml:11

• This system test only checks the record count, so it does not verify that the error transform extracts log.level, process.pid, or message (and would not catch a multiline/parser regression). Add field assertions for representative parsed error fields.

  hit_count: 3

packages/nginx_otel_integ/manifest.yml:29

• Both log streams select filelog_otel, whose rendered receiver has no storage setting, and the generated policy contains no file_storage extension. File offsets therefore are not persisted across an Agent restart, which can lose or replay log records; add a persistent storage binding to the input or to these streams before relying on this integration for continuous collection.

• Files reviewed: 29/32 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: ea27e76

History

• 💔 Build #49546 failed 1c47aed