feat: experimental traffic analysis#2848
Conversation
🦋 Changeset detectedLatest commit: cf2429b The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
sobanieca-redocly
changed the title
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||
Performance Benchmark (Lower is Faster)
|
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
@cursor review |
|
@cursor review |
|
@cursor review |
|
@cursor review |
|
@cursor review |
Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>
| type Options, | ||
| type ValidateFunction, | ||
| type Ajv2020 as Ajv2020Instance, | ||
| } from 'ajv/dist/2020.js'; |
There was a problem hiding this comment.
wouldn't it be better to use our fork @redocly/ajv, this could unify usage in the repo
| method: params.method, | ||
| url: params.forwardUrl.toString(), | ||
| httpVersion: `HTTP/${params.req.httpVersion}`, | ||
| cookies: parseCookieHeader(singleHeader(params.req.headers.cookie)), |
There was a problem hiding this comment.
maybe it's worth cleaning up sensitive data ?
There was a problem hiding this comment.
For now this command should be used for short-lived local capture (like during e2e tests etc). Not sure about use cases yet, I would wait until we get more feedback from users before designing proper sensitive data removal...
| findings.push(...ruleFindings); | ||
| } | ||
| } catch (error) { | ||
| findings.push({ |
There was a problem hiding this comment.
is it actually finding? maybe simply notify the user via the logger?
There was a problem hiding this comment.
Used logger instead. Thanks!
| * JSON object) and serialized through a promise chain so concurrent captures | ||
| * never interleave file writes. | ||
| */ | ||
| export class HarWriter { |
There was a problem hiding this comment.
keeps all entries in memory and re-serializes + rewrites the entire file on every request
Maybe it would be possible to add debouncing or stream the writes using something like stream-json:
https://www.npmjs.com/package/stream-json
There was a problem hiding this comment.
I've changed the idea here. We will store data in temporary file which get's converted into HAR. This should cut memory usage.
| return '***'; | ||
| } | ||
|
|
||
| return `${value.slice(0, 4)}…`; |
There was a problem hiding this comment.
Removed magic number and named it MASK_REVEALED_PREFIX_LENGTH. This is just arbitrary number on how many characters we want to show from masked value.
| reason: string; | ||
| } | ||
|
|
||
| interface SkippedSecurityCheck { |
There was a problem hiding this comment.
Duplicate removed. Thanks.
| return { specFiles: [specPath], fromDirectory: false }; | ||
| } | ||
|
|
||
| const YAML_OPENAPI_ROOT_KEY_RE = /^(['"]?)openapi\1\s*:/m; |
There was a problem hiding this comment.
is this changes necessary ?
There was a problem hiding this comment.
Yes - this is to avoid typescript issues between drift --format and lint --format
There was a problem hiding this comment.
But I'm checking for some alternatives. Maybe indeed we can avoid changes in lint 👀
There was a problem hiding this comment.
Renamed param to report-format with alias. This allows to get rid of overlap with lint command types.
There was a problem hiding this comment.
Is the plugin system compatible with the one that's already in the core? I think having two different plugin systems for the cli is a bad idea
There was a problem hiding this comment.
In this case we are dealing with different purpose of rules (in this case traffic data). This means different data contracts etc. I would defer this unification when this command goes out of experimental phase.
There was a problem hiding this comment.
can you add documentation for v2 for both commands?
There was a problem hiding this comment.
Added docs, but @JLekawa I will need your re-review on this one. Thanks!
Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7d65c90. Configure here.
4 participants
What/Why/How?
Added two new (experimental) commands -
driftandproxyfor collecting traffic data and validating it against provided OpenAPI definition.Reference
Testing
Screenshots (optional)
Check yourself
Security
Note
Medium Risk
Large new experimental surface (traffic parsing, security heuristics, live HTTP proxy forwarding) touches auth-related validation on captured traffic, though it reuses existing spec/validation stacks and is opt-in/experimental.
Overview
Adds two experimental CLI commands (minor
@redocly/clichangesets) for comparing real HTTP traffic to OpenAPI 3.x specs.driftingests traffic from a file or directory (HAR, Kong, Nginx/Apache JSON, NDJSON, with auto-detection and pluggable parsers), indexes specs via@redocly/openapi-core, matches exchanges to operations (--match-modeor--server), runs built-in rules (undocumented endpoints, schema consistency, security baseline; optionalowasp-api-top10), and emits pretty / JSON / CSV / SARIF reports with exit code1on error-level findings.proxyruns a local reverse proxy (undiciupstream) that streams HAR entries to disk and, when--apiis set, reuses the sameValidationSessionengine asdriftfor live findings plus a shutdown report. Captured HAR can be replayed throughdrift.Shared implementation includes OpenAPI operation matching, AJV schema validation with readOnly/writeOnly handling, rule/plugin registries, and documentation/sidebar updates plus e2e snapshot tests for formats and outputs.
Reviewed by Cursor Bugbot for commit cf2429b. Bugbot is set up for automated code reviews on this repo. Configure here.