Repository navigation
Conversation
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution! |
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
…ntegrations into datastream-compute
…grations into datastream-application
TL;DRThe Buildkite failure is caused by an auto-merge conflict during the Remediation
Investigation detailsRoot CauseThe failing job runs Evidence
A later Verification
Follow-up
Note 🔒 Integrity filter blocked 2 itemsThe following items were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | noneWhat is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
…ntegrations into datastream-application
| "created": "2024-12-25T22:17:25.000Z", | ||
| "kind": "event", | ||
| "original": "{\"asset_type\":\"urls\",\"internal_axon_id\":\"9c65d891077147892d0e632eb0cf6ebf\",\"adapters\":[\"okta_adapter\"],\"adapter_list_length\":1,\"event\":{\"accurate_for_datetime\":\"Thu, 13 Nov 2025 00:02:55 GMT\",\"adapter_categories\":[\"IAM\",\"SaaS Management\"],\"client_used\":\"67fd09cd782eb39db73d1af1\",\"data\":{\"domain\":\"example-domain\",\"accurate_for_datetime\":\"Thu, 13 Nov 2025 00:02:55 GMT\",\"activity_status_active\":0,\"activity_status_active_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.activity_status\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Active\"}],\"activity_status_inactive\":288,\"activity_status_inactive_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.activity_status\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Inactive\"}],\"app_id\":\"990f042e1d803894A3ae\",\"application_and_account_name\":\"okta/okta-demo\",\"auth_type\":[],\"created\":\"Wed, 25 Dec 2024 22:17:25 GMT\",\"extension_type\":\"SSO\",\"fetch_time\":\"Thu, 13 Nov 2025 00:02:18 GMT\",\"first_fetch_time\":\"Mon, 14 Apr 2025 13:27:09 GMT\",\"first_seen\":\"Mon, 22 Apr 2024 21:22:25 GMT\",\"from_last_fetch\":true,\"grant_types\":[],\"id\":\"cd2f40fe8670900112ab\",\"id_raw\":\"okta_adapter_0!990f042e1d803894A3ae\",\"integration_type\":\"Admin Managed\",\"is_admin\":16,\"is_admin_hyperlink\":[{\"compOp\":\"true\",\"field\":\"specific_data.data.permissions.is_admin\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0}],\"is_fetched_from_adapter\":true,\"is_identity\":0,\"is_identity_hyperlink\":[{\"compOp\":\"true\",\"field\":\"specific_data.data.permissions.is_identity\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0}],\"is_operational\":true,\"last_fetch_connection_id\":\"67fd09cd782eb39db73d1af1\",\"last_fetch_connection_label\":\"okta-demo\",\"last_seen\":\"Mon, 07 Apr 2025 13:05:30 GMT\",\"last_used\":\"Mon, 07 Apr 2025 13:05:30 GMT\",\"name\":\"Cloudflare\",\"not_fetched_count\":0,\"permissions\":[{\"is_admin\":true,\"name\":\"allow_api_access_to_account\",\"users_amount\":9}],\"redirect_uris\":[],\"related_vendor_name\":\"Cloudflare\",\"scope_tag_calendar\":0,\"scope_tag_calendar_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.app_id\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"990f042e1d803894A3ae\"}],\"scope_tag_drive\":0,\"scope_tag_drive_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.permissions.scope_tag\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Drive\"}],\"scope_tag_mail\":0,\"scope_tag_mail_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.permissions.scope_tag\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Mail\"}],\"source_application\":\"Okta\",\"type\":\"Extensions\",\"urls\":[],\"user_count\":288,\"user_count_link\":[{\"bracketWeight\":0,\"compOp\":\"IN\",\"field\":\"specific_data.data.mail\",\"leftBracket\":0,\"logicOp\":\"\",\"not\":false,\"rightBracket\":0,\"value\":\"ronald.mays@demo.local,justin.baugh@demo.local,brian.williamskaren.cox@demo.local,bill.mcnay@demo.local,billy.woodruff@demo.local,hazel.contreras@demo.local,eduardo.mandeville@demo.local,karen.neal@demo.local,troy.hooper@demo.local,andreas.rice@demo.local,janis.henry@demo.local,mary.tavares@demo.local,patrick.rowe@demo.local,kenneth.gardner@demo.local,paul.hendricks@demo.local,tanya.wolf@demo.local,kathleen.arroliga@demo.local,robert.blunt@demo.local,charles.duncan@demo.local,margarita.zapata@demo.local,charles.paredes@demo.local,floyd.conrad@demo.local,tammy.hawkins@demo.local,sam.chavez@demo.local,francis.rivera@demo.local,brandon.lilly@demo.local,matthew.wiley@demo.local,lacey.smith@demo.local,tracy.white@demo.local\"}],\"username_formats\":[],\"users_amount\":288},\"initial_plugin_unique_name\":\"okta_adapter_0\",\"plugin_name\":\"okta_adapter\",\"plugin_type\":\"Adapter\",\"plugin_unique_name\":\"okta_adapter_0\",\"quick_id\":\"okta_adapter_0!cd2f40fe8670900112ab\",\"type\":\"entitydata\"}}" | ||
| }, |
There was a problem hiding this comment.
Missing event.category, event.type. Can you please add them?
There was a problem hiding this comment.
Its still missing for some of the pipelines.
There was a problem hiding this comment.
Yes, this is because no other category fits for the other missing pipelines.
There was a problem hiding this comment.
Use configuration and info for the event.category and event.type respectively for the rest.
https://www.elastic.co/docs/reference/ecs/ecs-allowed-values-event-category#ecs-event-category-configuration
| - urls | ||
| - application_services | ||
| - application_resources | ||
| - secrets |
There was a problem hiding this comment.
Do we have test fixtures for all of them added? What about the pipelines, I see for some of them but not for all.
There was a problem hiding this comment.
We don't have logs for these asset types (not even any schema), hence not added in test logs.
There was a problem hiding this comment.
Are there any plans to get some documents? This is a concern.
There was a problem hiding this comment.
We had connected earlier with the Axonius team for logs, they had provided for few assets only.
There was a problem hiding this comment.
I thin we should only add the ones we have the samples for, because only those are mapped correctly. The rest can be added as enhancements later if we get samples.
There was a problem hiding this comment.
The common fields in default.yml will get populated in other asset types as well, hence I think we shouldn't remove support of those types.
There was a problem hiding this comment.
There are serious problems with shipping asset types you have no samples for:
-
No field mappings. The
fields.ymlhas 1018 lines of field definitions, but these were clearly derived from the asset types that have samples. For the 11 types without samples (admin_managed_extensions,user_initiated_extensions,application_addons,admin_managed_extension_instances,user_initiated_extension_instances,application_addon_instances,application_keys,urls,application_services,application_resources,secrets), we have no idea what fields those responses contain. Any type-specific fields will either fail mapping (wrong types) or fall through to dynamic mapping, creating untested index mappings. -
No type conversion. The sub-pipelines exist specifically to do type conversions (
convertto long/boolean/double,dateparsing) and ECS mappings for type-specific fields. Without a sub-pipeline, those fields stay as raw strings from the JSON parse — alongfield getting a string value, adatefield staying as a human-readable date string. This creates mapping conflicts or silent data quality issues. -
No test coverage. You can't validate what you can't see. Shipping untested asset types means any change to the default pipeline could silently break ingestion for those types without anyone noticing. The pipeline test fixture only covers 7-8 asset types. If something breaks for
secretsorapplication_keys, there's no test to catch it. -
False sense of coverage. Users see 18 asset types listed and assume the integration handles them all correctly. In reality, only ~7 are properly mapped, typed, and tested. That's worse than listing 7 types honestly and adding the rest when they're ready.
There was a problem hiding this comment.
After checking with the client again on this, got few more logs so adding them in test files, now only one asset type needed to be removed: application_services.
There was a problem hiding this comment.
Thanks. Let me know when it's ready to be reviewed again.
There was a problem hiding this comment.
I have added the missing types, please review @kcreddy
🚀 Benchmarks reportTo see the full report comment with |
kcreddy
left a comment
There was a problem hiding this comment.
Pending on #16658 (comment)
| name: '{{ IngestPipeline "pipeline_application_settings" }}' | ||
| tag: pipeline_application_settings | ||
| if: >- | ||
| ctx.axonius?.application?.asset_type != null && ctx.axonius.application.asset_type == 'application_settings' |
There was a problem hiding this comment.
| ctx.axonius?.application?.asset_type != null && ctx.axonius.application.asset_type == 'application_settings' | |
| ctx.axonius?.application?.asset_type == 'application_settings' |
That should work
| "created": "2024-12-25T22:17:25.000Z", | ||
| "kind": "event", | ||
| "original": "{\"asset_type\":\"urls\",\"internal_axon_id\":\"9c65d891077147892d0e632eb0cf6ebf\",\"adapters\":[\"okta_adapter\"],\"adapter_list_length\":1,\"event\":{\"accurate_for_datetime\":\"Thu, 13 Nov 2025 00:02:55 GMT\",\"adapter_categories\":[\"IAM\",\"SaaS Management\"],\"client_used\":\"67fd09cd782eb39db73d1af1\",\"data\":{\"domain\":\"example-domain\",\"accurate_for_datetime\":\"Thu, 13 Nov 2025 00:02:55 GMT\",\"activity_status_active\":0,\"activity_status_active_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.activity_status\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Active\"}],\"activity_status_inactive\":288,\"activity_status_inactive_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.activity_status\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Inactive\"}],\"app_id\":\"990f042e1d803894A3ae\",\"application_and_account_name\":\"okta/okta-demo\",\"auth_type\":[],\"created\":\"Wed, 25 Dec 2024 22:17:25 GMT\",\"extension_type\":\"SSO\",\"fetch_time\":\"Thu, 13 Nov 2025 00:02:18 GMT\",\"first_fetch_time\":\"Mon, 14 Apr 2025 13:27:09 GMT\",\"first_seen\":\"Mon, 22 Apr 2024 21:22:25 GMT\",\"from_last_fetch\":true,\"grant_types\":[],\"id\":\"cd2f40fe8670900112ab\",\"id_raw\":\"okta_adapter_0!990f042e1d803894A3ae\",\"integration_type\":\"Admin Managed\",\"is_admin\":16,\"is_admin_hyperlink\":[{\"compOp\":\"true\",\"field\":\"specific_data.data.permissions.is_admin\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0}],\"is_fetched_from_adapter\":true,\"is_identity\":0,\"is_identity_hyperlink\":[{\"compOp\":\"true\",\"field\":\"specific_data.data.permissions.is_identity\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0}],\"is_operational\":true,\"last_fetch_connection_id\":\"67fd09cd782eb39db73d1af1\",\"last_fetch_connection_label\":\"okta-demo\",\"last_seen\":\"Mon, 07 Apr 2025 13:05:30 GMT\",\"last_used\":\"Mon, 07 Apr 2025 13:05:30 GMT\",\"name\":\"Cloudflare\",\"not_fetched_count\":0,\"permissions\":[{\"is_admin\":true,\"name\":\"allow_api_access_to_account\",\"users_amount\":9}],\"redirect_uris\":[],\"related_vendor_name\":\"Cloudflare\",\"scope_tag_calendar\":0,\"scope_tag_calendar_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.app_id\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"990f042e1d803894A3ae\"}],\"scope_tag_drive\":0,\"scope_tag_drive_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.permissions.scope_tag\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Drive\"}],\"scope_tag_mail\":0,\"scope_tag_mail_hyperlink\":[{\"compOp\":\"equals\",\"field\":\"specific_data.data.permissions.scope_tag\",\"leftBracket\":0,\"logicOp\":\"and\",\"not\":false,\"rightBracket\":0,\"value\":\"Mail\"}],\"source_application\":\"Okta\",\"type\":\"Extensions\",\"urls\":[],\"user_count\":288,\"user_count_link\":[{\"bracketWeight\":0,\"compOp\":\"IN\",\"field\":\"specific_data.data.mail\",\"leftBracket\":0,\"logicOp\":\"\",\"not\":false,\"rightBracket\":0,\"value\":\"ronald.mays@demo.local,justin.baugh@demo.local,brian.williamskaren.cox@demo.local,bill.mcnay@demo.local,billy.woodruff@demo.local,hazel.contreras@demo.local,eduardo.mandeville@demo.local,karen.neal@demo.local,troy.hooper@demo.local,andreas.rice@demo.local,janis.henry@demo.local,mary.tavares@demo.local,patrick.rowe@demo.local,kenneth.gardner@demo.local,paul.hendricks@demo.local,tanya.wolf@demo.local,kathleen.arroliga@demo.local,robert.blunt@demo.local,charles.duncan@demo.local,margarita.zapata@demo.local,charles.paredes@demo.local,floyd.conrad@demo.local,tammy.hawkins@demo.local,sam.chavez@demo.local,francis.rivera@demo.local,brandon.lilly@demo.local,matthew.wiley@demo.local,lacey.smith@demo.local,tracy.white@demo.local\"}],\"username_formats\":[],\"users_amount\":288},\"initial_plugin_unique_name\":\"okta_adapter_0\",\"plugin_name\":\"okta_adapter\",\"plugin_type\":\"Adapter\",\"plugin_unique_name\":\"okta_adapter_0\",\"quick_id\":\"okta_adapter_0!cd2f40fe8670900112ab\",\"type\":\"entitydata\"}}" | ||
| }, |
There was a problem hiding this comment.
Its still missing for some of the pipelines.
@kcreddy , I didn't understand what is missing? |
| - audit_activities (endpoint: `/api/v2/assets/audit_activities`) | ||
| - business_applications (endpoint: `/api/v2/assets/business_applications`) | ||
| - urls (endpoint: `/api/v2/assets/urls`) | ||
| - application_services (endpoint: `/api/v2/assets/application_services`) |
There was a problem hiding this comment.
Remove application_services?
| - name: preserve_duplicate_custom_fields | ||
| required: false | ||
| title: Preserve duplicate custom fields | ||
| description: Preserve axonius.application.* fields that were copied to Elastic Common Schema (ECS) fields. | ||
| type: bool | ||
| multi: false | ||
| show_user: false | ||
| default: false |
There was a problem hiding this comment.
Using preserve_duplicate_custom_fields is a legacy pattern now. We should plan to remove this before GA.
There was a problem hiding this comment.
this change needs to be done in all the rest datastreams of Axonius and this one is a last datatsream PR pending for merge, one this gets merged, will have to create another PR, so will do this change there for all.
There was a problem hiding this comment.
yes before GA is the intention.
|
/test |
💚 Build Succeeded
History
|
|
Package axonius - 0.1.0 containing this change is available at https://epr.elastic.co/package/axonius/0.1.0/ |
Proposed commit message
The release includes application data stream and associated dashboard.
Axonius fields are mapped to their corresponding ECS fields where possible.
Test samples were derived from live data samples, which were subsequently
sanitized.
Checklist
changelog.ymlfile.How to test this PR locally
To test the axonius package:
Related issues
Screenshots