Cleanup build-scan, remove publish scan to elastic server#4
Merged
Conversation
nknize
added
:sanitize
peternied
pushed a commit
that referenced
this pull request
Mar 13, 2021
setiah
added a commit
that referenced
this pull request
Mar 18, 2021
* [PURIFY] Remove x-pack directory Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Remove docs directory (#3) This commit removes the doc directory Signed-off-by: Peter Nied <petern@amazon.com> * Cleanup build-scan, remove publish scan to elastic server (#1) (#4) Co-authored-by: Huan Jiang <huanji@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack eql (#5) This commit removes all trace of EQL from the sanitized fork. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove issue, pr tempalte to avoid confusion, we could add olater (#10) (#6) Co-authored-by: Huan Jiang <huanji@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] update build.gradle files to ensure build completes; gradle check fails (#7) Signed-off-by: Peter Nied <petern@amazon.com> * Cleanup build script to exclude security-authorization-engine (#8) (#8) * Cleanup build-scan, remove publish scan to elastic server * Cleanup build script to exclude security-authorization-engine which test has dependency on xpack * Cleanup build script to exclude security-authorization-engine which test has dependency on xpack Co-authored-by: Huan Jiang <huanji@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack enrichment processor (#9) This commit removes all trace of the Elastic licensed enrichment processor. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack async-search (#10) This commit removes all trace of Elastic licensed asyc-search Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack ccr (#11) This committ removes all trace of Elastic licensed CCR. Signed-off-by: Peter Nied <petern@amazon.com> * Remove the Elastic license file, all checks for this license and the license REST APIs. (#12) Co-authored-by: Rabi Panda <rabipanda@icloud.com> Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack graph (#13) This commit removes all trace of Elastic licensed graph feature Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack ml (#14) This commit removes all trace of x-pack ml. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Add InferenceConfig to org.elasticsearch.client.analytics (#15) This commits adds InferenceConfig back to org.elasticsearch.client.analytics for use in InferencePipelineAggregationBuilder. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack security (#16) This commit removes all trace of the security high level rest client and other reference to x-pack security Co-authored-by: Rabi Panda <rabipanda@icloud.com> Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack rollups (#17) This commit removes all trace of Elastic licensed rollups Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack sql (#18) This commit removes all trace of Elastic licensed SQL Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack migration (#19) This commit removes all trace of Elastic licensed Migration Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack index lifecycle management (#20) This commit removes all trace of Elastic licensed ILM. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack watcher (#21) This commit removes all trace of Elastic licensed watcher Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack monitoring (#22) This commit removes all trace of Elastic licensed monitoring Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Remove remaining x-pack license. (#25) Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Revert "Move data stream transport and rest action to xpack (#59593)" (#28) This commit reverts commit 2a89e13. Relicensing data streams from OSS to XPack. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] remove all trace of x-pack transforms (#31) This commit removes all trace of Elastic licensed transforms. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] fix GetDataStreamsRequestTests build failure This fixes the constructor for IndexNameExpressionResolver to pass in Settings.EMPTY to a ThreadContext used by the resolver. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Remove the AuthorizationEnginePlugin from examples. (#26) Signed-off-by: Peter Nied <petern@amazon.com> * Fix compilation issues for tests. (#29) Some of the tests that use the x-pack code are failing compilation. This PR cleans up the references to fix the issue. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Cleanup build and unblock HLRC integration tests (#33) This commit cleans up the following: * Remove unused imports * Remove ILM settings in hlrc testCluster formation * Comment out security users settings in ElasticsearchNode creation for build-tools tests Signed-off-by: Peter Nied <petern@amazon.com> * Adding initial CI workflow for search (#35) Signed-off-by: Peter Nied <petern@amazon.com> * [TEST] Fix unit test failure in RestHighLevelClientTests (#36) * fix testProvidedNamedXContents by modifying the assertion Signed-off-by: Peter Nied <petern@amazon.com> * [TEST] fix DeleteDataStreamRequestTests failure (#37) This commit fixes DeleteDataStreamRequestTests.testDeleteSnapshottingDataStream unit test failure by passing SnapshotsInProgress.Entry.SUCCESS in the createEntrymethod. Signed-off-by: Peter Nied <petern@amazon.com> * Remove license option in gradlew command (#41) * Remove license option in gradlew command * Remove "This username and password is part of trial license. Let's remove this too" from TESTING.asciidoc Signed-off-by: Peter Nied <petern@amazon.com> * Remove x-pack from build, distribution and packaging. (#43) This PR removes references to x-pack from buildSrc, distribution and qa modules. Signed-off-by: Peter Nied <petern@amazon.com> * Removing _reload_search_analyzers related changes since the related x-pack support is removed (#48) Signed-off-by: Peter Nied <petern@amazon.com> * Fixing Rest Converters Tests after x-pack removal (#54) Signed-off-by: Peter Nied <petern@amazon.com> * Remove license statement from CONTRIBUTING.md (#58) Signed-off-by: Peter Nied <petern@amazon.com> * Revert back refresh policy in RequestConverters. (#55) This PR reverts back the deleted code (#16, #54) related to refresh policies. Signed-off-by: Peter Nied <petern@amazon.com> * Remove unused imports in RemoteClustersIT and InternalTestCluster This commit removes unused imports that are causing checkStyle failures. Signed-off-by: Peter Nied <petern@amazon.com> * [DOCS] temporarily comment verifyDocsLuceneVersion in qa:verify-version-constants Docs have temporarily been removed. This commit can be reverted if the OSS docs are restored. Signed-off-by: Peter Nied <petern@amazon.com> * Mute AnalyticsAggsIT test failure AnalyticsAggsIT needs to be removed. This mutes the test until removal is complete. Signed-off-by: Peter Nied <petern@amazon.com> * [MUTE] AwaitsFix failing tests This commit mutes failing tests in: * IndicesClientIT * SearchIT (Freeze/UnFreeze) * IndicesClientDocumentationIT Fixes are identified and will be merged in a followup PR. Signed-off-by: Peter Nied <petern@amazon.com> * Remove packaging tests for the x-pack command line tools. (#56) Signed-off-by: Peter Nied <petern@amazon.com> * Remove x-pack aggregations. (#59) This PR removes the x-pack aggregations: string_stats, top_metrics and inference. Resolves #51 Relates #2 Signed-off-by: Peter Nied <petern@amazon.com> * Remove x-pack data-frame analytics hlrc. (#62) This PR removes the hlrc for x-pack data-frame analytics. Relates #2 Signed-off-by: Peter Nied <petern@amazon.com> * Remove ILM policy from GetDataStreamAction Response. (#63) Signed-off-by: Peter Nied <petern@amazon.com> * Ensure ReplicationOperation notify listener once (#68256) ReplicationOperation can notify the listener twice if the primary shard is demoted after it has completed the primary operation. Closes #68049 Signed-off-by: Peter Nied <petern@amazon.com> * Fix search template request (#43509) A seed was hit in (#43157) that caused mutateInstance to generate an identical instance. This change prevents that. Signed-off-by: Peter Nied <petern@amazon.com> * Lower skip version for token_cound yaml test (#68583) Signed-off-by: Peter Nied <petern@amazon.com> * Revert previous change to fix import issue. Signed-off-by: Peter Nied <petern@amazon.com> * Remove unused imports in ArchiveTests Signed-off-by: Peter Nied <petern@amazon.com> * Fix unit test for removal of x-pack aggregations. (#65) This PR fixes the unit test which failed after removal of the x-pack aggregations #59. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Remove ProtocolUtils, TimeUtils, and XContentSource from HLRC (#64) This commit removes the ProtocolUtils, TimeUtils, and XContentSource utility classes which is only used for xpack HLRC. Signed-off-by: Peter Nied <petern@amazon.com> * [PURIFY] Remove x-pack feature flag from yaml test (#68) This commit removes the xpack no_xpack feature flag from the yaml test suite. Signed-off-by: Peter Nied <petern@amazon.com> * Remove testcase testSearchWithBasicLicensedQuery since basic license is no longer applicable (#74) Signed-off-by: Peter Nied <petern@amazon.com> * Remove UnusedImports (#76) Signed-off-by: Peter Nied <petern@amazon.com> * Bring back the REST specs for data streams. (#78) Add back the REST specs for data streams which were moved to x-pack as part of the commit fe12217 Signed-off-by: Peter Nied <petern@amazon.com> * Remove unused imports after x-pack feature flag removed from yaml test (#81) Signed-off-by: Peter Nied <petern@amazon.com> * [TEST] Fix Feature Flags in Test Framework and SortTemplates yaml failure (#82) This commit adds parse logic to correctly parse feature flags in the test framework. It also fixes a test failure in cat.templates/Sort Templates yaml test. Signed-off-by: Peter Nied <petern@amazon.com> * Run precommit and unit tests as part of github actions. (#84) Signed-off-by: Peter Nied <petern@amazon.com> * Removing FreezeIndex related code since its x-pack counterpart is removed (#85) Signed-off-by: Peter Nied <petern@amazon.com> * Only run pre-commit checks in GitHub actions. (#94) Signed-off-by: Peter Nied <petern@amazon.com> * Fixing Bwc checks for 7.10.3 (#93) Signed-off-by: Peter Nied <petern@amazon.com> * Temporary fix for license check path for debian packaging. (#97) This currently unblocks the gradle check and subsequently will be removed by the meta issue #50 Signed-off-by: Peter Nied <petern@amazon.com> * Disable plugincli feature (#101) Plugins CLI - disable installing official plugins by name. Currently the plugin cli allows installation of a plugin by name in which case it downloads the plugin artifacts from the official elastic artifacts repository. We will enable it once we have the new official artifacts download URL (Tracking Issue: #100) Signed-off-by: Peter Nied <petern@amazon.com> * Support for continious integration with Jenkins (#96) Adding jenkinsfile to describe the build / test / deployment process for this repository Signed-off-by: Peter Nied <petern@amazon.com> * Remove any non oss from build, package, and distribution (#102) This commit changes the building, packaging, and testing framework to only support OSS on different distributions. Next steps: completely remove -oss flag dependencies in package and build tests move 6.x bwc testing to be an explicit option remove any references to elastic.co download site (or replace with downloads from the OSS website) Co-authored-by: Himanshu Setia <setiah@amazon.com> Co-authored-by: Rabi Panda <pandarab@amazon.com> Co-authored-by: Himanshu Setia <58999915+setiah@users.noreply.github.com> Co-authored-by: Sarat Vemulapalli <vemsarat@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * Remove x-pact from RESI API username and password (#117) Signed-off-by: Peter Nied <petern@amazon.com> * Update signoff message (#121) Signed-off-by: Harold Wang <harowang@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * Update CI workflow to work on new infra (#123) * Update CI workflow to work on new infra - Backward compatability tests are disabled during CI by default #113 - Added property to allow for disabling bwc tests - Added agent label to use specific hardware https://www.jenkins.io/doc/book/pipeline/syntax/#agent Signed-off-by: Peter Nied <petern@amazon.com> * Disable BWC checks. (#130) As part of this PR, we are disabling the BWC checks. Once we have finalized the versions for the fork, we can re-enable it with right configurations. Relates #105 Signed-off-by: Rabi Panda <pandarab@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * Create CODE_OF_CONDUCT.md (#124) Explicitly adding code of conduct from https://opendistro.github.io/for-elasticsearch/codeofconduct.html Signed-off-by: Peter Nied <petern@amazon.com> * Add script to perform signoff check between commits (#152) * Add script to perform signoff check between commits Signed-off-by: Peter Nied <petern@amazon.com> * [Rename] server/src/main/java/org/apache (#162) This commit refactors all instances of elasticsearch in server/src/main/java/org/apache to opensearch. Signed-off-by: Nicholas Knize <nknize@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * Revert "[Rename] server/src/main/java/org/apache (#162)" This reverts commit c50e8c8 which went should have merged to the rename branch instead of the main branch. Signed-off-by: Peter Nied <petern@amazon.com> * Update CODE_OF_CONDUCT.md replaced renameme Signed-off-by: Peter Nied <petern@amazon.com> * Dummy commit to test the CI/CD workflow Signed-off-by: Peter Nied <petern@amazon.com> * Update .gitignore Signed-off-by: Peter Nied <petern@amazon.com> * Fixing transport deserialization with oss distribution (#218) Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * Update LICENSE.txt (#227) updating to Apache 2.0 License from https://www.apache.org/licenses/LICENSE-2.0.txt Signed-off-by: Peter Nied <petern@amazon.com> * Update NOTICE.TXT with OpenSearch copyright (#232) This commit updates the NOTICE.txt file to include the OpenSearch copyright notice. Signed-off-by: Nicholas Knize <nknize@amazon.com> Signed-off-by: Peter Nied <petern@amazon.com> * fixed reference to old repo (#333) This commit fixes the url reference to bwc disabled message. Signed-off-by: Kyle Davis <kyledvs@amazon.com> Co-authored-by: nknize <nknize@gmail.com> Co-authored-by: Huan Jiang <huanji@amazon.com> Co-authored-by: Rabi Panda <rabipanda@icloud.com> Co-authored-by: Rabi Panda <adnapibar@gmail.com> Co-authored-by: nknize <nknize@amazon.com> Co-authored-by: Sarat Vemulapalli <vemsarat@amazon.com> Co-authored-by: Harold Wang <74381974+harold-wang@users.noreply.github.com> Co-authored-by: Himanshu Setia <58999915+setiah@users.noreply.github.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Jack Conradson <osjdconrad@gmail.com> Co-authored-by: Christoph Büscher <cbuescher@posteo.de> Co-authored-by: Abbas Hussain <abbas_10690@yahoo.com> Co-authored-by: Peter Nied <pnied@microsoft.com> Co-authored-by: Himanshu Setia <setiah@amazon.com> Co-authored-by: Rabi Panda <pandarab@amazon.com> Co-authored-by: Peter Nied <petern@amazon.com> Co-authored-by: CEHENKLE <henkle@amazon.com> Co-authored-by: Barani <bbarani@amazon.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Kyle J. Davis <halldirector@gmail.com>
Poojita-Raj
referenced
this pull request
in Poojita-Raj/OpenSearch
Nov 3, 2021
# This is the 1st commit message: changes to shardsearchreq tests Signed-off-by: Poojita Raj <poojiraj@amazon.com> # The commit message #2 will be skipped: # try to print # # Signed-off-by: Poojita Raj <poojiraj@amazon.com> # The commit message #3 will be skipped: # try # # Signed-off-by: Poojita Raj <poojiraj@amazon.com> # The commit message #4 will be skipped: # moved the class # # Signed-off-by: Poojita Raj <poojiraj@amazon.com> # The commit message #5 will be skipped: # try2 # # Signed-off-by: Poojita Raj <poojiraj@amazon.com> # The commit message #6 will be skipped: # only update # # Signed-off-by: Poojita Raj <poojiraj@amazon.com> # The commit message #7 will be skipped: # add fix # # Signed-off-by: Poojita Raj <poojiraj@amazon.com>
8 tasks
ritty27
pushed a commit
to ritty27/OpenSearch
that referenced
this pull request
May 12, 2024
Fixes the logic used to implement hasNext() that is needed because Jackson's parser doesn't allow testing if there's a next token without moving to that next token. Fixes opensearch-project#4
ritty27
pushed a commit
to ritty27/OpenSearch
that referenced
this pull request
May 12, 2024
Signed-off-by: Mital Awachat <mitalawachat@users.noreply.github.com>
neuenfeldttj
pushed a commit
to neuenfeldttj/OpenSearch
that referenced
this pull request
Jun 26, 2025
# This is the 1st commit message: Unregister rest and transport action handlers for HotToWarmTiering(opensearch-project#18247) Signed-off-by: Sandeep Kumawat <skumwt@amazon.com> # This is the commit message opensearch-project#2: got query and agg working with new abstraction poc Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> # This is the commit message opensearch-project#3: trying to refactor with a composite pb not done yet Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> # This is the commit message opensearch-project#4: need to fix SearchProfileShardResult Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> SearchProfileShardResult still needs to be fixed but tests pass Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> SearchProfileShardResult still needs to be fixed but tests pass Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com>
mengweieric
added a commit
to mengweieric/OpenSearch
that referenced
this pull request
May 7, 2026
Mutation UDF opensearch-project#4 — last walker reuse. Same push shape as json_append, but each paired value is first tried as a JSON-array parse: success → spread the elements; failure → push the whole string as one element (parity with legacy `JsonExtendFunctionImpl`'s `gson.fromJson(v, List.class)` try/fall-back). Rust side (rust/src/udf/json_extend.rs): * Helper `spread(raw) -> Vec<Value>`: returns the parsed items when `raw` is a JSON array, else `[Value::String(raw)]`. Scalars, objects, and malformed JSON all go through the single-push branch. * Terminal closure reuses json_append's array-target guards (Object field → Array, Array+Index → inner Array, Array+Wildcard → every array child). `Vec::extend(items.iter().cloned())` handles the spread and the single-push case uniformly. * Variadic arity matches every other mutation UDF. Invalid arity / any-NULL / malformed-doc / malformed-path → NULL. Deliberate divergence from legacy: integer-typed spread elements stay integers (serde_json preserves source type) rather than being widened to Double as Gson does. Documented in `json.md:555` but not covered by any legacy IT; we preserve the more useful default and will file a tracking issue for the wider Gson-compat decision. Java side: * JSON_EXTEND added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and `scalarFunctionAdapters`. * `JsonExtendAdapter` is a plain `AbstractNameMappingAdapter` rename. * Substrait YAML signature uses `variadic: {min: 1}` — same shape as the other variadic json_* UDFs. Tests: * 13 Rust unit tests for json_extend (3 legacy IT fixtures replayed: single-push on non-array value, plain-string push, JSON-array spread; plus empty-array-value / mixed-type-spread / wildcard-fan / non-array-noop / missing-path-noop / any-NULL / malformed-doc / malformed-path / coerce_types / return_type). * ScalarJsonFunctionIT gains `testJsonExtendParityWithLegacy` replaying all 3 legacy assertions with literal stringified JSON standing in for the nested constructor calls the legacy test uses. Parity-checked against legacy SQL plugin `CalcitePPLJsonBuiltinFunctionIT.testJsonExtend`. Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
mengweieric
added a commit
to mengweieric/OpenSearch
that referenced
this pull request
May 7, 2026
Mutation UDF opensearch-project#4 — last walker reuse. Same push shape as json_append, but each paired value is first tried as a JSON-array parse: success → spread the elements; failure → push the whole string as one element (parity with legacy `JsonExtendFunctionImpl`'s `gson.fromJson(v, List.class)` try/fall-back). Rust side (rust/src/udf/json_extend.rs): * Helper `spread(raw) -> Vec<Value>`: returns the parsed items when `raw` is a JSON array, else `[Value::String(raw)]`. Scalars, objects, and malformed JSON all go through the single-push branch. * Terminal closure reuses json_append's array-target guards (Object field → Array, Array+Index → inner Array, Array+Wildcard → every array child). `Vec::extend(items.iter().cloned())` handles the spread and the single-push case uniformly. * Variadic arity matches every other mutation UDF. Invalid arity / any-NULL / malformed-doc / malformed-path → NULL. Deliberate divergence from legacy: integer-typed spread elements stay integers (serde_json preserves source type) rather than being widened to Double as Gson does. Documented in `json.md:555` but not covered by any legacy IT; we preserve the more useful default and will file a tracking issue for the wider Gson-compat decision. Java side: * JSON_EXTEND added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and `scalarFunctionAdapters`. * `JsonExtendAdapter` is a plain `AbstractNameMappingAdapter` rename. * Substrait YAML signature uses `variadic: {min: 1}` — same shape as the other variadic json_* UDFs. Tests: * 13 Rust unit tests for json_extend (3 legacy IT fixtures replayed: single-push on non-array value, plain-string push, JSON-array spread; plus empty-array-value / mixed-type-spread / wildcard-fan / non-array-noop / missing-path-noop / any-NULL / malformed-doc / malformed-path / coerce_types / return_type). * ScalarJsonFunctionIT gains `testJsonExtendParityWithLegacy` replaying all 3 legacy assertions with literal stringified JSON standing in for the nested constructor calls the legacy test uses. Parity-checked against legacy SQL plugin `CalcitePPLJsonBuiltinFunctionIT.testJsonExtend`. Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
mengweieric
added a commit
to mengweieric/OpenSearch
that referenced
this pull request
May 9, 2026
Mutation UDF opensearch-project#4 — last walker reuse. Same push shape as json_append, but each paired value is first tried as a JSON-array parse: success → spread the elements; failure → push the whole string as one element (parity with legacy `JsonExtendFunctionImpl`'s `gson.fromJson(v, List.class)` try/fall-back). Rust side (rust/src/udf/json_extend.rs): * Helper `spread(raw) -> Vec<Value>`: returns the parsed items when `raw` is a JSON array, else `[Value::String(raw)]`. Scalars, objects, and malformed JSON all go through the single-push branch. * Terminal closure reuses json_append's array-target guards (Object field → Array, Array+Index → inner Array, Array+Wildcard → every array child). `Vec::extend(items.iter().cloned())` handles the spread and the single-push case uniformly. * Variadic arity matches every other mutation UDF. Invalid arity / any-NULL / malformed-doc / malformed-path → NULL. Deliberate divergence from legacy: integer-typed spread elements stay integers (serde_json preserves source type) rather than being widened to Double as Gson does. Documented in `json.md:555` but not covered by any legacy IT; we preserve the more useful default and will file a tracking issue for the wider Gson-compat decision. Java side: * JSON_EXTEND added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and `scalarFunctionAdapters`. * `JsonExtendAdapter` is a plain `AbstractNameMappingAdapter` rename. * Substrait YAML signature uses `variadic: {min: 1}` — same shape as the other variadic json_* UDFs. Tests: * 13 Rust unit tests for json_extend (3 legacy IT fixtures replayed: single-push on non-array value, plain-string push, JSON-array spread; plus empty-array-value / mixed-type-spread / wildcard-fan / non-array-noop / missing-path-noop / any-NULL / malformed-doc / malformed-path / coerce_types / return_type). * ScalarJsonFunctionIT gains `testJsonExtendParityWithLegacy` replaying all 3 legacy assertions with literal stringified JSON standing in for the nested constructor calls the legacy test uses. Parity-checked against legacy SQL plugin `CalcitePPLJsonBuiltinFunctionIT.testJsonExtend`. Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
mch2
pushed a commit
that referenced
this pull request
May 9, 2026
* [Analytics Engine] Port json_array_length to DataFusion backend
First PPL json_* function wired through PPL → Calcite → Substrait →
DataFusion. Scaffolds the pattern every follow-up UDF reuses: Rust kernel
+ YAML signature + ScalarFunction enum entry + JsonFunctionAdapters
rename + FunctionMappings.s(...) binding + STANDARD_PROJECT_OPS entry.
Rust UDF (rust/src/udf/json_array_length.rs) coerces the input to Utf8,
parses with serde_json, and returns Int32 to match PPL's
INTEGER_FORCE_NULLABLE declaration — returning Int64 would leak through
column-valued calls even though literal args const-fold via a narrowing
CAST. Malformed / non-array / NULL input → NULL, matching legacy
JsonArrayLengthFunctionImpl's NullPolicy.ANY + Gson parity.
ScalarFunction.CAST added to STANDARD_PROJECT_OPS so PPL's implicit CAST
around a UDF call (inserted when the UDF's declared return type differs
from the eval column's inferred type) doesn't fail OpenSearchProjectRule
with "No backend supports scalar function [CAST]". DataFusion handles
CAST natively — no UDF needed.
STANDARD_PROJECT_OPS and scalarFunctionAdapters reshaped to one-entry-
per-line (Map.ofEntries / Set.of) so parallel json_* PRs append without
touching neighbour lines.
Tests:
* 10 Rust unit tests (flat/nested arrays, non-array, malformed, NULL,
coerce_types accept/reject, arity guard, scalar-input fast path).
* JsonFunctionAdaptersTests guards adapter shape + return-type
preservation (BIGINT vs LOCAL_OP's INTEGER_NULLABLE).
* ScalarJsonFunctionIT covers happy path, empty array, non-array
object → NULL, malformed → NULL via /_analytics/ppl.
Parity-checked against legacy SQL plugin
CalcitePPLJsonBuiltinFunctionIT.testJsonArrayLength.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] JSON: introduce jsonpath-rust parser + shared helpers
Lands the parser crate + a small shared helpers module ahead of the per-
function json_* UDFs. Keeping this on its own commit lets reviewers sign
off on the crate choice (jsonpath-rust 0.7) and path-conversion behaviour
before 8 UDF bodies land on top.
* rust/Cargo.toml: add jsonpath-rust = "0.7".
* rust/src/udf/json_common.rs:
- convert_ppl_path: PPL path syntax (`a{i}.b{}`) -> JSONPath (`$.a[i].b[*]`).
Mirrors JsonUtils.convertToJsonPath in sql/core. Empty string maps
to "$" to match legacy root semantics.
- parse: serde_json wrapper returning None on malformed input, the
contract every json_* UDF will share.
- check_arity / check_arity_range: plan_err! wrappers for the
top-of-invoke guards.
* rust/src/udf/mod.rs: register the module (helpers are crate-private).
Consumers land in follow-up commits on the same PR (#21513); a module-
level #![allow(dead_code)] keeps this commit's cargo check clean.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] Port json_keys to DataFusion backend
Adds the second PPL json_* UDF on top of #21476 (json_array_length).
Matches the legacy SQL-plugin contract: object → JSON-array-encoded keys
in insertion order; non-object / malformed / scalar → SQL NULL.
- Rust UDF at rust/src/udf/json_keys.rs with scalar + columnar paths
- Shared rust/src/udf/json_common.rs helpers (parse, arity, Utf8 downcast,
PPL-path → JSONPath) seeded for later json_* UDFs
- serde_json preserve_order feature to preserve legacy LinkedHashMap ordering
- Java wiring: ScalarFunction.JSON_KEYS, JsonKeysAdapter, Substrait sig,
YAML signature, plugin project-op + adapter registration
- ScalarJsonFunctionIT parity test for the four legacy fixtures
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] Port json_extract to DataFusion backend
Rust UDF at rust/src/udf/json_extract.rs wraps jsonpath-rust: single path →
unquoted scalar or JSON-serialized container; multi-path → JSON array with
literal null slots for misses. < 2 args, malformed doc, malformed path, and
explicit-null matches all collapse to SQL NULL, matching legacy
JsonExtractFunctionImpl's calcite jsonQuery/jsonValue pair.
JsonExtractAdapter renames the PPL call to the Rust UDF name via the variadic
path; routing lives in FunctionMappings.s(...) in DataFusionFragmentConvertor
and the STANDARD_PROJECT_OPS allow-list.
Also fixes a pre-existing transport bug in DatafusionResultStream.getFieldValue:
VarCharVector.getObject returns Arrow Text, which StreamOutput.writeGenericValue
cannot serialize, so string-valued UDF results (json_keys, json_extract) were
dropped when shard results traveled back to the coordinator. Converting
VarCharVector cells to String at the source mirrors ArrowValues.toJavaValue
and unblocks every string-returning UDF.
Parity IT (ScalarJsonFunctionIT) replays four verbatim legacy cases covering
single-path scalar/container match, wildcard multi-match, multi-path with
missing path, and explicit-null resolution.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] Port json_delete to DataFusion backend
Mutation UDF #1. Introduces the shared mutation walker that json_set,
json_append, and json_extend will reuse on the same PR.
Rust side (rust/src/udf/json_delete.rs + json_common.rs):
* `parse_ppl_segments` tokenises PPL paths (a.b{0}.c{}) into Field /
Index / Wildcard segments without allocating field names.
* `walk_mut` drives a mutation closure against every terminal match in
a serde_json::Value; missing intermediate keys and out-of-range
indices are silent no-ops, matching Jayway's SUPPRESS_EXCEPTIONS
behaviour that legacy `JsonDeleteFunctionImpl` (→ Calcite
`JsonFunctions.jsonRemove`) relies on.
* `json_delete` terminal closure: `shift_remove` on Object (preserves
insertion order via serde_json's `preserve_order` feature),
`Vec::remove` on Array-with-Index, `Vec::clear` on Array-with-Wildcard.
Any-NULL-arg / malformed doc / malformed path → NULL.
The walker is generic enough that json_set / json_append / json_extend
are now pure terminal-closure swaps (set value, push value, extend
array) — no further traversal plumbing needed.
Java side:
* JSON_DELETE added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and
`scalarFunctionAdapters`.
* `JsonDeleteAdapter` is a plain `AbstractNameMappingAdapter` rename
(matches the other json_* adapters).
* Substrait YAML signature uses `variadic: {min: 1}` — same shape as
json_extract.
Tests:
* 10 Rust unit tests for json_delete (4 legacy IT fixtures replayed:
flat-key, nested, missing-path-unchanged, wildcard-array; plus
any-NULL / malformed / coerce_types / return_type).
* 4 new walker tests in json_common (tokeniser, flat-delete,
missing-noop, wildcard-fan-out, index-out-of-range-noop).
* ScalarJsonFunctionIT gains `testJsonDeleteParityWithLegacy`
replaying all 4 legacy assertions.
Parity-checked against legacy SQL plugin
`CalcitePPLJsonBuiltinFunctionIT.testJsonDelete*`.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] Port json_set to DataFusion backend
Mutation UDF #2. Reuses the walker introduced by #json_delete; this
commit is a pure terminal-closure swap on the Rust side (replace, not
remove) plus the usual 7-file Java/YAML wiring.
Rust side (rust/src/udf/json_set.rs):
* Terminal closure overwrites only existing keys on Object
(`map.contains_key` guard), in-range slots on Array-with-Index, and
every element on Array-with-Wildcard. This is the replace-only
semantics from legacy `JsonSetFunctionImpl` (→ Calcite
`JsonFunctions.jsonSet`, which guards `ctx.set` with
`ctx.read(k) != null`).
* Variadic arity: (doc, path1, val1, [path2, val2, ...]). Fewer than
3 args or an odd total (unpaired trailing path) short-circuits to
NULL, mirroring the "malformed input → NULL" convention the other
json_* UDFs follow.
* Values are always stored as `Value::String` because every arg is
coerced to Utf8 by `coerce_types` — matches the legacy fixture's
`"b":"3"` (stringified, not numeric).
* Root-path (`parse_ppl_segments` returns empty) is a no-op to match
Jayway's behaviour: `ctx.set("$", v)` silently fails because the
root is indelible and unreplaceable.
Java side:
* JSON_SET added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and
`scalarFunctionAdapters`.
* `JsonSetAdapter` is a plain `AbstractNameMappingAdapter` rename.
* Substrait YAML signature uses `variadic: {min: 1}` — same shape as
json_extract / json_delete.
Tests:
* 9 Rust unit tests for json_set (3 legacy IT fixtures replayed:
wildcard-replace, wrong-path-unchanged, partial-wildcard-set; plus
multi-pair / any-NULL / malformed-doc / malformed-path /
coerce_types / return_type).
* ScalarJsonFunctionIT gains `testJsonSetParityWithLegacy` replaying
all 3 legacy assertions.
Parity-checked against legacy SQL plugin
`CalcitePPLJsonBuiltinFunctionIT.testJsonSet*`.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] Port json_append to DataFusion backend
Mutation UDF #3. Another walker reuse: terminal closure pushes the
paired value onto array-valued targets (non-array / missing targets
are silent no-ops).
Rust side (rust/src/udf/json_append.rs):
* Terminal closure branches: Object+Field → look up field, if it's an
Array push the stringified value; Array+Index → if the indexed slot
is an Array, push; Array+Wildcard → push onto every array-valued
child. Non-array matches are skipped, matching legacy
`JsonFunctions.jsonInsert` via Jayway's Collection-parent branch
(`Collection.add`) which is how `JsonAppendFunctionImpl`'s
`.meaningless_key` suffix trick ultimately expands.
* Variadic arity (doc, path1, val1, [path2, val2, ...]). Fewer than 3
args or an odd total (unpaired trailing path) → NULL — the
malformed-input-to-NULL convention all other json_* UDFs share.
Matches legacy's `RuntimeException("needs corresponding path and
values")` observably-as-error via NULL surface.
* Pre-stringified values: all args are Utf8-coerced at `coerce_types`
entry, so nested `json_object(...)` / `json_array(...)` arrive here
already stringified. They are pushed as `Value::String`, which
reproduces the legacy IT's quoted-JSON-as-element rows without the
new engine having to implement `json_object`/`json_array` yet
(they ship in a follow-up PR).
Java side:
* JSON_APPEND added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and
`scalarFunctionAdapters`.
* `JsonAppendAdapter` is a plain `AbstractNameMappingAdapter` rename.
* Substrait YAML signature uses `variadic: {min: 1}` — same shape as
json_extract / json_delete / json_set.
Tests:
* 12 Rust unit tests for json_append (3 legacy IT fixtures replayed
with pre-stringified nested JSON: named-array push, nested-path
push, stringified-object push; plus multi-pair / wildcard-fan-out /
non-array-noop / missing-path-noop / any-NULL / malformed-doc /
malformed-path / coerce_types / return_type).
* ScalarJsonFunctionIT gains `testJsonAppendParityWithLegacy`
replaying all 3 legacy assertions with literal stringified JSON in
place of the nested constructor calls the legacy test uses.
Parity-checked against legacy SQL plugin
`CalcitePPLJsonBuiltinFunctionIT.testJsonAppend`.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
* [Analytics Engine] Port json_extend to DataFusion backend
Mutation UDF #4 — last walker reuse. Same push shape as json_append,
but each paired value is first tried as a JSON-array parse: success →
spread the elements; failure → push the whole string as one element
(parity with legacy `JsonExtendFunctionImpl`'s `gson.fromJson(v,
List.class)` try/fall-back).
Rust side (rust/src/udf/json_extend.rs):
* Helper `spread(raw) -> Vec<Value>`: returns the parsed items when
`raw` is a JSON array, else `[Value::String(raw)]`. Scalars,
objects, and malformed JSON all go through the single-push branch.
* Terminal closure reuses json_append's array-target guards (Object
field → Array, Array+Index → inner Array, Array+Wildcard → every
array child). `Vec::extend(items.iter().cloned())` handles the
spread and the single-push case uniformly.
* Variadic arity matches every other mutation UDF. Invalid arity /
any-NULL / malformed-doc / malformed-path → NULL.
Deliberate divergence from legacy: integer-typed spread elements stay
integers (serde_json preserves source type) rather than being widened
to Double as Gson does. Documented in `json.md:555` but not covered by
any legacy IT; we preserve the more useful default and will file a
tracking issue for the wider Gson-compat decision.
Java side:
* JSON_EXTEND added to `ScalarFunction`, `STANDARD_PROJECT_OPS`, and
`scalarFunctionAdapters`.
* `JsonExtendAdapter` is a plain `AbstractNameMappingAdapter` rename.
* Substrait YAML signature uses `variadic: {min: 1}` — same shape as
the other variadic json_* UDFs.
Tests:
* 13 Rust unit tests for json_extend (3 legacy IT fixtures replayed:
single-push on non-array value, plain-string push, JSON-array
spread; plus empty-array-value / mixed-type-spread / wildcard-fan
/ non-array-noop / missing-path-noop / any-NULL / malformed-doc /
malformed-path / coerce_types / return_type).
* ScalarJsonFunctionIT gains `testJsonExtendParityWithLegacy`
replaying all 3 legacy assertions with literal stringified JSON
standing in for the nested constructor calls the legacy test uses.
Parity-checked against legacy SQL plugin
`CalcitePPLJsonBuiltinFunctionIT.testJsonExtend`.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
---------
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
3 tasks
3 tasks
huyz0
added a commit
to huyz0/OpenSearch
that referenced
this pull request
Jul 8, 2026
…or WalChunkService One node-level WAL chunk mixing many shards' records meant one noisy shard could bloat the shared buffer and delay every other shard's flush/ack indefinitely, since flush() only fires on the caller's own timer/size trigger, not per shard. WalChunkService now takes an optional perShardBudgetBytes. A shard whose own cumulative buffered payload bytes since its last flush crosses that budget is immediately siphoned out of the shared buffer into its own dedicated chunk, written right away independent of the caller's flush trigger, leaving every other shard's buffered records and byte counters untouched. Disabled by default (<= 0), matching every other optional-feature-off default in this plugin, so every existing caller is unaffected. WalMirroringTranslog's one current caller already flushes after every single append (per-operation durability), so it never actually accumulates a multi-shard buffer today -- this is real infrastructure for the genuinely-batched, multi-shard use WalChunkService's own class javadoc already describes as its target, not yet exercised by an actual batching caller. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
huyz0
added a commit
to huyz0/OpenSearch
that referenced
this pull request
Jul 9, 2026
WalChunkService's perShardBudgetBytes (§18 risk opensearch-project#4's "one noisy shard can delay acks for others" mitigation) was built and unit-tested but ServerlessStoragePlugin always constructed the service via the 2-arg, budget-disabled overload -- there was no way for an operator to actually turn the mitigation on. Add serverless_storage.wal_per_shard_budget (a ByteSizeValue, zero/disabled by default) and thread it into the shared WalChunkService construction. Verified: the default still constructs a disabled budget, and an explicit value is threaded through unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cleanup build-scan, and remove publish scan to elastic server