Add a generic, extensible rest-endpoint provider SPI - #5656
Conversation
Add a leading `rest <endpoint>` command that exposes a curated, read-only, fixed-schema set of in-cluster management endpoints as a PPL table, modeled as a system row source bridged through visitRelation (the same seam as describe and the system-index family), so it runs on the Calcite engine without the unsupported table-function path. - Grammar/AST: REST/TIMEOUT tokens, restCommand rule, RestRelation, AstBuilder visit, query anonymizer. - Execution: RestSourceTable -> CalciteLogicalRestScan / CalciteEnumerableRestScan; RestEndpointRegistry (read-only allow-list + fixed schema + accepted args); RestEnumerator/RestRequest dispatch via OpenSearchClient (NodeClient in-cluster, RestClient standalone). - 9 endpoints: cluster health/state/settings, cat indices/nodes/cluster_manager/ plugins/shards, resolve/index. - Output shaping: numeric type normalization, id-to-name resolution, role-name expansion, structural flattening, graceful null degrade. - Args: count caps emitted rows; timeout reserved but rejected with 400; get-args applied server-side with per-arg value validation (local on health, health on cat/indices, expand_wildcards on resolve/index). Undeclared arg or out-of-domain value is rejected with a 400. level and include_defaults are deferred to a later release; flat_settings is dropped as redundant. - Error handling: blank endpoint, negative count, disallowed arg, and uncoercible value all surface clean 400s rather than 500s. Tests: CalcitePPLRestIT 25/25, RestEndpointRegistryTest 16, RestSourceTableTest 10. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
…xes; comment cleanup - /_cluster/settings: run the persistent and transient tiers through the node SettingsFilter (published via RestSettingsFilterHolder from SQLPlugin#getRestHandlers) so Property.Filtered and plugin-registered pattern settings are redacted exactly as the native GET /_cluster/settings endpoint. Remove the dead secretFields column-filter, which was the wrong shape for the (setting, value, tier) rows. - Parser: add TIMEOUT to searchableKeyWord so a bare 'timeout' term still matches searchLiteral. - coerce(): narrow the catch to IllegalArgumentException | ClassCastException; add empty-string guards in toNumber/toBoolean. - spotlessApply formatting; drop outdated and redundant comments. Tests: RestEndpointRegistryTest, RestSourceTableTest, OpenSearchNodeClientClusterSettingsFilterTest green. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
- clusterSettings: fail closed (throw IllegalStateException) when the node SettingsFilter is unavailable, instead of returning unredacted settings - collectSettings: handle list-type settings via getAsList fallback - decodeRestSpec: reject a blank/missing endpoint with a clear error - docs: correct rest.md allow-list table (9 endpoints + accepted args), quote endpoint literals, fix timeout/get-arg descriptions, add security note - register docs/user/ppl/cmd/rest.md in docs/category.json (deterministic single-node examples: number_of_nodes=1, cluster_manager count=1) Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
decodeRestSpec is only ever called behind an isRestSource gate today, but as a public decoder it must not assume its precondition. Without the guard a malformed token would surface an opaque StringIndexOutOfBoundsException from substring; now it throws a clear IllegalArgumentException instead. Addresses the PR opensearch-project#5599 Code Suggestions finding (importance 8). Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
…fetch - CalciteExplainIT.explainRestCommand: single-quote the endpoint literal; the explain harness inlines the query into a JSON body without escaping, so a double-quoted literal produced an invalid payload and a 400 (integration CI failure). - OpenSearchNodeClient.clusterSettings: resolve the SettingsFilter and fail closed BEFORE fetching cluster state, so settings are never read into memory when the redaction filter is unavailable. - OpenSearchRestClient.clusterState: narrow filter_path to nodes.*.name so node IPs/attributes are not over-fetched; manager-name resolution is preserved. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
… decode - AnalyticsEngineCompatIT: assert | rest '/_cluster/health' behaves identically with the analytics engine enabled (rest is never routed to DataFusion). - SystemIndexUtils.fromHex: reject an odd-length hex body so a crafted source name that passes the isRestSource suffix check fails clearly rather than silently dropping the trailing half-byte. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
On a cluster started with cluster.pluggable.dataformat=composite, RestUnifiedQueryAction.isAnalyticsIndex() routed every non-system-catalog PPL query to the analytics engine. The rest command's reserved in-cluster source (REST...__REST_SOURCE) has no backing index and only resolves on the Calcite path, so it was routed to DataFusion and failed with "Table 'REST...__REST_SOURCE' not found". Fix: exclude isRestSource(name) alongside isSystemCatalog(name) so a rest source falls back to the default (Calcite) pipeline and is never routed to the analytics engine. - RestUnifiedQueryActionTest: unit repro under cluster-composite. - integ-test analyticsEngineCompat testcluster set composite-default so the existing rest coexistence IT exercises this routing exclusion. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Collapse the two near-duplicate Calcite scan hierarchies -- SHOW/DESCRIBE system tables and the rest command -- into one generic OpenSearchCatalogTable whose per-endpoint behavior is supplied by a pluggable CatalogSource. - OpenSearchSystemIndex + RestSourceTable -> one OpenSearchCatalogTable backed by SystemIndexCatalogSource / RestCatalogSource. - Two Abstract/Logical/Enumerable scans + two enumerators + two converter rules -> AbstractCalciteCatalogScan / CalciteLogicalCatalogScan / CalciteEnumerableCatalogScan (+ rest-only CalciteScannableCatalogScan) / OpenSearchCatalogEnumerator / EnumerableCatalogScanRule. - Concerns stay per-source: system tables keep the real V2 implement() path; rest is Calcite-only (implement() throws) and opts into Scannable for the collect short-circuit. No behavior change: dispatch, schemas, V2 support, and the Scannable marker are preserved; this is pure de-duplication of the Calcite scan plumbing. Verified: opensearch compileJava/compileTestJava green; ppl and integ-test test-compile green; affected unit tests pass. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Two dynamic cluster settings gate the rest command per deployment: - plugins.ppl.rest.redaction.enabled (default false): mask network identifiers in _cat/* cells and availability-zone names in _cluster/settings values. - plugins.ppl.rest.allowed_endpoints (default all): restrict which endpoints are served; an empty list disables the rest command. Both default to open-source parity: all endpoints served, no masking. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
The shared language-grammar carried REST and TIMEOUT lexer tokens and the restCommand/restArgument parser rules with no consumer: async-query-core has no rest visitor, and the rest command grammar lives in the ppl module. Remove the dead rules. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Verify the rest command is subject to the security plugin fine grained access control: a caller without cluster:monitor privilege is denied the cat and cluster endpoints, a caller holding the privilege can run them, and the resolve index endpoint is filtered to the caller authorized indices. Test indices are created idempotently, and denials are asserted by the security denial reason in the response body because a denied action on the Calcite only rest path surfaces as a wrapped error carrying that reason. Calcite fallback is disabled so the denial reason is not replaced by an unsupported command error. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
plugins.ppl.rest.redaction.enabled and plugins.ppl.rest.allowed_endpoints were dynamic cluster settings, so they could be changed at runtime through _cluster/settings or the _plugins/_query/settings endpoint, which could override an operator-configured value. Drop Setting.Property.Dynamic so both are node-level settings set in the node config; the engine rejects runtime updates on both paths. Register them without an update consumer and read the node-configured value. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Flip plugins.ppl.rest.allowed_endpoints default from ["*"] to empty so open source ships with the rest command closed. Deployments opt specific endpoints in via the setting (AOS enables the ones it supports; AOSS leaves it empty and stays disabled). Enforcement already treats an empty or missing list as disabled. Opt the integration-test clusters into all endpoints so the rest ITs still exercise the enabled path. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
8bb45cd to
2c6901d
Compare
…ework Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com> # Conflicts: # ppl/src/main/antlr/OpenSearchPPLLexer.g4 # ppl/src/main/antlr/OpenSearchPPLParser.g4 # ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java
- ArgSpec: drop the unused csv-arg path (csvArg builder, csvArgs field, csv branch); no endpoint declares a csv arg (leftover from the deferred _cat endpoints). - RedactionRegistry: drop unused isEmpty() (consumers call mask()). - RestRequest: drop the unused 3-arg convenience ctor; fix a stale comment left by the ctx.client() refactor. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 5ed738d |
|
Persistent review updated to latest commit 84f5773 |
…e centralized Redactor The framework now exposes one Redactor seam ((endpoint, row) -> row, with a NONE no-op) published through RedactorHolder and applied once per raw row at the rest choke point before coercion; the default is NONE. Removes RedactionClass, RedactionRegistry, RedactionRegistryHolder and the per-column redaction tag on Column, so the framework stays field- and facet-agnostic, one coordinated masking pass avoids independent maskers interfering on the same value, and redaction has a single central owner. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
84f5773 to
7637f4d
Compare
|
Persistent review updated to latest commit 7637f4d |
…mmand index Shorten the ppl-rest-spi javadoc and the visitRestCommand comment to a concise, consistent style, correct the rest.md endpoint note, and list the rest command as 3.9 (experimental) in the PPL command index. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 7b63421 |
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 65d3f86 |
…tartup If two providers register the same endpoint name, keep the first registration and log a warning for the duplicate instead of throwing during createComponents. This bounds a naming collision to the offending endpoint rather than failing plugin/node bootstrap, and the first registration still wins so a later provider cannot replace it. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 8d371a8 |
The doctest, integTest, and RestCommandSecurityIT clusters allow-listed endpoints that are not registered in this version; only /_cluster/health exists (and is the default). Trim each to /_cluster/health and correct the comments, which wrongly said the command is disabled by default. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 6a4e66b |
…ssage Reword the resolve() error messages to "Only the following endpoints are supported: <list>", enumerating the registered allow-list (currently /_cluster/health) instead of promising a vaguer "read-only in-cluster endpoints" category. The "is not allow-listed" clause is kept so callers (and the IT assertion) still see that phrase. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 0ff4abc |
| } | ||
| registry.resolve(spec.getEndpoint()); | ||
| List<String> allowed = settings.getSettingValue(Settings.Key.PPL_REST_ALLOWED_ENDPOINTS); | ||
| if (allowed == null || !(allowed.contains("*") || allowed.contains(spec.getEndpoint()))) { |
There was a problem hiding this comment.
Should "*" apply to extension-provided endpoints? With the extensible registry, a cluster configured with ["*"] will automatically enable endpoints added by a newly installed or upgraded RestEndpointProvider, without an explicit allow-list change. This seems to weaken the endpoint-by-endpoint opt-in and security-review boundary. Could "*" be limited to built-in endpoints, with external endpoints requiring explicit names?
| LOG.warn( | ||
| "rest endpoint [{}] is already registered; ignoring the duplicate from provider [{}]", | ||
| definition.name(), | ||
| provider.getClass().getName()); | ||
| continue; |
There was a problem hiding this comment.
Endpoint names are global, while the operator allow-list identifies only the endpoint name and not the contributing provider. If two external providers register the same name, the current first-wins behavior binds that allow-list entry to whichever provider is loaded first. Was this ambiguity considered? Would it be safer to disable only the conflicting endpoint, while allowing the node and the remaining endpoints to start normally?
| The extension point for the PPL `rest` command. Any OpenSearch plugin can contribute read-only, | ||
| fixed-schema management endpoints that become queryable as `rest '<name>'`, without touching the | ||
| PPL grammar and without depending on the sql plugin's internals. The built-in `/_cluster/health` | ||
| endpoint is just one client of this same contract. |
There was a problem hiding this comment.
Could u make the statement more generic. It is not just extension point for the PPL. This solution provide a generic a way to let other plugin to integrate PPL without introduce new command.
The built-in /_cluster/health endpoint is just one client of this same contract. this is just example use case. mention it in example section.
| `loadExtensions` discovers your provider: | ||
|
|
||
| ```gradle | ||
| dependencies { compileOnly project(':ppl-rest-spi') } // or the published artifact |
There was a problem hiding this comment.
It should be clear step by step guide.
Other plugins need published artifact, right?
|
|
||
| ## Rules and guarantees | ||
|
|
||
| - Endpoints are read-only. The handler produces rows; the sql side coerces each value to the |
|
|
||
| - Endpoints are read-only. The handler produces rows; the sql side coerces each value to the | ||
| declared `ColumnType` and rejects an uncoercible value with a clear error. | ||
| - Endpoint names are global. If two providers register the same name, the first registration wins |
There was a problem hiding this comment.
In case all the endpoint is regestered at plugins.ppl.rest.allowed_endpoints. It should no conflict, right?
| and the duplicate is ignored with a logged warning. | ||
| - An endpoint must be both registered by a provider and present in `allowed_endpoints`; anything | ||
| else is rejected before any transport call. | ||
| - Redaction of sensitive values is applied centrally at the row-shaping choke point; a provider |
| // Composite-default cluster: PPL queries route to the analytics engine unless excluded. | ||
| setting 'cluster.pluggable.dataformat', 'composite' |
There was a problem hiding this comment.
This PR should not releated to analytics engine?
| throw new IllegalStateException( | ||
| "the rest command endpoint registry is not initialized on this node"); |
There was a problem hiding this comment.
this is not good user facing error.
|
|
||
| private Table restTable(String name) { | ||
| RestSpec spec = decodeRestSpec(name); | ||
| RestEndpointRegistry registry = RestEndpointRegistryHolder.get(); |
There was a problem hiding this comment.
nit: Why use static class? It is hard to mock and test. Does singleton better?
| List.of( | ||
| Column.of("cluster_name", STRING), | ||
| Column.of("status", STRING), | ||
| Column.of("number_of_nodes", INTEGER), | ||
| Column.of("number_of_data_nodes", INTEGER), | ||
| Column.of("active_primary_shards", INTEGER), | ||
| Column.of("active_shards", INTEGER), | ||
| Column.of("relocating_shards", INTEGER), | ||
| Column.of("initializing_shards", INTEGER), | ||
| Column.of("unassigned_shards", INTEGER), | ||
| Column.of("timed_out", BOOLEAN))) |
There was a problem hiding this comment.
It may not required to expose the schema. single string column should be fine. Then use spath command to extract.
Description
Rebuild of the reverted PPL
restcommand (#5599, reverted in #5635) as a separated, generic, extensible framework.rest <name>exposes read-only, fixed-schema management endpoints as a relational row source behind one operator allow-list, so results feed straight intowhere/stats/head.Key design points
ppl-rest-spimodule exposesRestEndpointProvider(anExtensiblePluginloadExtensionsSPI) plusRestEndpointDefinition/RestEndpointHandler/RestEndpointContextandColumn/ColumnType/ArgSpec. New endpoints are registrations, not new grammar keywords — the language stays a single genericrest <name>command./_cluster/healthships viaCoreEndpointsProvider, which fetches through the SPI context's transport node client (ctx.client()) exactly like an external plugin — there is no privileged in-repo path.ColumnTypekeeps the SPI module free of any dependency on the sql engine'sExprType.plugins.ppl.rest.allowed_endpointsis the singleresolve()gate; it defaults to["/_cluster/health"](health on, every other endpoint off until an operator opts in). Health stays gated by the cluster's owncluster:monitor/healthpermission.EXPLAINis side-effect free andwhere/head/statscompose and push down.RestRequestmirrors the system-indexOpenSearchSystemIndexScanbehind one shared catalog enumerator.Redactor((endpoint, row) -> row) at the choke point before coercion; the default isRedactor.NONE(no-op). Redaction lives at the platform choke point rather than in the endpoint SPI./_cluster/healthreference endpoint; further endpoints are left to follow-up PRs.Related Issues
Rebuild of #5599 (reverted in #5635) as a separated, extensible framework.
How it works:
handler+ctxflow (bootstrap to fetch)Each endpoint's
handleris captured into itsEndpointat bootstrap and travels bundled inside it; the per-queryctx(args + transport client) is assembled atsearch(); the two meet atEndpoint.toRows->handler.fetch(ctx). The registry and redactor cross the bootstrap-to-query Guice gap through static holders.flowchart TB subgraph BOOT["Phase 1 · Plugin bootstrap (once per node; loadExtensions runs before the Guice injector)"] direction TB prov["RestEndpointProvider SPI<br/>CoreEndpointsProvider.getEndpoints()<br/>→ RestEndpointDefinition(name, schema, argSpec, handler)"] load["SQLPlugin.loadExtensions<br/>loader.loadExtensions(RestEndpointProvider.class)"] build["publishRestCommandRegistries()<br/>new RestEndpointRegistry(providers)<br/>each Definition → new Endpoint(def)<br/>★ Endpoint.handler = def.handler()"] hold["RestEndpointRegistryHolder.set(registry)<br/>RedactorHolder.set(Redactor.NONE)"] prov -->|discovered| load load -->|providers| build build -->|publish| hold end subgraph QUERY["Phase 2 · Query execution (per query: rest '/_cluster/health' ...)"] direction TB gt["OpenSearchStorageEngine.restTable()<br/>registry = RestEndpointRegistryHolder.get()<br/>redactor = RedactorHolder.get()"] src["new RestCatalogSource(registry, spec, client, redactor)<br/>★ endpoint = registry.resolve(spec.endpoint) «allow-list gate»"] scan["CalciteEnumerableCatalogScan<br/>source.createRequest()"] req["RestCatalogSource.createRequest()<br/>→ new RestRequest(client, endpoint, spec, redactor)"] en["OpenSearchCatalogEnumerator<br/>request.search()"] se["RestRequest.search()<br/>★ ctx = RestEndpointContext.of(spec.args, nodeClient)<br/>endpoint.toRows(ctx, redactor)"] tr["Endpoint.toRows(ctx, redactor)<br/>★★ handler.fetch(ctx) «handler + ctx meet»<br/>→ redactor.redact(endpoint, raw) → coerce → rows"] gt --> src src --> scan scan --> req req --> en en --> se se --> tr end hold -.->|"registry / redactor (static bridge: bootstrap → query)"| gt build ==>|"handler travels bundled inside Endpoint"| trTesting
Redactorseam, storage engine, and settings.CalcitePPLRestITandRestCommandSecurityITcover/_cluster/healthplus the allow-list and per-arg gates.Check List
docs/user/ppl/cmd/rest.md).--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.