Aggregation return-type change breaks existing VTL templates accessing $results.aggregations

[fabrizzio-dotCMS]

Problem Statement

As part of the Elasticsearch → OpenSearch migration, the search aggregation API was made vendor-neutral. The return type of the aggregations exposed to Velocity changed, which silently breaks existing customer VTL templates that read aggregations from $estool.search(...) (the ESContentTool viewtool) / $contentlets results.

• Before: $results.aggregations returned org.elasticsearch.search.aggregations.Aggregations. Templates walked it as aggregations.<name>.buckets, then bucket.getKeyAsNumber(), bucket.getDocCount(), and bucket.getAggregations().get("top_content").getHits().getHits() for nested top_hits.
• After: $results.aggregations returns java.util.Map<String, java.util.List<AggregationBucket>>. AggregationBucket exposes only key() (String) and docCount() (long), and nested sub-aggregations are dropped entirely.

The query still executes and the data is present in the response — but the template can no longer reach the buckets, so the customer sees empty output / "no results" from the aggregation loop. No error is thrown (Velocity runtime.references.strict=false swallows the broken accessors), which makes it look like missing data.

Impact: Any customer/template relying on the documented Velocity aggregation walk is broken after the migration. Originally reported by a customer using a content_types terms aggregation with nested top_hits.

Findings — accessor-by-accessor breakage map

VTL access	Before (ES Aggregations)	After (neutral)	Status
$results.totalResults	getTotalResults()	getTotalResults()	✅ works
$results.aggregations	Aggregations object	Map<String,List<AggregationBucket>>	⚠️ type changed
...aggregations.content_types	Aggregations.get(name) → a Terms agg	Map.get(name) → already the List<AggregationBucket>	⚠️ semantics changed
...content_types.buckets	Terms.getBuckets()	a List has no .buckets → null	🔴 BREAKS
#foreach($g in $contentTypeGroups)	iterates buckets	iterates null → 0 iterations	🔴 nothing printed
$g.getKeyAsNumber()	exists on Terms.Bucket	AggregationBucket only has key() (String)	🔴 missing
$g.getDocCount()	exists	AggregationBucket has docCount() (no get prefix)	🔴 missing
$g.getAggregations().get("top_content")	nested sub-agg	AggregationBucket has no sub-aggregations	🔴 unrecoverable
$topHits.getHits().getHits()	top_hits hits	—	🔴 unrecoverable

Three distinct regressions (not one)

1. .buckets removed — aggregations.<name> is the bucket list now; templates must drop .buckets.
2. Accessors renamed and un-prefixed — AggregationBucket uses Immutables-style key() / docCount(). There is no getKeyAsNumber() / getDocCount(), and because the methods have no get/is prefix they are not reachable as Velocity properties ($g.key fails); they only work as explicit calls $g.key().
3. Nested aggregations lost — ContentSearchResponse.from(...) maps only first-level terms aggregations ("Only terms aggregations are mapped; other types are silently skipped"). The nested top_content / top_hits per bucket is gone. This cannot be fixed by migrating the template — the neutral model lacks the capability.

Steps to Reproduce

Customer VTL (verbatim):

$response.setContentType("text/plain")
$response.setHeader("Cache-Control", "no-cache")

#set($esQuery = '{"aggs":{"content_types":{"terms":{"field":"contentType","size":5},"aggs":{"top_content":{"top_hits":{"size":3}}}}},"size":5,"query":{"bool":{"filter":[{"term":{"live":true}}]}}}')

#set($results = $estool.search($esQuery))
totalResults: $!{results.totalResults}
aggregations: $!{results.aggregations}
CT: $!{results.aggregations.content_types}

#set($contentTypeGroups = $results.aggregations.content_types.buckets)
buckets: $!{contentTypeGroups}

#foreach($group in $contentTypeGroups)
  key: $!{group.getKeyAsNumber()}
  docCount: $!{group.getDocCount()}
  #set($topHits = $group.getAggregations().get("top_content"))
  #foreach($hit in $topHits.getHits().getHits())
    hit id: $!{hit.id}
  #end
#end

Reproduction steps:

1. Create a content type and publish a few live contentlets.
2. Render the template above through the dotCMS Velocity engine with $estool bound to ESContentTool and $response bound to the HTTP response.
3. Observe: totalResults and aggregations render with data (the content_types buckets are present in the map), but $contentTypeGroups (.buckets) is null, the #foreach runs 0 times, and no key: / docCount: / hit id: lines are emitted.

A reproduction integration test exists at dotcms-integration/.../viewtools/ContentSearchToolTest.java:

• test_customerVTL_verbatim_reproducesAggregationRegression — renders the verbatim template and asserts the regression.
• test_customerVTL_migratedToNeutralApi_emitsBuckets — shows the flat terms agg working once the template drops .buckets and calls key() / docCount() (still no path for nested top_hits).

Acceptance Criteria

• Existing VTL that accessed aggregations.<name>.buckets continues to work, OR a clearly documented migration path is published and the change is treated as a breaking change with release notes.
• A bucket exposes a key and doc count that are reachable from Velocity as both a method and a property (e.g. getKey() / getKeyAsNumber() / getDocCount() in addition to key() / docCount()).
• Nested sub-aggregations used by customers — at minimum top_hits inside a terms bucket — are representable through the neutral model (or an explicit, documented alternative is provided).
• The aggregation walk no longer fails silently: either it works, or it surfaces a meaningful error instead of empty output.
• ContentSearchToolTest (reproduction) passes against the fixed API.
• Decision recorded: backward-compatible neutral model (preferred) vs. documented breaking change.

dotCMS Version

Latest from main branch (Elasticsearch → OpenSearch migration work in progress).

Severity

Critical - System unusable

Links

NA (internal investigation; surfaced from a customer report — Freshdesk ticket TBD)

[github-actions]

PRs linked to this issue

• test(viewtools): reproduce VTL aggregation return-type regression (#36026) by @fabrizzio-dotCMS