Query Information
source=idx_int,idx_float | stats count() by amount
amount is long in idx_int, double in idx_float — typical after a rollover mapping change.
Expected: the aggregation, or a 400 naming the conflicting types.
Actual: no aggregation result. The query fails in the coordinating node's composite reduce:
at InternalComposite$InternalBucket.compareKey(InternalComposite.java:468)
at InternalComposite$BucketIterator.compareTo(InternalComposite.java:323)
at InternalComposite.reduce(InternalComposite.java:220)
at SearchPhaseController.reduceAggs(SearchPhaseController.java:561)
On a standard distribution build this surfaces as
ClassCastException: class java.lang.Double cannot be cast to class java.lang.Long (HTTP 500).
Composite aggregation does not support this case
Not PPL-specific — it reproduces with a raw _search, no plugin:
raw _search across both indices |
result |
terms agg |
works — [{key: 2.5}, {key: 5.0}] |
composite agg |
fails in reduce |
composite + value_type: "double" |
fails identically |
Current behaviour. The shard phase succeeds, keying each bucket by that shard's own mapping —
Long(5) from idx_int, Double(2.5) from idx_float. The coordinator merges those streams through a
PriorityQueue, comparing keys via Comparable.compareTo. Comparing Long to Double is invalid, and
the request fails there.
The contrast with terms is the point: terms normalises numeric types across shards. composite
carries each shard's raw key type into the reduce and normalises nothing, and value_type does not
change that. No request-level option makes it work.
Dataset
PUT idx_int { "mappings": { "properties": { "amount": { "type": "long" } } } }
PUT idx_float { "mappings": { "properties": { "amount": { "type": "double" } } } }
POST idx_int/_doc/1?refresh=true { "amount": 5 }
POST idx_float/_doc/1?refresh=true { "amount": 2.5 }
Each index alone returns 200, as does | fields amount — the merge is not the trigger.
refresh=true matters: both docs must be searchable.
Issue
PPL compiles stats ... by into a composite aggregation and pushes it across every index without
checking whether the group-by field has a consistent type. terms over the same data succeeds, so the
values are reconcilable — PPL picks a shape that cannot handle its own input.
Also reproduces via top and dedup (both compile to composite), and for long/keyword,
scaled_float/keyword, unsigned_long/long, alias→long/keyword.
The last two matter: unsigned_long is dropped by parseMapping
(OpenSearchDataType.java:126-129) and an alias is resolved away, so PPL's merged schema shows one
clean type — no conflict to detect. Detection needs per-index mappings, retained at
OpenSearchDescribeIndexRequest.lastIndexMappings.
Impact: in any index-pattern deployment, one rolled-over index with a changed field type makes
stats ... by on that field unusable, with no error explaining why.
Suggested fix
Do not push a composite aggregation when the group-by field's type differs across indices: partition into
index groups that agree and combine in-engine (PartialResultAggregatePushdown), fall back to terms,
decline the pushdown, or reject with a 400.
A companion core issue may be warranted: composite should normalise key types like terms, or reject
the request up front.
Environment: main @ 3f7048b21 (3.10.0-SNAPSHOT), also 07f079d3d. Reproduces with
plugins.calcite.enabled true and false.
Related: #5752 (same merge path), #5614 (same ClassCastException, dedup composite path),
#4383 (RFC).
Query Information
amountislonginidx_int,doubleinidx_float— typical after a rollover mapping change.Expected: the aggregation, or a
400naming the conflicting types.Actual: no aggregation result. The query fails in the coordinating node's composite reduce:
On a standard distribution build this surfaces as
ClassCastException: class java.lang.Double cannot be cast to class java.lang.Long(HTTP 500).Composite aggregation does not support this case
Not PPL-specific — it reproduces with a raw
_search, no plugin:_searchacross both indicestermsagg[{key: 2.5}, {key: 5.0}]compositeaggcomposite+value_type: "double"Current behaviour. The shard phase succeeds, keying each bucket by that shard's own mapping —
Long(5)fromidx_int,Double(2.5)fromidx_float. The coordinator merges those streams through aPriorityQueue, comparing keys viaComparable.compareTo. ComparingLongtoDoubleis invalid, andthe request fails there.
The contrast with
termsis the point:termsnormalises numeric types across shards.compositecarries each shard's raw key type into the reduce and normalises nothing, and
value_typedoes notchange that. No request-level option makes it work.
Dataset
Each index alone returns
200, as does| fields amount— the merge is not the trigger.refresh=truematters: both docs must be searchable.Issue
PPL compiles
stats ... byinto a composite aggregation and pushes it across every index withoutchecking whether the group-by field has a consistent type.
termsover the same data succeeds, so thevalues are reconcilable — PPL picks a shape that cannot handle its own input.
Also reproduces via
topanddedup(both compile to composite), and forlong/keyword,scaled_float/keyword,unsigned_long/long,alias→long/keyword.The last two matter:
unsigned_longis dropped byparseMapping(
OpenSearchDataType.java:126-129) and analiasis resolved away, so PPL's merged schema shows oneclean type — no conflict to detect. Detection needs per-index mappings, retained at
OpenSearchDescribeIndexRequest.lastIndexMappings.Impact: in any index-pattern deployment, one rolled-over index with a changed field type makes
stats ... byon that field unusable, with no error explaining why.Suggested fix
Do not push a composite aggregation when the group-by field's type differs across indices: partition into
index groups that agree and combine in-engine (
PartialResultAggregatePushdown), fall back toterms,decline the pushdown, or reject with a
400.A companion core issue may be warranted: composite should normalise key types like
terms, or rejectthe request up front.
Environment:
main@3f7048b21(3.10.0-SNAPSHOT), also07f079d3d. Reproduces withplugins.calcite.enabledtrue and false.Related: #5752 (same merge path), #5614 (same
ClassCastException,dedupcomposite path),#4383 (RFC).