Skip to content

[BUG] Composite aggregation pushdown does not support a field whose type differs across indices #5811

Description

@penghuo

Query Information

source=idx_int,idx_float | stats count() by amount

amount is long in idx_int, double in idx_float — typical after a rollover mapping change.

Expected: the aggregation, or a 400 naming the conflicting types.

Actual: no aggregation result. The query fails in the coordinating node's composite reduce:

at InternalComposite$InternalBucket.compareKey(InternalComposite.java:468)
at InternalComposite$BucketIterator.compareTo(InternalComposite.java:323)
at InternalComposite.reduce(InternalComposite.java:220)
at SearchPhaseController.reduceAggs(SearchPhaseController.java:561)

On a standard distribution build this surfaces as
ClassCastException: class java.lang.Double cannot be cast to class java.lang.Long (HTTP 500).

Composite aggregation does not support this case

Not PPL-specific — it reproduces with a raw _search, no plugin:

raw _search across both indices result
terms agg works — [{key: 2.5}, {key: 5.0}]
composite agg fails in reduce
composite + value_type: "double" fails identically

Current behaviour. The shard phase succeeds, keying each bucket by that shard's own mapping —
Long(5) from idx_int, Double(2.5) from idx_float. The coordinator merges those streams through a
PriorityQueue, comparing keys via Comparable.compareTo. Comparing Long to Double is invalid, and
the request fails there.

The contrast with terms is the point: terms normalises numeric types across shards. composite
carries each shard's raw key type into the reduce and normalises nothing, and value_type does not
change that. No request-level option makes it work.

Dataset

PUT idx_int    { "mappings": { "properties": { "amount": { "type": "long"   } } } }
PUT idx_float  { "mappings": { "properties": { "amount": { "type": "double" } } } }
POST idx_int/_doc/1?refresh=true    { "amount": 5 }
POST idx_float/_doc/1?refresh=true  { "amount": 2.5 }

Each index alone returns 200, as does | fields amount — the merge is not the trigger.
refresh=true matters: both docs must be searchable.

Issue

PPL compiles stats ... by into a composite aggregation and pushes it across every index without
checking whether the group-by field has a consistent type. terms over the same data succeeds, so the
values are reconcilable — PPL picks a shape that cannot handle its own input.

Also reproduces via top and dedup (both compile to composite), and for long/keyword,
scaled_float/keyword, unsigned_long/long, alias→long/keyword.

The last two matter: unsigned_long is dropped by parseMapping
(OpenSearchDataType.java:126-129) and an alias is resolved away, so PPL's merged schema shows one
clean type — no conflict to detect. Detection needs per-index mappings, retained at
OpenSearchDescribeIndexRequest.lastIndexMappings.

Impact: in any index-pattern deployment, one rolled-over index with a changed field type makes
stats ... by on that field unusable, with no error explaining why.

Suggested fix

Do not push a composite aggregation when the group-by field's type differs across indices: partition into
index groups that agree and combine in-engine (PartialResultAggregatePushdown), fall back to terms,
decline the pushdown, or reject with a 400.

A companion core issue may be warranted: composite should normalise key types like terms, or reject
the request up front.

Environment: main @ 3f7048b21 (3.10.0-SNAPSHOT), also 07f079d3d. Reproduces with
plugins.calcite.enabled true and false.

Related: #5752 (same merge path), #5614 (same ClassCastException, dedup composite path),
#4383 (RFC).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PPLPiped processing languagebugSomething isn't workinguntriaged

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions