Skip to content

minor: sync with upstream master (92fe6f5f71) - #3

Merged
Fryuni merged 1065 commits into
masterfrom
fryuni/sync-upstream-master
Oct 16, 2026
Merged

Fryuni merged 1065 commits into
masterfrom
fryuni/sync-upstream-master

Conversation

@Fryuni

@Fryuni Fryuni commented Oct 16, 2026

Copy link
Copy Markdown
Member

Description

Merge upstream/master (apache/druid@92fe6f5f71, 1064 commits) into the fork's master.

Important

Merge this PR with Create a merge commit. Squash or rebase would flatten upstream's history and break future syncs.

ADR review

  • ADR 0001 (Kafka offset lookups retry everything): upstream has not implemented this. Its KafkaRecordSupplier.getEarliestSequenceNumber() / getLatestSequenceNumber() still seek and call position() with no retry. The fork patch stays and the ADR is unchanged. No new upstream supervisor code path bypasses the patched lookups.

Conflict resolution

  • AGENTS.md: upstream added its own root AGENTS.md. Kept the fork-only "Fork Policy" and "Agent skills" sections on top of upstream's content.
  • KafkaSupervisor.updatePartitionLagFromStream(): upstream (Fix negative Kafka partition lag caused by inconsistent current/latest offsets apache/druid#18750) now builds a local offset map and publishes it via updateOffsetSnapshot(). Took upstream's structure and re-applied ADR 0001 by fetching latest offsets with the batch getLatestSequenceNumbers() (endOffsets()) lookup, not seekToLatest() + getPosition().
  • KafkaRecordSupplierTest: upstream migrated the class to JUnit 5 and added a metricBuilderSupplier argument to the consumer constructor. Ported the fork's four ADR 0001 tests to Assertions and new KafkaRecordSupplier(consumer, false, null).

KafkaRecordSupplier merged cleanly, and the retrying beginningOffsets() / endOffsets() lookups are intact. After the merge, git diff upstream/master HEAD contains only the fork's own changes (ADR, agent docs, ticket, and the Kafka patch).

Testing

  • mvn test -pl extensions-core/kafka-indexing-service -am -Dtest=KafkaRecordSupplierTest,KafkaSupervisorTest: 185 tests, 0 failures, 1 flake. The flake is KafkaSupervisorTest.testBadMetadataOffsets, which passed on rerun. On the failing attempt, a supervisor leaked from an earlier test (TaskRunner.getTaskLocation("id1")) emitted an alert first. That's an upstream test-isolation race, unrelated to this patch.
  • Checkstyle, PMD and forbidden-apis on druid-kafka-indexing-service (and its -am dependencies): clean.
  • Note: upstream now runs these tests against a Testcontainers Kafka broker and requires Java 25. My local Docker uses the journald log driver, so docker logs returns nothing and Testcontainers times out. To run the tests locally, I temporarily forced the broker container to the json-file log driver and disabled Ryuk. That tweak is not part of this PR.

Release note

No end-user impact beyond upstream's own changes.


Key changed/added classes in this PR
  • KafkaSupervisor
  • KafkaRecordSupplierTest
  • AGENTS.md

This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

🤖 Generated with Claude Code

dependabot Bot and others added 30 commits September 6, 2026 09:37
….3 (apache#20266)

* build(deps): bump org.checkerframework:checker-qual from 4.2.2 to 4.2.3

Bumps [org.checkerframework:checker-qual](https://github.com/typetools/checker-framework) from 4.2.2 to 4.2.3.
- [Release notes](https://github.com/typetools/checker-framework/releases)
- [Changelog](https://github.com/typetools/checker-framework/blob/master/docs/CHANGELOG.md)
- [Commits](typetools/checker-framework@checker-framework-4.2.2...checker-framework-4.2.3)

---
updated-dependencies:
- dependency-name: org.checkerframework:checker-qual
  dependency-version: 4.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build: update checker-qual license inventory

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
Bumps [io.grpc:grpc-bom](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.83.1...v1.84.0)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-bom
  dependency-version: 1.84.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ache#20263)

Bumps [io.grpc:grpc-netty-shaded](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.83.1...v1.84.0)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-netty-shaded
  dependency-version: 1.84.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump oshi.version from 7.6.0 to 7.6.1

Bumps `oshi.version` from 7.6.0 to 7.6.1.

Updates `com.github.oshi:oshi-core` from 7.6.0 to 7.6.1
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](oshi/oshi@oshi-parent-7.6.0...oshi-parent-7.6.1)

Updates `com.github.oshi:oshi-common` from 7.6.0 to 7.6.1
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](oshi/oshi@oshi-parent-7.6.0...oshi-parent-7.6.1)

---
updated-dependencies:
- dependency-name: com.github.oshi:oshi-core
  dependency-version: 7.6.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.github.oshi:oshi-common
  dependency-version: 7.6.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): update OSHI license metadata

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…gin (apache#20260)

Bumps [io.github.git-commit-id:git-commit-id-maven-plugin](https://github.com/git-commit-id/git-commit-id-maven-plugin) from 10.0.0 to 10.0.1.
- [Release notes](https://github.com/git-commit-id/git-commit-id-maven-plugin/releases)
- [Commits](git-commit-id/git-commit-id-maven-plugin@v10.0.0...v10.0.1)

---
updated-dependencies:
- dependency-name: io.github.git-commit-id:git-commit-id-maven-plugin
  dependency-version: 10.0.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump io.grpc:grpc-api from 1.83.1 to 1.84.0

Bumps [io.grpc:grpc-api](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.83.1...v1.84.0)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-api
  dependency-version: 1.84.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build: update grpc api license inventory

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…20257)

* build(deps): bump io.grpc:grpc-context from 1.83.1 to 1.84.0

Bumps [io.grpc:grpc-context](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.83.1...v1.84.0)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-context
  dependency-version: 1.84.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): align grpc-api with grpc-context

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…0256)

* build(deps): bump aws.sdk.v2.version from 2.54.8 to 2.54.10

Bumps `aws.sdk.v2.version` from 2.54.8 to 2.54.10.

Updates `software.amazon.awssdk:s3` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:s3-transfer-manager` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:sts` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:apache-client` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:auth` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:regions` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:ec2` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:aws-core` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:sdk-core` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:netty-nio-client` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:aws-crt-client` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:bom` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:retries-spi` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:retries` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:kinesis` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:http-client-spi` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:utils` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:rds` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:glue` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:kms` from 2.54.8 to 2.54.10

Updates `software.amazon.awssdk:bundle` from 2.54.8 to 2.54.10

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:apache-client
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:auth
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:aws-core
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:aws-crt-client
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:bom
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:bundle
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:ec2
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:glue
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:http-client-spi
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:kinesis
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:kms
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:netty-nio-client
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:rds
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:regions
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:retries
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:retries-spi
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:s3
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:s3-transfer-manager
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sdk-core
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sts
  dependency-version: 2.54.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:utils
  dependency-version: 2.54.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: update AWS SDK license registry

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
* build(deps): bump parquet.version from 1.18.0 to 1.18.1

Bumps `parquet.version` from 1.18.0 to 1.18.1.

Updates `org.apache.parquet:parquet-column` from 1.18.0 to 1.18.1
- [Release notes](https://github.com/apache/parquet-java/releases)
- [Changelog](https://github.com/apache/parquet-java/blob/master/CHANGES.md)
- [Commits](apache/parquet-java@apache-parquet-1.18.0...apache-parquet-1.18.1)

Updates `org.apache.parquet:parquet-hadoop` from 1.18.0 to 1.18.1
- [Release notes](https://github.com/apache/parquet-java/releases)
- [Changelog](https://github.com/apache/parquet-java/blob/master/CHANGES.md)
- [Commits](apache/parquet-java@apache-parquet-1.18.0...apache-parquet-1.18.1)

Updates `org.apache.parquet:parquet-avro` from 1.18.0 to 1.18.1
- [Release notes](https://github.com/apache/parquet-java/releases)
- [Changelog](https://github.com/apache/parquet-java/blob/master/CHANGES.md)
- [Commits](apache/parquet-java@apache-parquet-1.18.0...apache-parquet-1.18.1)

---
updated-dependencies:
- dependency-name: org.apache.parquet:parquet-column
  dependency-version: 1.18.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.apache.parquet:parquet-hadoop
  dependency-version: 1.18.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.apache.parquet:parquet-avro
  dependency-version: 1.18.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): align Parquet zstd dependency

* build: document parquet zstd exclusion

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…0269)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.14.2 to 3.15.2.
- [Changelog](https://github.com/nodeca/js-yaml/blob/3.15.2/CHANGELOG.md)
- [Commits](nodeca/js-yaml@3.14.2...3.15.2)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 3.15.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* test: wait for completed Dart reports

* test: use shared waiter for Dart reports
…pache#20127)

* perf(embedded-tests): reduce auto compaction test polling delay

* perf(embedded-tests): reduce MSQ fault tolerance test delay

* perf(embedded-tests): reduce auto scaler ingestion wait

* perf(processing): cache frame writer expected rows

* perf(indexing-service): reduce serial task polling interval

* docs(test): explain timing optimizations

* perf(test): reduce Kafka and Kinesis test setup time

* docs(test): explain Kinesis batch limit

* perf(test): reuse merged timestamps in iterator test

* docs(test): explain merged timestamp reuse

* perf(test): avoid exhausting large granularity iterator

* fix(test): preserve granularity interval count assertion

* test: make MSQ fault tolerance timing deterministic

* fix(test): handle task status delay timeout
* refactor!: remove deprecated SysMonitor

* refactor: remove remaining Sigar usage

* docs: remove upgrade notes entry

* refactor: remove Sigar PID discoverer

* fix: preserve JVM CPU baselines on process lookup failure

* docs: explain JVM process lookup
* ci: optimize Docker test pipeline

* ci: limit Docker build context
…9687)

Changes currently in this PR:
- UI code is generated by Claude and may contain mistakes
- Add a simulate API currently supported for Kafka supervisors only
- This API creates a `CostBasedAutoScaler` in "simulate" mode and generates the optimal task count for various input values of lag (based on criticalLag)
- Add a UI panel in the supervisor dialog which shows up only for "kafka" supervisors
- Add a single plot between task count vs lag
…apache#20289)

Bumps com.google.auth:google-auth-library-credentials from 1.51.0 to 1.52.0.

---
updated-dependencies:
- dependency-name: com.google.auth:google-auth-library-credentials
  dependency-version: 1.52.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#20288)

Bumps [com.squareup.okio:okio-jvm](https://github.com/lysine-dev/okio) from 3.18.1 to 3.18.2.
- [Changelog](https://github.com/lysine-dev/okio/blob/main/CHANGELOG.md)
- [Commits](lysine-dev/okio@parent-3.18.1...parent-3.18.2)

---
updated-dependencies:
- dependency-name: com.squareup.okio:okio-jvm
  dependency-version: 3.18.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…0285)

Bumps `bytebuddy.version` from 1.18.12 to 1.18.13.

Updates `net.bytebuddy:byte-buddy` from 1.18.12 to 1.18.13
- [Release notes](https://github.com/raphw/byte-buddy/releases)
- [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md)
- [Commits](raphw/byte-buddy@byte-buddy-1.18.12...byte-buddy-1.18.13)

Updates `net.bytebuddy:byte-buddy-agent` from 1.18.12 to 1.18.13
- [Release notes](https://github.com/raphw/byte-buddy/releases)
- [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md)
- [Commits](raphw/byte-buddy@byte-buddy-1.18.12...byte-buddy-1.18.13)

---
updated-dependencies:
- dependency-name: net.bytebuddy:byte-buddy
  dependency-version: 1.18.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: net.bytebuddy:byte-buddy-agent
  dependency-version: 1.18.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…20283)

Bumps [net.thisptr:jackson-jq](https://github.com/eiiches/jackson-jq) from 1.6.3 to 1.6.4.
- [Release notes](https://github.com/eiiches/jackson-jq/releases)
- [Commits](eiiches/jackson-jq@1.6.3...1.6.4)

---
updated-dependencies:
- dependency-name: net.thisptr:jackson-jq
  dependency-version: 1.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…21 (apache#20282)

Bumps [org.roaringbitmap:RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap) from 1.6.20 to 1.6.21.
- [Release notes](https://github.com/RoaringBitmap/RoaringBitmap/releases)
- [Commits](https://github.com/RoaringBitmap/RoaringBitmap/commits)

---
updated-dependencies:
- dependency-name: org.roaringbitmap:RoaringBitmap
  dependency-version: 1.6.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#20281)

Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/google-cloud-java) from 2.75.0 to 2.76.0.
- [Release notes](https://github.com/googleapis/google-cloud-java/releases)
- [Changelog](https://github.com/googleapis/google-cloud-java/blob/main/java-document-ai/CHANGELOG.md)
- [Commits](googleapis/google-cloud-java@gapic-generator-java/v2.75.0...gapic-generator-java/v2.76.0)

---
updated-dependencies:
- dependency-name: com.google.api.grpc:proto-google-common-protos
  dependency-version: 2.76.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `slf4j.version` from 2.0.18 to 2.0.19.

Updates `org.slf4j:jcl-over-slf4j` from 2.0.18 to 2.0.19

Updates `org.slf4j:slf4j-api` from 2.0.18 to 2.0.19

Updates `org.slf4j:slf4j-simple` from 2.0.18 to 2.0.19

---
updated-dependencies:
- dependency-name: org.slf4j:jcl-over-slf4j
  dependency-version: 2.0.19
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.slf4j:slf4j-api
  dependency-version: 2.0.19
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.slf4j:slf4j-simple
  dependency-version: 2.0.19
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Description

If a segment becomes unavailable after it has been added to the load queue of a historical, it might remain stuck in the queue while other segments are being loaded. Ideally, all unavailable segments should be prioritized.
Changes

    Simplify book-keeping in ServerHolder by getting rid of method simplify() and treat REPLICATE and LOAD actions as distinct
    In StrategicSegmentAssigner.updateReplicasInTier(), check if an unavailable segment needs to be prioritized and change all in-flight REPLICATE actions on that segment to LOAD.
    Update LoadQueuePeon.getSegmentsInQueue to return a List instead of a Set and verify the result in Coordinator simulations
Main changes:

1) Sync of Parser.jj (using dev/upgrade-calcite-parser) and
   CalcitePlanner.

2) Add DruidTypeFactory to work around undesirable behavior introduced
   by CALCITE-7358.

3) Add validation to prevent aliases in CLUSTERED BY, which have been
   added to the parser since CALCITE-5347. (The change is done at the
   validator instead of the parser to minimize parser diffs.)

4) Adjust test expectations for slightly different plans.
…che#20306)

PartialCompactionTest always builds its tuning config with
maxNumConcurrentSubTasks = 2, so it falls back to the 1000ms production
default for taskStatusCheckPeriodMs. Each of its three back-to-back
indexing/compaction rounds pays up to a full poll period per phase
transition, which dominates the test's wall-clock time.

Override newTuningConfig in this class to poll every 100ms. The override
mirrors the base builder exactly except for the poll interval. None of the
class's assertions depend on poll cadence, and the injected transient
task/API failures are retried by count, not time, so they behave the same.

CI (JDK 25): 42.00s -> 7.31s, 35.54s -> 5.71s, 24.13s -> 3.85s.
Bumps [svgo](https://github.com/svg/svgo) from 3.3.4 to 3.3.5.
- [Release notes](https://github.com/svg/svgo/releases)
- [Commits](svg/svgo@v3.3.4...v3.3.5)

---
updated-dependencies:
- dependency-name: svgo
  dependency-version: 3.3.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [joi](https://github.com/hapijs/joi) from 17.13.3 to 17.13.7.
- [Commits](hapijs/joi@v17.13.3...v17.13.7)

---
updated-dependencies:
- dependency-name: joi
  dependency-version: 17.13.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…he#20302)

Bumps [com.github.spotbugs:spotbugs-maven-plugin](https://github.com/spotbugs/spotbugs-maven-plugin) from 4.10.4.0 to 4.10.4.1.
- [Release notes](https://github.com/spotbugs/spotbugs-maven-plugin/releases)
- [Commits](spotbugs/spotbugs-maven-plugin@spotbugs-maven-plugin-4.10.4.0...spotbugs-maven-plugin-4.10.4.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-maven-plugin
  dependency-version: 4.10.4.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
clintropolis and others added 29 commits October 12, 2026 21:26
…ache#20541)

Addresses four problems:

1) Provide PhysicalSegmentColumnInspector from PartialQueryableIndexSegment.
   Previously it was not provided, so segmentMetadata queries were not
   able to report types.

2) Look at all cluster groups when determining capabilities, not just
   the first one. (They can differ in the details, like auto-determined
   type and hasMultipleValues.)

3) Use the DimensionIndexer's ColumnFormat when writing columns in
   cluster groups and projections. Previously the ColumnFormat present
   at creation time was used, which could differ in details (such as
   hasMultipleValues). This caused multi-valued columns to be
   incorrectly persisted, in situations where the persist logic did
   not realize they were multi-value.

4) Updates Capable.or and Capable.and to use correct 3-valued logic.
   This made certain new code simpler to write, and as far as I can
   tell, will only have positive effects on existing callers.
CodeQL Action v3 runs on Node.js 20 and is deprecated in December 2026. v4 runs on Node.js 24.
* Enable Dependabot version updates for GitHub Actions

* Exempt codeql-action from the GitHub Actions cooldown

* Trim cooldown comments

* Use a 7-day cooldown for GitHub Actions instead of exempting codeql-action
…he#20450)

* perf: cache incremental-index expression plans

* fix: avoid sharing mutable expression state

* fix: return fresh expressions from plan cache

* test: use daemon executor for expression cache concurrency

* docs: explain expression plan cache array

* perf: reuse expression selectors during ingestion

* test: cover expression selector reuse with arrays

* test: clarify expression ingestion expectation

* perf: add expression ingestion benchmark

* fix: reevaluate non-deterministic ingestion expressions
* refactor: migrate query context reference parameters

* fix: address review findings on query context descriptors

- Merge the user context over default contexts with QueryContexts.withDefaults, so a null user value falls back to the configured default on both the SQL and native paths.
- Declare maxSubqueryBytes as a string parameter that accepts the forms the subquery guardrail accepts.
- Deprecate the key constants replaced by descriptors for removal, and migrate their remaining uses.
- Escape default descriptions and ignore CRLF line endings in the generated documentation check.

* fix: address second review of query context descriptors
…ask groups are complete (apache#20505)

The changes in apache#19091 have shown that tasks often need to wait for their priors
to finish publishing as we often see logs like:

```
Task[xyz] needs to wait before publishing as other taskGroups[abc, def]
are currently publishing to partition[123].
```

Before this patch, if publish from streaming task A is slow, a later generation task B keeps
trying to publish repeatedly until eventually failing once the number of retries is exhausted.
It is difficult to fix the number of retries since publish times may vary across supervisors
and over time. It is also pointless to retry if a prior task is yet to finish publishing.

This patch addresses these problems by blocking the publish of later task B until A is complete.

Changes
---------
- Add `TaskActionClient.submitAsync` which returns a `ListenableFuture`
- Update API `/druid/indexer/v1/action` in `OverlordResource` to be async by employing
the new `submitAsync()` method
- Update task actions `segmentTransactionalAppend` and `segmentTransactionalReplace`
to implement `performAsync()`
- Update `LocalTaskActionClient.submitAsync()` to use `performAsync()` when supported
- Attempt segment publish only when the task is ready to publish
- Complete async API request only when the publish attempt is done
- For non-streaming tasks, attempt to publish immediately
- Add method `isTaskReadyToPublish()` in `SeekableStreamSupervisor`.
- Return a `ListenableFuture` that completes when all prior task groups have completed 
- Update `SegmentAllocateAction` and `SegmentAllocationQueue` to use `ListenableFuture`
Bumps the github-actions group with 4 updates: [actions/checkout](https://github.com/actions/checkout), [actions/setup-java](https://github.com/actions/setup-java), [actions/labeler](https://github.com/actions/labeler) and [actions/stale](https://github.com/actions/stale).


Updates `actions/checkout` from 6 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v6...v7)

Updates `actions/setup-java` from 5 to 6
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](actions/setup-java@v5...v6)

Updates `actions/labeler` from 6 to 7
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](actions/labeler@v6...v7)

Updates `actions/stale` from 10 to 11
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v10...v11)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/setup-java
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/labeler
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/stale
  dependency-version: '11'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#20588)

ScanQueryFrameProcessor currently includes the original offset and
limit when sending queries to data servers. It should instead send
an offset of zero and add the offset to the limit.
* build(deps): bump com.puppycrawl.tools:checkstyle from 12.3.1 to 14.3.0

Bumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle) from 12.3.1 to 14.3.0.
- [Release notes](https://github.com/checkstyle/checkstyle/releases)
- [Commits](checkstyle/checkstyle@checkstyle-12.3.1...checkstyle-14.3.0)

---
updated-dependencies:
- dependency-name: com.puppycrawl.tools:checkstyle
  dependency-version: 14.3.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix Javadoc that Checkstyle 14 cannot parse

Checkstyle 13 and later use a stricter Javadoc parser, and the UnusedImports
check reports every comment it cannot parse (and treats imports referenced
only from such comments as unused). Fix the malformed {@link}, @see and
@throws syntax, unbalanced {@code} braces, stray or mistyped HTML tags, and
unescaped generics and line-leading annotations in the affected comments.

* Point GuavaUtils.firstNonNull Javadoc link at MoreObjects

Guava's Objects#firstNonNull no longer exists, and javadoc cannot resolve the
(T, T) parameter list. Link to MoreObjects#firstNonNull(Object, Object).

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This patch adds the ability for SQL UNION to run over relations that
involve projections and filters of base tables, such as:

  SELECT dim1, SUM(m1), COUNT(*)
  FROM (SELECT dim1, m1 FROM foo2 WHERE dim1 = '1'
        UNION ALL
        SELECT dim1, m1 FROM foo)
  GROUP BY 1;

Such queries are planned into stacks of the existing "filter" datasource
and the new "project" datasource.

Other changes, outside the SQL planner:

1) DataSource method "createSegmentPruner" has DimFilter and
   VirtualColumns added as parameters. This enables DataSource
   implementations, such as FilteredDataSource and ProjectDataSource,
   to rewrite the filter and contribute additional virtual columns to
   the pruning logic. ExecutionVertex#isBaseColumn is removed and more
   scenarios are now pruneable.

2) DataSource gets "getReservedColumns" with a set of column names to
   avoid when generating higher-level virtual columns. This is used
   by the SQL planner to avoid name collisions.

3) DataSource gets "mayFilterRows", which allows time-boundary queries
   to properly recognize that filtering DataSources should go down the
   "query has filter" path.

4) Fix an issue where various CursorHolder implementations return
   lists from "getOrdering" that erroneously include base columns
   which were shadowed by virtual columns.

5) Updates to MSQ to call the new ExecutionVertex#splitUnion and
   splitUnionSegmentMapFunctions methods to provide each arm with its
   own SegmentPruner and  SegmentMapFunction.
…#20523)

fetchAllUsedSegmentsWithOvershadowedStatus hands out a ResourceHolder that closes the streamed response; MetadataSegmentView and SystemSchema close it, also on early exit, and QueryHandler now closes the bound Interpreter so scans under Filter/Project nodes are released too. No segment left behind.
List of suppressed CVEs:
- elasticache-java-cluster-client: client lib, not memcached server (CPE false positive)
- dev.cel:protobuf: CPE collision with google protobuf-java (CVE-2022-3171)
- async-http-client: no auto-redirect/cookie store in Druid's usage (CVE-2026-45300)
- netty-transport 4.1.100: hostname verification enabled in ChannelResourceFactory
(CVE-2023-4586)
- npm uuid: only uuidv4() used, no buf argument (GHSA-w5hq-g745-h8pq)
- kotlin-stdlib: build-toolchain only, no Kotlin source in Druid (CVE-2026-53914)
- azure-core/identity/json: Key Vault crypto path unused, blob storage auth only
(CVE-2026-33117)
- hive-storage-api: used only for Murmur3 hash, no Hive Metastore/HiveServer2
(CVE-2026-49845, -53561, -55976)
- jackson-databind shaded in hadoop-client-runtime/parquet-jackson
(CVE-2026-54512, -54513, -68497)
- jetty shaded in hadoop-client-runtime, used for Hadoop's internal HTTP only
(CVE-2026-2332, -10050)
- jline shaded in hadoop-client-runtime, Hadoop CLI only (CVE-2026-56741, -56740)
- opentelemetry-gcp-resources: CPE collision with OTel Go SDK
(CVE-2026-24051, -39883, -29181)
- httpclient5/httpcore5 shaded in docker-java-transport-zerodep: test-scope only
(CVE-2026-40542, -71290, -54399, -54428)
- zstd-jni: vulnerable paths not activated, pending upgrade to 1.5.7-14
(CVE-2026-87823, -87795)
- snappy-java: Druid uses Commons Compress, not xerial ByteBuffer API
(CVE-2026-90559)
- parquet-column/parquet-common: envelope encryption/crypto.keytools unused
(CVE-2026-73334)
- zookeeper/zookeeper-jute: used as client only, no server/quorum
(CVE-2026-79993, -59739, -59969)
…#20313)

Changes:
- Track in-flight requests in `DartWorkerClientImpl`
- Cancel in-flight requests when the client is closed
Bumps com.google.auth:google-auth-library-credentials from 1.53.0 to 1.54.0.

---
updated-dependencies:
- dependency-name: com.google.auth:google-auth-library-credentials
  dependency-version: 1.54.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…apache#20585)

Bumps [org.codehaus.woodstox:stax2-api](https://github.com/FasterXML/stax2-api) from 4.3.0 to 4.3.1.
- [Commits](FasterXML/stax2-api@stax2-api-4.3.0...stax2-api-4.3.1)

---
updated-dependencies:
- dependency-name: org.codehaus.woodstox:stax2-api
  dependency-version: 4.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump oshi.version from 7.6.1 to 7.7.0

Bumps `oshi.version` from 7.6.1 to 7.7.0.

Updates `com.github.oshi:oshi-core` from 7.6.1 to 7.7.0
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](oshi/oshi@oshi-parent-7.6.1...oshi-parent-7.7.0)

Updates `com.github.oshi:oshi-common` from 7.6.1 to 7.7.0
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](oshi/oshi@oshi-parent-7.6.1...oshi-parent-7.7.0)

---
updated-dependencies:
- dependency-name: com.github.oshi:oshi-core
  dependency-version: 7.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.github.oshi:oshi-common
  dependency-version: 7.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): update OSHI license metadata

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
….0 (apache#20583)

* build(deps): bump org.apache.maven:maven-artifact from 3.9.16 to 3.10.0

Bumps org.apache.maven:maven-artifact from 3.9.16 to 3.10.0.

---
updated-dependencies:
- dependency-name: org.apache.maven:maven-artifact
  dependency-version: 3.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): update Maven Artifact license metadata

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…7.3.0 (apache#20582)

* build(deps): bump com.fasterxml.woodstox:woodstox-core

Bumps [com.fasterxml.woodstox:woodstox-core](https://github.com/FasterXML/woodstox) from 7.2.2 to 7.3.0.
- [Commits](FasterXML/woodstox@woodstox-core-7.2.2...woodstox-core-7.3.0)

---
updated-dependencies:
- dependency-name: com.fasterxml.woodstox:woodstox-core
  dependency-version: 7.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): align stax2-api and license metadata

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…#20581)

* build(deps): bump guava.version from 33.7.1-jre to 33.7.2-jre

Bumps `guava.version` from 33.7.1-jre to 33.7.2-jre.

Updates `com.google.guava:guava` from 33.7.1-jre to 33.7.2-jre
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

Updates `com.google.guava:guava-testlib` from 33.7.1-jre to 33.7.2-jre
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-version: 33.7.2-jre
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.google.guava:guava-testlib
  dependency-version: 33.7.2-jre
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): update Guava license metadata

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…eam-master

Conflict resolution:

- AGENTS.md: keep the fork-only "Fork Policy" and "Agent skills"
  sections on top of upstream's content.
- KafkaSupervisor.updatePartitionLagFromStream(): upstream (apache#18750) now
  builds a local offset map and publishes it via updateOffsetSnapshot().
  Keep that structure, but re-apply ADR 0001 by fetching latest offsets
  with the batch getLatestSequenceNumbers() (endOffsets) lookup instead of
  seekToLatest() + getPosition().
- KafkaRecordSupplierTest: upstream migrated the class to JUnit 5 and
  added a metric builder supplier to the consumer constructor. Port the
  fork's ADR 0001 tests to Assertions and the 3-arg constructor.

KafkaRecordSupplier merged cleanly; the ADR 0001 retrying
beginningOffsets()/endOffsets() lookups are preserved. Upstream has not
implemented ADR 0001's behaviour, so it remains a fork patch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Fryuni
Fryuni merged commit 8ae4343 into master Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.