Repository navigation
minor: sync with upstream master (92fe6f5f71) - #3
Merged
Merged
Conversation
….3 (apache#20266) * build(deps): bump org.checkerframework:checker-qual from 4.2.2 to 4.2.3 Bumps [org.checkerframework:checker-qual](https://github.com/typetools/checker-framework) from 4.2.2 to 4.2.3. - [Release notes](https://github.com/typetools/checker-framework/releases) - [Changelog](https://github.com/typetools/checker-framework/blob/master/docs/CHANGELOG.md) - [Commits](typetools/checker-framework@checker-framework-4.2.2...checker-framework-4.2.3) --- updated-dependencies: - dependency-name: org.checkerframework:checker-qual dependency-version: 4.2.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build: update checker-qual license inventory --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
Bumps [io.grpc:grpc-bom](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.83.1...v1.84.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-bom dependency-version: 1.84.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ache#20263) Bumps [io.grpc:grpc-netty-shaded](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.83.1...v1.84.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-netty-shaded dependency-version: 1.84.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump oshi.version from 7.6.0 to 7.6.1 Bumps `oshi.version` from 7.6.0 to 7.6.1. Updates `com.github.oshi:oshi-core` from 7.6.0 to 7.6.1 - [Release notes](https://github.com/oshi/oshi/releases) - [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md) - [Commits](oshi/oshi@oshi-parent-7.6.0...oshi-parent-7.6.1) Updates `com.github.oshi:oshi-common` from 7.6.0 to 7.6.1 - [Release notes](https://github.com/oshi/oshi/releases) - [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md) - [Commits](oshi/oshi@oshi-parent-7.6.0...oshi-parent-7.6.1) --- updated-dependencies: - dependency-name: com.github.oshi:oshi-core dependency-version: 7.6.1 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: com.github.oshi:oshi-common dependency-version: 7.6.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): update OSHI license metadata --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…gin (apache#20260) Bumps [io.github.git-commit-id:git-commit-id-maven-plugin](https://github.com/git-commit-id/git-commit-id-maven-plugin) from 10.0.0 to 10.0.1. - [Release notes](https://github.com/git-commit-id/git-commit-id-maven-plugin/releases) - [Commits](git-commit-id/git-commit-id-maven-plugin@v10.0.0...v10.0.1) --- updated-dependencies: - dependency-name: io.github.git-commit-id:git-commit-id-maven-plugin dependency-version: 10.0.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump io.grpc:grpc-api from 1.83.1 to 1.84.0 Bumps [io.grpc:grpc-api](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.83.1...v1.84.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-api dependency-version: 1.84.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build: update grpc api license inventory --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…20257) * build(deps): bump io.grpc:grpc-context from 1.83.1 to 1.84.0 Bumps [io.grpc:grpc-context](https://github.com/grpc/grpc-java) from 1.83.1 to 1.84.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.83.1...v1.84.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-context dependency-version: 1.84.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): align grpc-api with grpc-context --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…0256) * build(deps): bump aws.sdk.v2.version from 2.54.8 to 2.54.10 Bumps `aws.sdk.v2.version` from 2.54.8 to 2.54.10. Updates `software.amazon.awssdk:s3` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:s3-transfer-manager` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:sts` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:apache-client` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:auth` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:regions` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:ec2` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:aws-core` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:sdk-core` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:netty-nio-client` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:aws-crt-client` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:bom` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:retries-spi` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:retries` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:kinesis` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:http-client-spi` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:utils` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:rds` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:glue` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:kms` from 2.54.8 to 2.54.10 Updates `software.amazon.awssdk:bundle` from 2.54.8 to 2.54.10 --- updated-dependencies: - dependency-name: software.amazon.awssdk:apache-client dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:auth dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:aws-core dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:aws-crt-client dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:bom dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:bundle dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:ec2 dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:glue dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:http-client-spi dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:kinesis dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:kms dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:netty-nio-client dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:rds dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:regions dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:retries dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:retries-spi dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:s3 dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:s3-transfer-manager dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:sdk-core dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:sts dependency-version: 2.54.10 dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: software.amazon.awssdk:utils dependency-version: 2.54.10 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * fix: update AWS SDK license registry --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
* build(deps): bump parquet.version from 1.18.0 to 1.18.1 Bumps `parquet.version` from 1.18.0 to 1.18.1. Updates `org.apache.parquet:parquet-column` from 1.18.0 to 1.18.1 - [Release notes](https://github.com/apache/parquet-java/releases) - [Changelog](https://github.com/apache/parquet-java/blob/master/CHANGES.md) - [Commits](apache/parquet-java@apache-parquet-1.18.0...apache-parquet-1.18.1) Updates `org.apache.parquet:parquet-hadoop` from 1.18.0 to 1.18.1 - [Release notes](https://github.com/apache/parquet-java/releases) - [Changelog](https://github.com/apache/parquet-java/blob/master/CHANGES.md) - [Commits](apache/parquet-java@apache-parquet-1.18.0...apache-parquet-1.18.1) Updates `org.apache.parquet:parquet-avro` from 1.18.0 to 1.18.1 - [Release notes](https://github.com/apache/parquet-java/releases) - [Changelog](https://github.com/apache/parquet-java/blob/master/CHANGES.md) - [Commits](apache/parquet-java@apache-parquet-1.18.0...apache-parquet-1.18.1) --- updated-dependencies: - dependency-name: org.apache.parquet:parquet-column dependency-version: 1.18.1 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.apache.parquet:parquet-hadoop dependency-version: 1.18.1 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.apache.parquet:parquet-avro dependency-version: 1.18.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): align Parquet zstd dependency * build: document parquet zstd exclusion --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…0269) Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.14.2 to 3.15.2. - [Changelog](https://github.com/nodeca/js-yaml/blob/3.15.2/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.14.2...3.15.2) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 3.15.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* test: wait for completed Dart reports * test: use shared waiter for Dart reports
…pache#20127) * perf(embedded-tests): reduce auto compaction test polling delay * perf(embedded-tests): reduce MSQ fault tolerance test delay * perf(embedded-tests): reduce auto scaler ingestion wait * perf(processing): cache frame writer expected rows * perf(indexing-service): reduce serial task polling interval * docs(test): explain timing optimizations * perf(test): reduce Kafka and Kinesis test setup time * docs(test): explain Kinesis batch limit * perf(test): reuse merged timestamps in iterator test * docs(test): explain merged timestamp reuse * perf(test): avoid exhausting large granularity iterator * fix(test): preserve granularity interval count assertion * test: make MSQ fault tolerance timing deterministic * fix(test): handle task status delay timeout
* refactor!: remove deprecated SysMonitor * refactor: remove remaining Sigar usage * docs: remove upgrade notes entry * refactor: remove Sigar PID discoverer * fix: preserve JVM CPU baselines on process lookup failure * docs: explain JVM process lookup
* ci: optimize Docker test pipeline * ci: limit Docker build context
…9687) Changes currently in this PR: - UI code is generated by Claude and may contain mistakes - Add a simulate API currently supported for Kafka supervisors only - This API creates a `CostBasedAutoScaler` in "simulate" mode and generates the optimal task count for various input values of lag (based on criticalLag) - Add a UI panel in the supervisor dialog which shows up only for "kafka" supervisors - Add a single plot between task count vs lag
…apache#20289) Bumps com.google.auth:google-auth-library-credentials from 1.51.0 to 1.52.0. --- updated-dependencies: - dependency-name: com.google.auth:google-auth-library-credentials dependency-version: 1.52.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#20288) Bumps [com.squareup.okio:okio-jvm](https://github.com/lysine-dev/okio) from 3.18.1 to 3.18.2. - [Changelog](https://github.com/lysine-dev/okio/blob/main/CHANGELOG.md) - [Commits](lysine-dev/okio@parent-3.18.1...parent-3.18.2) --- updated-dependencies: - dependency-name: com.squareup.okio:okio-jvm dependency-version: 3.18.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…0285) Bumps `bytebuddy.version` from 1.18.12 to 1.18.13. Updates `net.bytebuddy:byte-buddy` from 1.18.12 to 1.18.13 - [Release notes](https://github.com/raphw/byte-buddy/releases) - [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md) - [Commits](raphw/byte-buddy@byte-buddy-1.18.12...byte-buddy-1.18.13) Updates `net.bytebuddy:byte-buddy-agent` from 1.18.12 to 1.18.13 - [Release notes](https://github.com/raphw/byte-buddy/releases) - [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md) - [Commits](raphw/byte-buddy@byte-buddy-1.18.12...byte-buddy-1.18.13) --- updated-dependencies: - dependency-name: net.bytebuddy:byte-buddy dependency-version: 1.18.13 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: net.bytebuddy:byte-buddy-agent dependency-version: 1.18.13 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…20283) Bumps [net.thisptr:jackson-jq](https://github.com/eiiches/jackson-jq) from 1.6.3 to 1.6.4. - [Release notes](https://github.com/eiiches/jackson-jq/releases) - [Commits](eiiches/jackson-jq@1.6.3...1.6.4) --- updated-dependencies: - dependency-name: net.thisptr:jackson-jq dependency-version: 1.6.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…21 (apache#20282) Bumps [org.roaringbitmap:RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap) from 1.6.20 to 1.6.21. - [Release notes](https://github.com/RoaringBitmap/RoaringBitmap/releases) - [Commits](https://github.com/RoaringBitmap/RoaringBitmap/commits) --- updated-dependencies: - dependency-name: org.roaringbitmap:RoaringBitmap dependency-version: 1.6.21 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#20281) Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/google-cloud-java) from 2.75.0 to 2.76.0. - [Release notes](https://github.com/googleapis/google-cloud-java/releases) - [Changelog](https://github.com/googleapis/google-cloud-java/blob/main/java-document-ai/CHANGELOG.md) - [Commits](googleapis/google-cloud-java@gapic-generator-java/v2.75.0...gapic-generator-java/v2.76.0) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-version: 2.76.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `slf4j.version` from 2.0.18 to 2.0.19. Updates `org.slf4j:jcl-over-slf4j` from 2.0.18 to 2.0.19 Updates `org.slf4j:slf4j-api` from 2.0.18 to 2.0.19 Updates `org.slf4j:slf4j-simple` from 2.0.18 to 2.0.19 --- updated-dependencies: - dependency-name: org.slf4j:jcl-over-slf4j dependency-version: 2.0.19 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.slf4j:slf4j-api dependency-version: 2.0.19 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.slf4j:slf4j-simple dependency-version: 2.0.19 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Description
If a segment becomes unavailable after it has been added to the load queue of a historical, it might remain stuck in the queue while other segments are being loaded. Ideally, all unavailable segments should be prioritized.
Changes
Simplify book-keeping in ServerHolder by getting rid of method simplify() and treat REPLICATE and LOAD actions as distinct
In StrategicSegmentAssigner.updateReplicasInTier(), check if an unavailable segment needs to be prioritized and change all in-flight REPLICATE actions on that segment to LOAD.
Update LoadQueuePeon.getSegmentsInQueue to return a List instead of a Set and verify the result in Coordinator simulations
Main changes: 1) Sync of Parser.jj (using dev/upgrade-calcite-parser) and CalcitePlanner. 2) Add DruidTypeFactory to work around undesirable behavior introduced by CALCITE-7358. 3) Add validation to prevent aliases in CLUSTERED BY, which have been added to the parser since CALCITE-5347. (The change is done at the validator instead of the parser to minimize parser diffs.) 4) Adjust test expectations for slightly different plans.
…che#20306) PartialCompactionTest always builds its tuning config with maxNumConcurrentSubTasks = 2, so it falls back to the 1000ms production default for taskStatusCheckPeriodMs. Each of its three back-to-back indexing/compaction rounds pays up to a full poll period per phase transition, which dominates the test's wall-clock time. Override newTuningConfig in this class to poll every 100ms. The override mirrors the base builder exactly except for the poll interval. None of the class's assertions depend on poll cadence, and the injected transient task/API failures are retried by count, not time, so they behave the same. CI (JDK 25): 42.00s -> 7.31s, 35.54s -> 5.71s, 24.13s -> 3.85s.
Bumps [svgo](https://github.com/svg/svgo) from 3.3.4 to 3.3.5. - [Release notes](https://github.com/svg/svgo/releases) - [Commits](svg/svgo@v3.3.4...v3.3.5) --- updated-dependencies: - dependency-name: svgo dependency-version: 3.3.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [joi](https://github.com/hapijs/joi) from 17.13.3 to 17.13.7. - [Commits](hapijs/joi@v17.13.3...v17.13.7) --- updated-dependencies: - dependency-name: joi dependency-version: 17.13.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…he#20302) Bumps [com.github.spotbugs:spotbugs-maven-plugin](https://github.com/spotbugs/spotbugs-maven-plugin) from 4.10.4.0 to 4.10.4.1. - [Release notes](https://github.com/spotbugs/spotbugs-maven-plugin/releases) - [Commits](spotbugs/spotbugs-maven-plugin@spotbugs-maven-plugin-4.10.4.0...spotbugs-maven-plugin-4.10.4.1) --- updated-dependencies: - dependency-name: com.github.spotbugs:spotbugs-maven-plugin dependency-version: 4.10.4.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ache#20541) Addresses four problems: 1) Provide PhysicalSegmentColumnInspector from PartialQueryableIndexSegment. Previously it was not provided, so segmentMetadata queries were not able to report types. 2) Look at all cluster groups when determining capabilities, not just the first one. (They can differ in the details, like auto-determined type and hasMultipleValues.) 3) Use the DimensionIndexer's ColumnFormat when writing columns in cluster groups and projections. Previously the ColumnFormat present at creation time was used, which could differ in details (such as hasMultipleValues). This caused multi-valued columns to be incorrectly persisted, in situations where the persist logic did not realize they were multi-value. 4) Updates Capable.or and Capable.and to use correct 3-valued logic. This made certain new code simpler to write, and as far as I can tell, will only have positive effects on existing callers.
CodeQL Action v3 runs on Node.js 20 and is deprecated in December 2026. v4 runs on Node.js 24.
* Enable Dependabot version updates for GitHub Actions * Exempt codeql-action from the GitHub Actions cooldown * Trim cooldown comments * Use a 7-day cooldown for GitHub Actions instead of exempting codeql-action
…he#20450) * perf: cache incremental-index expression plans * fix: avoid sharing mutable expression state * fix: return fresh expressions from plan cache * test: use daemon executor for expression cache concurrency * docs: explain expression plan cache array * perf: reuse expression selectors during ingestion * test: cover expression selector reuse with arrays * test: clarify expression ingestion expectation * perf: add expression ingestion benchmark * fix: reevaluate non-deterministic ingestion expressions
* refactor: migrate query context reference parameters * fix: address review findings on query context descriptors - Merge the user context over default contexts with QueryContexts.withDefaults, so a null user value falls back to the configured default on both the SQL and native paths. - Declare maxSubqueryBytes as a string parameter that accepts the forms the subquery guardrail accepts. - Deprecate the key constants replaced by descriptors for removal, and migrate their remaining uses. - Escape default descriptions and ignore CRLF line endings in the generated documentation check. * fix: address second review of query context descriptors
…ask groups are complete (apache#20505) The changes in apache#19091 have shown that tasks often need to wait for their priors to finish publishing as we often see logs like: ``` Task[xyz] needs to wait before publishing as other taskGroups[abc, def] are currently publishing to partition[123]. ``` Before this patch, if publish from streaming task A is slow, a later generation task B keeps trying to publish repeatedly until eventually failing once the number of retries is exhausted. It is difficult to fix the number of retries since publish times may vary across supervisors and over time. It is also pointless to retry if a prior task is yet to finish publishing. This patch addresses these problems by blocking the publish of later task B until A is complete. Changes --------- - Add `TaskActionClient.submitAsync` which returns a `ListenableFuture` - Update API `/druid/indexer/v1/action` in `OverlordResource` to be async by employing the new `submitAsync()` method - Update task actions `segmentTransactionalAppend` and `segmentTransactionalReplace` to implement `performAsync()` - Update `LocalTaskActionClient.submitAsync()` to use `performAsync()` when supported - Attempt segment publish only when the task is ready to publish - Complete async API request only when the publish attempt is done - For non-streaming tasks, attempt to publish immediately - Add method `isTaskReadyToPublish()` in `SeekableStreamSupervisor`. - Return a `ListenableFuture` that completes when all prior task groups have completed - Update `SegmentAllocateAction` and `SegmentAllocationQueue` to use `ListenableFuture`
Bumps the github-actions group with 4 updates: [actions/checkout](https://github.com/actions/checkout), [actions/setup-java](https://github.com/actions/setup-java), [actions/labeler](https://github.com/actions/labeler) and [actions/stale](https://github.com/actions/stale). Updates `actions/checkout` from 6 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) Updates `actions/setup-java` from 5 to 6 - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@v5...v6) Updates `actions/labeler` from 6 to 7 - [Release notes](https://github.com/actions/labeler/releases) - [Commits](actions/labeler@v6...v7) Updates `actions/stale` from 10 to 11 - [Release notes](https://github.com/actions/stale/releases) - [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md) - [Commits](actions/stale@v10...v11) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-java dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/labeler dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/stale dependency-version: '11' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#20588) ScanQueryFrameProcessor currently includes the original offset and limit when sending queries to data servers. It should instead send an offset of zero and add the offset to the limit.
* build(deps): bump com.puppycrawl.tools:checkstyle from 12.3.1 to 14.3.0 Bumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle) from 12.3.1 to 14.3.0. - [Release notes](https://github.com/checkstyle/checkstyle/releases) - [Commits](checkstyle/checkstyle@checkstyle-12.3.1...checkstyle-14.3.0) --- updated-dependencies: - dependency-name: com.puppycrawl.tools:checkstyle dependency-version: 14.3.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Fix Javadoc that Checkstyle 14 cannot parse Checkstyle 13 and later use a stricter Javadoc parser, and the UnusedImports check reports every comment it cannot parse (and treats imports referenced only from such comments as unused). Fix the malformed {@link}, @see and @throws syntax, unbalanced {@code} braces, stray or mistyped HTML tags, and unescaped generics and line-leading annotations in the affected comments. * Point GuavaUtils.firstNonNull Javadoc link at MoreObjects Guava's Objects#firstNonNull no longer exists, and javadoc cannot resolve the (T, T) parameter list. Link to MoreObjects#firstNonNull(Object, Object). --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* close * doc * test * test
This patch adds the ability for SQL UNION to run over relations that
involve projections and filters of base tables, such as:
SELECT dim1, SUM(m1), COUNT(*)
FROM (SELECT dim1, m1 FROM foo2 WHERE dim1 = '1'
UNION ALL
SELECT dim1, m1 FROM foo)
GROUP BY 1;
Such queries are planned into stacks of the existing "filter" datasource
and the new "project" datasource.
Other changes, outside the SQL planner:
1) DataSource method "createSegmentPruner" has DimFilter and
VirtualColumns added as parameters. This enables DataSource
implementations, such as FilteredDataSource and ProjectDataSource,
to rewrite the filter and contribute additional virtual columns to
the pruning logic. ExecutionVertex#isBaseColumn is removed and more
scenarios are now pruneable.
2) DataSource gets "getReservedColumns" with a set of column names to
avoid when generating higher-level virtual columns. This is used
by the SQL planner to avoid name collisions.
3) DataSource gets "mayFilterRows", which allows time-boundary queries
to properly recognize that filtering DataSources should go down the
"query has filter" path.
4) Fix an issue where various CursorHolder implementations return
lists from "getOrdering" that erroneously include base columns
which were shadowed by virtual columns.
5) Updates to MSQ to call the new ExecutionVertex#splitUnion and
splitUnionSegmentMapFunctions methods to provide each arm with its
own SegmentPruner and SegmentMapFunction.
…#20523) fetchAllUsedSegmentsWithOvershadowedStatus hands out a ResourceHolder that closes the streamed response; MetadataSegmentView and SystemSchema close it, also on early exit, and QueryHandler now closes the bound Interpreter so scans under Filter/Project nodes are released too. No segment left behind.
List of suppressed CVEs: - elasticache-java-cluster-client: client lib, not memcached server (CPE false positive) - dev.cel:protobuf: CPE collision with google protobuf-java (CVE-2022-3171) - async-http-client: no auto-redirect/cookie store in Druid's usage (CVE-2026-45300) - netty-transport 4.1.100: hostname verification enabled in ChannelResourceFactory (CVE-2023-4586) - npm uuid: only uuidv4() used, no buf argument (GHSA-w5hq-g745-h8pq) - kotlin-stdlib: build-toolchain only, no Kotlin source in Druid (CVE-2026-53914) - azure-core/identity/json: Key Vault crypto path unused, blob storage auth only (CVE-2026-33117) - hive-storage-api: used only for Murmur3 hash, no Hive Metastore/HiveServer2 (CVE-2026-49845, -53561, -55976) - jackson-databind shaded in hadoop-client-runtime/parquet-jackson (CVE-2026-54512, -54513, -68497) - jetty shaded in hadoop-client-runtime, used for Hadoop's internal HTTP only (CVE-2026-2332, -10050) - jline shaded in hadoop-client-runtime, Hadoop CLI only (CVE-2026-56741, -56740) - opentelemetry-gcp-resources: CPE collision with OTel Go SDK (CVE-2026-24051, -39883, -29181) - httpclient5/httpcore5 shaded in docker-java-transport-zerodep: test-scope only (CVE-2026-40542, -71290, -54399, -54428) - zstd-jni: vulnerable paths not activated, pending upgrade to 1.5.7-14 (CVE-2026-87823, -87795) - snappy-java: Druid uses Commons Compress, not xerial ByteBuffer API (CVE-2026-90559) - parquet-column/parquet-common: envelope encryption/crypto.keytools unused (CVE-2026-73334) - zookeeper/zookeeper-jute: used as client only, no server/quorum (CVE-2026-79993, -59739, -59969)
…#20313) Changes: - Track in-flight requests in `DartWorkerClientImpl` - Cancel in-flight requests when the client is closed
Bumps com.google.auth:google-auth-library-credentials from 1.53.0 to 1.54.0. --- updated-dependencies: - dependency-name: com.google.auth:google-auth-library-credentials dependency-version: 1.54.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…apache#20585) Bumps [org.codehaus.woodstox:stax2-api](https://github.com/FasterXML/stax2-api) from 4.3.0 to 4.3.1. - [Commits](FasterXML/stax2-api@stax2-api-4.3.0...stax2-api-4.3.1) --- updated-dependencies: - dependency-name: org.codehaus.woodstox:stax2-api dependency-version: 4.3.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump oshi.version from 7.6.1 to 7.7.0 Bumps `oshi.version` from 7.6.1 to 7.7.0. Updates `com.github.oshi:oshi-core` from 7.6.1 to 7.7.0 - [Release notes](https://github.com/oshi/oshi/releases) - [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md) - [Commits](oshi/oshi@oshi-parent-7.6.1...oshi-parent-7.7.0) Updates `com.github.oshi:oshi-common` from 7.6.1 to 7.7.0 - [Release notes](https://github.com/oshi/oshi/releases) - [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md) - [Commits](oshi/oshi@oshi-parent-7.6.1...oshi-parent-7.7.0) --- updated-dependencies: - dependency-name: com.github.oshi:oshi-core dependency-version: 7.7.0 dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: com.github.oshi:oshi-common dependency-version: 7.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): update OSHI license metadata --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
….0 (apache#20583) * build(deps): bump org.apache.maven:maven-artifact from 3.9.16 to 3.10.0 Bumps org.apache.maven:maven-artifact from 3.9.16 to 3.10.0. --- updated-dependencies: - dependency-name: org.apache.maven:maven-artifact dependency-version: 3.10.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): update Maven Artifact license metadata --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…7.3.0 (apache#20582) * build(deps): bump com.fasterxml.woodstox:woodstox-core Bumps [com.fasterxml.woodstox:woodstox-core](https://github.com/FasterXML/woodstox) from 7.2.2 to 7.3.0. - [Commits](FasterXML/woodstox@woodstox-core-7.2.2...woodstox-core-7.3.0) --- updated-dependencies: - dependency-name: com.fasterxml.woodstox:woodstox-core dependency-version: 7.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): align stax2-api and license metadata --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…#20581) * build(deps): bump guava.version from 33.7.1-jre to 33.7.2-jre Bumps `guava.version` from 33.7.1-jre to 33.7.2-jre. Updates `com.google.guava:guava` from 33.7.1-jre to 33.7.2-jre - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) Updates `com.google.guava:guava-testlib` from 33.7.1-jre to 33.7.2-jre - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-version: 33.7.2-jre dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: com.google.guava:guava-testlib dependency-version: 33.7.2-jre dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): update Guava license metadata --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
…eam-master Conflict resolution: - AGENTS.md: keep the fork-only "Fork Policy" and "Agent skills" sections on top of upstream's content. - KafkaSupervisor.updatePartitionLagFromStream(): upstream (apache#18750) now builds a local offset map and publishes it via updateOffsetSnapshot(). Keep that structure, but re-apply ADR 0001 by fetching latest offsets with the batch getLatestSequenceNumbers() (endOffsets) lookup instead of seekToLatest() + getPosition(). - KafkaRecordSupplierTest: upstream migrated the class to JUnit 5 and added a metric builder supplier to the consumer constructor. Port the fork's ADR 0001 tests to Assertions and the 3-arg constructor. KafkaRecordSupplier merged cleanly; the ADR 0001 retrying beginningOffsets()/endOffsets() lookups are preserved. Upstream has not implemented ADR 0001's behaviour, so it remains a fork patch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Merge
upstream/master(apache/druid@92fe6f5f71, 1064 commits) into the fork'smaster.Important
Merge this PR with Create a merge commit. Squash or rebase would flatten upstream's history and break future syncs.
ADR review
KafkaRecordSupplier.getEarliestSequenceNumber()/getLatestSequenceNumber()still seek and callposition()with no retry. The fork patch stays and the ADR is unchanged. No new upstream supervisor code path bypasses the patched lookups.Conflict resolution
AGENTS.md: upstream added its own rootAGENTS.md. Kept the fork-only "Fork Policy" and "Agent skills" sections on top of upstream's content.KafkaSupervisor.updatePartitionLagFromStream(): upstream (Fix negative Kafka partition lag caused by inconsistent current/latest offsets apache/druid#18750) now builds a local offset map and publishes it viaupdateOffsetSnapshot(). Took upstream's structure and re-applied ADR 0001 by fetching latest offsets with the batchgetLatestSequenceNumbers()(endOffsets()) lookup, notseekToLatest()+getPosition().KafkaRecordSupplierTest: upstream migrated the class to JUnit 5 and added ametricBuilderSupplierargument to the consumer constructor. Ported the fork's four ADR 0001 tests toAssertionsandnew KafkaRecordSupplier(consumer, false, null).KafkaRecordSuppliermerged cleanly, and the retryingbeginningOffsets()/endOffsets()lookups are intact. After the merge,git diff upstream/master HEADcontains only the fork's own changes (ADR, agent docs, ticket, and the Kafka patch).Testing
mvn test -pl extensions-core/kafka-indexing-service -am -Dtest=KafkaRecordSupplierTest,KafkaSupervisorTest: 185 tests, 0 failures, 1 flake. The flake isKafkaSupervisorTest.testBadMetadataOffsets, which passed on rerun. On the failing attempt, a supervisor leaked from an earlier test (TaskRunner.getTaskLocation("id1")) emitted an alert first. That's an upstream test-isolation race, unrelated to this patch.druid-kafka-indexing-service(and its-amdependencies): clean.journaldlog driver, sodocker logsreturns nothing and Testcontainers times out. To run the tests locally, I temporarily forced the broker container to thejson-filelog driver and disabled Ryuk. That tweak is not part of this PR.Release note
No end-user impact beyond upstream's own changes.
Key changed/added classes in this PR
KafkaSupervisorKafkaRecordSupplierTestAGENTS.mdThis PR has:
🤖 Generated with Claude Code