FEAT: Add client_rack field to kafka task

[Shivang Nagta (ShivangNagta)]

Description

Adds an optional client_rack field to the kafka task that sets librdkafka's
client.rack on the consumer. This lets a consumer take advantage of Kafka's
rack-aware features like fetch-from-follower (KIP-392) and rack-aware partition
assignment (KIP-881), so it reads from a replica in its own availability zone
instead of the (possibly cross-AZ) leader, reducing inter-AZ data transfer cost.

The value is passed through to both the group consumer and standalone consumer
configs, and is omitted entirely when unset.

Note: this is the client-side requirement only. The benefit is realized when the
brokers are configured with broker.rack (and, for fetch-from-follower, a
rack-aware replica.selector.class such as RackAwareReplicaSelector).

Test

My tests were performed locally with 3 brokers (podman containers), with a single partition in a topic called t392 (replication set to 3, leader on broker-1)

Before

Without client_rack (consumer fetches from the leader broker (broker - 1))
Example yaml used:

  - name: kafka_read
    type: kafka
    bootstrap_server: localhost:19092
    topic: t392
    user_auth_type: none
    timeout: 2s
    end_after: 8s
  - name: echo_results
    type: echo
    only_data: true

After

With client_rack set to rack-c(same as broker -3) and replica.selector.class set to RackAwareReplicaSelector for brokers (consumer sends the initial request to leader broker (broker - 1), subsequent fetches migrate to broker - 3)
Example yaml used:

  - name: kafka_read
    type: kafka
    bootstrap_server: localhost:19092
    topic: t392
    user_auth_type: none
    timeout: 2s
    end_after: 8s
    client_rack: rack-c
  - name: echo_results
    type: echo
    only_data: true

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.

[Copilot]

Pull request overview

Adds support for configuring Kafka rack-awareness for consumers by introducing a client_rack task field that maps through to librdkafka’s client.rack, enabling rack-aware partition assignment and follower fetching when broker-side rack settings are present.

Changes:

• Add optional client_rack field to the Kafka task configuration struct.
• Propagate client_rack into both group and standalone consumer librdkafka config maps when set.
• Document the new field and provide an example configuration for rack-aware consumption.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
internal/pkg/pipeline/task/kafka/README.md	Documents client_rack and adds a rack-aware consumer example.
internal/pkg/pipeline/task/kafka/kafka.go	Adds ClientRack config field and sets client.rack for consumer configs when non-empty.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.