Add auto-detect for RoCE v2 GID for mlx5 NICs in MlxDevice#4304
Open
mkuchnik wants to merge 2 commits into
Open
Add auto-detect for RoCE v2 GID for mlx5 NICs in MlxDevice#4304mkuchnik wants to merge 2 commits into
mkuchnik wants to merge 2 commits into
Conversation
rdma-core's default gid_index (3) is not the routable GID on RoCE v2 fabrics,
and mlx5 does no auto-selection, so QPs come up on an unroutable GID.
This diff adds autoselection logic for MlxDevice:
1. MONARCH_IB_GID_INDEX, if set (as an override);
2. else auto-detect the first routable RoCE v2 GID from sysfs
(gid_attrs/types/<i> == "RoCE v2" and gids/<i> not link-local / all-zero);
3. else fall back to gid_index 3 (prior InfiniBand-mode NICs behavior).
The result is cached process-wide.
Tested on a GB300 node with ConnectX-8 SuperNIC, where the routable GID is index 7.
meta-cla
Bot
added
the
CLA Signed
samlurye
requested changes
Jun 26, 2026
|
|
||
| /// `MONARCH_IB_GID_INDEX` parsed as a `u8`, if set and valid. | ||
| fn env_gid_index() -> Option<u8> { | ||
| std::env::var("MONARCH_IB_GID_INDEX") |
Contributor
There was a problem hiding this comment.
The canonical way to define an env var option in monarch is using a hyperactor config attribute; the options for RDMA are defined here. Please call it RDMA_IBV_DEFAULT_GID_INDEX, and incorporate it into the code base by setting the value in IbvConfig's default implementation.
Contributor
Author
There was a problem hiding this comment.
Take a look. The downside of spreading autoconfig between 1) env 2) autoconfig 3) default is there is an unclear ordering of priority. For example, autoconfig is applied after env and default, so the priority order seems 2, 1, 3, which may be unexpected.
mkuchnik
force-pushed
the
mkuchnik/mlx-roce-v2-gid-autodetect
branch
from
84bed5c to
8119cdf
Compare
Addresses review of the auto-detect commit. Moves the GID-selection logic out of the standalone roce_gid module and onto primitives::Gid. The default is RDMA_IBV_DEFAULT_GID_INDEX (env MONARCH_RDMA_IBV_DEFAULT_GID_INDEX). Tested on a GB300 node with ConnectX-8 SuperNIC, where the routable GID is index 7.
mkuchnik
force-pushed
the
mkuchnik/mlx-roce-v2-gid-autodetect
branch
from
8119cdf to
47ceac0
Compare
samlurye
requested changes
Jun 30, 2026
| Some("MONARCH_RDMA_IBV_DEFAULT_GID_INDEX".to_string()), | ||
| Some("rdma_ibv_default_gid_index".to_string()), | ||
| )) | ||
| pub attr RDMA_IBV_DEFAULT_GID_INDEX: Option<u8> = None; |
Contributor
There was a problem hiding this comment.
Let's make this non-optional with a default of 0. The comment shouldn't mention mlx5 or any specific hardware.
| /// | ||
| /// Reads the kernel's GID table from sysfs | ||
| /// (`/sys/class/infiniband/<dev>/ports/<p>/gid_attrs/types/<i>` and | ||
| /// `gids/<i>`) rather than `ibv_query_gid*`: on some mlx5 / RoCE stacks the |
Contributor
There was a problem hiding this comment.
A few things:
- Comments in non-hardware-specific files shouldn't be mentioning specific hardware that uses it.
- Are you actually in practice running into cases where
ibv_query_gidreturns a bad value? - If
device_infodoesn't contain the right thing, then we should updateIbvDeviceInfoso that it does contain the right thing. Instead of having eachIbvPortcontain only a single gid, let's give it the full list of gids, read from sysfs (filtering out non-roce-v2 gids), and fill it in insidequery_device_info. ThenGid::index_ofcan just read fromIbvDeviceInfowithout having to go through sysfs or any additionalibv_query_gidcalls. - Please tell the LLM to be more concise in its comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rdma-core's default gid_index (3) is not the routable GID on RoCE v2 fabrics, and mlx5 does no auto-selection, so QPs come up on an unroutable GID. This diff adds autoselection logic for MlxDevice:
The result is cached process-wide.
Tested on a GB300 node with ConnectX-8 SuperNIC, where the routable GID is index 7.