Skip to content

enhancements/release: Go toolchain bump process for OCP release branches - #2077

Open
Prashanth684 wants to merge 1 commit into
openshift:masterfrom
Prashanth684:ep-go-bump
Open

Prashanth684 wants to merge 1 commit into
openshift:masterfrom
Prashanth684:ep-go-bump

Conversation

@Prashanth684

@Prashanth684 Prashanth684 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary:

  • Defines a repeatable 6-phase process for bumping the Go toolchain on OCP release branches
  • Follows upstream Kubernetes practice (KEP-3744): bump toolchain, keep go.mod go directive unchanged, rely on GODEBUG for runtime compatibility
  • Includes a Go 1.26 appendix with payload scan of 160 repos on release-4.22

Motivation:

  • Go goes EOL before OpenShift does.Update the Golang toolchain used to build and run OpenShift during the lifecycle of a given release. This greatly reduces the number of fixable CVEs since we can now fix them more easily by updating dependencies quicker instead of waiting for forks. Upstream Kube already does this.

What's in this PR:

  • enhancements/release/go-version-bump.md — version-agnostic process doc
  • enhancements/release/go-version-bump-go126.md — Go 1.25→1.26 appendix for OCP 4.22

Summary by CodeRabbit

  • Documentation
    • Added a repeatable process for upgrading Go versions across OpenShift release branches.
    • Documented compatibility considerations, FIPS requirements, validation, rollout monitoring, remediation, and version-skew handling.
    • Added a Go 1.25-to-1.26 assessment for OCP 4.22, including protobuf, GODEBUG, GOEXPERIMENT, and payload considerations.
    • Included guidance for repository-specific implementation and criteria for completing future Go upgrades.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5975f940-5a9a-48c3-b272-80afd25153da

📥 Commits

Reviewing files that changed from the base of the PR and between b6b3b86 and 226cead.

📒 Files selected for processing (1)
  • enhancements/release/go-version-bump.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • enhancements/release/go-version-bump.md

Walkthrough

The changes define a repeatable Go version bump process and add a Go 1.26 assessment for OCP 4.22. They document compatibility behavior, repository updates, FIPS review, validation, rollout, remediation, and implementation status.

Changes

Go version bump process and assessment

Layer / File(s) Summary
Version bump process and compatibility rules
enhancements/release/go-version-bump.md
Defines lifecycle alignment, GODEBUG behavior, scope, repository constraints, roles, risks, remediation, and version-skew handling.
Assessment, rollout, and remediation workflow
enhancements/release/go-version-bump.md
Defines dependency triage, builder preparation, staged rollout, CI and FIPS validation, remediation, and graduation criteria.
OCP 4.22 Go 1.26 assessment
enhancements/release/go-version-bump-go126.md, hack/lint-ignores
Documents repository requirements, protobuf fixes, Go 1.26 behavior, FIPS considerations, rollout checks, Go 1.27 lookahead, implementation status, and lint handling.

Estimated code review effort: 2 (Simple) | ~10 minutes

Mergeability Score: ⚪ Minimal · up to 226ce

This documentation-only change has no actionable merge-blocking risk beyond normal checks and review.

Suggested reviewers: cooktheryan, mrunalp

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: documenting a Go toolchain bump process for OpenShift release branches.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only two Markdown documents and lint configuration; no Ginkgo test declarations or dynamic test titles were added.
Test Structure And Quality ✅ Passed The usable PR diff changes only two lines in a Markdown document; it adds no Ginkgo tests or cluster operations subject to this check.
Microshift Test Compatibility ✅ Passed The PR adds only two enhancement documents and one lint-ignore entry; the diff adds no Ginkgo e2e tests or test declarations requiring MicroShift compatibility review.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR adds only two Markdown documents and one lint-ignore entry; the diff contains no new Ginkgo declarations or e2e test code, so SNO compatibility is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The PR changes only two Markdown documents and hack/lint-ignores; no manifests, operators, controllers, or scheduling constraints are added or modified.
Ote Binary Stdout Contract ✅ Passed The PR diff adds only Markdown documentation and a lint-ignore entry; it changes no Go or OTE process-level code and introduces no stdout writes.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR adds only two Markdown documents and a lint-ignore entry; the diff adds no Ginkgo tests or executable networking code. External URLs are documentation links.
No-Weak-Crypto ✅ Passed The diff adds only Markdown and a lint-ignore entry. SHA-1 appears only as a historical compatibility note; no weak-crypto implementation or secret comparison was added.
Container-Privileges ✅ Passed The PR changes only Markdown and lint-ignore files; no container/Kubernetes manifests or explicit privilege settings were added.
No-Sensitive-Data-In-Logs ✅ Passed The full PR diff adds only Markdown and a lint-ignore entry; searches found no logging constructs or sensitive values or collection instructions.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from cooktheryan and mrunalp August 12, 2026 21:53
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign pavolloffay for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🧹 Nitpick comments (2)
enhancements/release/go-version-bump-go126.md (2)

41-49: 🗄️ Data Integrity & Integration | 🔵 Trivial

Make the payload scan reproducible.

The appendix gives repository counts but no payload digest, repository commit SHAs, scan command, or machine-readable repository list. Add those inputs, including branch and module-path data. Later rollout decisions depend on the 160-repository inventory.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/release/go-version-bump-go126.md` around lines 41 - 49, Make the
repository inventory in the appendix reproducible by adding the payload digest,
scan command, repository commit SHAs, and a machine-readable list of all 160
repositories. Include each repository’s scanned branch and module path, while
preserving the existing category counts and release-4.22 scope.

66-74: 🚀 Performance & Scalability | 🔵 Trivial

Record the binary-size measurement conditions.

The table does not specify architecture, linker flags, stripping, compression, or the source artifacts. Add the exact build command and artifact references so the reported percentages can be reproduced and used as a validation gate.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/release/go-version-bump-go126.md` around lines 66 - 74, Document
the binary-size measurement conditions alongside the table: specify the
architecture, linker and stripping flags, compression state, and source
artifacts or revisions used for both Go versions. Add the exact reproducible
build command and artifact references so the reported sizes and percentages can
serve as a validation gate.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@enhancements/release/go-version-bump-go126.md`:
- Around line 56-58: Update the repository-count statements in the Go version
bump document to consistently report six repositories with .go-version, matching
the six entries in the inventory and the count referenced later in the document.
- Around line 146-150: Update the Phase 1 status text in the
implementation-status section to distinguish the scopes: state that 143
repositories require the protobuf fix, while all 151 Go repositories require the
builder-image/toolchain update.
- Around line 102-110: Update the “GODEBUG Expiration Lookahead (Go 1.27)”
section to state that Go 1.27-or-later binaries remove tlsrsakex and tls10server
regardless of the go directive. Add a rollout gate covering configurations that
use legacy RSA key exchange or --tls-min-version=VersionTLS10, and retain the
existing OCP exposure references.
- Around line 29-37: Update the protobuf pin instructions to account for each
consuming module’s go directive: use the specified revision only for modules
declaring go 1.23 or newer, and select a compatible protobuf revision for
modules declaring go 1.22 or lower while preserving their existing directive, or
explicitly document the required directive bump.

In `@enhancements/release/go-version-bump.md`:
- Around line 198-201: Update the appendix reference in the version-specific
analysis paragraph to link to go-version-bump-go126.md, matching the provided
filename while leaving the link text unchanged.
- Around line 379-386: Update the GODEBUG discussion and the
per-repository/version-skew sections in the release documentation to limit the
guarantee to wire compatibility, not identical runtime or build-time behavior.
State that compatibility is validated for active GOEXPERIMENT defaults and
unconditional standard-library changes, while retaining the existing explanation
that the unchanged go directive preserves GODEBUG-gated defaults.
- Around line 285-295: Document the native Go FIPS settings before attributing
FIPS behavior or module-version changes: record GOFIPS140, which selects the
linked module and default mode, and GODEBUG=fips140=on|off|only, which controls
runtime mode. In enhancements/release/go-version-bump.md lines 285-295, add this
conditional process to the FIPS review. In
enhancements/release/go-version-bump-go126.md lines 96-100, replace the
unconditional toolchain statement with the same settings-aware guidance.

---

Nitpick comments:
In `@enhancements/release/go-version-bump-go126.md`:
- Around line 41-49: Make the repository inventory in the appendix reproducible
by adding the payload digest, scan command, repository commit SHAs, and a
machine-readable list of all 160 repositories. Include each repository’s scanned
branch and module path, while preserving the existing category counts and
release-4.22 scope.
- Around line 66-74: Document the binary-size measurement conditions alongside
the table: specify the architecture, linker and stripping flags, compression
state, and source artifacts or revisions used for both Go versions. Add the
exact reproducible build command and artifact references so the reported sizes
and percentages can serve as a validation gate.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6a6336fa-333e-4f70-b0bc-7af94bc13ee7

📥 Commits

Reviewing files that changed from the base of the PR and between bff68b1 and 76e4721.

📒 Files selected for processing (2)
  • enhancements/release/go-version-bump-go126.md
  • enhancements/release/go-version-bump.md

Comment thread enhancements/release/go-version-bump-go126.md
Comment thread enhancements/release/go-version-bump-go126.md
Comment on lines +102 to +110
## GODEBUG Expiration Lookahead (Go 1.27)

Settings removed in Go 1.27 — will become active when the `go` directive
is eventually bumped past 1.27:

| Setting | OCP exposure |
|---------|--------------|
| `tlsrsakex` | RSA key exchange ciphers. Configurable via `--tls-cipher-suites`. |
| `tls10server` | TLS 1.0 server support. Affects clusters using `--tls-min-version=VersionTLS10`. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n --glob '*.go' --glob '*.yaml' --glob '*.yml' \
  'tlsrsakex|tls10server|tls-min-version|tls-cipher-suites' .

Repository: openshift/enhancements

Length of output: 160


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

file="enhancements/release/go-version-bump-go126.md"
sed -n '85,125p' "$file"
printf '\nRelated release-document references:\n'
rg -n -i --glob '*.md' 'GODEBUG|tlsrsakex|tls10server|Go 1\.27|go directive' enhancements/release enhancements | head -200

Repository: openshift/enhancements

Length of output: 12041


🌐 Web query:

site:go.dev/doc/go1.26 tlsrsakex tls10server Go 1.27 removed GODEBUG

💡 Result:

In Go 1.27, several legacy TLS-related GODEBUG settings that were introduced in Go 1.22 and 1.23 have been removed [1]. As a result, the behaviors controlled by these settings are now enforced permanently and cannot be reverted using GODEBUG [1]. Specifically for the settings you mentioned: - tlsrsakex: Legacy RSA-only key exchanges without ECDH are no longer enabled by default [1]. - tls10server: The default minimum TLS version for both clients and servers is now TLS 1.2, meaning support for TLS 1.0/1.1 is no longer enabled by default [1]. Starting with Go 1.27, these behaviors apply regardless of any GODEBUG setting or go.mod language version specified in your project [1].

Citations:


Correct the Go 1.27 GODEBUG expiration behavior.

Go 1.27 removes tlsrsakex and tls10server regardless of the go directive. Update the wording to state that Go 1.27-or-later binaries enforce the new behavior. Add a rollout gate for configurations using legacy RSA key exchange or --tls-min-version=VersionTLS10.

🧰 Tools
🪛 LanguageTool

[grammar] ~110-~110: Ensure spelling is correct
Context: ... server support. Affects clusters using --tls-min-version=VersionTLS10. | ## What to Watch During Rollout - ...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/release/go-version-bump-go126.md` around lines 102 - 110, Update
the “GODEBUG Expiration Lookahead (Go 1.27)” section to state that Go
1.27-or-later binaries remove tlsrsakex and tls10server regardless of the go
directive. Add a rollout gate covering configurations that use legacy RSA key
exchange or --tls-min-version=VersionTLS10, and retain the existing OCP exposure
references.

Source: MCP tools

Comment thread enhancements/release/go-version-bump-go126.md Outdated
Comment thread enhancements/release/go-version-bump.md
Comment thread enhancements/release/go-version-bump.md

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
enhancements/release/go-version-bump.md (1)

442-445: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Add focused TLS validation or narrow the claim.

The document says TLS behavior is validated implicitly and provides no dedicated cipher-suite tests. Go 1.26 changes default TLS behavior, including enabling new post-quantum key exchanges. Add deterministic handshake tests for the configured minimum version, cipher suites, signature algorithms, and FIPS profiles, or state that the current e2e coverage is only smoke coverage. (go.dev)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/release/go-version-bump.md` around lines 442 - 445, Update the
TLS validation section to either add deterministic handshake coverage for
configured minimum versions, cipher suites, signature algorithms, and FIPS
profiles, or narrow the existing claim to describe inter-component e2e tests as
smoke coverage only. Keep the statement aligned with the actual validation
implemented.

Source: MCP tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@enhancements/release/go-version-bump.md`:
- Line 41: Update all five diagram opening fences in the release documentation
to specify an appropriate language identifier, such as text, including the
fences referenced by the review. Leave the diagram contents unchanged.
- Around line 231-237: Update the dependency-pinning checklist near “If fixes
are not yet tagged upstream” to explicitly use Go pseudo-version terminology for
commit-based pins, while directing contributors to use a tagged pre-release
whenever one exists.
- Around line 212-223: Update the KEP-3744 safety controls in
enhancements/release/go-version-bump.md at lines 212-223 and 429-445: require Go
1.N to have been released for at least three months and used by a released
Kubernetes version for at least one month before adoption, and require unit and
integration test results using both Go 1.N and the Go minor version from the
branch’s .0 release as graduation criteria.

---

Nitpick comments:
In `@enhancements/release/go-version-bump.md`:
- Around line 442-445: Update the TLS validation section to either add
deterministic handshake coverage for configured minimum versions, cipher suites,
signature algorithms, and FIPS profiles, or narrow the existing claim to
describe inter-component e2e tests as smoke coverage only. Keep the statement
aligned with the actual validation implemented.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 145d2436-2b92-4154-b6e3-9880bd3105d2

📥 Commits

Reviewing files that changed from the base of the PR and between 76e4721 and 9476d13.

📒 Files selected for processing (1)
  • enhancements/release/go-version-bump.md

Comment thread enhancements/release/go-version-bump.md Outdated
Comment thread enhancements/release/go-version-bump.md
Comment on lines +231 to +237
- [ ] Check the upstream Kube release branch for dependency fixes related to
the Go bump (cherry-picks, version pins, disabled checks)
- [ ] Verify that openshift/kubernetes has inherited those fixes
- [ ] Identify any additional dependencies that break with the new Go version
- [ ] Track upstream fixes; determine if tagged releases are available
- [ ] If fixes are not yet tagged upstream, pin to specific commits with
pre-release versions in `go.mod`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- target lines ---'
sed -n '220,245p' enhancements/release/go-version-bump.md

printf '%s\n' '--- related terminology in repository ---'
rg -n -i 'pseudo-version|pre-release|prerelease|commit pin|specific commit' \
  enhancements go.mod go.work 2>/dev/null || true

printf '%s\n' '--- standalone terminology probe ---'
python3 - <<'PY'
import re
from pathlib import Path

path = Path("enhancements/release/go-version-bump.md")
text = path.read_text()
lines = text.splitlines()
for i in range(230, 237):
    print(f"{i+1}: {lines[i]}")
print("contains_commit_pin_and_prerelease_wording:",
      bool(re.search(r"pin to specific commits.*pre-release versions", text, re.I | re.S)))
PY

Repository: openshift/enhancements

Length of output: 9442


Use pseudo-version terminology for commit pins.

If fixes are not tagged upstream, pin the specific commit; Go records a pseudo-version in go.mod. Use a tagged pre-release when one exists.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/release/go-version-bump.md` around lines 231 - 237, Update the
dependency-pinning checklist near “If fixes are not yet tagged upstream” to
explicitly use Go pseudo-version terminology for commit-based pins, while
directing contributors to use a tagged pre-release whenever one exists.

Source: MCP tools

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@enhancements/release/go-version-bump.md`:
- Around line 346-353: Update the GODEBUG compatibility guidance to require
checking explicit overrides in the main go.mod or go.work godebug directives and
main-package //go:debug directives. In the version-specific appendix, record the
output of go list -f '{{.DefaultGODEBUG}}' and clarify that these settings can
override the go directive’s baseline behavior before claiming compatibility.

In `@hack/lint-ignores`:
- Line 2: Remove the enhancements/release/go-version-bump-go126.md entry from
hack/lint-ignores so the enhancement remains subject to Markdown linting and
metadata validation; do not modify the lint tooling or add an exception.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b7da3826-5cb0-477e-a5a6-73cc696a9cd0

📥 Commits

Reviewing files that changed from the base of the PR and between 9476d13 and 60c46bf.

📒 Files selected for processing (3)
  • enhancements/release/go-version-bump-go126.md
  • enhancements/release/go-version-bump.md
  • hack/lint-ignores

Comment on lines +346 to +353
- **GODEBUG directive**: The `go` directive in `go.mod` controls which Go
version's runtime behavior the binary uses. Because the `go` directive is
not bumped on release branches, GODEBUG automatically preserves the
original runtime behavior — no explicit GODEBUG configuration is needed.
- **GODEBUG does not cover all changes**: `GOEXPERIMENT` defaults and some
unconditional stdlib changes are active with the new toolchain regardless
of the `go` directive. These are documented in the version-specific
appendix.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu
file="enhancements/release/go-version-bump.md"
printf '%s\n' "== target section =="
sed -n '320,370p' "$file"
printf '%s\n' "== related GODEBUG references =="
rg -n -C 3 'GODEBUG|DefaultGODEBUG|go:debug|godebug|go\.work|go\.mod' "$file"
printf '%s\n' "== repository files mentioning this process =="
rg -n 'go-version-bump|DefaultGODEBUG|GODEBUG directive|GODEBUG does not cover' .

Repository: openshift/enhancements

Length of output: 9538


🌐 Web query:

site:go.dev/doc/godebug go.mod go.work //go:debug DefaultGODEBUG precedence

💡 Result:

When determining the effective GODEBUG settings for a Go program, the Go toolchain applies a hierarchical precedence order. Settings are derived from the following sources, listed from lowest to highest precedence [1]: 1. Toolchain Defaults: The default values for the Go toolchain currently being used to build the program [1]. 2. Go Version Compatibility: The toolchain amends these defaults to match the Go version specified in the main module's go.mod file (or the go.work file if a workspace is in use) [1]. 3. Configuration Directives: - In a single-module build, go.mod godebug directives are used [1]. - In a workspace build, go.work godebug directives take precedence, and go.mod directives are ignored [1]. 4. Environment Variables: The GODEBUG environment variable explicitly sets values at runtime [1]. 5. Source Code Directives: The //go:debug directive in a main package's source files provides the highest level of control, overriding all other settings [1]. Regarding the specific interaction between these components: - If a GODEBUG setting is not specified in the environment, its value is derived from the toolchain defaults, adjusted by the Go version in go.mod or go.work, and then overridden by //go:debug lines [1]. - Only the go.mod file of the main (work) module is consulted for godebug directives; directives in dependencies are ignored [1]. - When using a workspace, go.work is the sole source for godebug directives, and any directives in constituent go.mod files are disregarded [1]. - The //go:debug directives must appear at the top of a main package's source file, preceding the package statement [1]. It is an error to have multiple //go:debug lines for the same setting, or to use an unrecognized setting [1].

Citations:


🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' "== operational guidance around GODEBUG =="
sed -n '430,505p' enhancements/release/go-version-bump.md
printf '%s\n' "== Go 1.26 appendix GODEBUG guidance =="
rg -n -C 5 'GODEBUG|GOEXPERIMENT|DefaultGODEBUG|go:debug|godebug' enhancements/release/go-version-bump-go126.md

Repository: openshift/enhancements

Length of output: 6864


🌐 Web query:

site:go.dev/cmd/go go list DefaultGODEBUG template field

💡 Result:

The DefaultGODEBUG field is a template field available in the go list command that represents the default GODEBUG setting for a package [1][2][3]. This field is specifically populated for main packages [4][2][3]. It indicates the default debugging or runtime behavior settings that the package will adopt, which can be influenced by build-time configurations, such as the setting of environment variables like GOFIPS140 [5]. You can access this field when using the go list command with the -f (format) flag. For example: go list -f '{{.DefaultGODEBUG}}'./... Because this is an internal field used by the go command to describe package metadata, it is documented as part of the package structure in the internal Go toolchain source code [4][1][3].

Citations:


Inspect explicit GODEBUG overrides before claiming compatibility.

The go directive provides only the baseline. Inspect the main go.mod or go.work for godebug directives and main packages for //go:debug directives. Record go list -f '{{.DefaultGODEBUG}}' output because these settings can override the compatibility baseline.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@enhancements/release/go-version-bump.md` around lines 346 - 353, Update the
GODEBUG compatibility guidance to require checking explicit overrides in the
main go.mod or go.work godebug directives and main-package //go:debug
directives. In the version-specific appendix, record the output of go list -f
'{{.DefaultGODEBUG}}' and clarify that these settings can override the go
directive’s baseline behavior before claiming compatibility.

Source: MCP tools

Comment thread hack/lint-ignores
@@ -1 +1,2 @@
enhancements/authentication/direct-oidc-study/study-oidc-in-openshift.md
enhancements/release/go-version-bump-go126.md

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Do not exempt this enhancement from validation.

Line 2 adds enhancements/release/go-version-bump-go126.md to hack/lint-ignores. This bypasses both Markdown linting and metadata validation. It also conflicts with the hack/markdownlint.sh rule that ignored paths should only be non-enhancement Markdown files. Remove this entry, or add a narrowly scoped and documented exception in the lint tooling.

Proposed fix
 enhancements/authentication/direct-oidc-study/study-oidc-in-openshift.md
-enhancements/release/go-version-bump-go126.md
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/lint-ignores` at line 2, Remove the
enhancements/release/go-version-bump-go126.md entry from hack/lint-ignores so
the enhancement remains subject to Markdown linting and metadata validation; do
not modify the lint tooling or add an exception.

Comment thread enhancements/release/go-version-bump.md Outdated

@sdodson sdodson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to pause on the decisions around whether or not we're going to bump more than once, I'll try to drive a decision there ASAP.

Comment thread enhancements/release/go-version-bump.md Outdated
Comment thread enhancements/release/go-version-bump.md Outdated
Comment thread enhancements/release/go-version-bump.md Outdated
Comment on lines +129 to +137
- As a cluster administrator, I want OpenShift to ship with a supported Go
toolchain so that security scanners do not flag known CVEs in the Go
compiler that I must triage but cannot resolve.
- As a security reviewer, I want the Go toolchain used to build OpenShift
to receive upstream security patches so that critical Go vulnerabilities can
be addressed in a timely manner without requiring forks or backports.
- As a platform engineer maintaining an OpenShift operator, I want the
Go toolchain to stay current so that I can update dependencies whose CVE
fixes are compatible with the current toolchain.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like some of this is either worded in a confusing manner or outright misaligned with non-goal number 2. Can you try to resolve that in some way so it doesn't read as much like we're going to be current forever. Under the current release schedule we're going to be extending the period in which OpenShift remains on a supported version of Go by about 6 months. That will generally leave up to 4 years of supported life where we're on an EOL version of Go.

@Prashanth684 Prashanth684 Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe current was not the correct word - but in general - aligned with upstream kubernetes would make much more sense - i'll rework the wording. i left it more open in case we decide to bump further.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reworded this section and others to reflect that we will only match upstream kube initially.

Comment thread enhancements/release/go-version-bump.md
Comment thread enhancements/release/go-version-bump.md
Comment on lines +242 to +247
**Stage 2: Nightlies (~1-2 weeks)**
- [ ] Switch nightly builds to the new Go version
- [ ] Validate across the full product, not just unit/integration tests
- [ ] Monitor for TLS and FIPS-related regressions (see version-specific
appendix for what to watch)
- [ ] Run CI jobs with FIPS cluster profiles enabled

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're saying here that we expect to complete moving all builds from Go 1.N to 1.N+1 during this 1-2 weeks but we won't be stopping z-stream delivery during this time?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't have to ...the bumps will take place incrementally but shouldn't break anything as noted in the version skew section.

- Defines a repeatable 6-phase process for bumping the Go toolchain on OCP release branches
- Follows upstream Kubernetes practice (KEP-3744): bump toolchain, keep go.mod `go` directive unchanged, rely on GODEBUG for runtime compatibility
- Includes a Go 1.26 appendix with payload scan of 160 repos on release-4.22

Go goes EOL before OpenShift does.Update the Golang toolchain used to build and run OpenShift during the lifecycle of a given release.
This greatly reduces the number of fixable CVEs since we can now fix them more easily by updating dependencies quicker instead of waiting
for forks. Upstream Kube already does this.

- `enhancements/release/go-version-bump.md` — version-agnostic process doc
- `enhancements/release/go-version-bump-go126.md` — Go 1.25→1.26 appendix for OCP 4.22

Co-authored-by: Scott Dodson <sdodson@redhat.com>
@openshift-ci

openshift-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

@Prashanth684: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Roll out in two stages:

**Stage 1: CI repos (~1-2 weeks)**
- [ ] Switch CI repos to use the new Go builder image

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think from when we met with ART we determined we would need to pause reconciliation PRs while we move everyone and bump GO_LATEST/GO_EXTRA?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes i'll add that

@sdodson

sdodson commented Sep 11, 2026

Copy link
Copy Markdown
Member

@ashwindasr @dusk125 @ggiguash Do you three mind reviewing this? I know we identified etcd <-> microshift build root concerns, what we expect to do is as best as possible bump the version of Go in the buildroot at the same time as we flip nightlies over. If you have other suggestions or concerns with how we can manage the bump in 4.22 to 1.26 let us know.

Otherwise LGTM, at least good enough to get started on things. We've already started cleaning up main and release-5.0 branches in places where we're finding out of date builders in CI (some still building with 1.25 etc).

@Prashanth684

Prashanth684 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@ashwindasr @dusk125 @ggiguash Do you three mind reviewing this? I know we identified etcd <-> microshift build root concerns, what we expect to do is as best as possible bump the version of Go in the buildroot at the same time as we flip nightlies over. If you have other suggestions or concerns with how we can manage the bump in 4.22 to 1.26 let us know.

/cc @pacevedom

Comment on lines +141 to +142
1. Define a documented process for bumping Go on an OCP release branch to
match the upstream Kubernetes Go version.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So from the etcd side, upstream is accelerating its release cycle, but it's not a the 6-month window like Kube/Go; so, we get go.mod go version changes in patch releases on our minor versions 3.6, 3.5.

Since upstream Kube only supports to N-3 (if memory serves), etcd could be taking patches that increase go mod well after an upstream Kube version has been dropped.

It wasn't clear if this enhancement is just for keeping OCP and upstream Kube in sync (in terms of go build version), or are we going to continue to bump the builder for older - but still supported - OCP releases? If so, then the etcd bumps are a non-issue; if not, then we'll be limited in the etcd version bumps we can do (as we are today). This is also predicated on the etcd/microshift build root decoupling work mentioned below

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially at least, OCP releases will follow the same cadence of Go builder bump as upstream Kubernetes. Given the current Kubernetes and Go support cadences, this normally means one upstream-aligned toolchain bump per Kubernetes release. I think the main issue for any component being ahead is we might not know what issues exist that we are not able to catch - like the deadcode elimination issue that upstream found and which we would've missed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah makes sense, at least with etcd being ahead, that is upstream provided, so there's a little more certainty that for this one component that it's ok to build with the higher go version

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, where as we never bumped Go in the past we will now do it approximately once starting with 4.22, potentially skipping 5.0 until we've finalized all of the native Go FIPS migration and are comfortable with that happening in 5.0.z... which we may never be.

@ashwindasr

Copy link
Copy Markdown

Running it by Sid as well

/cc @thegreyd

@openshift-ci
openshift-ci Bot requested a review from thegreyd September 14, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants