Skip to content

fix(restapi): serve /shutdown only on a loopback listener (VLN-4882) - #195

Open
persona-cloudflare-app-oracle-se[bot] wants to merge 1 commit into
mainfrom
kyle.barton+editor/2026-10-26/loopback-shutdown-b741a837
Open

persona-cloudflare-app-oracle-se[bot] wants to merge 1 commit into
mainfrom
kyle.barton+editor/2026-10-26/loopback-shutdown-b741a837

Conversation

@persona-cloudflare-app-oracle-se

@persona-cloudflare-app-oracle-se persona-cloudflare-app-oracle-se Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

On behalf of: @sideshowbandana

Summary

Fixes VLN-4882 (unauthenticated ProxySQL shutdown).

The agent registered /shutdown on the same HTTP server as the kubelet probes. That server listens on :8080 on all interfaces, so any pod in the cluster could reach it on the pod IP. The handler did no auth, method or source check, so one request ran PreStopShutdown() (drain, PROXYSQL PAUSE, PROXYSQL SHUTDOWN) and stopped the pod for good. A source comment wrongly said the API was "only accessible inside the pod itself".

Changes

  • Probe server (api.port, default 8080, all interfaces): serves /healthz/started|ready|live only. The paths, port and handlers are unchanged. /shutdown now returns 404 here.
  • New shutdown server (127.0.0.1:api.shutdown_port, default 8081): serves POST /shutdown only, and returns 405 for any other method. The bind host is hard-coded to 127.0.0.1 and cannot be configured. Only the port can be set, via api.shutdown_port, AGENT_API_SHUTDOWN_PORT or --api.shutdown_port. Validation rejects ports outside 1–65535 and a port equal to api.port.
  • Both servers are stopped after the drain completes. SetHTTPServer became SetHTTPServers(...).
  • Replaced the misleading comment. nosemgrep stays on ListenAndServe with an accurate justification.
  • .github/workflows/golangci-lint.yml: pin setup-go to go.mod (see Testing).
  • Updated docs: README (HTTP API and preStop hook), config.yaml, CHANGELOG (marked breaking for deployments), CLAUDE.md, and the OrbStack smoke manifests (now using the exec hook).

Design: why no agent CLI subcommand

The preStop hook has to stay on the proxysql container. The agent runs as a native sidecar, so the kubelet runs the agent container's own preStop only after proxysql has exited, which is too late to drain. An exec hook therefore runs in the proxysql/proxysql image, not the agent image (debian:stable-slim, no curl or wget). The proxysql/proxysql image already ships wget: this repo's OrbStack smoke test runs kubectl exec -c proxysql -- wget … against the agent. A proxysql-agent shutdown subcommand would not be available in that container, so this PR doesn't add one.

Recommended preStop hook (on the proxysql container):

lifecycle:
  preStop:
    exec:
      command: ["wget", "-q", "-O", "-", "--post-data=", "http://127.0.0.1:8081/shutdown"]

Rollout order (persona-kubernetes)

⚠️ Once this is released, an httpGet preStop on /shutdown gets a 404 and pods stop without draining. Change the order like this:

  1. Merge this PR, then tag a release (vX.Y.Z). The tag triggers goreleaser, which publishes ghcr.io/persona-id/proxysql-agent:vX.Y.Z.
  2. In persona-kubernetes, bump the agent image and switch the preStop hook to the exec form above in the same change (lib/proxysql-stack-{green,blue}/deployment-{core,satellite}.libsonnet, lib/proxysql-blue/…). Switch the hook only after a release containing this change is deployed. Moving it earlier points the hook at a port the old agent doesn't serve.

Testing

  • New unit tests:
    • the probe server returns 404 for /shutdown on every method and starts no shutdown
    • shutdownAddr always resolves to loopback
    • shutdown server routing: POST runs PreStopShutdown (200, or 500 when it fails), GET and PUT return 405, /healthz/* returns 404
    • an end-to-end POST over a real loopback listener
    • shutdownHTTPServer stops every registered server
    • config defaults and validation for api.shutdown_port
  • Go 1.27 could not be downloaded in the authoring sandbox, so only gofmt ran locally. The Run Tests and golangci-lint workflows on this PR are the compile, test and lint gate. On the first push (89707d2), Run Tests passed.
  • CI fix: the lint workflow used go-version: stable. That now resolves to a Go newer than golangci-lint v2.13.2 supports, so lint failed in typecheck before it checked any code (export data version 5 is greater than maximum supported version 4). The workflow now pins go-version-file: go.mod.

🤖 Generated with Oracle

Co-authored-by: kyle.barton kyle.barton@withpersona.com
Co-authored-by: Claude noreply@anthropic.com

/shutdown was registered on the :8080 probe server, which listens on all
interfaces, so any pod in the cluster could drain and stop ProxySQL with
one unauthenticated request (VLN-4882).

/shutdown now lives on a separate server bound to 127.0.0.1 on
api.shutdown_port (default 8081) and accepts POST only. The probe server
keeps /healthz/* on :8080 unchanged. Both servers are stopped after the
drain completes. The preStop hook must switch from httpGet to an exec
hook on the proxysql container that POSTs to the loopback endpoint.

Refs: https://withpersona.atlassian.net/browse/VLN-4882

Oracle-Chat: https://oracle.withpersona.ai/editor/b741a8370f198022bbac60190d4f5cd745b4abbe0fa363d88c43d9d69356754a
Co-authored-by: Oracle Editor <oracle-selfedit[bot]@users.noreply.github.com>
@persona-cloudflare-app-oracle-se
persona-cloudflare-app-oracle-se Bot force-pushed the kyle.barton+editor/2026-10-26/loopback-shutdown-b741a837 branch from 89707d2 to bb24149 Compare October 26, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant