Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,30 @@

## [Unreleased]

### ✨ Added

- **`feat(https)`: mutual TLS (client certificate authentication) on HTTPS listeners** ([#1299](https://github.com/sozu-proxy/sozu/issues/1299)).
An HTTPS listener can now require or request a client certificate instead of
always disabling client auth. A new per-listener `client_auth` mode
(`none` / `optional` / `required`) drives the rustls `WebPkiClientVerifier`:
`required` aborts the handshake unless the client presents a certificate
chaining to a configured trusted CA, `optional` requests one but still
accepts connections that present none (a presented certificate must still
validate). New listener fields carry the PEM-encoded trusted CA bundle
(`client_ca_certificates`) and optional revocation lists (`client_ca_crls`).
Absent config decodes to `none`, preserving the previous behavior and keeping
existing state files loadable (new protobuf fields are appended, none
reordered, and default to empty when missing from an older record).
Configuration fails closed: an unknown `client_auth` value, a non-`none` mode
with no trusted CA, an unreadable CA/CRL file, a CA or CRL entry that yields no
certificate/revocation list, or an mTLS field set on a non-HTTPS listener is
rejected rather than silently accepting unauthenticated or unrevoked clients.
CRL expiration is enforced (a CRL past its `nextUpdate` is rejected instead of
trusted). In `none` mode CA/CRL paths are ignored entirely, so a stale path
never blocks the configuration from loading. `HttpsListenerConfig` gains the
`ClientAuthMode` enum and the three fields; `ListenerError::ClientAuth` and
`ConfigError::ClientAuthOnNonHttps` report misconfigured input.

### 🔐 Security

- **`fix(command)`: redact TLS certificate and private-key material from `Debug` output.**
Expand Down
5 changes: 5 additions & 0 deletions command/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,11 @@ pub fn main() {
.field_attribute("TcpListenerConfig.answers", "#[serde(default)]")
.field_attribute("RequestUdpFrontend.tags", "#[serde(default)]")
.field_attribute("RequestTcpFrontend.alpn", "#[serde(default)]")
.field_attribute(
"HttpsListenerConfig.client_ca_certificates",
"#[serde(default)]",
)
.field_attribute("HttpsListenerConfig.client_ca_crls", "#[serde(default)]")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 low — HttpsListenerConfig's hand-written Debug impl doesn't surface the new mTLS fields

HttpsListenerConfig is in the skip_debug list on the very next line, so prost's derive is replaced by the hand-written impl std::fmt::Debug for command::HttpsListenerConfig at command/src/proto/mod.rs:409. That impl enumerates every field explicitly — sensitive ones redacted (certificate, key), bulky ones summarised (_len/_count) — and ends at .finish() with hsts then h2_max_header_fields as the last entries. client_auth, client_ca_certificates, and client_ca_crls were not added, so a listener's entire mTLS configuration is invisible in Debug/log output, not even as a count.

Not a leak (omission, not over-exposure), but it breaks the established exhaustive pattern and removes the one observability handle for triaging mTLS config issues. Add .field("client_auth", &self.client_auth) plus client_ca_certificates_count/client_ca_crls_count (and total_string_len for the PEM bodies), matching the cipher_list/groups_list treatment in the same impl.


lenses: /codex • confidence: high • adversarially verified at 1518d93e

.skip_debug([
"CertificateAndKey",
"CertificateSummary",
Expand Down
32 changes: 32 additions & 0 deletions command/src/command.proto
Original file line number Diff line number Diff line change
Expand Up @@ -648,6 +648,23 @@ message HttpsListenerConfig {
// indexed-reference header bomb, where 1-byte indexed references are
// amplified into per-entry bookkeeping. Default: 128.
optional uint32 h2_max_header_fields = 47;
// Mutual TLS (client certificate authentication) mode for this listener.
// Absent or CLIENT_AUTH_NONE (the field-0 default) keeps the current
// behavior: no CertificateRequest is sent and any client certificate is
// ignored. OPTIONAL requests a client certificate but still accepts
// connections that present none; REQUIRED aborts the handshake when the
// client does not present a certificate chaining to a trusted CA.
optional ClientAuthMode client_auth = 48;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[security, high] Older workers silently ignore required mTLS and acknowledge the listener

UpgradeMain deliberately hands pre-upgrade workers to the new master, with worker replacement happening afterwards (bin/src/command/LIFECYCLE.md:188-203). Because this extends the existing AddHttpsListener payload rather than using a capability-distinguishable verb, a pre-PR worker recognizes the outer request while prost skips unknown nested tags 48-50. It then builds the listener through the historical .with_no_client_auth() path and returns OK, so client_auth = "required" can accept unauthenticated traffic until every old worker is gone. A failed worker upgrade can leave this state persistent because the CLI logs individual upgrade failures but still returns success. Please make mTLS listener creation fail closed on older workers—for example through a new request variant that old workers reject, or a verified per-worker capability gate before fan-out—and add a new-master/old-worker encoding regression test.

// PEM-encoded trusted CA certificates a client certificate must chain to
// for OPTIONAL/REQUIRED modes. Ignored when client_auth is NONE. Multiple
// entries build a single root store; each entry may itself hold a
// concatenated PEM chain.
repeated string client_ca_certificates = 49;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[regression, low] The default listener listing hides the active client-authentication policy

The normal ListListeners output formats HttpsListenerConfig through command/src/proto/display.rs:1257-1334, whose table has no row for client_auth or safe CA/CRL metadata. An operator can therefore apply required authentication successfully and then see output indistinguishable from a listener with client authentication disabled. The new Debug fields do not cover this default CLI path. Please display the mode plus non-secret CA/CRL counts or byte lengths, without rendering PEM contents, and cover that formatter.

// Optional PEM-encoded certificate revocation lists (CRLs). When present,
// a presented client certificate is rejected if it (or an intermediate)
// appears revoked. Ignored when client_auth is NONE or no trusted CA is
// configured.
repeated string client_ca_crls = 50;
}

// details of an TCP listener
Expand Down Expand Up @@ -1048,6 +1065,21 @@ enum TlsVersion {
TLS_V1_3 = 5;
}

// Mutual TLS client-certificate authentication policy for an HTTPS listener.
// NONE is field 0 so an absent `client_auth` decodes to the current
// no-client-auth behavior, keeping legacy state files byte-compatible.
enum ClientAuthMode {
// No CertificateRequest; client certificates are neither requested nor
// verified. The historical default.
CLIENT_AUTH_NONE = 0;
// Request a client certificate but accept connections that present none.
// Presented certificates must still chain to a trusted CA.
CLIENT_AUTH_OPTIONAL = 1;
// Require a client certificate chaining to a trusted CA; abort the
// handshake otherwise.
CLIENT_AUTH_REQUIRED = 2;
}

// How a UDP flow is keyed for backend affinity. SOURCE_IP keys on the
// client source IP only (all ports from one client pin to one backend);
// SOURCE_IP_PORT keys on the full 2-tuple (per-socket affinity).
Expand Down
186 changes: 185 additions & 1 deletion command/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ use crate::{
certificate::split_certificate_chain,
logging::AccessLogFormat,
proto::command::{
ActivateListener, AddBackend, AddCertificate, CertificateAndKey, Cluster,
ActivateListener, AddBackend, AddCertificate, CertificateAndKey, ClientAuthMode, Cluster,
CustomHttpAnswers, Header, HeaderPosition, HealthCheckConfig, HstsConfig,
HttpListenerConfig, HttpsListenerConfig, ListenerType, LoadBalancingAlgorithms,
LoadBalancingParams, LoadMetric, MetricDetail, MetricsConfiguration, PathRule,
Expand Down Expand Up @@ -393,6 +393,15 @@ pub enum ConfigError {
(RFC 6797 §7.2 forbids the header over plaintext HTTP)"
)]
HstsOnPlainHttp(String),
/// A client-certificate-authentication (mTLS) field was set on a non-HTTPS
/// listener. Client auth is a TLS-termination control; HTTP/TCP/UDP
/// listeners discard it, so sozu rejects it at load time rather than
/// silently starting an unauthenticated listener.
#[error(
"invalid client auth config at {0}: client_auth / client_ca_certificates / \
client_ca_crls are only valid on HTTPS listeners"
)]
ClientAuthOnNonHttps(String),
/// A TCP frontend's `hostname` (mapped to the wire `sni` field) is
/// neither an exact hostname nor a single leading `*.` wildcard label
/// (sozu-proxy/sozu#1279). Rejects `*.*.example.com`, an embedded `*`
Expand Down Expand Up @@ -521,6 +530,30 @@ pub enum ConfigError {
},
}

/// Config-facing mutual-TLS (client certificate authentication) mode, parsed
/// from the `client_auth` key of an HTTPS listener's TOML section. Serializes as
/// lowercase (`"none"` / `"optional"` / `"required"`) so operators write the
/// documented values rather than the protobuf enum's `SCREAMING_SNAKE_CASE`.
/// Maps to the wire [`ClientAuthMode`] in [`ListenerBuilder::to_tls`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")]
pub enum ClientAuthConfig {
#[default]
None,
Optional,
Required,
}

impl From<ClientAuthConfig> for ClientAuthMode {
fn from(value: ClientAuthConfig) -> Self {
match value {
ClientAuthConfig::None => ClientAuthMode::ClientAuthNone,
ClientAuthConfig::Optional => ClientAuthMode::ClientAuthOptional,
ClientAuthConfig::Required => ClientAuthMode::ClientAuthRequired,
}
}
}

/// An HTTP, HTTPS or TCP listener as parsed from the `Listeners` section in the toml
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
Expand Down Expand Up @@ -691,6 +724,19 @@ pub struct ListenerBuilder {
/// not exceed the global `buffer_size` (validated at config-load).
/// Defaults to [`DEFAULT_SNI_PREREAD_MAX_BYTES`].
pub sni_preread_max_bytes: Option<u32>,
/// HTTPS listener only: mutual TLS (client certificate authentication)
/// mode (`none` / `optional` / `required`). Absent and `none` both keep the
/// historical no-client-auth behavior.
pub client_auth: Option<ClientAuthConfig>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 medium — New client_auth / client_ca_certificates / client_ca_crls TOML keys are undocumented

doc/configure.md is the canonical [[listeners]] TOML reference and is untouched by this PR. Verified: git show 1518d93e:doc/configure.md | grep -n client_auth → no hits; the only mTLS mentions are pre-existing rows in the handshake-error tables (lines 1197, 2292, 2294) that assume the feature exists elsewhere.

CONTRIBUTING.md:74 says "Docs are code: a change to a public metric, config key, or CLI flag updates its documentation in the same changeset." Every comparable listener knob — hsts, sni_preread_max_bytes, the h2_* family, cipher suites, key-exchange groups — has a subsection under Options specific to HTTPS listeners (line 208) / Options specific to Rustls based HTTPS listeners (line 375) with an example TOML block and, for HSTS, a validation matrix. These three keys are the most security-sensitive of the set and an operator has no way to discover the accepted values (none/optional/required), that CA/CRL entries are filesystem paths resolved at config-materialization time, or the fail-closed failure modes (ConfigError::ClientAuthOnNonHttps, empty-CA rejection, unknown-enum rejection, CRL expiration enforcement).

Please add a mTLS subsection mirroring the HSTS treatment. Also stale in the same changeset: doc/upgrade/1.x-to-2.0.md:400 still lists "Frontend mTLS termination" ([#663]) as follow-up release, which this PR implements.


lenses: /review, /review-code, /guidelines, /codex • confidence: high • adversarially verified at 1518d93e

/// HTTPS listener only: filesystem paths to PEM-encoded trusted CA
/// certificates a client certificate must chain to. Loaded at
/// config-materialization time (like `certificate`/`key`) and inlined as
/// PEM into the resulting [`HttpsListenerConfig`].
pub client_ca_certificates: Option<Vec<String>>,
/// HTTPS listener only: filesystem paths to PEM-encoded CRLs used to
/// reject revoked client certificates. Loaded alongside
/// `client_ca_certificates`.
pub client_ca_crls: Option<Vec<String>>,
}

pub fn default_sticky_name() -> String {
Expand Down Expand Up @@ -785,6 +831,9 @@ impl ListenerBuilder {
max_flows: None,
sni_preread_timeout: None,
sni_preread_max_bytes: None,
client_auth: None,
client_ca_certificates: None,
client_ca_crls: None,
}
}

Expand Down Expand Up @@ -995,6 +1044,28 @@ impl ListenerBuilder {
self.request_timeout = Some(self.request_timeout.unwrap_or(config.request_timeout));
}

/// Reject mTLS fields on a non-HTTPS listener. Client auth is an
/// HTTPS-termination control; the HTTP/TCP/UDP conversions have no field to
/// carry it, so an operator who sets it there must get a typed error rather
/// than a silently unauthenticated listener.
fn reject_client_auth_fields(&self, listener_kind: &str) -> Result<(), ConfigError> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 low — New helper was inserted between an existing doc comment and its target, orphaning to_http's docs

Verified at head: config.rs:1047 is the pre-existing /// build an HTTP listener with config timeouts, using defaults if no config is provided, which documented pub fn to_http. This diff inserts reject_client_auth_fields (with its own doc block at 1048-1051) between the two, so rustdoc now renders that stray line as the first paragraph of reject_client_auth_fields's docs, while to_http at config.rs:1070 ends up with no doc comment at all — unlike its siblings to_tls/to_tcp/to_udp, which each keep theirs.

Fix: delete line 1047 and re-add it directly above pub fn to_http at 1070, leaving reject_client_auth_fields with only its own four-line block.


lenses: /review, /review-code, /simplify • confidence: high • adversarially verified at 1518d93e

let has_mtls = self
.client_auth
.is_some_and(|m| m != ClientAuthConfig::None)
|| self
.client_ca_certificates
.as_ref()
.is_some_and(|v| !v.is_empty())
|| self.client_ca_crls.as_ref().is_some_and(|v| !v.is_empty());
if has_mtls {
return Err(ConfigError::ClientAuthOnNonHttps(format!(
"{} listener {}",
listener_kind, self.address
)));
}
Ok(())
}

/// build an HTTP listener with config timeouts, using defaults if no config is provided
pub fn to_http(&mut self, config: Option<&Config>) -> Result<HttpListenerConfig, ConfigError> {
if self.protocol != Some(ListenerProtocol::Http) {
Expand All @@ -1003,6 +1074,7 @@ impl ListenerBuilder {
found: self.protocol.to_owned(),
});
}
self.reject_client_auth_fields("HTTP")?;

// RFC 6797 §7.2: `Strict-Transport-Security` MUST NOT appear on
// plaintext-HTTP responses. Reject an `[hsts]` block on an HTTP
Expand Down Expand Up @@ -1181,6 +1253,31 @@ impl ListenerBuilder {
.map(split_certificate_chain)
.unwrap_or_default();

// mTLS: load the trusted-CA and CRL PEM bundles from disk, inlining
// their contents (like `certificate`/`key` above). Any unreadable path
// aborts materialization: a dropped CA would weaken trust, and a
// dropped CRL would silently disable revocation for certificates the
// operator meant to reject. Failing closed keeps client auth honest.
//
// Files are read only for OPTIONAL/REQUIRED. In NONE (the default) the
// runtime ignores CA/CRL data, so a stale path left in the config must
// not block the whole configuration from loading.
let mode = self.client_auth.unwrap_or_default();
let client_auth = self
.client_auth
.map(|mode| ClientAuthMode::from(mode) as i32);
let load_pem_paths = |paths: &Option<Vec<String>>| -> Result<Vec<String>, ConfigError> {
if mode == ClientAuthConfig::None {
return Ok(Vec::new());
}
paths
.as_ref()
.map(|list| list.iter().map(|path| Config::load_file(path)).collect())
.unwrap_or_else(|| Ok(Vec::new()))
};
let client_ca_certificates = load_pem_paths(&self.client_ca_certificates)?;
let client_ca_crls = load_pem_paths(&self.client_ca_crls)?;

let http_answers = self.get_http_answers()?;
let answers = self.get_listener_answers()?;

Expand Down Expand Up @@ -1239,6 +1336,9 @@ impl ListenerBuilder {
Some(h) => Some(h.to_proto("listener")?),
None => None,
},
client_auth,
client_ca_certificates,
client_ca_crls,
};

// POST: the built listener binds the requested address and starts
Expand Down Expand Up @@ -1299,6 +1399,7 @@ impl ListenerBuilder {
found: self.protocol.to_owned(),
});
}
self.reject_client_auth_fields("TCP")?;

if let Some(config) = config {
self.assign_config_timeouts(config);
Expand Down Expand Up @@ -1360,6 +1461,7 @@ impl ListenerBuilder {
found: self.protocol.to_owned(),
});
}
self.reject_client_auth_fields("UDP")?;

let mut max_rx_datagram_size = self
.max_rx_datagram_size
Expand Down Expand Up @@ -5895,4 +5997,86 @@ mod tests {
"both cluster frontends on the udp listener must emit AddUdpFrontend"
);
}

#[test]
fn old_state_file_without_client_auth_fields_deserializes() {
// A state file written before mTLS landed carries no `client_ca_*`
// fields. `#[serde(default)]` (via build.rs) must let it round-trip,
// defaulting the new repeated fields to empty rather than erroring on a
// missing field.
let address = SocketAddress::new_v4(127, 0, 0, 1, 8443);
let config = ListenerBuilder::new_https(address)
.to_tls(None)
.expect("default HTTPS listener config");

let mut json: serde_json::Value =
serde_json::to_value(&config).expect("serialize HttpsListenerConfig");
let obj = json.as_object_mut().expect("object");
obj.remove("client_ca_certificates");
obj.remove("client_ca_crls");
obj.remove("client_auth");

let restored: HttpsListenerConfig =
serde_json::from_value(json).expect("old state file must deserialize");
assert!(restored.client_ca_certificates.is_empty());
assert!(restored.client_ca_crls.is_empty());
assert_eq!(restored.client_auth, None);
}

#[test]
fn client_auth_fields_rejected_on_non_https_listener() {
// A TCP listener carrying client_auth must be rejected, not silently
// started unauthenticated.
let address = SocketAddress::new_v4(127, 0, 0, 1, 9000);
let mut tcp = ListenerBuilder::new_tcp(address);
tcp.client_auth = Some(ClientAuthConfig::Required);
assert!(matches!(
tcp.to_tcp(None),
Err(ConfigError::ClientAuthOnNonHttps(_))
));

// A CA bundle without an explicit mode is equally a misconfiguration.
let mut http = ListenerBuilder::new_http(address);
http.client_ca_certificates = Some(vec!["ca.pem".to_string()]);
assert!(matches!(
http.to_http(None),
Err(ConfigError::ClientAuthOnNonHttps(_))
));
}

#[test]
fn none_mode_ignores_stale_ca_paths() {
// With client auth disabled, an unreadable CA/CRL path must not block
// the HTTPS listener from materializing.
let address = SocketAddress::new_v4(127, 0, 0, 1, 9443);
let mut https = ListenerBuilder::new_https(address);
https.client_auth = Some(ClientAuthConfig::None);
https.client_ca_certificates = Some(vec!["/nonexistent/ca.pem".to_string()]);
https.client_ca_crls = Some(vec!["/nonexistent/crl.pem".to_string()]);
let config = https.to_tls(None).expect("NONE mode must ignore CA paths");
assert!(config.client_ca_certificates.is_empty());
assert!(config.client_ca_crls.is_empty());
}

#[test]
fn client_auth_config_parses_documented_lowercase_names() {
// The documented TOML values (`none`/`optional`/`required`) must parse,
// not just the protobuf enum's SCREAMING_SNAKE_CASE.
assert_eq!(
serde_json::from_str::<ClientAuthConfig>("\"required\"").unwrap(),
ClientAuthConfig::Required
);
assert_eq!(
serde_json::from_str::<ClientAuthConfig>("\"optional\"").unwrap(),
ClientAuthConfig::Optional
);
assert_eq!(
serde_json::from_str::<ClientAuthConfig>("\"none\"").unwrap(),
ClientAuthConfig::None
);
assert_eq!(
ClientAuthMode::from(ClientAuthConfig::Required),
ClientAuthMode::ClientAuthRequired
);
}
}
14 changes: 14 additions & 0 deletions command/src/proto/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,20 @@ impl std::fmt::Debug for command::HttpsListenerConfig {
.field("send_x_real_ip", &self.send_x_real_ip)
.field("hsts", &self.hsts)
.field("h2_max_header_fields", &self.h2_max_header_fields)
.field("client_auth", &self.client_auth)
.field(
"client_ca_certificates_count",
&self.client_ca_certificates.len(),
)
.field(
"client_ca_certificates_len",
&total_string_len(&self.client_ca_certificates),
)
.field("client_ca_crls_count", &self.client_ca_crls.len())
.field(
"client_ca_crls_len",
&total_string_len(&self.client_ca_crls),
)
.finish()
}
}
Expand Down
Loading
Loading