identity: Proxy fails to validate certificates with name constraints

[olix0r]

Linkerd cannot be deployed when its CA certificate(s) use name constraints. Proxies fail to validate issued certificates.

Repro/Investigation

1. Generate a self-signed certificate with name constraints limiting issued
   certificates to foo.bar:

bar.tpl:

{
    "subject": {{ toJson .Subject }},
    "issuer": {{ toJson .Subject }},
    "keyUsage": ["certSign"],
    "basicConstraints": {
        "isCA": true,
        "maxPathLen": 1
    },
    "nameConstraints": {
        "critical": true,
        "permittedDNSDomains": ["foo.bar"]
    }
}

:; step certificate create foo.bar cert.pem key.pem --template=bar.tpl --not-after=24h --no-password --insecure

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 223894858428966857916253664437917250458 (0xa87094cb1d1a378f60aa362777a4f39a)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:07:42 2022 UTC
            Not After : Aug 31 20:07:42 2022 UTC
        Subject: CN=foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    10:f9:57:fa:30:30:93:4d:9b:e1:64:20:54:a1:1a:
                    20:76:28:5e:31:c4:6f:19:f6:62:9d:fe:9b:68:e8:
                    14:ee
                Y:
                    58:aa:27:73:08:24:78:a9:57:f1:35:fe:53:d9:2d:
                    83:01:d4:fd:c3:b6:44:26:16:06:a0:fb:6f:d8:4b:
                    6b:c8
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Subject Key Identifier:
                A3:E8:20:63:F8:F4:35:D0:0D:47:79:B7:AD:F3:CD:3B:EF:2C:74:AE
            X509v3 Name Constraints: critical
                Permitted:
                  DNS: foo.bar
    Signature Algorithm: ECDSA-SHA256
         30:45:02:20:60:b2:5c:96:44:30:0c:5b:fb:63:98:90:7d:60:
         7e:75:ce:88:81:70:8a:01:97:e8:f5:e4:29:30:bf:87:04:ad:
         02:21:00:ed:fa:f6:71:15:c0:26:99:ec:c7:9a:a3:4d:e6:5f:
         e0:e4:d0:fb:9e:3a:3e:30:26:0f:90:25:ba:0a:17:6f:52

2. Install Linkerd using this certificate:

:; linkerd install --crds |k apply -f -

:; linkerd install \
    --set identityTrustDomain="foo.bar" \
    --set-file identity.issuer.tls.crtPEM=cert.pem \
    --set-file identity.issuer.tls.keyPEM=key.pem \
    --set-file identityTrustAnchorsPEM=cert.pem \
    |k apply -f -

3. The control plane does not start.

The identity controller becomes 'running' and it appears to issue a certificate
to its local proxy:

:; k logs -n linkerd linkerd-identity-5c9d8bbb4c-n8n8n
Defaulted container "identity" out of: identity, linkerd-proxy, linkerd-init (init)
time="2022-08-30T20:13:40Z" level=info msg="running version stable-2.12.0"
time="2022-08-30T20:13:40Z" level=info msg="starting admin server on :9990"
time="2022-08-30T20:13:40Z" level=info msg="starting gRPC server on :8080"
time="2022-08-30T20:13:46Z" level=info msg="issued certificate for linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar until 2022-08-31 20:07:42 +0000 UTC: 4a28173a08c2e9629eeb982e8b2699b6c6007eef8b2377b31eeb2a6149bb371f"

But the proxy fails to validate the certificate:

:; k logs -n linkerd linkerd-identity-5c9d8bbb4c-n8n8n -c linkerd-proxy
...
[     0.011665s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=invalid peer certificate contents: invalid peer certificate: UnknownIssuer

The rest of the control plane fails to start because the identity controller never becomes ready.

---

This process works when name constraints are omitted from the certificate:

{
    "subject": {{ toJson .Subject }},
    "issuer": {{ toJson .Subject }},
    "keyUsage": ["certSign"],
    "basicConstraints": {
        "isCA": true,
        "maxPathLen": 1
    }
}

:; step certificate create foo.bar cert.pem key.pem --template=bar-unconstrained.tpl --not-after=24h --no-password --insecure

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 22516031807147259098312418109158923000 (0x10f06e13bbe38643d7ce95ec4cf91af8)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:33:12 2022 UTC
            Not After : Aug 31 20:33:12 2022 UTC
        Subject: CN=foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    74:e5:86:db:e6:68:1d:a5:c7:1d:fd:21:29:2d:69:
                    de:f1:0a:57:7c:30:35:f1:9d:c4:c2:d3:61:cf:42:
                    09:19
                Y:
                    ab:00:91:94:2c:df:e5:7f:4b:13:8e:40:e5:04:07:
                    e9:0e:db:c9:03:83:a2:fb:42:80:37:cb:35:df:f4:
                    17:7d
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Subject Key Identifier:
                60:42:74:A2:23:17:6A:BD:94:1E:0B:87:4C:A8:70:7B:B8:BD:AD:55
    Signature Algorithm: ECDSA-SHA256
         30:45:02:20:63:24:33:92:8a:71:65:d3:9f:fd:da:44:ab:40:
         1e:74:6e:0c:58:f5:8d:93:98:31:76:be:e4:96:7a:97:a5:e2:
         02:21:00:a1:a0:93:4d:0e:bc:96:0c:af:99:4a:34:0a:40:5f:
         b6:d1:72:d5:ed:16:85:8a:3a:c4:e1:5e:90:43:80:e0:15

:; linkerd identity -n linkerd linkerd-identity-75f7d4df68-wd8km

POD linkerd-identity-75f7d4df68-wd8km (1 of 1)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:38:17 2022 UTC
            Not After : Aug 31 20:38:27 2022 UTC
        Subject: CN=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    bb:43:be:97:a7:39:83:6c:e5:71:e1:17:71:c2:ef:
                    fc:64:fd:1e:cb:6f:5c:25:fe:8b:ce:1f:91:93:24:
                    40:bf
                Y:
                    07:89:d1:56:5d:8e:b7:b7:3c:9a:8a:a6:12:e3:19:
                    30:63:63:25:49:4b:8a:61:be:1f:5b:ed:f2:88:6c:
                    b8:79
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:5E:A4:79:4B:A0:EA:D3:8A:07:9F:14:8F:2A:77:E8:0E:0B:1A:4B:D7
            X509v3 Subject Alternative Name:
                DNS:linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar

    Signature Algorithm: ECDSA-SHA256
         30:45:02:20:4d:85:c6:af:4b:04:aa:eb:89:8f:c8:64:f3:49:
         42:60:b0:66:6d:1a:22:5d:e9:41:3c:40:9d:54:8f:e0:d8:1c:
         02:21:00:d5:ee:41:06:a1:ca:fb:1c:40:91:e4:35:75:68:ae:
         23:9e:f3:7b:24:b2:4d:14:a6:dd:66:0d:43:7c:5f:79:04

---

Note that we can manually create an end-entity certificate (as used by a proxy) with:

:; step certificate create linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar \
    ee-cert.pem ee-key.pem \
    --san=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar \
    --ca-key=key.pem --ca=cert.pem \
    --not-after=24h \
    --no-password --insecure

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 55110646295972328840139681094716422534 (0x2975eca5308fa2fde788e681d188f986)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:53:31 2022 UTC
            Not After : Aug 31 20:53:31 2022 UTC
        Subject: CN=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    52:b2:ad:f7:19:87:ef:f0:0c:e3:f4:4a:5c:3f:a1:
                    67:7b:b1:75:31:32:14:ba:42:56:f7:4a:58:b7:af:
                    fe:8c
                Y:
                    ed:fa:80:45:fb:30:51:8a:3e:86:bd:dd:0f:b8:75:
                    01:47:50:d7:c6:0a:24:ca:42:65:02:26:0a:36:db:
                    3d:4f
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Subject Key Identifier:
                0B:1B:29:6D:CF:76:77:A7:3E:40:7F:10:E6:0A:BD:69:69:BE:70:7B
            X509v3 Authority Key Identifier:
                keyid:5E:A4:79:4B:A0:EA:D3:8A:07:9F:14:8F:2A:77:E8:0E:0B:1A:4B:D7
            X509v3 Subject Alternative Name:
                DNS:linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar
    Signature Algorithm: ECDSA-SHA256
         30:44:02:20:3d:0a:2b:dd:49:21:a9:95:3c:49:48:17:54:d0:
         99:c9:f7:c7:7b:73:da:39:86:1b:27:8a:e6:9b:5c:c8:68:eb:
         02:20:3e:16:16:bd:ff:82:99:16:c8:f6:60:bb:e5:55:5f:f0:
         de:6e:02:73:37:eb:d8:3a:b4:37:96:31:4e:a8:ce:29

step happily verifies that the end-entity certificate is signed by the CA, even
when name constraints are present:

:; step certificate verify ee-cert.pem \
    --roots=cert.pem \
    --host=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar

If we generate an end entity certificate for an alternate suffix (idk.lol),
step properly fails to validate the certificate.

:; step certificate create linkerd-identity.linkerd.serviceaccount.identity.linkerd.idk.lol ee-cert.pem ee-key.pem --san=linkerd-identity.linkerd.serviceaccount.identity.linkerd.idk.lol --ca-key=key.pem --ca=cert.pem --not-after=24h --no-password --insecure
...
:; step certificate verify ee-cert.pem \
    --roots=cert.pem \
    --host=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bah
failed to verify certificate: x509: a root or intermediate certificate is not authorized to sign for this name: DNS name "linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bah" is not permitted by any constraint

Suggested next steps

Why doesn't the proxy validate its end entity certificates when name constraints are used?

To figure this out, I'd probably try to create a test (or repo) that attempts to
use webpki to validate these certificates. If this works in a standalone repo,
then we can try to figure out how Linkerd's usage differs from a simpler example
setup.

Note that it may be required to alter the format of the private key with, e.g.:

:; openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in key.pem -out pkcs8.key  

[olix0r]

It seems that there is a known bug in webpki and PR to fix it that's been open without review for >1 year:

• https://github.com/briansmith/webpki/issues/20
• UnknownIssuer with cert from private CA rustls/rustls#841
• Fix name constraints check briansmith/webpki#226

Given that we're already using a fork of webpki, we could consider merging some variation of the proposed fix...