external-builders: build env has /dev/ptmx but no /dev/pts, so os.openpty() fails with ENOENT (breaks nixosTest)

[robertwilcox267]

What happens

In a build running on the external-builders Linux builder, /dev/ptmx exists but /dev/pts is not mounted. Any os.openpty() call therefore fails with ENOENT, and no pseudo-terminal can be allocated.

The most visible casualty is nixosTest. NixOS test drivers that allocate a PTY for the VDE switch fail immediately after logging start vlan, with a misleading error:

start vlan
Traceback (most recent call last):
    raise OSError('out of pty devices')
OSError: out of pty devices

The message points at PTY exhaustion. Nothing is exhausted: /proc/sys/kernel/pty/nr reads 0 of 4096 at the moment of failure. Python's pty.openpty() catches the ENOENT from os.openpty(), falls back to scanning the legacy /dev/ptyXX names, finds none on Linux, and raises OSError('out of pty devices') from pty._open_terminal(). That sends people chasing kernel.pty.max and build concurrency for a missing mount.

Reproducer

devpts-repro.nix:

let
  pkgs = import <nixpkgs> { system = "aarch64-linux"; };
in
pkgs.runCommand "devpts-repro" { } ''
  echo "kernel:        $(uname -sr)"
  echo "pty/max:       $(cat /proc/sys/kernel/pty/max)"
  echo "pty/nr:        $(cat /proc/sys/kernel/pty/nr)"
  echo "devpts mounts: [$(grep devpts /proc/mounts || true)]"
  ls -ld /dev/ptmx || true
  ls -ld /dev/pts  || echo "/dev/pts is MISSING"
  ${pkgs.python3}/bin/python3 -c 'import os; os.openpty(); print("os.openpty OK")'
  touch $out
''

nix build --impure -f devpts-repro.nix -L

Actual result, on macOS via the external builder

Derivation /nix/store/zfqf66h0akf1xaiji2hq9gas01v02n5f-devpts-repro.drv:

kernel:        Linux 6.1.143
pty/max:       4096
pty/nr:        0
devpts mounts: []
crw-rw-rw- 1 root root 5, 2 Jan  1  1970 /dev/ptmx
ls: cannot access '/dev/pts': No such file or directory
/dev/pts is MISSING
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import os; os.openpty(); print("os.openpty OK")
               ~~~~~~~~~~^^
FileNotFoundError: [Errno 2] No such file or directory

/dev/ptmx is a real character device, 5:2. Without /dev/pts mounted, the master cannot get a slave.

Expected result

The same derivation, same store hash zfqf66h0akf1xaiji2hq9gas01v02n5f, built by a native Linux Nix inside a NixOS VM:

kernel:        Linux 6.12.81
pty/max:       4096
pty/nr:        1
devpts mounts: [none /dev/pts devpts rw,relatime,mode=620,ptmxmode=000 0 0]
lrwxrwxrwx 1 nobody nogroup 13 /dev/ptmx -> /dev/pts/ptmx
drwxr-xr-x 2 nobody nogroup  0 /dev/pts
os.openpty OK

Only the builder differs, so this isolates the fault to the external builder's build environment. A stock Linux Nix sandbox mounts devpts at /dev/pts and symlinks /dev/ptmx into it.

Notes

• --option sandbox true from the client makes no difference; the behaviour is the same.
• determinate-nixd builder --help exposes only --memory-size, --cpu-count, --kernel-loglevel, --kernel, and --initrd, so there is no client-side or config-side workaround for this.
• The workaround we use is to run nixosTest builds on a native Linux Nix instead.

Suggested fix

Mount devpts at /dev/pts in the external builder's build environment, and point /dev/ptmx at /dev/pts/ptmx, matching the stock Linux sandbox.

Environment

• macOS 26.5.2 (25F84), arm64
• Determinate Nix 3.21.1 (Nix 2.34.7)
• external-builders with systems = ["aarch64-linux","x86_64-linux"]
• Reproduced on both aarch64-linux and x86_64-linux