FNL: coil dev_assign_to_device faults above a grid-size threshold — transient host-copy doubling + discarded ierr

[szaghi]

Symptom (reported, cluster-only)

FNL/OpenACC PRISM run segfaults inside FUNDAL dev_assign_to_device, on the coil object's first host→device transfer. The fault appears only above a grid-size threshold, and that threshold is reported to be well below the VRAM limit. Not reproducible on the WSL dev box. No measurement was possible before the cluster went down, so the trigger below is derived from code inspection, not from a captured trace.

The path that actually runs

prism_fnl_object%copy_cpu_gpu calls the coil copy without buf6D:

! src/app/prism/fnl/adam_prism_fnl_object.F90:328
call self%coil_fnl%copy_cpu_gpu(coil=self%coil, grid=self%adam%grid)

buf6D is optional and is never passed by any caller (:328, :349, and adam_prism_fnl_coil_object.F90:133). So the buffered branch at adam_prism_fnl_coil_object.F90:63-66 is dead, and every coil transfer takes the fallback:

! src/app/prism/fnl/adam_prism_fnl_coil_object.F90:68
call dev_assign_to_device(src=coil%j_vec, dst=self%j_vec_gpu, ij=[1,5])

which resolves to DEV_ASSIGN_TO_DEVICE_KKP_6D_T (FUNDAL/src/lib/fundal_dev_assign_agnostic.INC:476-495):

if (associated(dst)) call dev_free(dst)                  ! 1. free device buffer
call transpose_array_alloc(bb=bb, ij=ij, a=src, t=srct)  ! 2. allocate FULL HOST COPY
call dev_alloc(fptr_dev=dst, ubounds=ubound(srct), lbounds=lbound(srct), ierr=ierr)
call dev_memcpy_to_device(dst=dst, src=srct)             ! 4. copy

Defect 1 — transient host-memory doubling (suspected trigger)

Step 2 allocates a second full-size host copy of j_vec. Between steps 2 and 4 the process holds j_vec and srct simultaneously.

j_vec is (3, 1-ngc:ni+ngc, 1-ngc:nj+ngc, 1-ngc:nk+ngc, nb, nc) (src/app/prism/common/adam_prism_coil_object.F90:134), so the transient scales as 3·(ni+2ngc)·(nj+2ngc)·(nk+2ngc)·nb·nc·8 bytes — a clean "grows with grid dimensions, crosses a threshold, dies" signature that is host RAM, not VRAM. That is consistent with the reported threshold sitting far below the VRAM limit.

Why cluster-only is plausible: WSL is one rank against a large flat pool; a cluster node runs N ranks sharing node RAM, multiplying the same per-rank transient.

transpose_array_alloc uses a plain allocate with no stat=, so exhaustion aborts inside a FUNDAL routine bracketed by device calls — which reads as a device fault.

This is a hypothesis consistent with the code and the reported symptom. It is not confirmed against a trace.

Defect 2 — ierr is discarded everywhere, converting OOM into SIGSEGV

dev_alloc's failure path sets fptr_dev => null() and returns FUNDAL_ERR_FPTR_DEV_NOT_ALLOCATED (fundal_dev_alloc_agnostic.INC:34-37). But:

• fundal_dev_assign_agnostic.INC:493 captures ierr into a local and never tests it, then unconditionally calls dev_memcpy_to_device on the possibly-null pointer.
• adam_prism_fnl_coil_object.F90:129-132 passes ierr=ierr four times and never tests it.
• The same holds at every dev_alloc call site in the FNL backend.

So any device allocation failure becomes a null-pointer acc_memcpy_to_device — a SIGSEGV inside the assign, with no diagnostic. This is the mechanism that makes an out-of-memory condition present as "segfault in FUNDAL assign", and it is why the crash carries no useful information.

This defect is independent of Defect 1 and worth fixing regardless: it is the difference between a silent segfault and a labelled failure with a byte count.

Defect 3 — the dev_alloc at :132 is wasted work

initialize allocates j_vec_gpu (adam_prism_fnl_coil_object.F90:132) and immediately calls copy_cpu_gpu (:133), whose assign branch frees that fresh buffer and reallocates it. Not a leak — dev_assign_to_device frees before allocating — but it is a full-size device alloc+free of the largest coil array on every init, and the declared lbounds=[1,1-ngc,1-ngc,1-ngc,1,1] is silently discarded in favour of lbound(srct) from the transposed host array. If those bound sets ever disagree, the device layout is whatever the assign chose, not what :132 declared.

Proposed fix

Route the coil transfer through the buffered path, mirroring the q-field at adam_prism_fnl_object.F90:319:

1. Allocate a dedicated buf_6D_R8P in allocate_gpu alongside buf_5D_R8P. It cannot reuse buf_5D_R8P — wrong rank and wrong extents; reusing a mismatched staging buffer is exactly the fWLayer failure mode already fixed under Inter-realm 1:1 mirror seam leaks div(B) on source-free pulse (same-resolution, refinement-divergent — distinct from #29) #31.
2. Store db6/hb6 as type components next to the existing db5/hb5 (:306-309).
3. Pass buf6D from adam_prism_fnl_object.F90:328 and :349.
4. Drop the now-redundant dev_alloc at adam_prism_fnl_coil_object.F90:132, or keep it and drop the assign — not both.

This removes the per-call host allocation entirely and makes the transfer allocation-free after init.

Independently: check ierr after every dev_alloc in the FNL backend.

Also noted in this file

• buf4D is declared in both copy_cpu_gpu (:41) and copy_gpu_cpu (:78) and referenced in neither body — dead parameter.
• A_gpu/f_gpu/phase_gpu are allocated lbounds=[0], ubounds=[nc] (:129-131) and filled by unbounded dev_memcpy_to_device (:60-62). The host arrays are 0:total_coils_number (adam_prism_coil_object.F90:130-132), so this is consistent — but by coincidence, not construction. Nothing asserts it.

Open question (decides priority vs the teardown issue)

Was the failing cluster case multi-realm? Per-realm device state is allocated N times and never freed (see the companion teardown issue), while initialize_prism:370 divides the device budget by realms_number. If the cluster manifest uses more realms than the local box, that is a second, independent path to the same symptom.

Adding the ierr check distinguishes the two immediately: Defect 1 fails in a host allocate; the multi-realm path fails in acc_malloc with a reportable byte count.

Verification plan (when the cluster returns)

1. Add ierr checks to every FNL dev_alloc and re-run — converts the segfault into a labelled error in one run.
2. Print bytes and the host high-water mark immediately before the coil assign.
3. Run under compute-sanitizer. Per Inter-realm 1:1 mirror seam leaks div(B) on source-free pulse (same-resolution, refinement-divergent — distinct from #29) #31, the WSL free/total memory print is /dev/dxg garbage and must not be used to rule out a device-side cause.

[szaghi]

Companion issue: #34 (no device-memory teardown on any FNL type). If the failing cluster case was multi-realm, #34's per-realm leak is a second, independent path to this symptom — and adding the ierr check proposed here distinguishes them in one run: this issue's suspected trigger fails in a host allocate, #34's fails in acc_malloc with a reportable byte count.