Compiler: $EXTERN pragma for calling C functions from BASIC (Level 2 FFI)

.github/workflows/ci.yml

@@ -30,6 +30,9 @@ jobs:
       - name: Compiler tests
         run: bash tests/run_compiler_tests.sh
 
+      - name: Cross-language (FFI) test
+        run: bash tests/run_ffi_test.sh
+
   dos-cross-compile:
     runs-on: ubuntu-latest
     steps:

docs/getting-started.md

@@ -229,8 +229,67 @@ gfortran driver.f90 greet.o -L./build -lgwrt -lm -lpthread -lpulse-simple
 
 The BASIC code shares the `gw` interpreter state with `libgwrt`, so a
 single binary runs at most one BASIC program at a time.  Calling BASIC
-from C / Fortran is always safe; calling C / Fortran from BASIC needs
-the foreign-function-declaration extension on the roadmap (Level 2).
+from C / Fortran is always safe; calling C / Fortran *from* BASIC uses
+the `'$EXTERN` pragma described next.
+
+### Foreign Functions from BASIC (`'$EXTERN`)
+
+A `'$EXTERN` pragma declares a C function that compiled BASIC can call
+directly.  It is written as an apostrophe comment, so the interpreter
+ignores it while the compiler picks it up:
+
+```basic
+10 '$EXTERN Cmul(DOUBLE, DOUBLE) AS DOUBLE
+20 '$EXTERN Greet(STRING) AS STRING
+30 '$EXTERN Getn AS INTEGER
+40 PRINT Cmul(2.5, 4)
+50 PRINT Greet("World")
+60 PRINT Getn
+```
+
+Type mapping at the boundary:
+
+| BASIC type | C type        |
+|------------|---------------|
+| `INTEGER`  | `int16_t`     |
+| `SINGLE`   | `float`       |
+| `DOUBLE`   | `double`      |
+| `STRING`   | `const char *` (NUL-terminated) |
+
+The C side supplies the symbols at link time:
+
+```c
+#include <stdint.h>
+int16_t     Getn(void)                  { return 42; }
+double      Cmul(double a, double b)    { return a * b; }
+const char *Greet(const char *who) {
+    static char buf[128];
+    snprintf(buf, sizeof buf, "Hello, %s!", who);
+    return buf;   /* callee owns the buffer; BASIC copies it */
+}
+```
+
+Build and link as in the `--emit-obj` example above:
+
+```bash
+build/gwbasic-compile --emit-obj --runtime . demo.bas   # -> demo.o
+gcc -c lib.c -o lib.o
+gcc demo.o lib.o -L./build -lgwrt -lm -lpthread -lpulse-simple -o demo
+```
+
+Notes and constraints:
+
+- The function name is matched case-insensitively at the call site (BASIC
+  convention) but emitted as the C symbol with the **case written in the
+  pragma**, so `Cmul` calls C's `Cmul`, not `cmul`.
+- Names must be BASIC-legal identifiers (letters and digits) because the
+  call site is tokenized as ordinary BASIC.  To call a C function whose
+  name contains underscores or other characters (e.g. `sqlite3_open`),
+  write a thin C wrapper with a BASIC-legal name.
+- String arguments cross as `const char *` (the compiler converts and frees
+  a temporary copy); a `STRING` return value is copied into the BASIC
+  string pool and the callee retains ownership of its own buffer.
+- For Fortran callees, declare the routine `bind(c)` with a matching name.
 
 ## Building for DOS / FreeDOS
 

docs/roadmap.md

@@ -91,97 +91,45 @@ Cross-compiles to DOS using OpenWatcom V2.  Two targets:
 
 Tested on FreeDOS 1.4 via QEMU.
 
-## Next Up
-
-### Compiler Optimization Flags
-- **`--inline-arrays`** -- emit direct array indexing for statically-DIMmed arrays
-  instead of runtime `gwrt_array_elem()` lookup
-- **`-O0` through `-O3`** -- compiler-level optimization tiers mapping to different
-  sets of codegen optimizations (constant folding, dead code elimination, FOR
-  step=1 elision, fast-path expressions)
-
-### Cross-Language Linking
-
-Three levels of integration with C and Fortran.  Level 1 is implemented;
-Level 2 is the natural follow-up; Level 3 is deferred unless a concrete
-use case appears.
-
-- **Level 1 -- Link BASIC objects into a larger C/Fortran project (done)**
-  -- `gwbasic-compile prog.bas --emit-obj --main-name=run_basic` produces
-  `prog.o` with the entry point renamed.  The host project links it
-  alongside its own objects against `libgwrt`.  From Fortran, declare
-  the entry with `bind(c)`.
-
-- **Level 2 -- Foreign function declarations from BASIC**: extend the
-  language with a `'$EXTERN NAME(ARGS) AS TYPE` pragma (or a new
-  `EXTERNAL` statement) so BASIC code can call C functions directly.
-  Type mapping: `INTEGER` <-> `int16_t`, `SINGLE` <-> `float`,
-  `DOUBLE` <-> `double`, `STRING` <-> `char *` (NUL-terminated, owned
-  by `gw_str_to_cstr`) or `gw_string_t` for richer interop.  Fortran
-  callees must use `bind(c)`; legacy F77/F90 mangling out of scope --
-  users write a thin C shim instead.
-
-- **Level 3 (deferred) -- Embed individual BASIC SUBs/FUNCTIONs as
-  C-callable functions**.  Compile each labeled SUB or DEF FN to a
-  separate C function with a stable signature; emit a header so C
-  drivers can call them.  Useful when BASIC is the configuration /
-  rule-engine language for a larger application.  Bigger scope:
-  needs export annotations, header generation, and a way to share
-  state between calls.  Defer until a specific use case appears.
-
-### IDE Integration
-- **VS Code extension** -- syntax highlighting (TextMate grammar), snippets,
-  run/debug tasks, integrated terminal runner
-- **JetBrains plugin (IntelliJ/CLion)** -- syntax highlighting, code completion,
-  run configurations, debugger integration (breakpoints via `STOP`, variable
-  inspection), structure view (line number outline)
-
-### Formatted I/O Extensions
-
-Beyond the existing `PRINT` / `PRINT USING` / `PRINT#`, expose two
-formatted-I/O styles familiar from neighbouring languages.  Both write
-through the existing HAL output path so they work in interactive mode
-and in compiled binaries.
-
-- **FORTRAN-style `WRITE`** -- `WRITE (#unit, "(format-spec)") args`
-  with Fortran format-spec language: `I5`, `F8.3`, `E12.4`, `A`, `X`,
-  `/` (newline), repeat counts, slashes, parenthesized groups.  Useful
-  for porting numerical code; Fortran formats are denser than
-  PRINT USING.
-- **C-style `PRINTF`** -- `PRINTF format$, arg, arg, ...` accepting C's
-  `%d` / `%f` / `%e` / `%g` / `%s` / `%c` / `%x` / `%o` / width / precision
-  / flags.  Cheaper to learn for users coming from C / Python.  Goes to
-  stdout; `FPRINTF #unit, ...` for file output.
-
-Both share an underlying formatter (probably a C function in
-`libgwrt`) that the codegen calls directly; the interpreter tokenizes
-and dispatches the same way.
-
-### Numerical / Data Standard Library
-
-Substantial scope -- treat as a separate sub-project, possibly a
-companion repo.  All three modules build on top of GW-BASIC arrays
-(or new dynamically-typed buffers via DEF SEG / virtual memory).
-Likely written partly in BASIC and partly in C for the inner loops.
-
-- **NumPy-style array module** -- `NDARRAY` type with shape, dtype,
-  broadcasting; element-wise ops (`+`, `*`, `SIN`, `EXP`); reductions
-  (`SUM`, `MIN`, `MAX`, `MEAN`); slicing; basic linear algebra (`MATMUL`,
-  `INV`, `EIG`).  The existing single-typed BASIC arrays are a starting
-  point; the new module needs a proper shape/dtype descriptor.
-- **DataFrame module (pandas-like)** -- column-oriented table with
-  named columns and heterogeneous dtypes; CSV / TSV load and save;
-  filter, sort, group-by, aggregate, join.  Builds on the array
-  module.
-- **Plotting module (matplotlib-like)** -- high-level wrappers
-  (`PLOT x, y`, `SCATTER`, `BAR`, `HIST`) with axes, labels, legend,
-  title.  Backend: existing CGA / Sixel rendering for terminals; PNG
-  output via stb_image_write or libpng for files; SVG as a third
-  backend that needs no library.  Output format selectable
-  (`SET BACKEND` or per-call argument).
-
-Each module wants its own design pass before implementation; the
-sketches above are the rough shapes.
+### Cross-Language Linking (Levels 1 & 2)
+
+- **Level 1 (v0.17.0)** -- `gwbasic-compile prog.bas --emit-obj
+  --main-name=run_basic` produces `prog.o` with a renamed entry point, so a
+  host C/Fortran project can link BASIC objects alongside its own against
+  `libgwrt`.  From Fortran, declare the entry with `bind(c)`.
+- **Level 2 -- `'$EXTERN` FFI pragma** -- `'$EXTERN NAME(ARGTYPES) AS RET`
+  declares a C function callable from compiled BASIC, with INTEGER/SINGLE/
+  DOUBLE/STRING ⇄ C type coercion at the boundary.  Case-preserving C symbol,
+  BASIC-legal call name.  See *Foreign Functions from BASIC* in
+  getting-started.md; test at `tests/run_ffi_test.sh`.  Arbitrary-C-symbol
+  aliasing and string-result comparison are follow-ups (git-bug).
+
+Level 3 (export BASIC routines as C-callable) remains deferred -- see git-bug.
+
+## Planned
+
+Actionable planned work is tracked in **git-bug** (`git-bug bug`), grouped
+by priority/theme labels rather than duplicated here.  Release and outreach
+items (FreeDOS package, Show HN writeup, etc.) live in git-bug only; this
+file keeps the shipped-feature history and the known limitations.  Current
+dev highlights:
+
+| Theme | Item | git-bug | Priority |
+|-------|------|---------|----------|
+| compiler | `$EXTERN` follow-ups -- aliasing, INSTR/WRITE dispatch, validation | `8329647` | P2 |
+| compiler | `--inline-arrays` direct array indexing | `e6d977c` | P2 |
+| compiler | `-O0..-O3` codegen optimization tiers | `fecc17f` | P2 |
+| compiler | Level 3 -- export BASIC SUBs/FUNCs as C-callable (deferred) | `1b7d59c` | P2 |
+| language | FORTRAN-style `WRITE` formatted I/O | `a6e99af` | P2 |
+| language | C-style `PRINTF` / `FPRINTF` | `cd8750c` | P2 |
+| ide | VS Code extension (+ JetBrains follow-up) | `32a637c` | P2 |
+| stdlib | Numerical/Data stdlib -- NDArray + DataFrame + Plotting (sub-project) | `55a9d14` | P2 |
+
+Recently shipped: Level 2 `'$EXTERN` FFI pragma (`56b96e0`, closed).
+
+Run `git-bug bug show <id>` for the full design notes on any item.  The
+numerical/data stdlib (`55a9d14`) is the main enabler for the
+Jupyter-kernel data-analysis use case.
 
 ## Known Limitations
 

include/analysis.h

@@ -11,13 +11,27 @@
 #define MAX_DATA      4096
 #define MAX_GOSUB_RET 1024
 
+#define MAX_EXTERNS      64
+#define MAX_EXTERN_ARGS  8
+#define EXTERN_NAME_MAX  32
+
 typedef struct {
     uint16_t line_num;
     bool is_target;       /* referenced by GOTO/GOSUB/etc. */
     bool has_data;        /* contains DATA statement */
     int data_start;       /* index into data pool */
 } line_info_t;
 
+/* A C function declared via the '$EXTERN NAME(ARGS) AS RET pragma.
+ * name is stored case-preserving (emitted as the C symbol); matching
+ * against BASIC call sites is case-insensitive. */
+typedef struct {
+    char name[EXTERN_NAME_MAX];
+    gw_valtype_t ret_type;
+    gw_valtype_t arg_types[MAX_EXTERN_ARGS];
+    int argc;
+} extern_func_t;
+
 typedef struct {
     char name[2];
     gw_valtype_t type;
@@ -42,6 +56,9 @@ typedef struct {
     int data_line_count;
 
     gw_valtype_t def_type[26];  /* from DEFINT/DEFSNG/DEFDBL/DEFSTR */
+
+    extern_func_t externs[MAX_EXTERNS];  /* '$EXTERN FFI declarations */
+    int extern_count;
 } analysis_t;
 
 /* Run analysis pass over the loaded program */
@@ -56,6 +73,9 @@ int analysis_add_var(analysis_t *a, const char name[2], gw_valtype_t type);
 /* Check if a line number is a jump target */
 bool analysis_is_target(analysis_t *a, uint16_t line_num);
 
+/* Find a declared extern function by name (case-insensitive), or NULL */
+const extern_func_t *analysis_find_extern(analysis_t *a, const char *name);
+
 /* Emit static analysis warnings to stderr */
 void analysis_warnings(analysis_t *a);