lanl · awadell1 · Jul 30, 2026 · Jul 30, 2026 · Jul 30, 2026 · Jul 30, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,9 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## Changed
+### Added
+
+- Added optional SciFM background tasks with configurable Docket storage and bounded Nomad queue admission ([#7])
+
+### Breaking
+
+- Migrated to the FastMCP v4 beta and MCP 2, with FastMCP owning MCP telemetry and Nomad retaining domain metrics and child spans ([#7])
+
+### Changed
 
 - Enabled [OTEL](https://opentelemetry.io/) logging within the [nomad demo image](ghcr.io/lanl/nomad). Added a docker-compose with a reference observability stack ([#9]).
+- Changed `max_pending_per_tool` to a finite, non-nullable limit and added derived Docket worker concurrency controls ([#7])
 
 ## [v0.2.0] - 2026-07-29
 
@@ -43,5 +52,6 @@ Initial Public Release of Nomad
 [#4]: https://github.com/lanl/nomad/pull/4
 [#5]: https://github.com/lanl/nomad/pull/5
 [#6]: https://github.com/lanl/nomad/pull/6
+[#7]: https://github.com/lanl/nomad/pull/7
 [#8]: https://github.com/lanl/nomad/pull/8
 [#9]: https://github.com/lanl/nomad/pull/9
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 <p align="center">
-  <img src="assets/icon.png" alt="Nomad icon" width="96">
+  <img src="https://raw.githubusercontent.com/lanl/nomad/main/assets/icon.png" alt="Nomad icon" width="96">
 </p>
 
 # Nomad
@@ -10,9 +10,9 @@ PyTorch-backed scientific tools over the [Model Context Protocol
 typed Python callables inside a code-execution sandbox.
 
 [Documentation site](https://lanl.github.io/nomad) |
-[Getting started](docs/guides/getting-started.md) |
-[Model builder guide](docs/guides/model-builder.md) |
-[Reference](docs/reference/index.md)
+[Getting started](https://lanl.github.io/nomad/guides/getting-started.html) |
+[Model builder guide](https://lanl.github.io/nomad/guides/model-builder.html) |
+[Reference](https://lanl.github.io/nomad/reference/index.html)
 
 ## What Nomad does
 
@@ -52,20 +52,20 @@ npx @modelcontextprotocol/inspector -- \
 ```
 
 For the HTTP workflow and matching [URSA](https://github.com/lanl/ursa) configs, see
-[Starting a Nomad server](docs/guides/getting-started.md#starting-a-nomad-server)
+[Starting a Nomad server](https://lanl.github.io/nomad/guides/getting-started.html#starting-a-nomad-server)
 and
-[Connect to a hosted Nomad server](docs/guides/getting-started.md#connect-to-a-hosted-nomad-server).
+[Connect to a hosted Nomad server](https://lanl.github.io/nomad/guides/getting-started.html#connect-to-a-hosted-nomad-server).
 
 ## Start here by task
 
 | If you want to... | Start here |
 | --- | --- |
-| Connect to a running Nomad server | [Getting started](docs/guides/getting-started.md) |
-| Host a new SciFM | [Model builder guide](docs/guides/model-builder.md) |
-| Use Nomad for Inference | [Nomad inference notebook](docs/guides/nomad_inference.ipynb) |
-| Browse CLI, config, and API docs | [Reference](docs/reference/index.md) |
-| Run the demo deployment | [Deployments](docs/deployments/index.md) |
-| Work on Nomad itself | [Developer docs](docs/guides/developer.md) |
+| Connect to a running Nomad server | [Getting started](https://lanl.github.io/nomad/guides/getting-started.html) |
+| Host a new SciFM | [Model builder guide](https://lanl.github.io/nomad/guides/model-builder.html) |
+| Use Nomad for Inference | [Nomad inference notebook](https://lanl.github.io/nomad/guides/nomad_inference.html) |
+| Browse CLI, config, and API docs | [Reference](https://lanl.github.io/nomad/reference/index.html) |
+| Run the demo deployment | [Deployments](https://lanl.github.io/nomad/deployments/index.html) |
+| Work on Nomad itself | [Developer docs](https://lanl.github.io/nomad/guides/developer.html) |
 
 ## Development
 

diff --git a/container/demo/pyproject.toml b/container/demo/pyproject.toml
@@ -27,6 +27,7 @@ where = ["src"]
 nomad-scifm = { path = "../..", editable = true }
 
 [tool.uv]
+prerelease = "allow"
 override-dependencies = [
     "nomad-scifm[otel]",
 ]

diff --git a/container/demo/uv.lock b/container/demo/uv.lock
diff --git a/docs/reference/api-tool-manager.md b/docs/reference/api-tool-manager.md
@@ -9,6 +9,26 @@ GPU memory can be shared across a larger set of models.
 It also queues requests, batches calls for tools that support batching, and can
 register the managed tools with FastMCP.
 
+FastMCP background-task workers feed requests into this queue; they do not
+schedule accelerator work themselves. Nomad derives concurrency by adding the
+total per-tool pending capacity to the
+device pipeline capacity: `number_of_tools * max_pending_per_tool + devices *
+max_batch_size * device_queue_depth`. It then applies
+{py:attr}`~nomad.config.ToolManagerConfig.task_min_concurrency` as a floor.
+`max_pending_per_tool` defaults to 65,536. Docket limits each SciFM task to 75%
+of that capacity before invoking Nomad, leaving headroom for foreground calls;
+each admitted background task also atomically reserves a real Nomad queue slot.
+If no slot is available, Docket reschedules the task without removing it from
+the queue. The Nomad limit remains a defensive boundary that returns a `Server
+busy` tool error. The device queue depth and minimum default to two and ten,
+respectively. This keeps enough work available for batching without moving GPU
+scheduling out of the tool manager.
+
+Docket retains a completed or failed task's result payload for 15 minutes by
+default. This lifetime starts when the task completes or fails, not when it is
+submitted; for example, a task that runs for an hour still has its result
+available for approximately 15 minutes after it finishes.
+
 For managed Torch tools, the manager keeps one resident CPU instance when the
 tool is loaded from configuration and may create additional tool instances from
 the configured source when a busy tool is assigned to more than one device slot.

diff --git a/docs/reference/config.md b/docs/reference/config.md
@@ -20,7 +20,10 @@ tool_manager:
   idle_seconds: 120
   gc_idle_seconds: 300
   disk_idle_seconds: 600
-  max_pending_per_tool: 50000
+  max_pending_per_tool: 65536
+  task_min_concurrency: 10
+  device_queue_depth: 2
+  task_backend_url: memory://
 search_tool:
   expose: true
 telemetry:
@@ -94,9 +97,8 @@ The `workspace_root` default controls where sandboxed scripts run. If that
 workspace contains `.venv/bin/python` or `.venv/Scripts/python.exe`, the gateway
 uses it for sandbox execution.
 
-The gateway emits OpenTelemetry metrics and spans for MCP tool entrypoints,
-sandbox runs, and upstream MCP tool calls. See {doc}`otel` for the full metric
-reference.
+FastMCP emits MCP boundary spans, while the gateway adds domain metrics and
+sandbox execution spans. See {doc}`otel` for the full telemetry reference.
 
 Nomad configures OTLP/gRPC trace and metric exporters when `telemetry.enabled`
 is `true`, `NOMAD_OTEL_ENABLED=true`, or standard OTel environment variables

diff --git a/docs/reference/otel.md b/docs/reference/otel.md
@@ -1,8 +1,12 @@
 # OpenTelemetry
 
-Nomad emits OpenTelemetry metrics and spans for model serving and the code-mode
-gateway when telemetry export is enabled. Install the optional dependency extra
-to include the OTel SDK and OTLP/gRPC exporter:
+FastMCP owns MCP tracing and trace-context propagation. Nomad adds domain
+metrics and child spans for work below that MCP boundary, such as code-mode
+sandbox execution. Both use the same process-wide OpenTelemetry provider.
+
+FastMCP instrumentation is active by default and is a no-op until an OTel SDK
+is configured. Install Nomad's optional dependency extra to let Nomad configure
+that SDK with OTLP/gRPC trace and metric exporters:
 
 ```bash
 pip install "nomad-scifm[otel]"
@@ -28,16 +32,28 @@ service name. Set `telemetry.otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` to
 control the collector endpoint. Set `OTEL_SDK_DISABLED=true` to disable SDK
 setup.
 
+Nomad configures a provider only when none exists. If the process was launched
+with `opentelemetry-instrument` or an embedding application already installed a
+provider, Nomad and FastMCP use that provider without replacing it. Nomad only
+flushes providers it configured itself.
+
+Use `FASTMCP_TELEMETRY_MODE` to control FastMCP tracing: `native` (the default)
+creates MCP spans and propagates context, `propagation_only` propagates context
+without MCP spans, and `off` disables both. Nomad's custom spans use FastMCP's
+tracer and therefore follow the same mode.
+
 For OTel environment variable behavior and exporter configuration, see the
 [OpenTelemetry Python documentation](https://opentelemetry.io/docs/languages/python/)
 and
 [Python exporter documentation](https://opentelemetry.io/docs/languages/python/exporters/).
 
 ## Spans
 
-`nomad serve` creates spans around managed Torch tool requests and batch
-execution. The code-mode gateway creates spans for MCP tool entrypoints, sandbox
-runs, and upstream MCP tool calls.
+FastMCP creates the server and client MCP spans, including `tools/call` spans
+for SciFM tools, code-mode entrypoints, and upstream calls. Nomad does not wrap
+those boundaries a second time. The code-mode gateway adds the
+`nomad.gateway.sandbox.run` child span for sandbox execution details; Nomad
+metrics continue to cover entrypoints, sandbox runs, and upstream calls.
 
 ## Model-serving metrics
 

diff --git a/pyproject.toml b/pyproject.toml
@@ -5,9 +5,9 @@ description = "MCP-Server for SciFMs"
 readme = "README.md"
 requires-python = ">=3.12"
 dependencies = [
-    "fastmcp>=3.0.0",
+    "fastmcp[tasks]>=4.0.0b1,<5.0",
     "filelock>=3.13.0",
-    "mcp>=1.20.0,<2.0",
+    "mcp>=2.0.0,<3.0",
     "pydantic>=2.12.0,<3.0",
     "pyyaml>=6.0.3",
     "tantivy>=0.25.0",

diff --git a/src/nomad/_torch_module_compat.py b/src/nomad/_torch_module_compat.py
@@ -4,8 +4,10 @@
 from collections.abc import Callable
 from typing import Any
 
+from docket import ConcurrencyLimit
 from fastmcp import FastMCP
 from fastmcp.tools import FunctionTool
+from fastmcp.utilities.tasks import TaskConfig
 
 
 def _compatible_torch_module_tool_types() -> tuple[type[Any], ...]:
@@ -30,10 +32,22 @@ def build_torch_module_fastmcp_tool(
     tool: Any,
     *,
     invoke: Callable[[Any], Any] | None = None,
+    task_concurrency_limit: int | None = None,
+    task_admission: Any = None,
 ) -> FunctionTool:
     """Build a FastMCP tool from a TorchModuleTool-like object."""
 
-    async def fn(**input_data: Any) -> Any:
+    concurrency = (
+        ConcurrencyLimit(max_concurrent=task_concurrency_limit)
+        if task_concurrency_limit is not None
+        else None
+    )
+
+    async def fn(
+        _task_concurrency: ConcurrencyLimit | None = concurrency,
+        _task_admission: Any = task_admission,
+        **input_data: Any,
+    ) -> Any:
         args = tool.args_schema(**input_data)
         result = invoke(args) if invoke is not None else tool(args)
         if inspect.isawaitable(result):
@@ -46,6 +60,7 @@ async def fn(**input_data: Any) -> Any:
         description=tool.description,
         parameters=tool.args_schema.model_json_schema(),
         output_schema=tool.output_schema.model_json_schema(),
+        task_config=TaskConfig(mode="optional"),
     )
 
 

diff --git a/src/nomad/cli.py b/src/nomad/cli.py
@@ -10,6 +10,7 @@
 import click
 import typer
 from fastmcp import FastMCP
+from fastmcp_tasks import TasksExtension
 from typer.core import TyperCommand
 
 from ._torch_module_compat import add_torch_module_tool_to_fastmcp
@@ -27,7 +28,6 @@
 
 LogLevelName = Literal["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
 ServeTransport = Literal["stdio", "http", "streamable-http", "streamable_http"]
-
 app = typer.Typer(
     no_args_is_help=True,
     help=(
@@ -77,6 +77,22 @@ def _format_visible_devices(manager: Any) -> str:
     return "none"
 
 
+def _background_task_concurrency(
+    manager_config: Any,
+    manager: Any | None,
+    *,
+    number_of_tools: int,
+    max_batch_size: int,
+) -> int:
+    minimum = getattr(manager_config, "task_min_concurrency", 10)
+    device_queue_depth = getattr(manager_config, "device_queue_depth", 2)
+    max_pending_per_tool = getattr(manager_config, "max_pending_per_tool", 2**16)
+    pending_capacity = number_of_tools * max_pending_per_tool
+    device_count = len(getattr(manager, "devices", ())) if manager is not None else 0
+    pipeline_capacity = device_count * max(1, max_batch_size) * device_queue_depth
+    return max(minimum, pending_capacity + pipeline_capacity)
+
+
 def run_code_mode_script(
     config_path: Path,
     script_path: Path,
@@ -244,6 +260,8 @@ def serve(
 
     manager = manager_cfg.instantiate() if use_manager else None
     LOGGER.info("Visible devices: %s", _format_visible_devices(manager))
+    registered_fmod_count = 0
+    max_batch_size = 1
     for fm_config in config.fmod_models:
         fm_name = fm_config.tool_name or fm_config.name_or_path
         try:
@@ -262,6 +280,10 @@ def serve(
             else:
                 add_torch_module_tool_to_fastmcp(server, tool)
 
+            registered_fmod_count += 1
+            tool_batch_size = getattr(tool, "batch_size", 1) or 1
+            max_batch_size = max(max_batch_size, max(1, tool_batch_size))
+
             card_locator.register(
                 tool.name or fm_config.name_or_path,
                 source,
@@ -274,6 +296,21 @@ def serve(
     if manager:
         manager.add_to_fastmcp(server)
 
+    if registered_fmod_count:
+        task_concurrency = _background_task_concurrency(
+            manager_cfg,
+            manager,
+            number_of_tools=registered_fmod_count,
+            max_batch_size=max_batch_size,
+        )
+        LOGGER.info("FastMCP background task concurrency: %s", task_concurrency)
+        server.add_extension(
+            TasksExtension(
+                url=getattr(manager_cfg, "task_backend_url", "memory://"),
+                concurrency=task_concurrency,
+            )
+        )
+
     if config.search_tool.expose:
         register_search_tool(
             server,

diff --git a/src/nomad/config.py b/src/nomad/config.py
@@ -227,8 +227,17 @@ class ToolManagerConfig(BaseModel):
     idle_seconds: float | None = Field(default=300.0, ge=0)
     """Idle seconds before reducing a tool's device allocation by one slot. ``None`` disables device-slot idle eviction."""
 
-    max_pending_per_tool: int | None = Field(default=None, ge=1)
-    """Maximum queued requests per tool. ``None`` disables the queue limit."""
+    max_pending_per_tool: int = Field(default=2**16, ge=1)
+    """Maximum queued requests per tool."""
+
+    task_min_concurrency: int = Field(default=10, ge=1)
+    """Minimum derived FastMCP background-task concurrency."""
+
+    device_queue_depth: int = Field(default=2, ge=1)
+    """Maximum batches per device included in derived task concurrency."""
+
+    task_backend_url: str = "memory://"
+    """Docket backend URL used for FastMCP background tasks."""
 
     max_devices_per_tool: int | None = Field(default=None, ge=1)
     """Maximum device slots one tool may occupy. ``None`` uses all managed slots."""

diff --git a/src/nomad/gateway/middleware/logging.py b/src/nomad/gateway/middleware/logging.py
@@ -39,7 +39,7 @@ async def after_tool(
     ) -> CallToolResult:
         start_ns = ctx.metadata.pop("start_ns", None)
         duration_ms = (time.perf_counter_ns() - start_ns) / 1e6 if start_ns else None
-        payload = getattr(result, "structuredContent", None)
+        payload = result.structured_content
         size = 0
         if payload is not None:
             try: