Skip to content

Show the result builder as a node in the Hamilton UI - #1678

Open
charitarthchugh wants to merge 5 commits into
apache:mainfrom
charitarthchugh:feature/ResultBuilder-node-in-ui
Open

Show the result builder as a node in the Hamilton UI#1678
charitarthchugh wants to merge 5 commits into
apache:mainfrom
charitarthchugh:feature/ResultBuilder-node-in-ui

Conversation

@charitarthchugh

@charitarthchugh charitarthchugh commented Aug 4, 2026

Copy link
Copy Markdown

Implements #1150

The combined result returned by execute() is assembled by a result builder (DictResult,
PandasDataFrameResult, ...) in do_build_result, after the last node finishes. That runs
outside the dataflow, so there is no node to attach a data summary to. Every individual output
is profiled in the UI; the object the caller actually receives is not.

This adds a synthetic node named _result_builder to stand in for that object, so it gets the
same data observability as any other node. Nothing is needed to enable it — attach a
HamiltonTracker as usual.

Before / after

Hamilton UI Hamilton UI-2

Before: 4 nodes, the task table ends at average_squared. After: 5 nodes, with
_result_builder (typing.Any, success) alongside them. Tags, outputs and duration are
otherwise identical.

nodes with recorded runs
before average_squared, input_numbers, squared, sum_squared
after average_squared, input_numbers, _result_builder, squared, sum_squared

Opening the node shows the object the caller received:

evidence-after-node-summary
_result_builder — Result summary
{ 2 items
  "sum_squared":     int 55
  "average_squared": int 11
}

A PandasDataFrameResult gives a full DataFrame profile in the same view, since the built
result goes through the existing process_result path unchanged.

Dataflow used for the comparison
import pandas as pd

def input_numbers() -> pd.Series:
    return pd.Series([1, 2, 3, 4, 5])

def squared(input_numbers: pd.Series) -> pd.Series:
    return input_numbers**2

def sum_squared(squared: pd.Series) -> float:
    return float(squared.sum())

def average_squared(sum_squared: float, input_numbers: pd.Series) -> float:
    return sum_squared / len(input_numbers)
dr = (
    driver.Builder()
    .with_modules(dag)
    .with_adapters(HamiltonTracker(...), base.DictResult())
    .build()
)
dr.execute(["average_squared", "sum_squared"])

Run twice against the same local UI and project — once with the SDK from main, once with
this branch. Both returned {'average_squared': 11.0, 'sum_squared': 55.0}.

Changes

Five commits, one per layer.

Tracking server. NodeTemplate.classifications is an ArrayField constrained by
TextChoices, not a free-form field, so the SDK cannot send a value the server does not know:

class NodeTemplate(TimeStampedModel):
    class NodeType(models.TextChoices):
        transform = "transform", _("Transform")
        ...
        placeholder = "placeholder", _("Placeholder")

    classifications = ArrayField(models.CharField(choices=NodeType.choices))
  • Add result_builder to those choices.
  • New postgres migration. migrations_sqlite/0001_initial.py is edited in place, matching how
    0002's unique_together is already carried there.

Frontend. The Classification union mirrors that server field, so it has to widen with it.

  • Add result_builder to the union. No styling, icon or filter — the node renders through the
    existing paths.

SDK, node template. Which nodes feed the result varies per run, so a fixed dependency list
on the template would be wrong for any subset run.

  • Declare no dependencies on the template; record the real ones per run.
  • Use typing.Any for the output type, since it varies with the result builder. The real type
    shows up in the per-run summary.
  • Make registration opt-in. Registering the node without also emitting a task run renders it as
    never-executed on every run, which is what the legacy Driver in that module would do.
  • Fold the node into the DAG hash when registered. register_dag_template_if_not_exists matches
    on that hash alone without inspecting the nodes posted with it, so otherwise a template
    registered before this node existed gets reused and runs log against a node it does not have.
  • Skip the node with a warning if the dataflow already has one by that name. NodeTemplate is
    unique on (name, dag_template), so registering both would fail the run.

SDK, task run. The combined result only exists at post_graph_execute time.

  • Log the task run there, in both the sync and async trackers.
  • Build the payload once in a shared, I/O-free helper; the trackers differ only in how they
    reach their client.
  • Log and swallow emission failures. This runs immediately before log_dag_run_end, so an
    exception escaping here would leave a successful run rendering as still-running.

Docs. Add a "The result builder node" subsection to docs/hamilton-ui/ui.rst.

How I tested this

  • pytest ui/sdk/tests/test_driver.py ui/sdk/tests/test_adapters.py -q — 22 passed, 3 skipped.
  • pre-commit run --files <changed files>
  • Ran the suite at each of the five commits, so the branch bisects.
  • End-to-end against a local UI backend with the migration applied, each result confirmed in
    postgres:
Case Result
DictResult Node renders; summary is the combined dict; deps narrowed to the requested outputs
PandasDataFrameResult Full DataFrame profile
materialize() Deps recorded correctly, materializer id excluded
Failed run Nothing emitted; node renders not-executed
Name collision Warning logged, run succeeded, nothing recorded against the user's node

Notes

Dependency narrowing. Driver.materialize asks pre_graph_execute for
final_vars + materializer_vars but hands post_graph_execute only the final_vars slice, so
recording the requested list unfiltered would credit the node with materializers it never saw.
Narrowing is only sound when the result is keyed by node name, so a dataframe or a custom
builder's own dict keeps the full list rather than be credited with nothing.

Not every path sees a built result. Only Driver.execute() does. raw_execute() and
materialize() never call a result builder, and async drivers build the result after the tracker
has logged. The node is still emitted in those cases, summarizing the raw dict of outputs. No
lifecycle hook reports which result builder ran, or whether one ran at all.

The node is part of the DAG hash, so the first tracked run after upgrading registers a new
version of each dataflow. Existing versions and their runs are untouched.

It renders unconnected in the DAG view. The template carries no dependencies by design and
the DAG view builds edges from template dependencies, so the node sits off to one side. Noted in
the docs.

The result is profiled twice. Each output node is already summarized and the combined result
contains those same objects, so expect roughly twice the profiling time and payload.

Not refactored: _result_attribute/_result_attributes duplicate the inline
attribute-shaping already in both trackers' post_node_execute. Folding those together means
editing working code outside this change, so I left it. Happy to do it as a follow-up.

cc @skrawcz

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

The SDK tracker is about to register a synthetic node standing in for the
driver's built result. NodeTemplate.classifications is a constrained choice
field, so the server has to know the value before it can be sent one.

Postgres gets a new migration; the sqlite initial migration is edited in
place, matching how 0002's unique_together is already carried there.
The union is the frontend's mirror of the server's choice field, so it has
to widen with it or the new classification fails to type-check on arrival.

No styling, icon or filter is added for it -- the node renders through the
existing paths like any other classification.
The combined result a driver returns is assembled by a result builder after
the last node finishes, outside the dataflow. There is no node to hang a
data summary on, so what a run actually produced never reached the UI.

Add a synthetic node template standing in for it. The template carries no
dependencies -- which nodes fed the result varies per run, so they are
recorded per run instead -- and typing.Any for its output, since the type
varies with the result builder.

Registering it is opt-in. A caller that registers the node without also
emitting a task run for it would render it as never-executed on every run,
as the legacy Driver in this module would.

The node is folded into the DAG hash when it is registered, because
register_dag_template_if_not_exists matches on that hash alone: without it,
a template registered before this node existed would be reused and runs
would log against a node it does not have.

Underscore-prefixed functions never become nodes, but names that do not
come from a function do -- an external input, a decorator-generated name.
NodeTemplate is unique on (name, dag_template), so a collision would fail
registration outright. Detect it and skip the node with a warning instead:
skipping costs the run its node, failing would cost the user their run.
Register the synthetic node and log a run for it on every successful
tracked run, so the combined result gets the same data observability as
any other node. Both the sync and async trackers do this; the payload is
built once in a shared, I/O-free helper and each sends it its own way.

Dependencies are recorded per run, from the result itself where it can say
what went into it. Driver.materialize asks pre_graph_execute for
final_vars + materializer_vars but hands post_graph_execute only the
final_vars slice, so recording the requested list unfiltered would credit
the node with materializers it never saw. Narrowing is only sound when the
result is keyed by node name, so a dataframe or a custom builder's own
dict keeps the full list rather than be credited with nothing.

Nothing is emitted when the name collided and no template was registered,
or when the run failed -- a failed run leaves the node not-executed, like
any node the run never reached. A builder that returns None still counts
as having run.

Emission failures are logged and swallowed. This runs just before
log_dag_run_end, so an exception escaping here would leave an otherwise
successful run rendering as still-running forever.
Covers what the node is, that nothing is needed to enable it, and the four
things worth knowing: the result is profiled twice, only Driver.execute
sees a real built result, failed runs leave it not-executed, and a name
collision makes the tracker step aside.

Also notes that the node is part of what identifies a DAG version, so the
first tracked run after upgrading registers a new version of each dataflow.
charitarthchugh added a commit to charitarthchugh/su26-ai301-contribution that referenced this pull request Aug 4, 2026
The Hamilton contribution shipped: PR apache/hamilton#1678 is open against
upstream main with five per-layer commits, 15 new tests and a green suite.
The README still described Phase III as in progress and pointed at commit
hashes that no longer exist after the branch was rebuilt.

Rewrites it against the branch as submitted: real commit hashes and file
lists, test names and verified counts (148/7 vs a 133/7 baseline on main),
the end-to-end cases and before/after evidence, and a Pull Request section
with the PR summary, acceptance criteria and a dated maintainer-feedback
log.

Also corrects two things earlier phases got wrong: the frontend touchpoint
is friendlyApi.ts rather than DAGViz.tsx, and the backend enum is
NodeTemplate.NodeType. Documents the two visible history gaps -- the
analysis window waiting on maintainer approval, and the pre-PR rebase that
dates all five commits to one day -- rather than leaving a reader to find
them in the reflog.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant