[feature] Node Overlay Influence Tracking via Node Labels

# [feature] Node Overlay Influence Tracking via Node Labels

## Problem Statement

When Veneer creates NodeOverlays to influence Karpenter provisioning, there is **no record** of which nodes were actually launched under that influence. This makes it impossible to:

1. Validate that NodeOverlays are having the intended effect
2. Track how many nodes were provisioned as on-demand due to RI/SP availability
3. Distinguish between "on-demand because of overlay" vs "on-demand for other reasons"
4. Build dashboards correlating overlay activity with actual node provisioning outcomes

### The Gap

```
Veneer creates NodeOverlay  -->  Karpenter sees modified pricing  -->  Node launches
                                                                            |
                                                    No record of WHY it was chosen
```

Neither Karpenter nor Veneer currently tracks "this node was launched because a NodeOverlay influenced the decision."

## Proposed Solution

Implement an **optional Node Watcher** in Veneer that:

1. Watches for newly created on-demand nodes using Kubernetes informers (event-driven)
2. Determines if any active NodeOverlay matches the node's characteristics
3. Labels the node with overlay influence metadata
4. Exposes metrics for tracking labeled nodes

### Design Goals

| Goal | Approach |
|------|----------|
| **Optional** | Disabled by default, opt-in via config |
| **Modular** | Separate package, clean interfaces, independent of metrics reconciler |
| **Performant** | Event-driven using informers, not polling; minimal API calls |
| **Non-blocking** | Node creation is never blocked; labels added post-creation |

---

## Architecture

### Event-Driven Node Watching

Use controller-runtime's informer/watch mechanism instead of polling:

```go
// pkg/nodewatcher/controller.go

// NodeWatcher watches for Node events and labels nodes influenced by overlays
type NodeWatcher struct {
    client.Client
    Log     logr.Logger
    Config  *config.Config
    Scheme  *runtime.Scheme

    // Cache overlay data to avoid repeated API calls
    overlayCache *OverlayCache
}

// Reconcile is triggered by Node create/update events
func (r *NodeWatcher) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Event-driven: only called when nodes change
}

// SetupWithManager sets up the controller with the Manager
func (r *NodeWatcher) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&corev1.Node{}).
        WithEventFilter(r.nodeFilter()).  // Filter to only on-demand nodes
        Complete(r)
}
```

### Why Event-Driven vs Polling

| Polling (30s interval) | Event-Driven (Informers) |
|------------------------|--------------------------|
| Always runs every 30s | Only runs when nodes change |
| Lists all nodes each cycle | Receives individual events |
| Higher API load at scale | Minimal API calls |
| Latency up to 30s | Sub-second latency |
| Simpler implementation | Slightly more complex |

For a cluster with high node churn, event-driven is significantly more performant.

---

## Implementation Plan

### Phase 1: Configuration & Package Structure

**File:** `pkg/config/config.go`

Add configuration for node watcher:

```go
// NodeWatcherConfig controls the optional node labeling feature
type NodeWatcherConfig struct {
    // Enabled controls whether the node watcher is active
    // Default: false (opt-in feature)
    Enabled bool `yaml:"enabled,omitempty"`

    // LabelNamespace is the prefix for node labels
    // Default: "veneer.io"
    LabelNamespace string `yaml:"labelNamespace,omitempty"`

    // OverlayCacheTTL controls how long overlay data is cached
    // Default: 60s
    OverlayCacheTTL time.Duration `yaml:"overlayCacheTTL,omitempty"`

    // MaxConcurrentReconciles limits concurrent node processing
    // Default: 5
    MaxConcurrentReconciles int `yaml:"maxConcurrentReconciles,omitempty"`
}
```

**File structure:**
```
pkg/nodewatcher/
├── controller.go       # Main controller implementation
├── controller_test.go  # Unit tests
├── cache.go           # Overlay cache for performance
├── cache_test.go
├── filter.go          # Event filtering logic
├── filter_test.go
├── labels.go          # Label constants and helpers
└── metrics.go         # Node watcher specific metrics
```

### Phase 2: Overlay Cache

Cache overlay data to avoid querying the API on every node event:

**File:** `pkg/nodewatcher/cache.go`

```go
package nodewatcher

import (
    "sync"
    "time"

    karpenterv1alpha1 "sigs.k8s.io/karpenter/pkg/apis/v1alpha1"
)

// OverlayCache caches NodeOverlay data for efficient lookups
type OverlayCache struct {
    mu       sync.RWMutex
    overlays map[string]*CachedOverlay
    ttl      time.Duration
}

// CachedOverlay holds overlay data with metadata for matching
type CachedOverlay struct {
    Name           string
    CapacityType   string
    InstanceFamily string
    InstanceType   string
    Region         string
    Requirements   []corev1.NodeSelectorRequirement
    CachedAt       time.Time
}

// NewOverlayCache creates a new cache with the specified TTL
func NewOverlayCache(ttl time.Duration) *OverlayCache {
    return &OverlayCache{
        overlays: make(map[string]*CachedOverlay),
        ttl:      ttl,
    }
}

// Refresh updates the cache from the API
func (c *OverlayCache) Refresh(ctx context.Context, client client.Client) error {
    overlayList := &karpenterv1alpha1.NodeOverlayList{}
    if err := client.List(ctx, overlayList,
        client.MatchingLabels{overlay.LabelManagedBy: overlay.LabelManagedByValue}); err != nil {
        return fmt.Errorf("failed to list overlays: %w", err)
    }

    c.mu.Lock()
    defer c.mu.Unlock()

    // Clear old entries
    c.overlays = make(map[string]*CachedOverlay)

    for _, ov := range overlayList.Items {
        c.overlays[ov.Name] = &CachedOverlay{
            Name:           ov.Name,
            CapacityType:   ov.Labels[overlay.LabelCapacityType],
            InstanceFamily: ov.Labels[overlay.LabelInstanceFamily],
            InstanceType:   ov.Labels[overlay.LabelInstanceType],
            Region:         ov.Labels[overlay.LabelRegion],
            Requirements:   ov.Spec.Requirements,
            CachedAt:       time.Now(),
        }
    }

    return nil
}

// FindMatchingOverlay returns the overlay that matches the given node, if any
func (c *OverlayCache) FindMatchingOverlay(node *corev1.Node) *CachedOverlay {
    c.mu.RLock()
    defer c.mu.RUnlock()

    for _, cached := range c.overlays {
        if c.nodeMatchesOverlay(node, cached) {
            return cached
        }
    }
    return nil
}

// nodeMatchesOverlay checks if a node satisfies all overlay requirements
func (c *OverlayCache) nodeMatchesOverlay(node *corev1.Node, overlay *CachedOverlay) bool {
    for _, req := range overlay.Requirements {
        if !nodeMatchesRequirement(node, req) {
            return false
        }
    }
    return true
}
```

### Phase 3: Event Filter

Filter events to only process relevant nodes:

**File:** `pkg/nodewatcher/filter.go`

```go
package nodewatcher

import (
    "sigs.k8s.io/controller-runtime/pkg/event"
    "sigs.k8s.io/controller-runtime/pkg/predicate"
    corev1 "k8s.io/api/core/v1"
)

const (
    // KarpenterCapacityTypeLabel is the label Karpenter uses for capacity type
    KarpenterCapacityTypeLabel = "karpenter.sh/capacity-type"

    // KarpenterNodePoolLabel indicates Karpenter-managed nodes
    KarpenterNodePoolLabel = "karpenter.sh/nodepool"
)

// nodeFilter returns predicates that filter node events
func (r *NodeWatcher) nodeFilter() predicate.Predicate {
    return predicate.Funcs{
        CreateFunc: func(e event.CreateEvent) bool {
            node, ok := e.Object.(*corev1.Node)
            if !ok {
                return false
            }
            return r.shouldProcessNode(node)
        },
        UpdateFunc: func(e event.UpdateEvent) bool {
            // Only process updates if the node wasn't already labeled
            newNode, ok := e.ObjectNew.(*corev1.Node)
            if !ok {
                return false
            }
            // Skip if already labeled
            if _, exists := newNode.Labels[r.labelKey("overlay-influenced")]; exists {
                return false
            }
            return r.shouldProcessNode(newNode)
        },
        DeleteFunc: func(e event.DeleteEvent) bool {
            return false // Never process deletes
        },
        GenericFunc: func(e event.GenericEvent) bool {
            return false
        },
    }
}

// shouldProcessNode determines if a node should be processed
func (r *NodeWatcher) shouldProcessNode(node *corev1.Node) bool {
    // Must be a Karpenter-managed node
    if _, exists := node.Labels[KarpenterNodePoolLabel]; !exists {
        return false
    }

    // Must be on-demand (spot nodes are never overlay-influenced)
    capacityType := node.Labels[KarpenterCapacityTypeLabel]
    if capacityType != "on-demand" {
        return false
    }

    // Skip if already labeled by us
    if _, exists := node.Labels[r.labelKey("overlay-influenced")]; exists {
        return false
    }

    return true
}

// labelKey returns the full label key with namespace
func (r *NodeWatcher) labelKey(suffix string) string {
    ns := r.Config.NodeWatcher.LabelNamespace
    if ns == "" {
        ns = DefaultLabelNamespace
    }
    return ns + "/" + suffix
}
```

### Phase 4: Main Controller

**File:** `pkg/nodewatcher/controller.go`

```go
package nodewatcher

import (
    "context"
    "fmt"
    "time"

    "github.com/go-logr/logr"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller"

    "veneer/pkg/config"
    "veneer/pkg/overlay"
)

const (
    DefaultLabelNamespace       = "veneer.io"
    DefaultOverlayCacheTTL      = 60 * time.Second
    DefaultMaxConcurrentReconciles = 5
)

// NodeWatcher watches for Node events and labels nodes influenced by overlays
type NodeWatcher struct {
    client.Client
    Log          logr.Logger
    Config       *config.Config
    Scheme       *runtime.Scheme
    overlayCache *OverlayCache
}

// Reconcile processes a single node event
func (r *NodeWatcher) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := r.Log.WithValues("node", req.Name)

    // Get the node
    node := &corev1.Node{}
    if err := r.Get(ctx, req.NamespacedName, node); err != nil {
        if client.IgnoreNotFound(err) != nil {
            return ctrl.Result{}, err
        }
        // Node was deleted, nothing to do
        return ctrl.Result{}, nil
    }

    // Double-check filters (in case of race conditions)
    if !r.shouldProcessNode(node) {
        return ctrl.Result{}, nil
    }

    // Ensure overlay cache is fresh
    if err := r.refreshCacheIfNeeded(ctx); err != nil {
        log.Error(err, "Failed to refresh overlay cache")
        // Requeue to try again
        return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
    }

    // Find matching overlay
    matchingOverlay := r.overlayCache.FindMatchingOverlay(node)
    if matchingOverlay == nil {
        // No overlay matched - this on-demand node wasn't influenced by Veneer
        log.V(1).Info("No matching overlay found for on-demand node")

        // Label as NOT influenced (for completeness)
        if err := r.labelNodeNoOverlay(ctx, node); err != nil {
            log.Error(err, "Failed to label node as not overlay-influenced")
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, nil
    }

    // Label the node with overlay information
    if err := r.labelNodeWithOverlay(ctx, node, matchingOverlay); err != nil {
        log.Error(err, "Failed to label node with overlay information")
        return ctrl.Result{}, err
    }

    log.Info("Labeled node with overlay information",
        "overlay", matchingOverlay.Name,
        "capacity-type", matchingOverlay.CapacityType)

    // Update metrics
    NodeLabeledTotal.WithLabelValues(
        matchingOverlay.CapacityType,
        matchingOverlay.InstanceFamily,
    ).Inc()

    return ctrl.Result{}, nil
}

// labelNodeWithOverlay adds overlay influence labels to a node
func (r *NodeWatcher) labelNodeWithOverlay(
    ctx context.Context,
    node *corev1.Node,
    overlay *CachedOverlay,
) error {
    patch := client.MergeFrom(node.DeepCopy())

    if node.Labels == nil {
        node.Labels = make(map[string]string)
    }

    // Core labels
    node.Labels[r.labelKey("overlay-influenced")] = "true"
    node.Labels[r.labelKey("matching-overlay")] = overlay.Name
    node.Labels[r.labelKey("capacity-type")] = overlay.CapacityType
    node.Labels[r.labelKey("labeled-at")] = time.Now().UTC().Format(time.RFC3339)

    // Additional metadata if available
    if overlay.InstanceFamily != "" {
        node.Labels[r.labelKey("instance-family")] = overlay.InstanceFamily
    }
    if overlay.InstanceType != "" {
        node.Labels[r.labelKey("instance-type")] = overlay.InstanceType
    }
    if overlay.Region != "" {
        node.Labels[r.labelKey("region")] = overlay.Region
    }

    return r.Patch(ctx, node, patch)
}

// labelNodeNoOverlay marks a node as NOT influenced by any overlay
func (r *NodeWatcher) labelNodeNoOverlay(ctx context.Context, node *corev1.Node) error {
    patch := client.MergeFrom(node.DeepCopy())

    if node.Labels == nil {
        node.Labels = make(map[string]string)
    }

    node.Labels[r.labelKey("overlay-influenced")] = "false"
    node.Labels[r.labelKey("labeled-at")] = time.Now().UTC().Format(time.RFC3339)

    return r.Patch(ctx, node, patch)
}

// refreshCacheIfNeeded updates the overlay cache if it's stale
func (r *NodeWatcher) refreshCacheIfNeeded(ctx context.Context) error {
    if r.overlayCache.IsStale() {
        return r.overlayCache.Refresh(ctx, r.Client)
    }
    return nil
}

// SetupWithManager sets up the controller with the Manager
func (r *NodeWatcher) SetupWithManager(mgr ctrl.Manager) error {
    // Initialize the overlay cache
    ttl := r.Config.NodeWatcher.OverlayCacheTTL
    if ttl == 0 {
        ttl = DefaultOverlayCacheTTL
    }
    r.overlayCache = NewOverlayCache(ttl)

    // Configure controller options
    maxConcurrent := r.Config.NodeWatcher.MaxConcurrentReconciles
    if maxConcurrent == 0 {
        maxConcurrent = DefaultMaxConcurrentReconciles
    }

    return ctrl.NewControllerManagedBy(mgr).
        For(&corev1.Node{}).
        WithEventFilter(r.nodeFilter()).
        WithOptions(controller.Options{
            MaxConcurrentReconciles: maxConcurrent,
        }).
        Complete(r)
}
```

### Phase 5: Node Watcher Metrics

**File:** `pkg/nodewatcher/metrics.go`

```go
package nodewatcher

import (
    "github.com/prometheus/client_golang/prometheus"
    "sigs.k8s.io/controller-runtime/pkg/metrics"
)

const metricsNamespace = "veneer"

var (
    // NodeLabeledTotal counts nodes labeled with overlay information
    NodeLabeledTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Namespace: metricsNamespace,
        Subsystem: "nodewatcher",
        Name:      "nodes_labeled_total",
        Help:      "Total nodes labeled with overlay influence information",
    }, []string{"capacity_type", "instance_family"})

    // NodeLabelingErrorsTotal counts labeling failures
    NodeLabelingErrorsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Namespace: metricsNamespace,
        Subsystem: "nodewatcher",
        Name:      "labeling_errors_total",
        Help:      "Total errors when attempting to label nodes",
    }, []string{"error_type"})

    // NodesInfluencedGauge tracks current count of overlay-influenced nodes
    NodesInfluencedGauge = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Namespace: metricsNamespace,
        Subsystem: "nodewatcher",
        Name:      "nodes_influenced_current",
        Help:      "Current number of nodes with overlay-influenced=true label",
    }, []string{"capacity_type"})

    // OverlayCacheRefreshTotal counts cache refreshes
    OverlayCacheRefreshTotal = prometheus.NewCounter(prometheus.CounterOpts{
        Namespace: metricsNamespace,
        Subsystem: "nodewatcher",
        Name:      "cache_refresh_total",
        Help:      "Total overlay cache refreshes",
    })

    // ReconcileDuration tracks reconciliation latency
    ReconcileDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
        Namespace: metricsNamespace,
        Subsystem: "nodewatcher",
        Name:      "reconcile_duration_seconds",
        Help:      "Duration of node watcher reconciliation",
        Buckets:   prometheus.DefBuckets,
    })
)

func init() {
    metrics.Registry.MustRegister(
        NodeLabeledTotal,
        NodeLabelingErrorsTotal,
        NodesInfluencedGauge,
        OverlayCacheRefreshTotal,
        ReconcileDuration,
    )
}
```

### Phase 6: Integration in main.go

**File:** `cmd/main.go`

```go
import (
    "veneer/pkg/nodewatcher"
)

func main() {
    // ... existing setup ...

    // Conditionally enable node watcher
    if cfg.NodeWatcher.Enabled {
        setupLog.Info("Node watcher enabled")

        nodeWatcher := &nodewatcher.NodeWatcher{
            Client: mgr.GetClient(),
            Log:    ctrl.Log.WithName("node-watcher"),
            Config: cfg,
            Scheme: mgr.GetScheme(),
        }

        if err := nodeWatcher.SetupWithManager(mgr); err != nil {
            setupLog.Error(err, "unable to create node watcher controller")
            os.Exit(1)
        }
    } else {
        setupLog.Info("Node watcher disabled (set nodeWatcher.enabled=true to enable)")
    }

    // ... rest of main.go ...
}
```

---

## Labels Applied to Nodes

When a matching overlay is found:

| Label | Example Value | Description |
|-------|---------------|-------------|
| `veneer.io/overlay-influenced` | `true` | Node was provisioned with overlay influence |
| `veneer.io/matching-overlay` | `cost-aware-compute-sp-m5` | Name of the matching overlay |
| `veneer.io/capacity-type` | `compute_savings_plan` | Type of RI/SP that triggered overlay |
| `veneer.io/instance-family` | `m5` | Instance family from overlay |
| `veneer.io/region` | `us-west-2` | Region from overlay |
| `veneer.io/labeled-at` | `2026-01-12T15:30:00Z` | When the label was applied |

When no overlay matches (on-demand but not influenced):

| Label | Value |
|-------|-------|
| `veneer.io/overlay-influenced` | `false` |
| `veneer.io/labeled-at` | timestamp |

---

## Configuration

```yaml
# config.yaml

nodeWatcher:
  # Enable the node watcher (default: false)
  enabled: true

  # Label namespace prefix (default: "veneer.io")
  labelNamespace: "veneer.io"

  # How long to cache overlay data (default: 60s)
  overlayCacheTTL: "60s"

  # Max concurrent node reconciliations (default: 5)
  maxConcurrentReconciles: 5
```

Environment variables:
```bash
VENEER_NODE_WATCHER_ENABLED=true
VENEER_NODE_WATCHER_LABEL_NAMESPACE=veneer.io
VENEER_NODE_WATCHER_OVERLAY_CACHE_TTL=60s
VENEER_NODE_WATCHER_MAX_CONCURRENT_RECONCILES=5
```

---

## Example Queries

### kubectl

```bash
# List all nodes influenced by overlays
kubectl get nodes -l veneer.io/overlay-influenced=true

# List nodes by capacity type
kubectl get nodes -l veneer.io/capacity-type=compute_savings_plan

# Count influenced nodes
kubectl get nodes -l veneer.io/overlay-influenced=true --no-headers | wc -l
```

### Prometheus/Grafana

```promql
# Total nodes labeled with overlay influence
veneer_nodewatcher_nodes_labeled_total

# Current influenced nodes by capacity type
veneer_nodewatcher_nodes_influenced_current

# Labeling error rate
rate(veneer_nodewatcher_labeling_errors_total[5m])

# Using kube-state-metrics for node labels
count(kube_node_labels{label_veneer_io_overlay_influenced="true"})

# Influenced nodes by instance family
count by (label_veneer_io_instance_family) (
  kube_node_labels{label_veneer_io_overlay_influenced="true"}
)
```

---

## Important Caveats

### 1. Heuristic, Not Causal

This is a **heuristic** - we determine "an active overlay matched this node's characteristics at the time it was created." We cannot prove Karpenter chose this instance *because of* the overlay.

Karpenter does not expose which overlays influenced its decisions.

### 2. Race Condition Window

There's a small window between node creation and labeling where the label won't exist. Queries should account for this:

```promql
# Include recently created nodes that haven't been labeled yet
count(kube_node_labels{label_karpenter_sh_capacity_type="on-demand"})
- count(kube_node_labels{label_veneer_io_overlay_influenced=~"true|false"})
```

### 3. Overlay Changes

If an overlay is deleted/updated after a node is labeled, the node's labels remain unchanged. The labels represent "state at provisioning time."

---

## Acceptance Criteria

### Phase 1: Configuration & Structure
- [ ] Add `NodeWatcherConfig` to `pkg/config/config.go`
- [ ] Add config validation (defaults, bounds checking)
- [ ] Create `pkg/nodewatcher/` package structure
- [ ] Add CLI flags for node watcher configuration

### Phase 2: Overlay Cache
- [ ] Implement `OverlayCache` with TTL-based refresh
- [ ] Add cache metrics (refresh count, staleness)
- [ ] Unit tests for cache operations
- [ ] Benchmark cache performance with 100+ overlays

### Phase 3: Event Filter
- [ ] Implement `nodeFilter()` predicate
- [ ] Filter for Karpenter-managed nodes only
- [ ] Filter for on-demand capacity type only
- [ ] Skip already-labeled nodes
- [ ] Unit tests for all filter conditions

### Phase 4: Controller
- [ ] Implement `Reconcile()` method
- [ ] Implement `SetupWithManager()` with options
- [ ] Handle node not found (deleted during reconcile)
- [ ] Handle overlay cache refresh failures gracefully
- [ ] Unit tests with mock client

### Phase 5: Metrics
- [ ] Add `veneer_nodewatcher_*` metrics
- [ ] Register metrics with controller-runtime registry
- [ ] Add reconcile duration histogram

### Phase 6: Integration
- [ ] Register controller in `cmd/main.go` conditionally
- [ ] Add E2E tests for full flow
- [ ] Update Helm chart with new config options
- [ ] Update documentation

### Validation
- [ ] Deploy to dev1-test with `nodeWatcher.enabled=true`
- [ ] Verify labels appear on new on-demand nodes within seconds
- [ ] Verify spot nodes are NOT labeled
- [ ] Verify nodes without matching overlay get `overlay-influenced=false`
- [ ] Verify metrics are exposed and accurate
- [ ] Load test: 100 concurrent node creates

---

## Testing

### Unit Tests

```go
// pkg/nodewatcher/controller_test.go

func TestNodeWatcher_ShouldProcessNode(t *testing.T) {
    tests := []struct {
        name     string
        labels   map[string]string
        expected bool
    }{
        {
            name: "on-demand karpenter node",
            labels: map[string]string{
                "karpenter.sh/nodepool":      "default",
                "karpenter.sh/capacity-type": "on-demand",
            },
            expected: true,
        },
        {
            name: "spot node",
            labels: map[string]string{
                "karpenter.sh/nodepool":      "default",
                "karpenter.sh/capacity-type": "spot",
            },
            expected: false,
        },
        {
            name: "already labeled",
            labels: map[string]string{
                "karpenter.sh/nodepool":      "default",
                "karpenter.sh/capacity-type": "on-demand",
                "veneer.io/overlay-influenced": "true",
            },
            expected: false,
        },
        {
            name: "non-karpenter node",
            labels: map[string]string{
                "node.kubernetes.io/instance-type": "m5.xlarge",
            },
            expected: false,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            node := &corev1.Node{
                ObjectMeta: metav1.ObjectMeta{Labels: tt.labels},
            }
            w := &NodeWatcher{Config: &config.Config{}}
            result := w.shouldProcessNode(node)
            assert.Equal(t, tt.expected, result)
        })
    }
}

func TestOverlayCache_FindMatchingOverlay(t *testing.T) {
    // Test overlay requirement matching logic
}
```

### E2E Tests

```go
// test/e2e/nodewatcher_test.go

var _ = Describe("Node Watcher", Ordered, func() {
    BeforeAll(func() {
        // Enable node watcher in config
    })

    It("should label on-demand nodes with matching overlay", func() {
        By("creating a NodeOverlay for m5 instances")
        overlay := createTestOverlay("m5")

        By("creating an on-demand m5 node")
        node := createTestNode("test-node", map[string]string{
            "karpenter.sh/nodepool":           "default",
            "karpenter.sh/capacity-type":      "on-demand",
            "node.kubernetes.io/instance-type": "m5.xlarge",
        })

        By("waiting for node to be labeled")
        Eventually(func(g Gomega) {
            err := k8sClient.Get(ctx, client.ObjectKeyFromObject(node), node)
            g.Expect(err).NotTo(HaveOccurred())
            g.Expect(node.Labels["veneer.io/overlay-influenced"]).To(Equal("true"))
            g.Expect(node.Labels["veneer.io/matching-overlay"]).To(Equal(overlay.Name))
        }, 30*time.Second, 1*time.Second).Should(Succeed())
    })

    It("should NOT label spot nodes", func() {
        // ...
    })

    It("should label on-demand nodes without overlay as not influenced", func() {
        // ...
    })
})
```

---

## Performance Considerations

### Event-Driven Efficiency

Using informers means we only process nodes when they're created/updated, not on a polling interval. For a cluster with 1000 nodes and 10 node creates/hour:

| Approach | API Calls/Hour |
|----------|----------------|
| Polling (30s) | 120 List + patches |
| Event-driven | 10 Get + patches |

### Cache TTL Tuning

The overlay cache prevents querying overlays on every node event. Default 60s TTL balances freshness with API load.

If overlays change frequently, reduce TTL. If overlays are stable, increase TTL.

### Concurrent Reconciles

Default of 5 concurrent reconciles handles burst node creation. Increase for clusters with high node churn.

---

## References

- [Controller-Runtime Controllers](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller)
- [Predicate Filters](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate)
- [Karpenter Node Labels](https://karpenter.sh/docs/concepts/nodeclaims/)
- Related: Veneer Metrics Issue (see veneer-metrics-issue.md)

---

## Dependencies

- Requires Veneer metrics implementation (for consistent metric patterns)
- No external dependencies beyond controller-runtime

---

**Priority:** Medium - valuable for validation but not blocking core functionality
**Complexity:** Medium - event-driven controller with caching
**Risk:** Low - opt-in, additive feature, no changes to core overlay logic


Label	Example Value	Description
`veneer.io/overlay-influenced`	`true`	Node was provisioned with overlay influence
`veneer.io/matching-overlay`	`cost-aware-compute-sp-m5`	Name of the matching overlay
`veneer.io/capacity-type`	`compute_savings_plan`	Type of RI/SP that triggered overlay
`veneer.io/instance-family`	`m5`	Instance family from overlay
`veneer.io/region`	`us-west-2`	Region from overlay
`veneer.io/labeled-at`	`2026-01-12T15:30:00Z`	When the label was applied

Goal	Approach
Optional	Disabled by default, opt-in via config
Modular	Separate package, clean interfaces, independent of metrics reconciler
Performant	Event-driven using informers, not polling; minimal API calls
Non-blocking	Node creation is never blocked; labels added post-creation

Polling (30s interval)	Event-Driven (Informers)
Always runs every 30s	Only runs when nodes change
Lists all nodes each cycle	Receives individual events
Higher API load at scale	Minimal API calls
Latency up to 30s	Sub-second latency
Simpler implementation	Slightly more complex

Label	Value
`veneer.io/overlay-influenced`	`false`
`veneer.io/labeled-at`	timestamp

Uh oh!

[feature] Node Overlay Influence Tracking via Node Labels #28

Description

[feature] Node Overlay Influence Tracking via Node Labels

Problem Statement

The Gap

Proposed Solution

Design Goals

Architecture

Event-Driven Node Watching

Why Event-Driven vs Polling

Implementation Plan

Phase 1: Configuration & Package Structure

Phase 2: Overlay Cache

Phase 3: Event Filter

Phase 4: Main Controller

Phase 5: Node Watcher Metrics

Phase 6: Integration in main.go

Labels Applied to Nodes

Configuration

Example Queries

kubectl

Prometheus/Grafana

Important Caveats

1. Heuristic, Not Causal

2. Race Condition Window

3. Overlay Changes

Acceptance Criteria

Phase 1: Configuration & Structure

Phase 2: Overlay Cache

Phase 3: Event Filter

Phase 4: Controller

Phase 5: Metrics

Phase 6: Integration

Validation

Testing

Unit Tests

E2E Tests

Performance Considerations

Event-Driven Efficiency

Cache TTL Tuning

Concurrent Reconciles

References

Dependencies

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions