feno

Anomaly detector specifications, written as OpenSLO-shaped YAML in git, compiled into the alert rules your stack already evaluates, and replayed over history before they ship.

The name is for phenology, the study of when recurring natural events happen. Seasonality is the whole problem: a metric is not anomalous because it is high, it is anomalous because it is high for a Tuesday at 03:00.

Status: three commands run against Prometheus

feno suggest drafts a detector from an SLI somebody already wrote, which is the cheapest useful thing in the tool: a team keeping SLOs in git has already said where the metrics live and which query matters, so the first detector costs them a review rather than an authoring session.

feno suggest --from-sli slis/ -f specs/ --url http://mimir/prometheus -o detectors/

It profiles each SLI's series over three weeks and decides from what it measured: seasonality from the autocorrelation at a day and at a week, strategy from how far the standard deviation sits above the interquartile scale, class from the names in the query. Every one of those lines, and the evidence behind it, is a comment in the draft, so the first thing a reviewer reads is the argument rather than the conclusion. window, minCoverage and minWidth are left absent so feno's own defaults apply, because a number in a draft that nothing measured is a number nobody should trust.

A ratio SLI contributes its total, the traffic, not the ratio. The burn alert already watches the ratio against the objective; what no ratio can see is its own denominator collapsing while every remaining request succeeds. A series too sparse, too spiky or too flat to band is refused with the measurement behind it, and a run where nothing was worth drafting exits 1.

sloRef is set when exactly one SLO in the tree names the SLI. Two SLOs over one SLI is a link no draft can resolve, so the field is left out and the candidates go in the header for a person to pick from. Nothing in the drafting path calls out to a model.

feno backtest replays a detector over real history and prints the report. It needs a URL and read access, writes nothing, and exits 1 when a detector is over the noise budget in feno.yaml, so it can gate a pull request. A runner that cannot reach the datasource exits 3 instead, which says the budget was never checked rather than that it held.

Every request carries a deadline, --request-timeout, defaulting to 30 seconds, so a server that accepts a connection and never answers ends the run instead of hanging the job. It is per request rather than per run: a run that gave up halfway reported some detectors and not others, and the code for a datasource nobody reached would be a claim about the whole run that is not true of it. Ctrl-C keeps the reports already printed and exits 3 for the same reason. What each exit code means, and the workflow to copy, is Running the gate in CI.

feno backtest -f detectors/ --url http://mimir/prometheus --from -30d

The budget lives in feno.yaml beside the specifications, because a limit somebody can drop by editing the workflow that checks it is not a limit. One default for the repository, and a detector that needs its own says so in a line:

targets:
  - engine: prometheus
    output: rules/anomaly

budget:
  maxEpisodes: 4
  maxFiringRatio: 0.02
  detectors:
    checkout-request-rate:
      maxEpisodes: 12

feno generate compiles each specification into a rule file, one file per specification, carrying a provenance header naming the tool version, the specification and a hash of it. --check writes nothing and fails when the committed files are not what the specifications compile to, which is the drift gate a pull request runs offline.

--url probes that server once for the optional functions it was started with, so the band is compiled against what the server has rather than against a flag somebody had to know the answer to. Without it the compilation is offline and --experimental-functions decides, which is what keeps the drift gate working on a fork pull request with no path to production.

Each detector is held to the strictest limit in the files above it, from the path -f names down to its own directory, so a repository with a team per directory gives each team a feno.yaml it owns and nobody can loosen the ceiling above them:

alerting/
  feno.yaml                   # the ceiling. maxEpisodes: 4
  teams/checkout/
    feno.yaml                 # maxEpisodes: 2, and may not say 12
    detectors/
  teams/search/
    feno.yaml
    detectors/

$ feno backtest -f alerting/ --url http://mimir/prometheus --from -14d
checkout-request-rate: 10 paging episodes, limit 2 (alerting/teams/checkout/feno.yaml)
search-latency: 10 paging episodes, limit 4 (alerting/feno.yaml)

A finding names the file that set the limit it broke, because in a repository with a budget per team that file is often not the one the author is looking at. -f walks the tree, and -config names one file instead and reads nothing else.

A detector that has to stay over budget for now says so with a waiver, which is the third way through a pull request and the one that does not outlive the incident:

budget:
  maxEpisodes: 4
  detectors:
    checkout-request-rate:
      waiver:
        reason: black friday traffic shape, retuning after the freeze
        until: 2026-12-05

One waiver covers both limits on that detector, both fields are required, and a run it covers exits 0 with the finding still printed:

$ feno backtest -f alerting/ --url http://mimir/prometheus --from -14d
waived: checkout-request-rate: 10 paging episodes, limit 4 (alerting/teams/checkout/feno.yaml)
the waiver in alerting/teams/checkout/feno.yaml runs out on 2026-12-05T00:00:00Z: black friday traffic shape, retuning after the freeze

Past the date it is a finding of its own and the build fails, whether or not the detector is still over budget, so a waiver is renewed in front of a reviewer or deleted. Unlike a limit, a waiver is not merged with the files above it: the nearest one wins, so a team can waive a ceiling it does not own for a detector it does. Nothing in feno.yaml waives generate --check or promtool, which are correctness rather than noise.

A detector that names the SLO it protects gets the number the report exists for: how much of its paging coincided with that SLO's error budget actually being spent.

apiVersion: anomaly.openslo.io/v1alpha
kind: AnomalyDetector
metadata:
  name: checkout-request-rate
spec:
  sloRef: checkout-availability    # an SLO in the same tree
  indicator:
    metadata:
      name: checkout-requests
    spec:
      thresholdMetric:
        metricSource:
          metricSourceRef: mimir-prod
          spec:
            query: sum(rate(http_requests_total{service="checkout"}[5m]))
  detection:
    strategy: adaptive
    seasonality: [daily, weekly]
    class: requests
    sensitivity: balanced
  alerting:
    for: 10m

  paging episodes       2, plus 0 flaps that crossed the band without reaching for
  paged wall clock      3h  (12.50% of the reported window)
  budget overlap        0 of 2 paging episodes coincided with the budget burning
                        0s of 3h paged wall clock  (0.00%)
  linked SLO            checkout-availability, objective 99.9%
  budget burning        120 of 10081 instants: 0 absent, 0 undefined

That report says delete the detector, and it says it without ambiguity: the budget did burn over the window, and none of the pages landed on it. The burn signal is compiled from the SLO's own objective and indicator, so nothing has to be deployed and no recording rule has to be named a particular way. A detector with no sloRef says it has none rather than printing a zero, which is a different thing.

-o is the target's output in feno.yaml when it is left off, read relative to the file that declared it.

feno generate -f detectors/ -o rules/anomaly/
feno generate -f detectors/ -o rules/anomaly/ --check
feno generate -f detectors/ -o rules/anomaly/ --url http://mimir/prometheus

Package	What it does
internal/spec	Reads a directory of OpenSLO and detector YAML, resolves each detector's datasource and SLO references, and turns a detector into a band
internal/prometheus	Reads a range query from anything speaking /api/v1/query_range, chunked under the server's resolution limit and stitched back together, and probes one instant query for the optional functions the server has
internal/detect	The band: one structured representation, every target compiles it and the backtest evaluates it
internal/episode	Turns band breaches into the episodes a person would have been paged for, matching how Prometheus runs for, and intersects those episodes with another timeline
internal/rules	Compiles a band into Prometheus recording and alerting rules, and a linked SLO into the burn query the budget overlap is read from, refusing what PromQL cannot express rather than approximating it
internal/config	Reads feno.yaml: the repository's targets and its noise budget, with a limit per detector where one default will not do and a waiver that expires where a detector has to stay over it
internal/suggest	Profiles an SLI's series over a stated window, decides the fields nobody can fill in from the document alone, and renders the draft a person reviews
internal/backtest	Ties the others together: the query range a band's lookback demands, the second read the budget overlap is an intersection with, the aggregation across the series one query returns, and the report a person reads

Specification to band to episodes to report is backtest, and specification to band to rules is generate. SLI to profile to draft is suggest, which is where a specification comes from in the first place. What is missing is the ClickHouse target.

spec.md is the source of truth for intended behaviour and carries the research behind it. Read it before proposing a design change.

The problem

Anomaly detection on metrics is standard practice and barely adopted, and the algorithms were never the reason. Z-score, MAD, STL decomposition and Random Cut Forest all work well enough on operational metrics. Tuning them is the work, and keeping them tuned is the work after that.

Nothing in the open source tool chain answers the question an engineer has before merging a detector, which is "how often would this have paged me last month". Prometheus can unit test a rule against fixtures it was handed. Nothing replays a candidate rule over real history and reports what it would have done.

What it will do

Three commands over one specification.

feno suggest drafts that specification from an SLI already in the repository, profiling the series it names and writing down the evidence behind every field it decided. It is an authoring aid and not part of the pipeline: the draft is reviewed, edited and committed, and the committed document is the only source of truth.

feno generate compiles a detector into recording and alerting rules for an engine you already run. Prometheus, Mimir, Thanos and the stores that speak PromQL first. ClickHouse, through clickhouse-ruler, after that. No new service, no daemon, no anomaly score to take on trust. The output is rules a person can read while holding a pager.

feno backtest replays a detector over history and reports what it would have done: separate paging episodes, total firing time, the longest one, which series caused them, and how much of it coincided with an error budget actually burning. It writes nothing and needs only read access, so it is useful before anybody agrees to deploy anything. A detector that links to no SLO is told apart from one whose pages never landed on a burning budget, because the second is a finding and the first is a missing line in a file.

Why the specification is OpenSLO's

Because a team that already keeps SLOs in git has already declared the two hard things. A DataSource says where the metrics live and how to connect. An SLI says which query matters. feno reuses both rather than inventing a second way to say them, which means the cheapest useful command is deriving a detector from an SLI that already exists.

Anyone complying with the specification benefits, whatever store is underneath.

Build order

1. Specification types, the loader, and a test proving oslo validate still passes on the OpenSLO half of a mixed directory
2. The band representation, with an explicit model of what each target can and cannot express, which is internal/detect
3. The Prometheus target, with golden tests against grafana/promql-anomaly-detection and promtool in CI
4. backtest over /api/v1/query_range, with the episode report
5. generate, which is the command over internal/rules
6. The budget overlap, which is a second read of the same window against the burn signal compiled from the SLO a detector links to
7. suggest, which is where the specification the other five steps read comes from, and which could not be built earlier: a draft is only worth having if the thing it drafts can be backtested, so the backtester had to exist first
8. The ClickHouse target, which depends on four changes in clickhouse-ruler

Steps 2 and 4 came before step 1 because both are pure functions over numbers and a time range, so they could be pinned by tests without a specification format to argue about first.

Prometheus is first on purpose. It is the restricted dialect: mad_over_time exists but is experimental and off unless the server was started with --enable-feature=promql-experimental-functions, and there is no seasonal decomposition at all. Building against that forces the capability model to be honest about the deployment rather than the dialect. ClickHouse has seriesDecomposeSTL and seriesOutliersDetectTukey built in and would have let that stay implicit.

Development

just          # list recipes
just check    # what CI runs: lint, build, tests, promtool, markdownlint
just test     # unit tests with -race
just smoketest # the capability probe against two real Prometheus servers
just backtest-history # the backtest against a month of generated history
just init     # mise tool versions and pre-commit hooks

go run ./cmd/feno version
go run ./cmd/feno backtest -f detectors/ --url http://localhost:9090 --from -30d

The tests that need a real Prometheus are behind a build tag and say what they want in their own skip message.

FENO_PROMETHEUS_URL=http://localhost:9090 go test -tags integration ./internal/backtest/

License

Apache-2.0.