fix: warn when documented columns are missing from the relation in persist_docs

[r-jais]

Ports dbt-labs/dbt-adapters#1684 (closes dbt-labs/dbt-adapters#1690) into dbt-databricks. dbt-databricks overrides persist_docs with its own native handling, so the upstream macro change does not reach this adapter — the warning is added here directly.

Description

When persist_docs.columns is enabled, columns documented in a model's schema.yml that are not present in the relation being commented (typo, renamed column, stale docs) are silently skipped — no comment is applied and no feedback is given.

This PR adds a warning that names those columns on the two column-comment comparison paths, where the adapter diffs documented columns against the relation's actual columns to build the comment ALTER:

• V1 materialization — DatabricksAdapter.get_persist_doc_columns (impl.py), which iterates only the columns that actually exist in the relation.
• V2 materialization — ColumnCommentsConfig.get_diff (relation_configs/column_comments.py), which previously emitted the missing column into the diff (targeting a nonexistent column on the ALTER).

Filtering behavior is unchanged (missing columns are still skipped so the ALTER never errors); the columns are now surfaced via warn_or_error, so the warning respects --warn-error like other adapter warnings.

Example warning:

The following columns are specified in the schema but are not present in the database and will be skipped: col2

Scope

Covered: the two column-comment comparison paths above (V1 get_persist_doc_columns, V2 ColumnCommentsConfig.get_diff). These run when documented columns are diffed against an existing relation to build the comment ALTER:

• V1 — post-write on table and incremental materializations.
• V2 — the relation-config changeset path, reached by incremental (and materialized-view / streaming-table) re-runs. Note a V2 table rebuilds via CREATE OR REPLACE and applies comments inline every run, so it does not go through get_diff — V2 table coverage is the create-time follow-up below.

Out of scope / follow-ups

Deliberately not addressed here to keep the change focused on the comparison paths. Filed as follow-ups:

• Create-time inline comments (parse_columns_and_constraints). This builds inline CREATE/replace comments and never warns. It covers V2 table models (every run) and the first create / --full-refresh of any model. A warning here must be gated explicitly on config.persist_column_docs() — an unconditional warning in parse_columns_and_constraints would fire even when column persistence is disabled, and --warn-error would turn those false positives into failures.
• View create (V1 + V2). get_persist_docs_column_list iterates query columns only, so YAML-only columns stay silent.
• Typed persist gate. ColumnCommentsProcessor gates column comments on persist_docs.relation; it should gate on persist_docs.columns (config.persist_column_docs()).
• V2 with incremental_apply_config_changes: false. Skips get_diff entirely, with no V1-style persist_docs fallback, so no warning is emitted.
• Duplicate warnings (not a problem today). The v1 and v2 warning sites are mutually exclusive per model run (use_materialization_v2 selects one), and each fires once, so a run warns exactly once — asserted in the functional tests. Duplication would only become possible if the warning were later added to an additional helper that can co-fire with these (e.g. create-time parse_columns_and_constraints); at that point a centralized/deduplicated warning would be preferable.

Tests

• Unit — both comparison paths, warn and no-warn branches (tests/unit/test_adapter.py, tests/unit/relation_configs/test_column_comments_config.py).
• Functional (tests/functional/adapter/persist_docs/test_persist_docs.py) — V1 warns and still comments present columns; V2 warns on the changeset path (second run of an incremental model — a table rebuild re-applies comments inline and never reaches get_diff); --warn-error escalates the warning to a run failure. Verified against a live UC SQL warehouse.

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests (unit + functional; see above)
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
impl.py
dbt/adapters/databricks/relation_configs
column_comments.py
Project Total

This report was generated by python-coverage-comment-action

[sd-db]

Changes look good, but the current change just is for the materialisation-v1 path. We need to make similar changes for mat_v2 path as well

[sd-db]

Current changes look good, had a minor comment on changelog but the current coverage is limited to mostly V1 persist_docs + V2 alter/config-diff

[sd-db]

@r-jais — follow-up coverage review of this PR at head a248840e. Sharing so you can decide what to fix here versus document as out of scope / follow-up.

Coverage report: missing-column persist_docs warning

Scenario reviewed: exactly persist_docs: {columns: true} (with relation unset/false) plus a YAML column that is not present on the relation/query.

Overall: PARTIAL

Important config caveat: ColumnCommentsProcessor.from_relation_config currently derives its persist flag from persist_docs.relation, not persist_docs.columns. The matrix below follows the stated columns-only scenario; outcomes that change when both flags are enabled are listed separately.

Matrix — columns: true, relation: false

Materialization	Scenario	Verdict	Path
table	V1 create/replace	WARNS	persist_docs → get_persist_doc_columns
table	V2 create/replace	SILENT	create_table_at → parse_columns_and_constraints
incremental	V1 create/full refresh	WARNS	post-create persist_docs
incremental	V1 subsequent, config changes on	WARNS once	typed diff is gated off; later persist_docs warns
incremental	V1 subsequent, config changes off	WARNS once	config diff is skipped; persist_docs still runs
incremental	V2 create/full refresh	SILENT	create_table_at → parse_columns_and_constraints
incremental	V2 subsequent, config changes on	SILENT	ColumnCommentsConfig.persist is false because relation is false
incremental	V2 subsequent, config changes off	SILENT	no config diff and no persist_docs fallback
view	V1 create/replace	SILENT	get_persist_docs_column_list iterates query columns only
view	V2 create/replace_with_view	SILENT	same shared create helper
view	V2 comment-only alter	SILENT	typed diff is gated off by relation: false
view	V2 query-changing alter/reapply	WARNS once	get_persist_doc_columns after ALTER VIEW AS
snapshot	after materialize	WARNS	persist_docs
clone	table path	WARNS	persist_docs
clone	view fallback	SILENT	shared view-create helper
seed	create/reset	SILENT	iterates agate/query columns only
materialized_view	create/replace/full refresh	SILENT	parse_columns_and_constraints
materialized_view	in-place alter	N/A	no ColumnCommentsProcessor
streaming_table	create/replace/full refresh	SILENT	parse_columns_and_constraints
streaming_table	in-place alter	N/A	no ColumnCommentsProcessor

I removed the headline counts because the earlier matrix combined multiple scenarios and flag combinations, making the totals non-reproducible.

Outcomes when both relation: true and columns: true

These rows change:

• V1 incremental subsequent, config changes on → DOUBLE warning risk. It runs model_config.get_changeset(...) and later persist_docs.
• V2 incremental subsequent, config changes on → WARNS. The typed get_diff gate is now enabled.
• V2 view comment-only alter → WARNS. The typed get_diff gate is now enabled.
• V2 view query-changing alter/reapply → DOUBLE warning risk. Config diff warns, then query reapplication calls get_persist_doc_columns.

The wrong-key gate also has the inverse effect: relation: true, columns: false can warn through the typed diff even though column persistence is disabled. Fixing the gate is therefore needed to avoid both false negatives and false positives.

What this PR already covers well

• V1 table/incremental/snapshot/clone-table paths through get_persist_doc_columns
• Relation-config diff paths when their current persist gate is enabled
• V2 view query reapplication through get_persist_doc_columns
• Filtering missing columns out of the V2 alter diff so no ALTER targets a nonexistent column

Highest-priority gaps to consider

1. V2 table/incremental create-time — parse_columns_and_constraints never warns; first create and full refresh remain silent.
2. View create (V1 + V2) — get_persist_docs_column_list iterates query columns only, so YAML-only columns remain silent.
3. Typed persist gate — ColumnCommentsProcessor should gate column comments on persist_docs.columns, not persist_docs.relation.
4. V2 + incremental_apply_config_changes: false — skips get_diff with no V1-style persist_docs fallback.
5. Duplicate warnings — warning in multiple low-level comparison helpers can emit twice during one model run.

Suggested next steps

• If keeping the current scope: narrow the PR description/CHANGELOG to the comparison points actually covered, and call out typed-create and view-create as follow-ups.
• If expanding scope here: centralize or otherwise deduplicate the missing-column warning; fix the typed gate to use columns; cover view create; and gate any typed-create warning explicitly on config.persist_column_docs().
• Do not add an unconditional warning inside parse_columns_and_constraints: that helper runs even when column persistence is disabled, and --warn-error would turn those false positives into failures.

Happy to clarify any row in the matrix if useful.

[sd-db]

@r-jais — follow-up coverage review of this PR at head a248840e. Sharing so you can decide what to fix here versus document as out of scope / follow-up.

I would also recommend to add functional tests to verify on behaviour as well as there can be cases on Silent/Double warnings so need to balance coverage/correctness