[DRAFT][Feature] Add scalar quantization support for vector extension

vector/docs/scalar_quantization_hnsw.md

@@ -0,0 +1,257 @@
+# Scalar Quantization For HNSW Indexes
+
+This document describes the scalar quantization design for the vector extension HNSW index. The
+implementation supports SQ8 and SQ16 quantized distance evaluation while keeping quantized
+embeddings in Ladybug storage instead of a raw sidecar file.
+
+## Goals
+
+- Reduce memory and distance-computation cost for HNSW search by storing quantized vectors.
+- Keep quantized embeddings transactionally consistent with the base node table and HNSW graph.
+- Support both `cache_embeddings := true` and `cache_embeddings := false`.
+- Preserve existing HNSW graph storage: upper and lower graph layers remain internal rel tables.
+- Keep full-precision rerank optional through `use_full_precision_rerank := true`.
+
+## Non-Goals
+
+- Quantization is not supported for dot-product metrics. It is available for L2, L2SQ, and cosine.
+- The HNSW graph itself is not quantized. Only the embedding representation used for distance
+  evaluation is quantized.
+- The in-memory quantized cache is not the durable source of truth. It is a query-time accelerator.
+
+## User-Facing Configuration
+
+Scalar quantization is enabled through `CREATE_VECTOR_INDEX`:
+
+```cypher
+CALL CREATE_VECTOR_INDEX(
+    'embeddings',
+    'idx',
+    'vec',
+    metric := 'l2',
+    quantization := 'sq8'
+);
+```
+
+Supported values are:
+
+- `quantization := 'sq8'`
+- `quantization := 'sq16'`
+
+When quantization is enabled, query distance evaluation uses the quantized representation unless
+full-precision rerank is enabled. With `use_full_precision_rerank := true`, HNSW candidate
+generation uses quantized distances, then final candidate ordering is recomputed against the
+original full-precision embedding column.
+
+## Physical Storage
+
+Each quantized HNSW index owns an internal node table in addition to its existing graph tables.
+
+For base table id `T` and index name `idx`, the table name is:
+
+```text
+_<T>_<idx>_QEMB
+```
+
+The table schema is:
+
+```cypher
+CREATE NODE TABLE _<T>_<idx>_QEMB (
+    id INT64,
+    valid UINT8,
+    scale FLOAT,
+    norm_sq FLOAT,
+    payload INT8[dimension] | INT16[dimension],
+    PRIMARY KEY (id)
+);
+```
+
+Column meaning:
+
+- `id`: source node offset. The quantized table row identity mirrors the base table offset.
+- `valid`: `1` when the source embedding is non-null and visible to the writer, `0` for null rows.
+- `scale`: per-vector scalar. For L2 and L2SQ this is the reconstruction scale
+  `max_abs / max_quantized_value`. For cosine this is the normalized-dot scale
+  `1 / sqrt(sum(q_i^2))`.
+- `norm_sq`: squared norm of the quantized vector representation. L2 and L2SQ use this in the
+  reconstructed distance formula. Cosine keeps it for metadata compatibility but uses the
+  normalized-dot scale in the hot path.
+- `payload`: SQ8 or SQ16 vector payload.
+
+This table is an internal catalog entry. It is created and dropped with the vector index and is
+not part of the public user schema.
+
+## Catalog And Index Metadata
+
+`HNSWStorageInfo` stores the table ids for all durable index-owned structures:
+
+- upper HNSW graph rel table
+- lower HNSW graph rel table
+- quantized embeddings node table, when quantization is enabled
+
+The quantized table id is appended to the serialized storage info. Deserialization treats it as
+optional so older non-quantized or pre-quantized index metadata remains readable.
+
+## In-Memory Types
+
+The implementation uses three embedding providers:
+
+- `QuantizedInMemEmbeddings`
+  Used while building a new in-memory HNSW graph during `CREATE_VECTOR_INDEX` when embeddings are
+  cached. It quantizes from the input vectors and avoids storage lookups during the hot build loop.
+
+- `TableBackedQuantizedEmbeddings`
+  Reads and writes quantized embeddings through the internal `_QEMB` node table. This is the
+  durable, transactional representation used when populating or maintaining quantized rows. It is
+  also the on-disk query provider when `cache_embeddings := false`. Query-time reads use storage
+  `lookup`/`lookupMultiple`, so hot pages are managed by the normal buffer manager.
+
+- `CachedQuantizedEmbeddings`
+  A reader over `CachedQuantizedColumn`, which materializes `_QEMB` rows into an aligned dense
+  quantized cache for on-disk queries when `cache_embeddings := true`. This mirrors the baseline
+  cached embedding path: storage remains the source of truth, but HNSW neighbor distance
+  evaluation reads contiguous in-memory views.
+
+The durable source of truth is `_QEMB`. With `cache_embeddings := false`, query-time reads stay in
+the storage layer. With `cache_embeddings := true`, `_QEMB` is used to hydrate a transaction-local
+dense cache before serving hot HNSW distance calls.
+
+## Create Index Flow
+
+`CREATE_VECTOR_INDEX` creates all internal storage in one rewrite:
+
+1. Create the upper graph rel table.
+2. Create the lower graph rel table.
+3. If quantization is enabled, create the `_QEMB` node table.
+4. Build the HNSW graph.
+5. Register the `OnDiskHNSWIndex` with `HNSWStorageInfo`, including the quantized table id.
+6. Rebuild quantized rows by scanning the base embedding column and writing one `_QEMB` row per
+   source offset.
+7. Force checkpoint so the index-owned tables are persisted together.
+
+Null embeddings are written as `valid = 0` rows. This preserves the invariant that source offset
+`N` maps to quantized row `N`, even when the source embedding is null.
+
+## Query Flow
+
+When a query uses a quantized index:
+
+1. The query vector is quantized into the same SQ8 or SQ16 format.
+2. `OnDiskHNSWIndex` chooses an embedding provider:
+   - `cache_embeddings := true` uses `CachedQuantizedEmbeddings`, backed by a dense
+     transaction-local `CachedQuantizedColumn` populated from `_QEMB`.
+   - `cache_embeddings := false` uses `TableBackedQuantizedEmbeddings`.
+   - `TableBackedQuantizedEmbeddings::getEmbeddings` batches neighbor vector reads with
+     `_QEMB` `lookupMultiple`, matching the baseline storage-backed access pattern more closely
+     than one point lookup per neighbor.
+3. HNSW graph traversal reads neighbors from the existing graph rel tables.
+4. Distance evaluation reads quantized embedding views from the selected provider.
+5. If full-precision rerank is enabled, the final candidates are rescored using `OnDiskEmbeddings`
+   over the original embedding column.
+
+For quantized on-disk query, `cache_embeddings := true` does not cache full-precision embeddings.
+It caches the quantized representation in memory after hydrating it from durable `_QEMB` rows. If
+full-precision rerank is enabled, the original embedding column is read separately for reranking.
+
+## Distance Evaluation
+
+L2 and L2SQ use scalar reconstruction from the quantized dot product:
+
+```text
+l2sq = scale_l^2 * norm_l + scale_r^2 * norm_r - 2 * scale_l * scale_r * dot(q_l, q_r)
+```
+
+`metric := 'l2'` returns `sqrt(max(0, l2sq))`; `metric := 'l2sq'` returns `max(0, l2sq)`.
+
+Cosine uses a MariaDB-style normalized quantized dot product. During quantization for cosine
+indexes, `scale` is stored as:
+
+```text
+scale = 1 / sqrt(sum(q_i^2))
+```
+
+The hot path then evaluates cosine distance as:
+
+```text
+distance = 1 - clamp(scale_l * scale_r * dot(q_l, q_r), -1, 1)
+```
+
+This avoids computing a denominator from both vector norms for every candidate comparison. Query
+vectors are quantized with the same cosine-specific scale semantics as `_QEMB` rows, so cached,
+table-backed, and in-memory build paths use the same formula.
+
+## Insert, Update, And Finalize
+
+Committed inserts write both graph state and quantized state in the same transaction:
+
+1. Read the inserted embedding from the insert vector.
+2. Insert the vector into the HNSW graph if it is non-null.
+3. Write the matching `_QEMB` row through `TableBackedQuantizedEmbeddings`.
+4. Write `valid = 0` for null embeddings.
+5. Remove any transaction-local derived qemb cache entry.
+
+Updates use the existing HNSW update model: the old table row version becomes invisible and the new
+vector is inserted as a new version. The quantized table follows the inserted version.
+
+`finalize()` catches up rows that were not part of the checkpointed graph. For quantized indexes it
+writes quantized rows only for the offsets it is finalizing. It does not rebuild the full `_QEMB`
+table, which avoids duplicate primary-key writes for rows already maintained by insert.
+
+## Checkpoint And Recovery
+
+Checkpoint includes:
+
+- upper graph rel table
+- lower graph rel table
+- quantized embeddings node table, when present
+
+Because `_QEMB` is a normal internal node table, it uses the same storage, WAL, MVCC, checkpoint,
+and recovery paths as other node tables. This is the main reason the design uses an internal table
+instead of a raw `.qemb` sidecar file.
+
+After reload, `OnDiskHNSWIndex::load` reconstructs the index from serialized storage info and
+loads the quantized table pointer from `quantizedEmbeddingsTableID`.
+
+## Drop Index Flow
+
+`DROP_VECTOR_INDEX` drops all index-owned internal structures:
+
+1. upper graph rel table
+2. lower graph rel table
+3. `_QEMB` node table, when the index config has quantization enabled
+4. index catalog entry
+
+There is no filesystem sidecar cleanup because quantized embeddings are stored through Ladybug
+storage.
+
+## Cache And Buffer Semantics
+
+`cache_embeddings := true` does not mean cache full-precision embeddings for quantized indexes.
+For on-disk quantized query, it creates a transaction-local dense qemb cache from `_QEMB` and uses
+that cache for HNSW distance evaluation. This is analogous to the baseline cached-embedding path,
+but the cached payload is SQ8/SQ16 rather than full-precision floats.
+
+The durable source of truth remains the `_QEMB` table. `TableBackedQuantizedEmbeddings` uses
+transactional `NodeTable` lookup APIs, including `lookupMultiple` for batched neighbor reads, when
+`cache_embeddings := false`. `CachedQuantizedEmbeddings` still checks source-table visibility while
+serving dense cached views when `cache_embeddings := true`.
+
+Within one query execution, local HNSW scan states share a mutex while obtaining the quantized
+embedding provider. This prevents duplicate dense cache construction and keeps storage-backed
+provider construction serialized.
+
+This gives the cached quantized path the same hot-query shape as the regular cached baseline while
+retaining a smaller quantized representation.
+
+## Tradeoffs And Follow-Ups
+
+- `TableBackedQuantizedEmbeddings` uses storage lookups during search. This is correct,
+  transactional, page-aware, and buffer-pool-backed, but it still pays lookup/list-decoding overhead
+  compared with the dense cache used by `cache_embeddings := true`.
+- A dense `CachedQuantizedColumn` avoids storage lookup in the hot path, but it adds cache
+  population cost and transaction-local invalidation requirements.
+- `norm_sq` is currently stored as `FLOAT`, matching the table schema. SQ16 with very high
+  dimensionality may benefit from wider norm storage if precision becomes measurable in recall or
+  ranking tests.
+- Dead HNSW graph edges can still accumulate under the existing delete/update model. Quantization
+  follows existing visibility rules but does not compact the graph.

vector/src/catalog/hnsw_index_catalog_entry.cpp

@@ -40,9 +40,20 @@ std::string HNSWIndexAuxInfo::toCypher(const IndexCatalogEntry& indexEntry,
     auto propertyName = tableEntry->getProperty(indexEntry.getPropertyIDs()[0]).getName();
     auto metricName = HNSWIndexConfig::metricToString(config.metric);
     cypher += std::format("CALL CREATE_VECTOR_INDEX('{}', '{}', '{}', mu := {}, ml := {}, "
-                          "pu := {}, metric := '{}', alpha := {}, efc := {});",
+                          "pu := {}, metric := '{}', alpha := {}, efc := {}",
         tableName, indexEntry.getIndexName(), propertyName, config.mu, config.ml, config.pu,
         metricName, config.alpha, config.efc);
+    if (!config.cacheEmbeddingsColumn) {
+        cypher += ", cache_embeddings := false";
+    }
+    if (config.quantization != QuantizationType::NONE) {
+        cypher += std::format(", quantization := '{}'",
+            HNSWIndexConfig::quantizationToString(config.quantization));
+    }
+    if (config.storeFullPrecisionEmbeddings) {
+        cypher += ", use_full_precision_rerank := true";
+    }
+    cypher += ");";
     return cypher;
 }
 

vector/src/function/create_hnsw_index.cpp

@@ -34,7 +34,7 @@ CreateInMemHNSWSharedState::CreateInMemHNSWSharedState(const CreateHNSWIndexBind
                     ->getTable(bindData.tableEntry->getTableID())
                     ->cast<storage::NodeTable>()},
       numNodes{bindData.numRows}, bindData{&bindData} {
-    storage::IndexInfo dummyIndexInfo{"", "", bindData.tableEntry->getTableID(),
+    storage::IndexInfo dummyIndexInfo{bindData.indexName, "", bindData.tableEntry->getTableID(),
         {bindData.tableEntry->getColumnID(bindData.propertyID)}, {PhysicalTypeID::ARRAY}, false,
         false};
     hnswIndex = std::make_shared<InMemHNSWIndex>(bindData.context, dummyIndexInfo,
@@ -285,15 +285,39 @@ static void finalizeHNSWTableFinalizeFunc(const ExecutionContext* context,
         {columnID}, {PhysicalTypeID::ARRAY},
         hnswIndexType.constraintType == storage::IndexConstraintType::PRIMARY,
         hnswIndexType.definitionType == storage::IndexDefinitionType::BUILTIN};
+    auto storageManager = storage::StorageManager::Get(*clientContext);
     auto upperTableID = upperTableEntry.getSingleRelEntryInfo().oid;
     auto lowerTableID = lowerTableEntry.getSingleRelEntryInfo().oid;
+    auto quantizedEmbeddingsTableID = common::INVALID_TABLE_ID;
+    storage::NodeTable* quantizedEmbeddingsTable = nullptr;
+    if (bindData->config.quantization != QuantizationType::NONE) {
+        auto quantizedEmbeddingsTableName =
+            HNSWIndexUtils::getQuantizedEmbeddingsTableName(nodeTableID, bindData->indexName);
+        auto quantizedEmbeddingsTableEntry =
+            catalog->getTableCatalogEntry(transaction, quantizedEmbeddingsTableName, true)
+                ->ptrCast<catalog::NodeTableCatalogEntry>();
+        quantizedEmbeddingsTableID = quantizedEmbeddingsTableEntry->getTableID();
+        quantizedEmbeddingsTable =
+            storageManager->getTable(quantizedEmbeddingsTableID)->ptrCast<storage::NodeTable>();
+    }
     auto storageInfo = std::make_unique<HNSWStorageInfo>(upperTableID, lowerTableID,
-        index->getUpperEntryPoint(), index->getLowerEntryPoint(), bindData->numRows);
+        quantizedEmbeddingsTableID, index->getUpperEntryPoint(), index->getLowerEntryPoint(),
+        bindData->numRows);
     auto onDiskIndex = std::make_unique<OnDiskHNSWIndex>(context->clientContext, indexInfo,
         std::move(storageInfo), bindData->config.copy());
-    auto storageManager = storage::StorageManager::Get(*clientContext);
     auto nodeTable = storageManager->getTable(nodeTableID)->ptrCast<storage::NodeTable>();
     nodeTable->addIndex(std::move(onDiskIndex));
+    if (bindData->config.quantization != QuantizationType::NONE) {
+        DASSERT(quantizedEmbeddingsTable != nullptr);
+        const auto& columnType = nodeTable->getColumn(columnID).getDataType();
+        const auto typeInfo =
+            columnType.getExtraTypeInfo()->constPtrCast<common::ArrayTypeInfo>();
+        auto quantizedEmbeddings = TableBackedQuantizedEmbeddings(context->clientContext,
+            common::ArrayTypeInfo{typeInfo->getChildType().copy(), typeInfo->getNumElements()},
+            *nodeTable, columnID, *quantizedEmbeddingsTable, bindData->config.quantization,
+            bindData->config.metric, bindData->numRows);
+        quantizedEmbeddings.rebuild(context->clientContext);
+    }
     index->moveToPartitionState(*hnswSharedState->partitionerSharedState);
     transaction->setForceCheckpoint();
 }
@@ -346,8 +370,21 @@ static std::string rewriteCreateHNSWQuery(main::ClientContext& context,
         HNSWIndexUtils::getUpperGraphTableName(tableID, indexName), tableName, tableName);
     query += std::format("CREATE REL TABLE {} (FROM {} TO {}) WITH (storage_direction='fwd');",
         HNSWIndexUtils::getLowerGraphTableName(tableID, indexName), tableName, tableName);
-    std::string params;
     auto& config = hnswBindData->config;
+    if (config.quantization != QuantizationType::NONE) {
+        const auto& columnType =
+            hnswBindData->tableEntry->getProperty(hnswBindData->propertyID).getType();
+        const auto typeInfo =
+            columnType.getExtraTypeInfo()->constPtrCast<common::ArrayTypeInfo>();
+        const auto payloadType =
+            config.quantization == QuantizationType::SQ8 ? "INT8" : "INT16";
+        query += std::format(
+            "CREATE NODE TABLE {} (id INT64, valid UINT8, scale FLOAT, norm_sq FLOAT, payload "
+            "{}[{}], PRIMARY KEY (id));",
+            HNSWIndexUtils::getQuantizedEmbeddingsTableName(tableID, indexName), payloadType,
+            typeInfo->getNumElements());
+    }
+    std::string params;
     params += std::format("mu := {}, ", config.mu);
     params += std::format("ml := {}, ", config.ml);
     params += std::format("efc := {}, ", config.efc);
@@ -356,8 +393,15 @@ static std::string rewriteCreateHNSWQuery(main::ClientContext& context,
     params += std::format("pu := {}, ", config.pu);
     params +=
         std::format("cache_embeddings := {}", config.cacheEmbeddingsColumn ? "true" : "false");
+    if (config.quantization != QuantizationType::NONE) {
+        params += std::format(", quantization := '{}'",
+            HNSWIndexConfig::quantizationToString(config.quantization));
+    }
+    if (config.storeFullPrecisionEmbeddings) {
+        params += ", use_full_precision_rerank := true";
+    }
     auto columnName = hnswBindData->tableEntry->getProperty(hnswBindData->propertyID).getName();
-    if (config.cacheEmbeddingsColumn) {
+    if (config.cacheEmbeddingsColumn && config.quantization == QuantizationType::NONE) {
         query +=
             std::format("CALL _CACHE_ARRAY_COLUMN_LOCALLY('{}', '{}');", tableName, columnName);
     }