Knn to ANN - #9
Merged
Merged
Conversation
- Introduced CumulativeTriTopic class for cumulative, batch-wise clustering on top of TriTopic. - Added configuration options for reclustering strategies, triggers, and memory management. - Implemented batch processing with automatic reclustering based on novelty detection and manual triggers. - Created StreamingCorpus utility for generating realistic streaming datasets with LSA embeddings. - Developed evaluation metrics for comparing cumulative clustering against full-batch baselines, including ARI, NMI, and keyword overlap. - Added pluggable recluster strategies: global_refit, coreset, and batch_merge. - Enhanced metrics utility with functions for computing ARI, NMI, and Jaccard overlap for keyword stability.
…ove drift detection
- Introduced GPU acceleration for compute-intensive steps using PyTorch, FAISS, and RAPIDS cuML. - Updated README.md to include GPU installation instructions and performance benefits. - Added `min_cluster_fraction` parameter to TriTopicConfig for better scaling with corpus size. - Refactored clustering methods to utilize effective min_cluster_size based on document count. - Implemented GPU-accelerated cosine similarity and embedding refinement functions. - Updated cumulative benchmark notebook to use new configuration options. - Enhanced graph builder to select between FAISS and HNSW backends based on sample size.
… relevant dependencies
- Introduced LLM-based topic alignment in `alignment.py` with functions `_build_align_prompt` and `_parse_alignment_response`. - Added `llm_align_topics` function to facilitate LLM-driven topic alignment, allowing for flexible mapping of new topics to existing global topics. - Updated `CumulativeTriTopic` to support LLM alignment methods, including "cosine", "llm", and "both", with appropriate configuration options. - Implemented stratified coreset selection in `strategies.py` to ensure representation of rare topics, enhancing robustness against tail-collapse. - Added metrics for evaluating rare topic recall in `evaluation.py`, providing insights into the preservation of small topics during cumulative modeling. - Enhanced `LLMLabeler` to support OpenRouter as a provider, allowing for broader compatibility with LLM services.
- Implemented `llm_merge_topics` method in `TriTopic` and `CumulativeTriTopic` classes to semantically merge topics using a large language model (LLM). - Introduced a new module `llm_merger.py` for handling LLM-driven topic merging, including prompt building and response parsing. - Enhanced `LLMLabeler` with structured output capabilities for LLM calls, allowing for better integration with Google and OpenAI APIs. - Added JSON schema enforcement for structured responses from the LLM, improving the reliability of the merging process. - Updated documentation and added warnings for potential stale states in the cumulative model after merging.
- Implemented sensitivity weights for lightweight-coreset sampling, favoring outliers. - Added tests for sensitivity sampling and micro-cluster coreset strategies. - Enhanced README with details on new coreset selection methods: sensitivity and microcluster. - Updated coreset configuration options to include `coreset_sampling` and `reserve_novel_docs`.
Weighted coresets (stratified_coreset/sensitivity_weights) previously only reached small-cluster pruning, centroids, and keyword extraction — the Leiden partition objective itself drew cluster boundaries as if every sampled point represented one document, blunting the point of the provably-bounded sampling distributions upstream. ConsensusLeiden now switches to RBERVertexPartition with node_sizes=node_weights whenever node weights are present, scoped only to that branch so every unweighted fit (including global_refit) keeps using RBConfigurationVertexPartition exactly as before, with zero behavior change. An earlier attempt that scaled edge weights by w_i*w_j instead of switching objectives was discarded after benchmarking showed it amplifies kNN-graph noise near sparse regions and measurably hurt real coreset fidelity (ARI vs. full-batch baseline dropped); the native node_sizes mechanism avoids that failure mode and raised ARI from ~0.88-0.95 to ~0.92-0.99 across seeds on a realistic streaming benchmark.
Adds a third, opt-in path for choosing the Leiden resolution parameter alongside modularity-maximization and binary-search-to-target-count. Follows ClusterLLM (Zhang, Wang & Shang, EMNLP 2023): candidate resolutions are scored by how well their partitions agree with LLM judgments on sampled same/different-cluster document triplets. - tritopic/labeling/llm_granularity.py: triplet sampling, prompt building, tiered-fallback response parsing, and candidate scoring - TriTopic.tune_resolution_with_llm(): callable after fit(), reuses the existing graph/refresh helpers, never invoked automatically - Unit + integration tests with a mocked labeler (no real API calls) - README section documenting the method and its cost profile Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LaDUfD19BrzSkqbv1bfqbx
- Thread node_weights through llm_select_resolution/_partition_at_resolution so candidate scoring uses the same RBER+node_sizes objective as the final weighted consensus re-fit (previously always unweighted RBConfiguration) - Guard tune_resolution_with_llm against models missing graph_/documents_/ embeddings_ (e.g. after save()/load(), which never persists graph_) with a clear error instead of a deep leidenalg crash - Validate n_candidates/batch_size are >= 1 in llm_select_resolution - Drop unused NearestNeighbors distance outputs in _sample_triplets - Laplace-smooth _triplet_agreement so a candidate informative on only one or two triplets can't outrank one informative across many at a slightly lower but more reliable agreement rate - Replace np.argmax's implicit first-index tie-break with an explicit middle-candidate fallback for tied/all-zero scores - Persist the calibrated resolution back to config.resolution and _clusterer.resolution so later resolution-dependent defaults (build_hierarchy, divide, _auto_resolve_topic_count) stay consistent Adds 10 new tests covering each fix; all verified empirically before being asserted (e.g. Leiden partition invariance for the tie-break test). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LaDUfD19BrzSkqbv1bfqbx
Verifies the node_weights forwarding fixed in the previous commit (model.py:1871) actually reaches leidenalg end-to-end as the weighted RBER objective across the full call chain (candidate scoring, per-run consensus, and the co-occurrence re-clustering step), not just as a passed-but-unused kwarg. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LaDUfD19BrzSkqbv1bfqbx
New section 5b (opt-in, off by default via TUNE_RESOLUTION_WITH_LLM) demonstrates tune_resolution_with_llm() as an alternative to manually hand-tuning RESOLUTION. Uses the existing OpenRouter labeler pattern already established in this notebook (OPENROUTER_MODEL defaults to google/gemini-2.5-flash-lite; DeepSeek is equally suitable) rather than Claude, since B-or-C triplet judgments don't need a stronger model. Outputs stripped before commit to match the other notebooks in this repo and avoid committing real dataset content into git history. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LaDUfD19BrzSkqbv1bfqbx
…ation-9c6nrj Add LLM-guided granularity calibration (ClusterLLM-style)
- Added a check for empty content in the LLM labeler, raising a ValueError with a descriptive message if the response is empty. - Updated the LLM merger to issue a warning and return all topics as singletons when the LLM response is empty or consists only of whitespace.
- Introduced a minimum cluster size parameter to suppress small clusters during candidate partitioning, aligning with final fit behavior. - Improved triplet sampling efficiency with a global k-NN query, reducing computational cost. - Added diagnostics to capture detailed metrics during resolution selection, including triplet counts and unparsed responses. - Implemented a two-stage resolution selection process: a coarse sweep followed by a fine grid search around the best candidate. - Enhanced triplet response parsing to handle unparsed responses more robustly, avoiding bias in scoring. - Updated documentation to reflect new features and improvements in the LLM granularity approach.
- Introduced a new feature to allow LLM-guided calibration of the RESOLUTION parameter. - Added markdown explanation for the new feature, detailing its benefits and usage. - Implemented code to utilize the LLM for tuning resolution based on boundary case judgments. - Included diagnostics output to assess the effectiveness of the calibration.
…ation docs New tritopic.adaptation subpackage implementing ClusterLLM-style triplet fine-tuning: sample LLM-judged (anchor, positive, negative) triplets from a fitted model, adapt the embedder to them via a pure-numpy linear transform or a real sentence-transformers fine-tune, and refit TriTopic on the result. Also includes the cheaper Few-Shot-Clustering extras (LLM keyphrase expansion, low-confidence correction) and a compare_embedders evaluation harness for judging whether adaptation actually helped on a given corpus. - tritopic/adaptation/: config, triplet sampling/bank/cache, LinearAdapter + EmbeddingAdapter, evaluation harness, keyphrase/correction extras, adapt_and_refit() pipeline - TriTopic.adapt_embeddings_with_llm(): in-place delegate mirroring tune_resolution_with_llm's ergonomics - benchmarks/adaptation_quality_report.py: oracle-labeler POC report (synthetic + real 20 Newsgroups scenarios) - notebooks/embedding_adaptation_demo.ipynb: executed end-to-end demo/test notebook with inline assertions, real 20NG data - 34 new tests across tests/test_adaptation_*.py - pyproject.toml: new `adaptation` extra (datasets, accelerate) - README: document the new feature; expand tune_resolution_with_llm docs with two-stage/bias-mitigation details and a diagnostics example Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- run_benchmark.py: reproduces the README's TriTopic vs BERTopic/NMF/LDA benchmark table (20ng, BBC News, AG News, Arxiv) with a --quick synthetic smoke-test mode for CI. The file was previously referenced by the README but never committed because a blanket `run*` .gitignore rule silently swallowed it; added a `!run_benchmark.py` exception. - Verify quoted phrases in LLM-generated report-theme narratives against the source documents shown to the LLM (tritopic/utils/quote_verification.py), flagging quotes that can't be traced back to a real document so a fabricated participant quote doesn't silently ship in a qualitative research report. Surfaced via ReportTheme.unverified_quotes and an inline warning in export_report()'s Markdown output. - Add GitHub Actions CI: pytest on push/PR (py3.10/3.12), a fast benchmark harness smoke test, and a manually-triggered full benchmark reproduction job. - Fix README code samples using the nonexistent `n_topics_target` kwarg; the actual TriTopic constructor param is `n_topics`.
…ules
CI (.github/workflows/ci.yml):
- persist-credentials: false on all checkout steps
- add a read-only top-level permissions block
- timeout-minutes on the full-benchmark job
run_benchmark.py:
- embedding cache key now includes sample_seed, so a rerun with a different
subsampling seed can't silently reuse embeddings for a different set of docs
tritopic/adaptation/adapter.py:
- _resolve_mode now runs the same accelerate/datasets/sentence-transformers
version check for explicit adapter_mode="finetune" that the "auto" branch
already ran, raising the same actionable message instead of failing later
with a raw ImportError out of _finetune_sentence_transformer
- EmbeddingAdapter.encode()'s normalize flag is now threaded through to
LinearAdapter.transform() in linear mode instead of being ignored
tritopic/adaptation/correction.py:
- reassign_low_confidence drops any topic that a reassignment empties out
entirely before recomputing centroids/probabilities, instead of averaging
an empty embedding slice (was a real NaN-propagation crash, reproduced and
confirmed fixed with a new regression test)
- clarify batch_size's docstring: it chunks iteration only, doesn't batch
LLM requests (kept the signature as-is; a real multi-doc-per-call rewrite
is a bigger change than this pass warrants)
tritopic/adaptation/keyphrase.py:
- generate_keyphrases' cache loader now skips malformed/partial JSONL lines
instead of aborting the whole read
- keyphrase_expand_embeddings no longer blends in encode("") for documents
with an empty keyphrase list; also merges the doc+keyphrase encode calls
into one, halving request overhead for API-backed encoders
tritopic/adaptation/pipeline.py:
- adapt_and_refit carries the original model's explicit n_topics through to
the refit model instead of silently resetting it to "auto"
- warns when use_metadata_view was enabled, since the fit-time metadata
DataFrame isn't persisted on the model and can't be recovered for the refit
tritopic/core/model.py:
- AdaptationConfig is now imported under TYPE_CHECKING so the annotation on
adapt_embeddings_with_llm resolves for static analysis (was a bare
"name not defined" for mypy/Ruff)
Skipped (not still valid / not worth the risk-to-value tradeoff):
- Renaming/removing correction.py's batch_size outright — no call site or
doc example depends on the "real batching" framing; docstring fix covers
the misleading-cost-expectation concern without an API break.
All 232 tests pass (224 pre-existing + 8 new regression tests), including a
reproduction of the correction.py crash against the pre-fix code to confirm
the new test actually exercises the bug.
HF's datasets/huggingface_hub stopped resolving the legacy script-based "ag_news" repo id (HfUriError: repo id must be namespace/name), breaking the full benchmark CI job. Point at fancyzhx/ag_news, the parquet-based mirror with the same text/label schema.
The manual full-benchmark job (4 datasets x 3 seeds x k-grid x 4 models, including BERTopic) was hitting the 45-minute timeout and getting cancelled before finishing, not failing outright.
…mark-error-juhs58 Update AG News dataset source to fancyzhx/ag_news
…teness-ipnli2 Add LLM-guided embedding adaptation module; polish granularity calibration docs
Mirrors the CSV_PATHS config pattern from challenges_clustering_kaggle.ipynb: set CSV_PATHS (or the CSV_PATHS/CSV_PATH env var) to load your own documents from CSV instead of the built-in 20 Newsgroups demo. An optional LABEL_COL enables the ground-truth oracle self-test in Section 3; without it, that section is skipped and the notebook points to the real-LLM path in Section 4.
…-input-kvq90t Add LLM-guided embedding adaptation module; polish granularity calibration docs
- Added a new notebook `embedding_adaptation_kaggle.ipynb` for real fine-tuning on Kaggle using OpenRouter and `all-MiniLM-L6-v2`. - Updated `README.md` to include links to the new notebook and clarify the adaptation process. - Modified `adapt_and_refit` function to include the trained `EmbeddingAdapter` in the report, allowing for persistence of fine-tuned weights. - Added a test to ensure the `EmbeddingAdapter` can be saved and loaded correctly after adaptation.
- Updated CHEAT_SHEET.md to reflect default memory-safe configuration for large datasets. - Changed README.md to specify file locations in a clearer format. - Adjusted README.md to correct the OOM crash mitigation advice. - Modified __init__.py to reorder exports for better organization. - Enhanced graphweave_full_demo.ipynb by clarifying LLM labeling options and comments. - Improved run_benchmark.py to provide clearer warnings when BERTopic is not installed.
- Updated CHEAT_SHEET.md to clarify memory usage for default consensus_method="graph" and legacy hierarchical path. - Revised README.md to emphasize the memory-safe nature of the default path and provide clearer installation instructions for optional features. - Enhanced VISUAL_GUIDE.md to reflect changes in memory usage timelines and consensus steps for both paths. - Adjusted integration tests to remove unnecessary low_memory=True settings when using the default graph consensus. - Modified model configuration to clarify the effect of low_memory on hierarchical consensus only. - Updated cumulative benchmark notebook to reflect changes in consensus method and memory management.
Introduce the GraphWeave brand and top-level API
… improve error handling in adaptation modules - Updated CI to support Python 3.9 in addition to 3.10 and 3.12. - Clarified installation instructions in README for LLM-guided embedding fine-tuning. - Improved error handling in adapter.py and correction.py to provide more informative warnings. - Enhanced keyphrase.py to include normalization options for embeddings. - Updated triplet sampling logic for better performance and clarity. - Added tests for quote verification and reasoning field handling in LLM labeler.
- Changed co-occurrence matrix accumulation from float32 to int16 to optimize memory usage. - Updated documentation to reflect changes in data types and memory safety. - Modified error handling in `adapt_and_refit` to raise ValueError when metadata is not passed, ensuring metadata view is preserved. - Added tests to verify that metadata is correctly handled during adaptation and refitting processes. - Adjusted installation instructions in notebooks to point to the correct repository branch.
Update embedding adaptation, metrics, CI, and GraphWeave metadata
Add LLM-guided embedding adaptation and report quote verification
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.