Skip to content

feat: add LiteLLM as a unified LLM gateway provider#198

Open
RheagalFire wants to merge 1 commit into
cased:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as a unified LLM gateway provider#198
RheagalFire wants to merge 1 commit into
cased:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Summary

Adds LiteLLM as a new LLM provider across all of kit's LLM call paths (Summarizer, PR Review, Agentic Review, Commit Generation, Deep Research, and CLI). LiteLLM acts as a unified gateway that routes to 100+ LLM providers (OpenAI, Anthropic, Azure, AWS Bedrock, Vertex AI, Ollama, Groq, etc.) through a single litellm.completion() interface.

Changes

11 files changed, +458 / -11 lines

File What changed
pyproject.toml Added litellm>=1.80.0,<1.87.0 optional dep group, included in all
src/kit/summaries.py Added LiteLLMConfig dataclass + litellm.completion() branches in summarize_file, summarize_function, summarize_class
src/kit/__init__.py Exported LiteLLMConfig
src/kit/llm_client_factory.py Added create_litellm_client() factory + config/review-config dispatch
src/kit/pr_review/config.py Added LLMProvider.LITELLM enum, litellm/ prefix detection, config loading, example config
src/kit/pr_review/reviewer.py Added _analyze_with_litellm_enhanced() with cost tracking
src/kit/pr_review/agentic_reviewer.py Added _run_agentic_analysis_litellm() with OpenAI-compatible tool calling
src/kit/pr_review/commit_generator.py Added _generate_with_litellm()
src/kit/deep_research.py Added LiteLLMConfig branch in _init_llm_client() and research()
src/kit/cli.py Added LiteLLM API key handling in 3 model-switching locations
tests/test_llm_client_factory.py Added 5 unit tests for LiteLLM client creation

Usage

Python SDK (Summarizer)

from kit import Repository, Summarizer, LiteLLMConfig

config = LiteLLMConfig(
    model="anthropic/claude-sonnet-4-6",  # any litellm-supported model
    api_key="sk-...",                       # optional if set in env
    api_base="http://localhost:4000",       # optional: LiteLLM proxy URL
)

repo = Repository("/path/to/repo")
summarizer = Summarizer(repo, config)
summary = summarizer.summarize_file("src/main.py")

PR Review (config file)

llm:
  provider: litellm
  model: bedrock/anthropic.claude-3-sonnet  # Bedrock via LiteLLM
  # api_key: optional
  # api_base_url: http://localhost:4000     # optional proxy

CLI

# Direct provider routing
kit summarize file src/main.py --model litellm/anthropic/claude-sonnet-4-6

# Via LiteLLM proxy
LITELLM_API_KEY=sk-proxy-key kit summarize file src/main.py --model litellm/openai/gpt-4

Design decisions

  • Lazy import: import litellm happens at call time, not module load. Kit imports cleanly without the optional dep installed
  • drop_params=True: Silently drops provider-unsupported kwargs (e.g., seed on Anthropic). Critical for cross-provider compatibility without per-provider param filtering
  • OpenAI-compatible response parsing: LiteLLM returns ModelResponse with the same .choices[0].message.content shape as OpenAI, so response handling mirrors the existing OpenAI provider
  • Optional dep group: pip install 'cased-kit[litellm]' keeps the base install lightweight
  • Version pin >=1.80.0,<1.87.0: Floor covers all features used (drop_params, tool calling, async). Ceiling avoids surprise breakage from LiteLLM's weekly releases

Tests

Unit tests (25 tests touching LiteLLM paths, all pass)

tests/test_llm_client_factory.py  - 20 passed (5 new LiteLLM tests + 15 existing)
tests/test_summaries.py           - 46 passed
tests/test_pr_review.py           - 63 passed
tests/test_cli_parsing.py         - 27 passed
tests/test_ollama_client.py       - 8 passed

Ruff lint

ruff check src/ tests/ -- All checks passed

Live E2E (Summarizer path)

Model: anthropic/claude-sonnet-4-6 via Azure Foundry endpoint
Input: src/kit/llm_client_factory.py
Output: 810-char structured summary (## Summary, ### Purpose, ### Key Components)
Chain: LiteLLMConfig -> Summarizer.summarize_file() -> litellm.completion() -> Azure Anthropic -> parsed response

Live E2E (DeepResearch path)

Model: anthropic/claude-sonnet-4-6 via Azure Foundry endpoint
Query: "What is the factory design pattern in 2 sentences?"
Output: 684-char research result
Time: 5.88s
Chain: LiteLLMConfig -> DeepResearch.research() -> litellm.completion() -> Azure Anthropic -> ResearchResult

Testing instructions

pip install -e '.[litellm]'

# Summarizer
python -c "
from kit import Repository, Summarizer, LiteLLMConfig
config = LiteLLMConfig(model='openai/gpt-4o-mini', api_key='sk-...')
repo = Repository('.')
print(repo.summarize_file('README.md', config=config))
"

# PR Review
kit review pr --model litellm/openai/gpt-4o-mini <pr-url>

Route to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock,
Vertex AI, Ollama, etc.) through a single interface via LiteLLM.

Adds LiteLLMConfig for Summarizer and DeepResearch, LLMProvider.LITELLM
for PR review, agentic review, and commit generation, plus CLI support.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant