Skip to content

feat: add LiteLLM as LLM provider - #60

Open
RheagalFire wants to merge 2 commits into
stepfun-ai:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as LLM provider#60
RheagalFire wants to merge 2 commits into
stepfun-ai:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a new model provider in ask_llm_anything(), enabling access to 100+ LLM providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, etc.) via a single unified SDK
  • When model_provider="litellm", routes through litellm.completion() with drop_params=True instead of the OpenAI client
  • No proxy server needed, uses litellm SDK directly

Changes

  • tools/ask_llm_v2.py - added import litellm at top level, added litellm code path in ask_llm_anything(): when model_provider == "litellm", calls litellm.completion() with drop_params=True
  • model_config.yaml - added litellm provider entry (uses "EMPTY" placeholder following repo convention, litellm falls back to provider env vars)
  • requirements.txt - added litellm>=1.60.0,<2.0.0
  • tests/test_litellm_provider.py - 12 unit tests (all passing)

Proof of work

Unit tests (12/12 passing):

$ python -m pytest tests/test_litellm_provider.py -v
test_litellm_branch_exists PASSED
test_uses_drop_params_true PASSED
test_uses_litellm_completion PASSED
test_imports_litellm_at_top PASSED                                                                                                                                                                                 
test_skips_empty_api_key PASSED
test_skips_empty_api_base PASSED                                                                                                                                                                                   
test_litellm_in_config PASSED
test_litellm_config_has_api_base PASSED
test_litellm_config_has_api_key PASSED                                                                                                                                                                             
test_completion_called_with_drop_params PASSED
test_completion_forwards_api_key PASSED                                                                                                                                                                            
test_completion_response_has_content PASSED
12 passed in 0.04s                                                                                                                                                                                                 

Live E2E (Azure Foundry, Claude):

Model: anthropic/claude-sonnet-4-6                                                                                                                                                                                 
Response: "4"   
Tokens: prompt=20, completion=5
E2E SUCCESS

Example usage

# model_config.yaml - set your key or leave EMPTY to use env vars
litellm:                                                                                                                                                                                                           
    api_base: "EMPTY"
    api_key: "EMPTY"                                                                                                                                                                                               
from tools.ask_llm_v2 import ask_llm_anything

# LiteLLM reads provider env vars automatically
# (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.)
response = ask_llm_anything(                                                                                                                                                                                       
    "litellm",
    "anthropic/claude-sonnet-4-20250514",                                                                                                                                                                          
    [{"role": "user", "content": "Describe the GUI elements on screen."}],
)                                                                                                                                                                                                                  

See https://docs.litellm.ai/docs/providers for 100+ supported model strings.

Impact

  • Additive only, existing providers (local, stepfun) untouched
  • litellm added to requirements.txt as a dependency
  • drop_params=True silently drops provider-unsupported kwargs
  • "EMPTY" api_key/api_base values are skipped so litellm falls back to provider env vars

@RheagalFire

Copy link
Copy Markdown
Author

cc @softsweetengineer

@RheagalFire

Copy link
Copy Markdown
Author

@softsweetengineer do you have any update on this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant