Skip to content

Commit 68e4bbd

Browse files
jiarong0907yl231
andauthored
[Feat] Reorganize the evaluation pipeline (#7)
* fix white space * try fixing CI errors * types: mypy clean across repo * style: apply ruff/ruff-format fixes * ci: retrigger checks * chore: sync README.md, pyproject.toml, uv.lock, .python-version with main; apply pre-commit fixes * Update the pipeline to reflect the changes in HF datasets. * ci: retrigger pre-commit * [Major update] Easier new router evaluation pipeline; Updates on README.md * chore: fixed some typo * remove type ignore * restore local python version for pre-commit --------- Co-authored-by: yl231 <yifan21lu@gmail.com>
1 parent aa09708 commit 68e4bbd

42 files changed

Lines changed: 3290 additions & 185945 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,9 @@ OPENAI_API_KEY='<YOUR_KEY>'
55
HF_ACCESS_TOKEN='<YOUR_KEY>'
66
HF_TOKEN = '<YOUR_KEY>'
77

8-
# Volcengine API Configuration
9-
VOLCENGINE_ACCESS_TOKEN='<YOUR_KEY>'
10-
118
# DeepSeek API Configuration
129
DEEPSEEK_API_KEY='<YOUR_KEY>'
1310

14-
# NotDiamond API Configuration
15-
NOTDIAMOND_API_KEY='<YOUR_KEY>'
16-
1711
# Anthropic API Configuration
1812
ANTHROPIC_API_KEY='<YOUR_KEY>'
1913

@@ -35,9 +29,6 @@ AZURE_API_KEY = "<YOUR_KEY>"
3529
# Replicate API Configuration
3630
REPLICATE_API_TOKEN = "<YOUR_KEY>"
3731

38-
# Clarifai API Configuration
39-
CLARIFAI_PAT = "<YOUR_KEY>"
40-
4132
# AWS
4233
AWS_ACCESS_KEY_ID = "<YOUR_KEY>"
4334
AWS_SECRET_ACCESS_KEY = "<YOUR_KEY>"
@@ -46,3 +37,6 @@ AWS_BEARER_TOKEN_BEDROCK = "<YOUR_KEY>="
4637

4738
# xAI
4839
XAI_API_KEY = "xai-<YOUR_KEY>"
40+
41+
# ZHIPU_API_KEY
42+
ZHIPU_API_KEY = "<YOUR_KEY>"

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ __pycache__/
77
outputs/
88
results/
99

10+
# datasets
11+
dataset/
12+
1013
# Logs
1114
*.log
1215
logs/

‎README.md‎

Lines changed: 176 additions & 109 deletions
Large diffs are not rendered by default.

‎cached_results/claude-3-haiku-20240307.jsonl‎

Lines changed: 202 additions & 0 deletions
Large diffs are not rendered by default.

‎cached_results/gemini-2.0-flash-001.jsonl‎

Lines changed: 202 additions & 0 deletions
Large diffs are not rendered by default.

‎cached_results/gpt-4o-mini.jsonl‎

Lines changed: 203 additions & 0 deletions
Large diffs are not rendered by default.

‎cached_results/gpt-4o.jsonl‎

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎cached_results/mistral-medium.jsonl‎

Lines changed: 202 additions & 0 deletions
Large diffs are not rendered by default.

‎images/leaderboard.png‎

247 KB
Loading

‎images/routerarena_diagram.png‎

610 KB
Loading

0 commit comments

Comments
 (0)