What this skill does

# SOTA Tracker **The definitive open-source database of State-of-the-Art AI models.** Auto-updated daily from [LMArena](https://lmarena.ai), [Artificial Analysis](https://artificialanalysis.ai), and [HuggingFace](https://huggingface.co). ## Why This Exists AI models are released weekly. Keeping track is impossible. This project: 1. **Curates authoritative data** - LMArena Elo rankings, manual curation for video/image/audio models 2. **Updates daily** via GitHub Actions 3. **Exports to JSON/CSV/SQLite** - Use in your own projects 4. **Provides multiple interfaces** - Static files, REST API, or MCP server ## Quick Start: Use the Data ### Option 1: Download JSON/CSV ```bash # Latest data (updated daily) curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.json curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.csv ``` ### Option 2: Clone and Query Locally ```bash git clone https://github.com/romancircus/sota-tracker-mcp.git cd sota-tracker-mcp # Query with sqlite3 sqlite3 data/sota.db "SELECT name, sota_rank FROM models WHERE category='llm_api' ORDER BY sota_rank LIMIT 10" # List forbidden/outdated models sqlite3 data/sota.db "SELECT name, reason, replacement FROM forbidden" ``` ### Option 3: Use with Claude Code (Recommended) The recommended approach for Claude Code users is **static file embedding** (lower token cost than MCP): ```bash # Set up daily auto-update of CLAUDE.md cp scripts/update_sota_claude_md.py ~/scripts/ # Enable systemd timer (runs at 6 AM daily) systemctl --user enable --now sota-update.timer # Or run manually python ~/scripts/update_sota_claude_md.py --update ``` This embeds a compact SOTA summary directly in your `~/.claude/CLAUDE.md` file. ### Option 4: REST API ```bash # Start the API server uvicorn rest_api:app --host 0.0.0.0 --port 8000 # Query endpoints curl "http://localhost:8000/api/v1/models?category=llm_api" curl "http://localhost:8000/api/v1/forbidden" curl "http://localhost:8000/api/v1/models/FLUX.1-dev/freshness" ``` ### Option 5: MCP Server (Optional) MCP support is available but disabled by default (higher token cost). To enable: ```bash # Edit .mcp.json to add the server config cat > .mcp.json << 'EOF' { "mcpServers": { "sota-tracker": { "command": "python", "args": ["server.py"] } } } EOF ``` ## Data Sources | Source | Data | Update Frequency | |--------|------|------------------| | [LMArena](https://lmarena.ai/leaderboard) | LLM Elo rankings (6M+ human votes) | Daily | | [Artificial Analysis](https://artificialanalysis.ai) | LLM benchmarks, pricing, speed | Daily | | [HuggingFace](https://huggingface.co) | Model downloads, trending | Daily | | Manual curation | Video, Image, Audio, Video2Audio models | As needed | ## Categories | Category | Description | Top Models (Feb 2026) | |----------|-------------|----------------------| | `llm_api` | Cloud LLM APIs | Gemini 3 Pro, Grok 4.1, Claude Opus 4.5 | | `llm_local` | Local LLMs (GGUF) | Qwen3, Llama 3.3, DeepSeek-V3 | | `llm_coding` | Code-focused LLMs | Qwen3-Coder, DeepSeek-V3 | | `image_gen` | Image generation | Z-Image-Turbo, FLUX.2-dev, Qwen-Image | | `video` | Video generation | LTX-2, Wan 2.2, HunyuanVideo 1.5 | | `video2audio` | Video-to-audio (foley) | MMAudio V2 Large | | `tts` | Text-to-speech | ChatterboxTTS, F5-TTS | | `stt` | Speech-to-text | Whisper Large v3 | | `embeddings` | Vector embeddings | BGE-M3 | ## REST API Endpoints | Endpoint | Description | |----------|-------------| | `GET /api/v1/models?category=X` | Get SOTA for a category | | `GET /api/v1/models/:name/freshness` | Check if model is current or outdated | | `GET /api/v1/forbidden` | List outdated models to avoid | | `GET /api/v1/compare?model_a=X&model_b=Y` | Compare two models | | `GET /api/v1/recent?days=30` | Models released in past N days | | `GET /api/v1/recommend?task=chat` | Get recommendation for a task | | `GET /health` | Health check | ## Run Your Own Scraper ```bash # Install dependencies pip install -r requirements.txt pip install playwright playwright install chromium # Run all scrapers python scrapers/run_all.py --export # Output: # data/sota_export.json # data/sota_export.csv # data/lmarena_latest.json ``` ## GitHub Actions (Auto-Update) This repo uses GitHub Actions to: - **Daily**: Scrape all sources, update database, commit changes - **Weekly**: Create a tagged release with JSON/CSV exports To enable on your fork: 1. Fork this repo 2. Go to Settings → Actions → Enable workflows 3. Data will auto-update daily at 6 AM UTC ## File Structure ``` sota-tracker-mcp/ ├── server.py # MCP server (optional) ├── rest_api.py # REST API server ├── init_db.py # Database initialization + seeding ├── requirements.txt # Dependencies ├── data/ │ ├── sota.db # SQLite database │ ├── sota_export.json # Full JSON export │ ├── sota_export.csv # CSV export │ └── forbidden.json # Outdated models list ├── scrapers/ │ ├── lmarena.py # LMArena scraper (Playwright) │ ├── artificial_analysis.py # AA scraper (Playwright) │ └── run_all.py # Unified runner ├── fetchers/ │ ├── huggingface.py # HuggingFace API │ └── cache_manager.py # Smart caching └── .github/workflows/ └── daily-scrape.yml # GitHub Actions workflow ``` ## Contributing Found a model that's missing or incorrectly ranked? 1. **For manual additions**: Edit `init_db.py` and submit a PR 2. **For scraper improvements**: Edit files in `scrapers/` 3. **For new data sources**: Add a new scraper and update `run_all.py` See [CONTRIBUTING.md](CONTRIBUTING.md) for full developer setup and PR process. ## OpenCode / Agents.md Integration The repo now supports updating `agents.md` files for OpenCode agents: ```bash # Update your agents.md with latest SOTA data python update_agents_md.py # Minimal version (top 1 model per category, lightweight) python update_agents_md.py --minimal # Custom categories and limit python update_agents_md.py --categories llm_local image_gen --limit 3 # Force refresh from sources first python update_agents_md.py --refresh ``` ### Automation Add to your cron or systemd timer for daily updates: ```cron # ~: crontab -e @daily python ~/Apps/sota-tracker-mcp/update_agents_md.py ``` Or systemd: ```bash # ~/.config/systemd/user/sota-update.service [Unit] Description=Update SOTA models for agents After=network.target [Service] ExecStart=%h/Apps/sota-tracker-mcp/update_agents_md.py [Install] WantedBy=default.target # ~/.config/systemd/user/sota-update.timer [Unit] Description=Daily SOTA data update OnCalendar=daily AccuracySec=1h [Install] WantedBy=timers.target # Enable systemctl --user enable --now sota-update.timer ``` See [CONTRIBUTING.md](CONTRIBUTING.md) for full setup guide ## Data Attribution & Legal This project aggregates **publicly available benchmark data** from third-party sources. We do not claim ownership of rankings, Elo scores, or benchmark results. ### Data Sources (Used With Permission) | Source | Data | Permission | |--------|------|------------| | [LMArena](https://lmarena.ai) | Chatbot Arena Elo rankings | `robots.txt: Allow: /` | | [Artificial Analysis](https://artificialanalysis.ai) | LLM quality benchmarks | `robots.txt: Allow: /` (explicitly allows AI crawlers) | | [HuggingFace](https://huggingface.co) | Model metadata, downloads | Public API | | [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) | Open-source LLM benchmarks | CC-BY license | ### Disclaimer - All benchmark scores and rankings are the intellectual work of their respective sources - This project provides aggregation and tooling, not original benchmark data - Data is scraped once daily to minimize server load - If you are a data source and wish to be excluded, please open an issue ### Fair Use This project: - Aggregates factual data (not copyrightable) - Adds value through tooling (API server, unified format, forbidden list) - Attributes all sources with links - Does not compete commercially with sources - Respects robots.txt permissions ## License MIT - See [LICENSE](LICENSE) for details. The **code** in this repository is MIT licensed. The **data** belongs to its respective sources (see attribution above).