Skill-MAS

Run locally with demo_inference.py

1. Clone the Skill_MAS repository and enter the project root.
2. Activate your conda environment (example):

   source /path/to/miniconda3/etc/profile.d/conda.sh
   conda activate skill_mas
   python --version

3. Install dependencies (if needed):

   pip install openai httpx aiohttp tiktoken

4. Set API credentials via environment variables (recommended — do not commit keys into model_config.json):

   export OPENAI_API_KEY="your-key"
   export OPENAI_API_BASE="https://your-endpoint/v1"   # optional
   export SKILL_MAS_DEMO_MODEL="deepseek-v4-flash"      # optional, model id in skill_mas/model_config.json

5. Pick a skill file (5 available):
   • init_skill/SKILL.md — generic init skill
   • optimized_skill/hlemath.md — HLE-Math
   • optimized_skill/drb.md — Deep Research Bench
   • optimized_skill/bcp.md — BrowseComp-Plus (needs BM25 index)
   • optimized_skill/vitabench.md — VitaBench
6. Run inference on your custom question (build + execute):

   cd Skill_MAS   # repository root containing demo_inference.py
   
   python demo_inference.py \
     --skill init_skill/SKILL.md \
     --question "Your custom task description here" \
     --model deepseek-v4-flash \
     --dataset hlemath

   Or use the shell launcher (model, skill path, question as arguments):

   bash demo_inference.sh deepseek-v4-flash init_skill/SKILL.md "Your question here"

   Optional flags: --save-mas-code path/to/workflow.py, --verbose
7. The script prints each build stage, generated MAS code, and the final answer to stdout. Use --dataset when auto-inference from the skill filename is insufficient (hlemath, drb, bcp, vita).
8. Full dataset / batch runs: when evolving or evaluating on an entire dataset (e.g. run_hlemath.sh, run_drb.sh, run_bcp.sh), increase --max-concurrency — the second argument to those shell scripts, or the flag on python -m Skill_MAS evolve. Higher concurrency (within your API quota) shortens total wall-clock time substantially compared to running one sample at a time.