Skip to main content
Benchmarking is an engineering task that needs stability and reproducibility. You’ll be calling the model thousands of times; even tiny drifts in system setup or network latency can compromise result accuracy. Here’s what we’ve learned to keep things reproducible and trustworthy. Quick notes
  • For any unlisted or closed-source benchmark: settemperature = 1.0, stream = true, top_p = 0.95
  • Reasoning benchmarks: max_tokens = 128k, and run at least 500–1000 samples to get low variance (e.g. AIME 2025: 32 runs -> 30 × 32 = 960 questions)
  • Coding benchmarks: max_tokens = 256k
  • Agentic task benchmarks:
    • For multi-hop search: max_tokens = 256k + context management
    • Others: max_tokens ≥ 16k–64k
Benchmark CategoryBenchmarkTemperatureRecommended max tokensRecommended runsTop-pOthers (e.g. test log)
Multi-modalMMMU-Pro1.0max tokens = 96k3top_p=0.95thinking=
MMMU-Pro w/ python1.0per step tokens = 64k;
total max tokens = 256k		3top_p=0.95Recommended max steps = 50
thinking=
CharXiv (RQ)1.0max tokens = 96k3top_p=0.95thinking=
CharXiv (RQ) w/ python1.0per step tokens = 64k;
total max tokens = 256k		3top_p=0.95Recommended max steps = 50
thinking=
MathVision1.0max tokens = 96k3top_p=0.95thinking=
MathVision w/ python1.0per step tokens = 64k;
total max tokens = 256k		3top_p=0.95Recommended max steps = 50
thinking=
V* w/ python1.0per step tokens = 64k;
total max tokens = 256k		3top_p=0.95Recommended max steps = 50
thinking=
AgentHLE-Full w/ tools1.0per step tokens = 48k;
total max tokens = 256k		1top_p=0.95Recommended max steps = 300
thinking=
BrowseComp1.0per step tokens = 48k;
total max tokens = 256k		1top_p=0.95Recommended max steps = 300
thinking=
DeepSearchQA1.0per step tokens = 48k;
total max tokens = 256k		1top_p=0.95Recommended max steps = 300
thinking=
WideSearch1.0per step tokens = 48k;
total max tokens = 256k		4top_p=0.95Recommended max steps = 300
thinking=
Toolathlon1.0per step tokens = 48k;
total max tokens = 256k		4top_p=0.95Recommended max steps = 300
thinking=
MCPMark1.0per step tokens = 48k;
total max tokens = 256k		4top_p=0.95Recommended max steps = 300
thinking=
Claw Eval1.0per step tokens = 48k;
total max tokens = 256k		4top_p=0.95Recommended max steps = 300
thinking=
APEX-Agents1.0per step tokens = 48k;
total max tokens = 256k		4top_p=0.95Recommended max steps = 300
thinking=
CodingTerminal-Bench 2.0 (Terminus-2)1.0max tokens = 256k3top_p=0.95thinking=
SWE-Bench Pro1.0per step tokens = 32k;
total max tokens = 256k		5top_p=0.95Recommended max steps = 300
thinking=
SWE-Bench Multilingual1.0per step tokens = 32k;
total max tokens = 256k		5top_p=0.95Recommended max steps = 300
thinking=
SWE-Bench Verified1.0per step tokens = 32k;
total max tokens = 256k		5top_p=0.95Recommended max steps = 300
thinking=
SciCode1.0max tokens = 96k4top_p=0.95thinking=
OJBench (python)1.0max tokens = 96k8top_p=0.95thinking=
LiveCodeBench (v6)1.0max tokens = 96k1top_p=0.95thinking=
MathAIME 20261.0max tokens = 96k32top_p=0.95thinking=
HMMT 2026 (Feb)1.0max tokens = 96k32top_p=0.95thinking=
IMO-AnswerBench1.0max tokens = 96k4top_p=0.95thinking=
KnowledgeHLE-Full1.0max tokens = 96k1top_p=0.95thinking=
GPQA-Diamond1.0max tokens = 96k8top_p=0.95thinking=

​
API Recommendations & Notes

  • Use the official API: some 3rd-party endpoints show noticeable accuracy drift.
  • Use the recommended models for testing
    • For K2.6: use kimi-k2.6 for testing
  • Must set: stream = true
    • Non-streaming mode can lead to random mid-connection interruptions that are hard to control.
  • Current API default settings:
    • Kimi K2.6:
      • default max_tokens = 32768
      • default thinking = {"type": "enabled", "keep": null}
      • default temperature = 1.0
      • default top_p = 0.95
      • default n = 1
      • default presence_penalty = 0.0
      • default frequency_penalty = 0.0
  • Timeouts:
    • With stream = false, api.moonshot.ai timeout = 2 hours, but some ISPs may terminate earlier.
    • So again we recommend you to set stream = true
  • Concurrency:
    • Keep concurrency low to avoid rate limiting
  • Retry logic is not optional:
    • handle overloaded
    • handle unexpected finish reason due to random server issues
    • handle errors due to complicated network issues

​
FAQ

Q1. Is the temperature setting consistent across models? A. No. Different model families use different recommended temperatures:
  • k2.6 model: temperature = 1.0
Q2. Why use stream = true? A. Long outputs can take minutes. Idle TCP connections may be terminated by firewalls, load balancers, or NAT gateways. Streaming keeps the connection alive and significantly improves reliability. In production, requests with stream = false fail far more often than with stream = true. Q3. How much concurrency should I use? A. Your API account has specific rate limits (see Recharge and Rate Limits). Start low. If you hit HTTP 429 (rate limit), your concurrency is too high. Accuracy > speed, so tune concurrency to stay within limits. Q5. Why should I add retry? A. Even with streaming, requests can fail due to transient network issues. Retry on temporary faults (network jitter, server overload, rate limiting) to avoid avoidable failures. Q6. Why should multi-turn or multi-step tasks include full context and reasoning? A. The model needs full context to stay logically consistent. Without previous reasoning steps, later turns can go off track or produce incomplete answers.

​
Contact Us

Hit any issues? Drop us an email at [email protected] with your logs. We’ll take a look!