Comparison Framework
1) Keep Comparisons Fair
- Use the same trait and dataset set for each model.
- Group runs under one experiment for traceability.
- Avoid comparing scores across unrelated datasets.
2) Use Leaderboard for Ranking
Leaderboard helps identify top-performing models quickly:- Compare relative ordering.
- Look for score gaps, not only rank position.
- Re-check runs with small score differences.
3) Use Explorer for Qualitative Validation
After ranking, inspect sample-level outputs:- Validate prompt understanding.
- Check response consistency.
- Confirm failures are acceptable for your use case.
4) Track Operational Signals
Include non-score context from run history:- Run duration
- Completion/failure frequency
- Trait-level variance across reruns