Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)

LLM evaluation is critical for comparing models and tracking progress, employing four main approaches: multiple-choice benchmarks, verifiers, leaderboards, and LLM judges. Multiple-choice benchmarks offer quantifiable accuracy for knowledge recall but lack real-world utility and reasoning evaluation. Verifiers allow free-form answers in verifiable domains like math or code, providing objective accuracy for reasoning, though they shift complexity to external tools. Leaderboards, such as LM Arena, rank models by user preferences, capturing subjective qualities like style and helpfulness but are prone to biases and lack instant feedback. LLM judges use a strong language model and a predefined rubric to automatically score responses against a reference, offering scalability and consistency, despite potential biases from the judge model itself.