Humanity’s Last Exam: A New Benchmark for AI

The rapid advancement of artificial intelligence has prompted the Center for AI Safety (CAIS) and Scale AI to launch an ambitious initiative called “Humanity’s Last Exam.” This project aims to create the world’s most challenging AI benchmark, setting a new standard for evaluating AI capabilities.

Purpose and Goals

The primary objective of Humanity’s Last Exam is to measure how close AI systems are to achieving human expert-level capabilities. With existing benchmarks becoming too easy for advanced models, this new exam seeks to provide questions that challenge even highly skilled humans. By doing so, it encourages the development of AI systems with deeper cognitive skills, such as abstract reasoning and advanced problem-solving.

Submission Process and Review

The exam will feature at least 1,000 crowd-sourced questions, with submissions due by November 1, 2024. These questions will undergo a rigorous peer review process to ensure their quality and relevance. A subset of questions will be kept private to prevent AI systems from merely memorizing answers, allowing for a more accurate assessment of their true capabilities.

Participation Incentives

To attract high-quality submissions, a $500,000 prize pool has been established. The top 50 questions will earn $5,000 each, while the next 500 questions will receive $500 each. In addition to monetary rewards, successful contributors will gain co-authorship on the resulting paper, earning recognition in the academic and AI research communities.

Question Guidelines

Submissions should be complex and original, challenging non-experts and not easily answerable through quick online searches. Ideally, question-writers should have at least five years of experience in a technical industry job or hold PhD-level academic training. Questions must be objective and free from personal taste or ambiguity, with answers that other experts would accept.

This initiative has already attracted submissions from researchers at prestigious institutions like MIT, UC Berkeley, and Stanford. By setting a new bar for AI assessment, Humanity’s Last Exam aims to influence both market leaders and startups in their AI research and development efforts.

In conclusion, Humanity’s Last Exam represents a significant step forward in evaluating AI capabilities. By fostering collaboration between AI researchers and domain experts across various fields, it promises to provide valuable insights into the current capabilities and limitations of frontier AI models.