Vals Aims to Revolutionize AI Benchmarking

Vals, a startup launched in 2024, is on a mission to enhance the flawed benchmarking systems currently used in the AI industry. With established credibility over the past two years, Vals recently secured a significant $40 million in a series A funding round led by Andreessen Horowitz, following a successful seed round co-led by 8VC and Bloomberg Beta.

The company’s co-founder, Rayan Krishnan, a Stanford alumnus with internship experience at major tech firms like Palantir and Microsoft, recognized a pressing need for more relevant benchmarks in evaluating AI capabilities. “With a plethora of new models emerging, traditional benchmarks are lagging behind,” he stated, emphasizing the importance of verification in AI, especially as these technologies proliferate throughout society.

During a recent tour of Vals’ office in an old brewery building on San Francisco’s Folsom Street, Krishnan elaborated on the shortcomings of current evaluation methods, noting that they often assess knowledge in a rather abstract manner. He believes that a more comprehensive approach should evaluate models on their practical capabilities rather than just their theoretical knowledge.

In stark contrast to public benchmarking tests that can be manipulated, Vals maintains the confidentiality of its test materials. This approach enables the company to assess AI models on complex tasks relevant to specific fields, including law, finance, and software development. Krishnan highlighted the necessity of understanding the implications of AI performance, including potential risks, saying, “We want to analyze what the consequences could be if these models were implemented unchecked.”

The scope of Vals’ evaluations is expanding, targeting innovative areas such as mental health, cybersecurity, biosecurity, and even applications relevant to international law. Companies are incentivized to engage Vals despite the possibility of receiving unfavorable evaluations, as these assessments offer critical insights for improving model performance. Krishnan compares this model to students paying for standardized tests like the SAT.

Vals has shown impressive growth, with reported revenue increasing eightfold compared to the previous year and staff expanding from eight to twenty-five. The startup plans to relocate to a larger office as well and intends to bring on an additional 10 to 15 employees. Furthermore, Vals has initiated a program to provide model evaluations for federal agencies.

Krishnan envisions that Vals’ benchmarking system will play a crucial role in shaping how AI companies approach growth and foster public trust. He notes, “As companies in the AI sector begin to go public, the models of measuring and evaluating we develop will become integral to their reporting and investment strategies.”