Irene Solaiman, Head of Policy at Hugging Face, stated that AI model evaluation methods are complex and still maturing, and embedded AI evaluators are not an easy solution.

Solaiman noted at the TechCrunch Disrupt conference that AI labs rely on model benchmarking to measure product quality, but current evaluation methods lack reproducibility, comparability, and standardization. She believes that evaluation is expanding from the models themselves to include entire systems, such as harnesses and agents, making evaluation more difficult and expensive, and requiring a more unified industry approach.