NIST Launches AI Technology Evaluation Program

The National Institute of Standards and Technology (NIST) has launched the Artificial Intelligence Technology Evaluation (AITE), a new program from its Technology Test and Evaluation Division that provides researchers with a sequestered testbed environment for assessing AI model performance across diverse datasets, modalities, and domains.

AITE offers volunteer testing of AI models on blind data, with a sequestered environment designed to mitigate the risk of train/test data contamination and support rigorous, objective assessment. The three initial tasks focus on image analysis using large vision language models (VLMs) in the areas of quantum science, genomics, and public safety, with additional tasks to be added over time.

Participation is open to both data providers and model providers. NIST cites the following benefits for participants:

• For those providing data for testing: “Receive careful measurements of top models on your data conducting your task without having to conduct the evaluation yourself.”
• For those submitting a model for testing: “Learn how your model performs on an increasing number of datasets and tasks without having to coordinate your own evaluation,” and “learn how other models perform on the same data using the same metrics, thus improving comparability while also ensuring the data used for evaluation is not part of the training set for any model.”

Results will be made available publicly on the AITE website.

Interested parties can request participation or ask questions by contacting [email protected]. Additional information, including the AITE evaluation overview and specifications for the initial three tasks, is available on the AITE website.