Singapore’s IMDA Launches Gen AI Evaluation Sandbox

by Kumar Gandharv

Published: November 02, 2023 at 8:10 am Updated: November 02, 2023 at 8:10 am

by Victor Dey

Edited and fact-checked:
November 02, 2023 at 8:10 am

To improve your local-language experience, sometimes we employ an auto-translation plugin. Please note auto-translation may not be accurate, so read original article for precise information.

In Brief

Singapore’s Infocomm Media Development Authority of Singapore Partners AI Verify Foundation to Unveil Generative AI Evaluation Sandbox.

In a collaborative effort, Infocomm Media Development Authority of Singapore (IMDA), in partnership with the AI Verify Foundation unveiled the Generative AI Evaluation Sandbox.

The initiative aims to serve as a platform for assessing trusted artificial intelligence (AI) products and unveiling potential gaps in their performance.

The Sandbox leverages a recently introduced Evaluation Catalogue, providing a comprehensive framework with standardized methods and recommendations for the evaluation of generative AI products. Guided by this catalogue, which assembles widely utilized technical testing tools, the initiative organizes them based on their testing objectives and methodologies.

Furthermore, it recommends a fundamental suite of tests, ultimately enhancing the evaluation process of AI products.

In an era characterized by the rapid evolution of AI technologies, ensuring the trustworthiness of these systems has emerged as a paramount concern. The Gen AI Evaluation Sandbox is set to bridge the gap between the realm of theoretical trustworthiness and the practical evaluation of AI systems.

At its core, this groundbreaking initiative is designed to create a common standard approach for assessing Generative AI (Gen AI).

Gen AI refers to the sophisticated AI models known as Large Language Models (LLMs), such as GPT-3, which have demonstrated the potential for both creativity and controversy, making standardized evaluation more critical than ever.

Addressing AI Risks and Harms

The foundation for this initiative was laid in a discussion paper titled “Generative AI: Implications for Trust and Governance”. The paper identified key risks and harms associated with Large Language Models. In response to these concerns, IMDA and the AI Verify Foundation have launched the Gen AI Evaluation Sandbox.

The Sandbox seeks to foster a collaborative ecosystem for evaluating AI products by engaging a wide array of stakeholders. IMDA has extended an open invitation to industry partners to join forces and contribute to the development of evaluation tools and capabilities within the Sandbox. This approach ensures that the responsibility of evaluating AI extends beyond just model developers to include application developers and third-party testers.

It will provide a standardized language for Gen AI evaluation through its Evaluation Catalogue. This Catalogue categorizes existing evaluation benchmarks and methods while recommending a baseline set of evaluation tests for Gen AI products. It provides a structured framework to assess the capabilities and limitations of these AI models.

The launch of the Gen AI Evaluation Sandbox has already seen key industry players joining the cause. Notable participants include technology giants like Google, Microsoft, Anthropic, IBM, NVIDIA, Stability.AI and Amazon Web Services (AWS). Third-party testers including Resaro.AI, Deloitte, EY, and TÜV SÜD will also lending their expertise.

Further, it aims to involve regulators such as the Singapore Personal Data Protection Commission (PDPC), to maintain transparency and compliance at all stages of AI development and deployment.

AWS’ country manager, worldwide public sector, Elsie Tan, said, “The responsible use of generative AI technologies will transform entire industries and reimagine how work gets done. We look forward to being a part of IMDA’s Generative AI Evaluation Sandbox to provide businesses with the tools and guidance needed to build artificial intelligence and machine learning applications responsibly.”

Opportunities for Participation

Singapore has taken significant strides in the responsible AI domain, exemplified by the introduction of the AI Verify Foundation. The Gen AI Evaluation Sandbox marks the next phase of this journey, leveraging global contributions and open-source community support.

The AI Verify Foundation and IMDA have extended an invitation to model and app developers, as well as third-party testers, to participate in the Gen AI Evaluation Sandbox. This is a unique opportunity for organizations and individuals to contribute to the development of a more robust testing environment for AI models.

The global collaboration fostered by this Sandbox represents a significant step towards a standardized and transparent approach to evaluating Gen AI.