Procurement of AI Systems: Is your procurement policy fit for purpose?

Organisations are increasingly procuring standalone AI systems or services that have embedded AI tools. AI systems have unique characteristics as compared to non-AI systems that require careful consideration during the buying process.

Consequently, an organisation should assess whether its procurement policy is fit for purpose and adequately addresses the specific issues that flow from procuring AI systems or services with embedded AI tools.

This article summarises 5 key principles which should be reflected in organisations' procurement policies.

A clear business case and use case is established before an AI system is purchased

Before procuring an AI system, organisations must critically assess whether they have a business case and need for an AI system. An organisation should have a clear understanding of why it needs to purchase an AI system, its intended purpose, use case and the setting in which the AI system will be deployed. In short, any AI procurement should be led by business need, not by the availability of technology. Non-AI options which may address the underlying need should also be considered and explored.

Without this clear understanding, an organisation will likely struggle to procure an AI system that delivers a return on investment.

Market approach is informed by AI specific risks and impacts

The way in which the market is approached for a particular procurement is commonly informed by spend or likely contract value. For example, a procurement policy may require a purchaser obtain one quote for goods or services valued under $50,000, with minimal due diligence and evaluation.

A low value AI system may still be high risk and require greater scrutiny than is normally warranted for the purchase of a low value non-AI system. An assessment of the AI specific risk and impact in each use case should inform the method of market approach, and the level of due diligence and evaluation required for the purchase of an AI system.

Assessment of AI system extends beyond functional and non-functional requirements

Normally an organisation procuring an IT system will assess how well the proposed system matches the organisation's functional and non-functional requirements.

An AI system should undergo greater scrutiny beyond assessment of functional and non-functional requirements. For instance, an AI algorithm or model should be scrutinised, data provenance examined, trustworthiness characteristics evaluated, privacy and intellectual property issues deeply considered and third-party software and other rights properly understood.

AI system impacts are assessed before AI system is purchased

Understanding an AI system's impact (both positive and negative) is imperative for effective AI governance and responsible use of an AI system. An AI system impact assessment should be completed as an integral step in the procurement process, prior to the purchase of an AI system.

An AI system impact assessment assesses the potential consequences for individuals or groups of individuals (or both) and society that can result from the development, provision or use of AI systems.

Contract considers AI specific risks

Organisations must be alive to the specific issues and risks arising from the adoption and use of third-party AI systems. Before an organisation selects an AI system and commences system implementation, AI specific issues and risks should be addressed in a written contract between the organisation and supplier.

In particular, a contract should be carefully crafted to address issues including intellectual property rights in AI inputs and outputs, privacy of any personal information input into or generated by AI systems, information security, use and ownership of customer data, transparency obligations, model training and retraining (including whether customer data is permitted to be used for training purposes), processes for AI system updates, and liability for AI outputs.

If your organisation needs help updating its procurement policy or informing AI sourcing strategy, please contact our team of experienced technology lawyers to learn how we can help you to procure AI systems responsibly and navigate AI specific risks throughout the entire procurement lifecycle.