Algorithm Diagnostic Tool: Short-form Assessment
Algorithm Diagnostic Questionnaire
Please answer the questions to assess the current state of algorithm and governance capabilities within your organization
AI Strategy
Have you developed and implemented a firm-wide AI Strategy?
1 - The organization has not defined its AI Strategy
2 - The organization has defined an AI Strategy but has not deployed it
3 - The organziation has deployed an AI Strategy across the AI/Data function
4 - The organization has deployed an AI Strategy across the organization
5 - The organization has deployed an AI Strategy consistently throughout the organization, and has obtained buy-in and commitment from all stakeholders
Has the organization developed specific considerations and guidance related to Generative AI and its utilization across the organization?
1 - The AI Strategy does not cover Generative AI
2 - The AI Strategy has been updated to incorporate Generative AI considerations
3 - There has been consistent communication and guidance related to Generative AI shared across the organization
4 - The leadership has communicated clear guidance and go-to specialists to discuss Generative AI matters
5 - In addition to a clear leadership positioning on the matter, detailed guidance and learning programmes have been set up across the organization
Next
AI Governance
Does the organisation have a documented definition of AI?
1 - The organization has defined what algorithms are
2 - The organziation has defined what algorithms are and how they are used in tools and applications
3 - The organization has defined what algorithms are, how they interact with tools and how they can be combined to build models
4 - The organization has built a comprehensive dictionary of algorithms, tools and models, as well as machine learning algorithms
5 - The organization has defined and deployed a dictionary that provides a comprehensive view of all concepts related to AI (algorithms, tools, models, machine learning, deep learning and Generative AI)
Has the organization developed and deployed an AI Governance framework?
1 - The organization has no specific AI Governance Framework in place
2 - The organization has defined an AI Governance Framework within the AI function under the Chief AI Officer
3 - The organization has defined and deployed an AI Governance Framework across both the AI and Data functions
4 - The organization has deployed AI Governance Framework across all functions
5 - The organization has efficiently deployed an AI Governance Framework with detailed roles, responsibilities, commitment from stakeholders and accountability from leadership
Next
Processes & Controls
Has the organisation defined and documented processes across the AI lifecycle?
1 - There is no defined process across the AI lifecycle
2 - The organization has defined processes to cover the AI lifecycle, managed by the data science team
3 - The organization has defined and deployed processes that involve risk functions and other impacted stakeholders across the AI lifecycle
4 - The organization has deployed processes throughout all functions
5 - All process have been defined and deployed across all functions throught the AI lifecycle. Processes are constantly monitored and improved on a continuous basis
Has the organization defined and implemented controls across the AI lifecycle?
1 - The organization has defined high-level controls around AI
2 - The organization has detailed controls in place, implemented throughout the AI lifecycle
3 - The controls implemented comply with existing standards and local regulations, and are performed on an annual basis by the internal audit function
4 - The organization's controls are implemented across the AI lifecycle and performed on an annula basis by the internal audit function. In addition, the organization gets certified by a third-party on an annual basis
5 - The organization has redefined all processes across the AI lifecycle to make them Responsible-by-design, with controls implemented natively and also providing a real-time monitoring of those controls
Next
Monitoring & Reporting
Has the organization implemented a Responsible AI Monitoring and Reporting process, including defined thresholds and incident reporting implemented?
1 - No monitoring and reporting implemented
2 - Monitoring and reporting implemented through manual controls around Responsible AI
3 - Manual monitoring and reporting implemented for Responsible AI/Trustworthy AI on all principles
4 - Limited automatic monitoring and reporting process to flag discrepancies and perform remediations when thresholds are met
5 - Automatic monitoring and reporting process implemented across all pillars of Responsible/Trustworthy AI, with predictive analysis of risks to meet thresholds, enabling a proactive management across the AI lifecycle
Is there a standard method of capturing and monitoring relevant standards and regulations that impact AI?
1 - The organisation has no established process to capture and monitor regulations and rely on ad hoc analysis
2 - The organisation has an established compliance team which regularly captures and monitors the regulatory agencies and websites to keep on-track of any market developments
3 - The organisation has strong relationships with regulators in the market which enables them to keep a track of the relevant regulations in the market
4 - The organisation has an in-built compliance tracking software which is responsible for tracking relevant regulations in the market
5 - The organisation has sufficient resources and processes for compliance monitoring which includes documentation, formulation, implementation and monitoring strategies
Next
Human Capital
Has the organisation clearly considered future impacts of AI when it comes to resources, capabilities and skillsets?
1 - The organization has established a data-driven culture and strategy to scale AI across the organisation
2 - The organization has developed a program to improve AI literacy by emphasising on upskilling and reskilling based on changing AI landscape
3 - The organization encourages AI working groups and committees to recognise the need of professionalising AI for both technical and non-technical individuals
4 - The organization has enabled a transition from siloed work to align to interdisciplinary collaboration to ensure business, technical, operational, IT work side-by-side
5 - The organization has established strong AI/ML/GenAI ops through allocation of defined roles and responsibilities
Does the organisation ensure third party resources are appropriately skilled and experienced?
1 - The organization has no defined process for evaluating the skills and experience of third-party resources. Third-party resources may not have the necessary qualifications or experience for the tasks they are assigned
2 - The organization has a basic process for evaluating skills and experience, but it may not be well-defined or consistently applied. There may be some concerns about the qualifications or experience of some third-party resources.
3 - The organization has a defined process for evaluating the skills and experience of third-party resources, and it is generally applied consistently. Most third-party resources appear to have the necessary qualifications and experience for the tasks they are assigned
4 - The organization has a thorough process for evaluating the skills and experience of third-party resources, and it is consistently applied. The organization selects third-party resources with a strong track record of success in relevant areas.
5 - The organization has a rigorous process for evaluating the skills and experience of third-party resources, and it goes beyond basic qualifications to consider specific expertise and cultural fit. The organization prioritizes selecting highly skilled and experienced third-party resources who consistently exceed expectations.
Next
Data
Has the organization deployed a consistent collaboration between AI teams and Data Management teams?
1 - There is no formal or informal collaboration between AI and Data Management teams. Teams operate independently and may have conflicting priorities.
2 - Collaboration happens occasionally, driven by specific projects or issues. Communication is inconsistent and there may be confusion about roles and responsibilities.
3 - Some efforts are made to collaborate, such as through occasional meetings or shared platforms. However, collaboration is not yet fully integrated into the workflow and challenges remain.
4 - AI and Data Management teams have defined processes and structures for collaboration. Information is shared regularly, and teams work together effectively to achieve common goals.
5 - Collaboration is seamless and well-integrated into the organizational culture. Teams have clear roles and responsibilities, and communication is open and transparent. This leads to improved data quality, faster AI development, and better overall outcomes.
Has the organization deployed specific processes to comply with data security and privacy standards and local regulations?
1 - The organization lacks defined processes for data security and privacy compliance. Reliance may be on ad-hoc measures or individual knowledge, leading to high risk of non-compliance.
2 - Some basic procedures are in place, but they may be incomplete, outdated, or inconsistently applied. The organization may face significant challenges in demonstrating compliance.
3 - Defined processes cover key aspects of data security and privacy, but there may be gaps or areas for improvement. Compliance efforts are somewhat managed, but risks remain.
4 - Comprehensive processes address data security, privacy, and local regulations. Regular risk assessments and audits are conducted. The organization can demonstrate a strong commitment to compliance.
5 - The organization goes beyond basic compliance, actively implementing best practices and seeking continuous improvement. Data security and privacy are embedded in the organizational culture.
Next
Technology
Does the organization have a contingency plan in place, in the event that infrastructure is offline?
1 - The organization lacks any documented contingency plan for infrastructure outages. This leaves them unprepared for potential disruptions and may lead to significant downtime and consequences.
2 - A rudimentary plan might exist, but it is incomplete, outdated, or untested. It may lack critical details or fail to address key scenarios, leaving the organization vulnerable.
3 - The organization is actively creating a contingency plan, but it is still under development and may not yet be fully functional. Testing and refinement are needed to ensure preparedness.
4 - A documented and tested plan outlines procedures for handling infrastructure outages. It identifies critical systems, defines roles and responsibilities, and outlines recovery steps. The organization can demonstrate its ability to respond effectively
5 - The organization possesses a comprehensive and regularly tested contingency plan. It considers diverse outage scenarios, incorporates lessons learned from previous incidents, and prioritizes rapid recovery with minimal disruption. The organization can confidently adapt and respond to unexpected situations.
How does the organization ensure that the AI system being designed is protected against intentionally motivated results and harmful system attacks?
1 - The organization lacks any mechanisms to address intentional manipulation or attacks on the AI system. This makes it highly vulnerable to manipulation, bias, and potential harm.
2 - Basic security measures like password protection and access control are in place, but these are insufficient to prevent determined attackers. Vulnerability assessments and penetration testing are not conducted regularly.
3 - The organization is starting to implement safeguards, such as data integrity checks and anomaly detection. Awareness of potential threats is growing, but a comprehensive approach is still under development.
4 - The organization has established security measures that address known vulnerabilities and malicious intent. These include threat modeling, adversarial testing, and robust monitoring systems. The organization can demonstrate its ability to detect and mitigate potential attacks.
5 - The organization goes beyond basic safeguards, actively seeking and implementing best practices in AI security and attack prevention. Regular security audits and threat monitoring ensure ongoing adaptation and preparedness. The organization is highly resilient against intentional manipulation and harmful attacks.
Please enter the following details to get started
Name
Company