1. Framing AI Risks and Key Challenges

Organizations can frame AI-related risks by considering the composite measure of an event’s probability and the magnitude of the consequences. This involves understanding and addressing the impacts, which can be both positive and negative, resulting in opportunities or threats. AI risk management offers a path to minimize potential negative impacts, such as threats to civil liberties and rights, while maximizing positive impacts.

Key Challenges in AI Risk Management

1. Risk Measurement

Risk measurement in AI risk management involves various challenges and considerations, that can significantly impact the effectiveness of risk management efforts and the trustworthiness of AI systems.

2. Risk Tolerance

Risk tolerance in the context of AI risk management refers to an organization’s or AI actor’s readiness to bear risk to achieve its objectives. It is highly contextual and specific to the application and use case. Various factors influence risk tolerance, including legal or regulatory requirements, organizational priorities, and resource considerations.

• Contextual and Use-Case Specificity:
  Risk tolerance varies greatly depending on the specific context in which an AI system is deployed. Different applications and use cases present unique risks and thus require tailored approaches to risk tolerance. For instance, AI systems used in healthcare might have a lower risk tolerance due to potential impacts on human health compared to AI systems used for marketing purposes.
• Influence of Policies and Norms:
  The policies and norms established by AI system owners, organizations, industries, communities, or policymakers significantly influence risk tolerance. These influences are dynamic and may evolve over time as AI systems, policies, and societal norms change.
• Legal and Regulatory Requirements:
  Legal and regulatory frameworks play a crucial role in defining acceptable risk levels. These requirements help ensure that AI systems operate within boundaries that protect public interest and safety. Organizations must align their risk management strategies with these legal requirements to maintain compliance and avoid legal repercussions.
• Harm/Cost-Benefit Trade-offs:
  Organizations must continually develop and debate methods to better inform harm/cost-benefit trade-offs. This ongoing process involves businesses, governments, academia, and civil society working together to specify AI risk tolerances. It acknowledges that challenges remain in fully specifying AI risk tolerances, which can lead to contexts where a risk management framework may not be fully applicable.
• Organizational Priorities and Resources:
  Different organizations may have varied risk tolerances due to their specific priorities and the resources at their disposal. Larger organizations might have more resources to manage higher risk levels, whereas smaller organizations may need to adopt more conservative risk management strategies.
• Evolving Knowledge and Practices:
  As knowledge about AI systems and their impacts evolves, so too will the methods for managing risks. This evolution requires organizations to stay informed and adaptable, ensuring that their risk management practices remain relevant and effective over time.

3. Risk Prioritization

Risk prioritization is a crucial aspect of AI risk management, ensuring that resources are allocated efficiently to address the most significant risks first. The AI Risk Management Framework (AI RMF) outlines several key principles and considerations for effective risk prioritization.

• Recognizing the Inevitability of Some Risks:
  Attempting to eliminate all negative risks is impractical. Organizations must recognize that not all incidents and failures can be completely eradicated. Unrealistic expectations about risk can lead to inefficient resource allocation and ineffective risk triage. A culture of risk management helps organizations understand that not all AI risks are equal, allowing for purposeful resource allocation.
• Assessment of Trustworthiness:
  Actionable risk management efforts should provide clear guidelines for assessing the trustworthiness of each AI system. Policies and resources should be prioritized based on the assessed risk level and potential impact of the AI system. This includes considering the extent to which an AI system may be customized or tailored to its specific context of use.
• Urgent Prioritization for High-Risk Systems:
  AI systems that present the highest risks within a given context of use should receive the most urgent prioritization and the most thorough risk management process. In cases where an AI system poses unacceptable negative risk levels, such as imminent severe harms or catastrophic risks, development and deployment should cease in a safe manner until these risks are sufficiently managed.
• Differentiation Based on Interaction with Humans:
  AI systems that directly interact with humans may require higher initial prioritization compared to those that do not. This is especially true in settings where AI systems are trained on large datasets containing sensitive or protected data, or where their outputs have direct or indirect impacts on humans. Conversely, AI systems interacting only with computational systems and trained on non- sensitive datasets might call for lower initial prioritization.
• Regular Assessment and Contextual Adjustment: Regular assessment and prioritization of risks based on context are vital. Even non-human-facing AI systems can have downstream safety or social implications that necessitate ongoing risk evaluation and adjustment. Residual risk, defined as the risk remaining after treatment, directly impacts end users and affected communities. Documenting these risks ensures that the system provider fully considers the risks of deploying the AI product and informs end users about potential negative impacts.

4. Organizational Integration and Management of Risk

Effective AI risk management necessitates an integrated approach that aligns with broader enterprise risk management strategies. The AI Risk Management Framework (AI RMF) highlights several key principles and practices for integrating AI risk management into organizational processes:

Microsoft Word – CI 5 – Artificial Intelligence Risk Management Framework

• Holistic Approach:
  AI risks should not be considered in isolation. Instead, they should be integrated into broader enterprise risk management strategies. This includes aligning AI risk management with other critical risk areas such as cybersecurity and privacy. Treating AI risks alongside these other risks can yield more integrated outcomes and organizational efficiencies.
• Accountability Mechanisms:
  Establishing and maintaining appropriate accountability mechanisms is crucial. This involves defining roles and responsibilities, creating a culture of risk management, and setting up incentive structures that promote effective risk management. Senior leadership must commit to these processes, as organizational commitment at higher levels is essential for success.
• Cultural Change:
  Implementing effective risk management may require a cultural change within the organization or industry. This change is necessary to foster a purpose-driven culture focused on understanding and managing risks. Continuous execution of risk management functions, aligned with evolving knowledge, cultures, and expectations from AI actors, is imperative.
• Integration with Existing Frameworks:
  The AI RMF can be utilized alongside related guidance and frameworks for managing AI system risks or broader enterprise risks. Many risks related to AI systems overlap with those of traditional software development and deployment, such as privacy concerns, security issues, and the environmental impact of resource-heavy computing demands.
• Processes and Procedures:
  Organizations need to have policies, processes, procedures, and practices in place across the organization to effectively map, measure, and manage AI risks. This includes understanding legal and regulatory requirements, integrating trustworthy AI characteristics into organizational policies, and ensuring transparency in risk management activities.
• Ongoing Monitoring and Review:
  Regular monitoring and periodic review of the risk management process and its outcomes are essential. This involves clearly defining organizational roles and responsibilities and determining the frequency of reviews. Mechanisms should also be in place to inventory AI systems and decommission them safely when necessary.
• Diverse and Inclusive Teams:
  Decision-making related to AI risk management should involve a diverse team with various demographics, disciplines, experiences, expertise, and backgrounds. This diversity enhances the ability to surface problems and identify both existing and emergent risks.

2. AI Actors

The AI Risk Management Framework (AI RMF) outlines several key AI actors who play significant roles throughout the AI lifecycle. These actors are responsible for different tasks in various phases, ensuring that AI systems are developed, deployed, and managed in a trustworthy and responsible manner. The following are the primary AI actors and their roles across the lifecycle dimensions:

1. AI Design Actors:

• Tasks: Creating the concept and objectives of AI systems, planning, designing, data collection, and processing.
• Roles: Data scientists, domain experts, socio-cultural analysts, diversity experts, human factors experts (e.g., UX/UI design), governance experts, data engineers, data providers, system funders, product managers, third-party entities, evaluators, and legal and privacy governance experts.
• Phase: Application Context and Data and Input phases.
• Responsibilities: Ensuring the AI system is lawful and fit-for-purpose, documenting the system’s concept and objectives, gathering and cleaning data, and documenting metadata and dataset characteristics.

2. AI Development Actors:

• Tasks: Model building and interpretation, creation, selection, calibration, training, and/or testing of models or algorithms.
• Roles: Machine learning experts, data scientists, developers, third-party entities, legal and privacy governance experts, and experts in socio-cultural and contextual factors associated with the deployment setting.
• Phase: AI Model phase.
• Responsibilities: Providing the initial infrastructure of AI systems, ensuring model accuracy and reliability, and addressing legal and privacy concerns.

3. AI Deployment Actors:

• Tasks: Contextual decisions on AI system usage, piloting, compatibility checks, regulatory compliance, organizational change management, and user experience evaluation.
• Roles: System integrators, software developers, end users, operators and practitioners, evaluators, and domain experts in human factors, socio-cultural analysis, and governance.
• Phase: Task and Output phase.
• Responsibilities: Ensuring the system is deployed into production safely and effectively, managing organizational changes, and evaluating the user experience to ensure the system meets user needs and regulatory requirements

4. Operation and Monitoring Actors:

• Tasks: Regular assessment of system output and impacts, operating the AI system.
• Roles: System operators, domain experts, AI designers, users, product developers, evaluators, auditors, compliance experts, organizational management, and researchers.
• Phase: Application Context/Operate and Monitor phase.
• Responsibilities: Ongoing monitoring and periodic updates, tracking incidents or errors, and managing detected emergent properties and impacts.

5. Test, Evaluation, Verification, and Validation (TEVV) Actors:

• Tasks: Examining the AI system or its components, detecting and remediating problems, validating assumptions for design and data, model validation and assessment, system validation and integration in production, ongoing monitoring, testing, and recalibration.
• Roles: TEVV experts, separate from those performing development tasks.
• Phase: Through out the AI lifecycle.
• Responsibilities: Ensuring the system meets technical, societal, legal, and ethical standards, providing mid-course remediation, and post-hoc risk management.

6. Human Factors Actors:

• Tasks: Human-centred design practices, active involvement of end users and other stakeholders, incorporating context-specific norms and values, evaluating and adapting user experiences.
• Roles: Human factors professionals, UX/UI designers, sociologists, psychologists, and other relevant experts.
• Phase: Through out the AI lifecycle.
• Responsibilities: Ensuring that AI systems align with human needs and values, designing user- friendly interfaces, and performing human-centred evaluation and testing.

7. Domain Experts:

• Tasks: Providing knowledge or expertise in specific industry sectors, economic sectors, or application areas.
• Roles: Multidisciplinary practitioners or scholars.
• Phase: Through out the AI lifecycle.
• Responsibilities: Guiding AI system design and development, interpreting outputs, and supporting TEVV and impact assessment teams.

8. AI Impact Assessment Actors:

• Tasks: Assessing and evaluating AI system accountability, combating harmful bias, examining impacts, ensuring product safety, and managing liability and security.
• Roles: Impact assessors, evaluators with technical, humanfactor, socio-cultural, and legal expertise.
• Phase: Through out the AI lifecycle.
• Responsibilities: Ensuring that AI systems are accountable and do not introduce harmful biases, maintaining product safety, and managing potential legal and security issues.

9. Governance and Oversight Actors:

• Tasks: Oversight of AI system performance, ensuring compliance with policies and regulations.
• Roles: Organizational management, senior leadership, Board of Directors.
• Phase: Through out the AI lifecycle.
• Responsibilities: Ensuring the organization’s AI systems align with its values and principles, overseeing the implementation of risk management practices, and maintaining organizational accountability.

Collaboration for Trustworthy and Responsible AI

These AI actors work together across the lifecycle to manage risks and achieve the goals of trustworthy and responsible AI by:

• Integrating TEVV Tasks: TEVV-specific expertise is integrated throughout the AI lifecycle, providing insights relative to technical, societal, legal, and ethical standards, anticipating impacts, and assessing and tracking emergent risks. Regular TEVV processes allow for mid-course remediation and post-hoc risk management.
• Engaging Diverse Perspectives: Successful risk management depends on collective responsibility among AI actors with diverse perspectives, disciplines, and experiences. This diversity promotes open sharing of ideas and assumptions, surfacing problems, and identifying existing and emergent risks.
• Promoting Human-Centred Design: Human factors actors ensure that AI systems are designed with the end-user in mind, promoting usability and alignment with societal values and norms. This human- centred approach is critical for achieving trustworthy AI.
• Ensuring Continuous Monitoring and Feedback: Operation and monitoring actors provide ongoing assessment of AI systems, incorporating feedback from end users and other stakeholders to ensure the system continues to perform as intended and to address any emergent issues promptly.

By integrating these diverse roles and responsibilities, organizations can manage AI risks effectively and develop AI systems that are not only technically sound but also socially responsible and trustworthy.

3. Characteristics of Trustworthy AI and Proposed Guidance

The AI Risk Management Framework (AI RMF) identifies several key characteristics of trustworthy AI systems and provides guidance on how to address these characteristics. The main characteristics of trustworthy AI include being valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful biases managed.

1. Valid and Reliable

• Description: Validity and reliability ensure that AI systems operate correctly and as intended under the expected conditions of use. This includes accuracy, robustness, and the ability to generalize beyond training data.
• Guidance: Ongoing testing and monitoring are essential to confirm that AI systems continue to perform as intended. Validation and reliability assessments should include computational-centric measures and human-AI teaming evaluations, ensuring external validity.

2. Safe

• Description: AI systems should operate without causing harm to human life, health, property, or the environment under defined conditions.
• Guidance: Implement responsible design, development, and deployment practices, provide clear information to deployers, and ensure robust decision-making by deployers and end users. Safety risks should be managed with rigorous simulation, in-domain testing, real-time monitoring, and the ability to shut down or modify systems as needed.

3. Secure and Resilient

• Description: AI systems should withstand unexpected adverse events and maintain their functionality and structure. Security involves maintaining confidentiality, integrity, and availability.
• Guidance: Follow guidelines from the NIST Cybersecurity Framework and Risk Management Framework. Security concerns such as adversarial examples and data poisoning must be addressed with appropriate protection mechanisms.

4. Accountable and Transparent

• Description: Transparency ensures that information about the AI system and its outputs is available, and accountability ensures that AI actors are responsible for the AI system’s outcomes.
• Guidance: Promote transparency by providing access to information based on the stage of the AI lifecycle and the role of AI actors. Enhance accountability by maintaining organizational practices and governing structures that support harm reduction and risk management.

5. Explainable and Interpretable

• Description: Explainability refers to understanding the mechanisms of AI operation, while interpretability refers to understanding the meaning of AI outputs in context.
• Guidance: Provide descriptions of AI system functionality tailored to different users, and ensure that explanations are clear and contextually appropriate. Transparency, explainability, and interpretability should be interlinked to support deeper insights into AI systems.

6. Privacy-Enhanced

• Description: AI systems should safeguard human autonomy, identity, and dignity by adhering to privacy norms and practices.
• Guidance: Implement privacy-enhancing technologies(PETs) and data minimization methods such as de-identification and aggregation. Balance privacy with other characteristics like accuracy and fairness.

Microsoft Word – CI 5 – Artificial Intelligence Risk Management Framework

7. Fair with Harmful Bias Managed

• Description: Fairness in AI involves addressing harmful biases and discrimination to promote equality and equity.
• Guidance: Recognize and manage different types of biases, including systemic, computational, and human-cognitive biases. Address these biases throughout the AI lifecycle to ensure that AI systems do not exacerbate existing disparities or systemic biases.

Proposed Guidance to Address Overall Trustworthiness Characteristics

• Holistic Evaluation: Address each characteristic individually and collectively, recognizing that trade-offs may be necessary. Decisions should be made transparently and justifiably based on the context.
• Ongoing Monitoring and Adaptation: Continuously monitor AI systems and update risk management practices as technology and societal norms evolve.
• Inclusive and Diverse Teams: Engage diverse stakeholders and experts throughout the AI lifecycle to inform context-sensitive evaluations and identify benefits and risks.
• Documentation and Communication: Maintain thorough documentation and transparent communication to support accountability and trust.

By following these guidelines, organizations can enhance the trustworthiness of their AI systems and ensure they are developed and deployed responsibly.

4. Benefits of Evaluating AI Risk Management Effectiveness

Periodically evaluating the effectiveness of AI risk management processes offers numerous benefits, enhancing the trustworthiness and overall performance of AI systems. The following are the potential benefits as outlined in the AI Risk Management Framework (AI RMF):

1. Enhanced Processes for Governing, Mapping, Measuring, and Managing AI Risk:

• Regular evaluations improve the processes used to govern, map, measure, and manage AI risks. Thiscontinuous improvement ensures that these processes remain effective and relevant as AI technologies and their applications evolve.

2. Improved Awareness of Relationships and Trade-offs:

• Evaluations help organizations better understand the relationships and trade-offs among various trustworthiness characteristics, socio-technical approaches, and AI risks. This awareness enables more informed decision-making and balanced risk management strategies.

3. Explicit Processes for Decision Making:

• Periodic evaluations establish clear processes for making go/no-go decisions regarding the commissioning and deployment of AI systems. This helps organizations manage risks more effectively by ensuring that critical decisions are based on comprehensive risk assessments.

4. Enhanced Organizational Accountability:

• Regular assessments of AI risk management practices enhance organizational accountability. By establishing and adhering to documented policies, processes, and procedures, organizations can demonstrate their commitment to managing AI risks responsibly.

5. Improved Organizational Culture:

• Periodic evaluations contribute to the development of an organizational culture that prioritizes the identification and management of AI system risks. This cultural shift encourages proactive risk management and continuous improvement in AI practices.

6. Better Information Sharing:

• Evaluations promote better information sharing within and across organizations about risks, decision-making processes, responsibilities, common pitfalls, and best practices. This collaborative approach enhances the overall effectiveness of AI risk management efforts.

7. Increased Awareness of Downstream Risks:

• Regular evaluations provide contextual knowledge that increases awareness of downstream risks. Understanding these risks enables organizations to take appropriate measures to mitigate potential negative impacts on end users and affected communities.

8. Strengthened Engagement with Stakeholders:

• Evaluations foster stronger engagement with relevant stakeholders and AI actors. This engagement ensures that diverse perspectives are considered in risk management practices, leading to more robust and inclusive AI systems.

9. Augmented Capacity for Testing, Evaluation, Verification, and Validation (TEVV):

• Periodic evaluations enhance an organization’s capacity for TEVV of AI systems and associated risks. This ongoing assessment ensures that AI systems continue to meet safety, security, and reliability standards throughout their lifecycle.

By periodically evaluating AI risk management effectiveness, organizations can ensure that their AI systems remain trustworthy, reliable, and aligned with evolving best practices and societal expectations. This continuous improvement cycle is essential for managing the inherent uncertainties and dynamic risks associated with AI technologies.

5. Special Functions to Address AI Risks in Practice

The AI Risk Management Framework (AI RMF) outlines four specific functions to help organizations address AI risks effectively: GOVERN, MAP, MEASURE, and MANAGE. Each function has categories and subcategories designed to ensure comprehensive risk management across the AI lifecycle.

1. GOVERN:

• Purpose: Cultivates and implements a culture of risk management within organizations involved in designing, developing, deploying, evaluating, or acquiring AI systems.
• Key Activities:

• Establishing and documenting organizational policies, processes, and procedures related to AI risk management.
• Ensuring compliance with legal and regulatory requirements.
• Integrating trustworthy AI characteristics into organizational policies.
• Defining roles and responsibilities for managing AI risks.
• Implementing mechanisms for ongoing monitoring and periodic review of risk management processes.

• Outcome: Enhanced organizational accountability, transparency, and a culture that prioritizes theidentification and management of AI risks.

2. MAP:

• Purpose: Establishes the context to frame risks related to an AI system by defining and understanding the system’s context, including its purpose, environment, and potential impacts.
• Key Activities:

• Defining and documenting intended purposes, beneficial uses, context-specific laws, norms, and prospective deployment settings.
• Identifying interdisciplinary AI actors and competencies.
• Categorizing the AI system’s tasks and methods, understanding system knowledge limits, and documenting human oversight requirements.

• Outcome: Improved understanding of AI system impacts, enabling informed decisions about AI design, development, and deployment.

Microsoft Word – CI 5 – Artificial Intelligence Risk Management Framework

3. MEASURE:

• Purpose: Quantifies and assesses AI risks, trustworthiness, and impacts using appropriate metrics and methodologies.
• Key Activities:

• Establishing quantitative and qualitative metrics for AI performance and trustworthiness. o Conducting risk assessments and impact analyses.
• Monitoring AI systems for emergent risks and performance degradation over time.

• Outcome: Reliable and valid measurement of AI risks and impacts, ensuring that AI systems perform as intended and remain trustworthy throughout their lifecycle.

4. MANAGE:

• Purpose: Implements risk treatment strategies and controls to mitigate identified risks and optimize AI system performance.
• Key Activities:

• Developing and applying risk mitigation strategies.
• Implementing controls to manage and reduce AI system risks.
• Continuously monitoring and adjusting risk management practices based on feedback and evolving risks.
• Outcome: Effective management of AI risks, leading to safer, more reliable, and trustworthy AI systems.

These functions provide a structured approach for organizations to manage AI risks comprehensively. By integrating these functions into their risk management practices, organizations can ensure that their AI systems are developed and deployed responsibly, maintaining high standards of trustworthiness and minimizing potential negative impacts.