U.S. DEPARTMENT OF DEFENSE RESPONSIBLE ARTIFICIAL INTELLIGENCE STRATEGY AND IMPLEMENTATION PATHWAYTo ensure that our citizens, warfighters, and leaders can trust the outputs of DoD AJ capabilities, DoD must demonstrate that our military's steadfast commitment to lawful and ethical behavior apply when designing, developing, testing, procuring, deploying, and using Al. The Responsible AI (RAI) Strategy and Implementation (S&I) Pathway illuminates our path forward by defining and communicating our framework for harnessing AI. It helps to eliminate uncertainty and hesitancy - and enables us to move faster. Integrating ethics from the start also empowers the DoD to maintain the trust of our allies and coalition partners as we work alongside them to promote democratic norms and international standards. Prepared by the DoD Responsible AI Working Council in accordance with the memorandum issued by Deputy Secretary of Defense Kathleen Hicks on May 26, 2021, Implementing Responsible Artificial Intelligence in the Department of Defense. June 2022U.S. Department of DefenseDoD_5e8dcfdc-5d6a-11df-839d-400e7a64ea2aKathleen H. HicksDeputy Secretary of Defense Chief Digital and Artificial Intelligence Officer (CDAO)As outlined in the RAl Memo, the Office of the Chief Digital and Artificial Intelligence Officer (CDAO), as the successor organization to the Joint Artificial Intelligence Center (JAJC), serves as the Depaitment's lead for coordinating the implementation and oversight of guidance and policy on Al, including RAI and the DoD Al Ethical Principles. The CDAO is responsible for enabling, assessing, and tracking the implementation of a DoD RAI ecosystem, with support from the DoD Components... ^ As the DoD's lead for AI, the CDAO is designated within this Pathway as the OPR for many Department-wide Al activities, especially those that involve the creation of Al-enabling tools to be used across the DoD in support of its RAI approach. The CDAO should leverage the existing technical foundation set by the Do D's R&D enterprise and the commercial sector to the greatest extent possible as it continues to coordinate the development of these tools with DoD Component end users.DoD ComponentsThis includes the Office of the Assistant to the Secretary of Defense for Privacy, Civil Liberties, and Transparency (ATSD(PCLT)), and the Joint Staff, and Military Departments, all of whom are represented through the RAl Working Council... ^ All DoD Components must ensure that their Al capabilities are in alignment with the DoD Ethical Principles, and that their policies and practices enable RAI implementation. The outputs of this RAI S&l Pathway will. offer tools and guidance for DoD Components to accomplish this task. For the purpose of this Pathway, select DoD Components have been designated as OPRs to serve as a lead coordinating entity for their respective Line of Effort (LOEs). Many of the LOEs are complex, cross-cutting, and involve multiple external stakeholders and effectively completing them will require Department-wide input based on existing policy authorities, staffing requirements, and well-founded subject matter expertise. Those LOEs have been designated to OPRs believed appropriate to execute such coordination. RAI Working CouncilThe RAI Working Council was created via the RAI Memo as an initial working body to ensure Department-wide input and coordination on the development oftrus RAI S&I Pathway. With the chartering of the CDAO and its governing processes, the RAI Working Council will be a permanent working group that reports to tbe CDAO's governing council, a 4-star level governance body run by the CDAO to oversee all aspects of data, analytics, and AI for the Department. This Pathway directs member organizations that comprise the current RAI Working Council to designate or hire RAI Leads who will be responsible for implementing this DoD RAI S&l Pathway in their respective organizations and provide reporting on such progress, and relevant barriers for removal, to the RAJ Working Council, and up to the Deputy Secretary of Defense, when necessary, through the CDAO governing council. The RAI Working Council should update its membership to reflect the designated RAI Leads, fonnalize its role through a charter as approved by the CDAO, and assist the CDAO in reporting on the progress of DoD-wide RAl implementation.Trust in DoD Al_be417f74-f596-11ec-a68d-e5082683ea00To enable the Department to modernize its warfighting capability across a range of combat and non-combat applications_be418000-f596-11ec-a68d-e5082683ea00ResponsibilityRESPONSIBLE ~ DoD personnel will exercise appropriate levels of judgment and care, while remaining responsible for the development, deployment, and use of Al capabilities.EquityEQUITABLE ~ The Department will take deliberate steps to minimize unintended bias in Al capabilities.TraceabilityTRACEABLE ~ The Department's AI capabilities will be developed and deployed such that relevant personnel possess an appropriate understanding of the technology, development processes, and operational methods applicable to AI capabilities, including with transparent and auditable methodologies, data sources, and design procedures and documentation.ReliabilityRELIABLE ~ The Department's Al capabilities will have explicit, well-defined uses, and the safety, security, and effectiveness of such capabilities will be subject to testing and assurance within those defined uses across their entire life-cycles.GovernanceGOVERNABLE ~ The Department will design and engineer AI capabilities to fulfill their intended functions while possessing the ability to detect and avoid unintended consequences, and the ability to disengage or deactivate deployed systems that demonstrate unintended behavior.RAI GOVERNANCEEnsure disciplined governance structure and processes at the Component and DoDwide levels for oversight and accountability and clearly articulate DoD guidelines and policies on RAI and associated incentives to accelerate RAI adoption within the DoD._be418082-f596-11ec-a68d-e5082683ea00TENET 1Goal: Modernize governance structures and processes that allow for continuous oversight of DoD use of Al, taking into account the context in which the technology will be used. ^ Governance structures and processes will enable the appropriate assessment of risks and the mitigation of unintended consequences or bias in AI capabilities. Users or developers will also have clear mechanisms to implement the DoD AI Ethical Principles and to report potential concerns. OVERSIGHT & ACCOUNTABILITYBUILD ORGANIZATIONAL CAPACITY TO IMPROVE OVERSIGHT AND ACCOUNTABILITY OF DOD AI. _be418208-f596-11ec-a68d-e5082683ea00LOE 1.1StaffingFully staff the CDAO RAI Office with expertise in Al technology, policy, acquisition, workforce, and governance. _a06ebcae-f759-11ec-b196-a1f52b83ea00LOE 1.1.1CDAORAI LeadsRAI Working Council member organizations will designate or hire DoD Component RAI Leads to lead the implementation of this DoD RAJ S&I Pathway within their respective Components._a06ebee8-f759-11ec-b196-a1f52b83ea00LOE 1.1.2RAI Working Council MembersDoD Component RAI Leads will be provided with the adequate authorities, staffing, and resources to fulfill all relevant duties. Transformation MetricsAs part of DoD-wide metrics to measure Al transformation, develop metrics to measure RAI adoption, including progress on individual LOEs identified in this DoD RAl S&I Pathway, and report progress as required to the CDAO governing council. _a06ebfba-f759-11ec-b196-a1f52b83ea00LOE 1.1.3CDAOComponent-Specific MetricsDevelop DoD Component-specific metrics as informed RAI by LOE 1.1.3 to measure RAJ adoption, including progress on individual LOEs identified in th.is DoD RAI S&T Pathway, and establish reporting mechanisms. _a06ec0c8-f759-11ec-b196-a1f52b83ea00LOE 1.1.4DoD Component LeadsExpertise & StaffingAppropriately staff DoD- and Component-level internal oversight bodies to ensure appropriate expertise is in place to conduct robust and effective oversight of DoD's use of AI under their roles and authorities. _a06ec38e-f759-11ec-b196-a1f52b83ea00LOE 1.1.5DoD Inspector GeneralAssistant to the Secretary of Defense for Privacy, Civil Liberties, and Transparency (ATSD(PCLT))DOT &EDDirector for Developmental Test, Evaluation, and Assessments (DTE&A)ConcernsIdentify methods for users and developers to report concerns about the implementation of the DoD Al Ethical Principles. _a06ec47e-f759-11ec-b196-a1f52b83ea00LOE 1.1.6DoD Component RAI LeadsEnsure such methods are clearly communicated to users and developers.GovernanceUpdate the Department's existing governance framework for Al development and delivery. _a06ec546-f759-11ec-b196-a1f52b83ea00LOE 1.1.7CDAOTOOLS & RESOURCESPROVIDE TOOLS AND RESOURCES TO SUPPORT A DOD-WIDE RAJ GOVERNANCE STRUCTURE, INCLUDING THROUGH COORDINATED AND REGULAR KNOWLEDGE-SHARING. _a06ec5fa-f759-11ec-b196-a1f52b83ea00LOE 1.2ReportingReport to the CDAO (no less than once per year) exemplary Al use cases, identifying best practices, failure modes, and risk mitigation strategies, including after-action reports as appropriate._a06ec6ae-f759-11ec-b196-a1f52b83ea00LOE 1.2.1DoD Component RAI LeadsIdentify capability use cases, training in system capabilities, and docwnentation on how to employ and retire systems responsibly, in order to support the safe and responsible employment of AI capabilities by operators. Include any reports of concerns per LOE 1.1.6. RepositoryCreate and maintain a DoD-wide central repository of exemplary AI use cases and any supplementary information to support coordinated and regular knowledge-sharing of best practices and risk mitigation. _a06ec76c-f759-11ec-b196-a1f52b83ea00LOE 1.2.2CDAOBarriersWithin six months of this Pathway's approval, report to the CDAO any perceived significant barriers - including gaps in infrastructure required to support traceability, auditability, risk analysis, and forensics, and recommended hardware, software, or other infrastructure needs - necessary to fulfill RAI and AI requirements and best practices. _a06ec82a-f759-11ec-b196-a1f52b83ea00LOE 1.2.3DoD Component RAl LeadsUpdate annually. InventoryTo maintain the DoD AI Inventory in accordance with Public Law 116-260 House Report Division C, identify and report all DoD AI activities to the CDAO, including program's appropriation, and project, and budget line number; current and future years' defense program funding; names of academic or industry mission partners, if applicable; and any planned transition partners, if applicable. _a06ec8de-f759-11ec-b196-a1f52b83ea00LOE 1.2.4DoD Component RAI Leadsin coordination with ...CDAOOUSD(A&S)Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E))STRATEGIC PLANNINGENSURE THAT RAl IS INCORPORATED IN DOD'S STRATEGIC PLANNING EFFORTS. _a06ec9a6-f759-11ec-b196-a1f52b83ea00LOE 1.3StrategiesIncorporate RAl and elements of the DoD RAI S&l Pathway as appropriate into the DoD strategies for data, analytics, and Al adoption. _a06eca64-f759-11ec-b196-a1f52b83ea00LOE 1.3.1CDAOPlanning GuidanceEnsure RAT is incorporated into the Defense Planning Guidance for resourcing planning purposes including but not limited to manpower, tools, education and training, and post-deployment monitoring and retraining. _a06ecb2c-f759-11ec-b196-a1f52b83ea00LOE 1.3.2Office of the Under Secretary of Defense for Policy (OUSD(P))PrioritizationIdentify implementing RAI as a priority in DoD Military Department AI strategies and plans and incorporate Service-specific actions as appropriate. _a06ecc44-f759-11ec-b196-a1f52b83ea00LOE 1.3.3Military Department RAI LeadsWARFIGHTINGUPDATE DOD'S REVIEW PROCESSES FOR WARFIGHTING CAPABILITIES TO SUPPORT IMPLEMENTATION OF THE DOD AI ETHICAL PRINCIPLES. _a06ecd20-f759-11ec-b196-a1f52b83ea00LOE 1.4Review ProcedureExplore whether a review procedure is needed to ensure DoD warfighting capabilities will be consistent with the DoD AI Ethical Principles._a06ecdf2-f759-11ec-b196-a1f52b83ea00LOE 1.4.1CDAOin coordination with the ...Military Department RAI LeadsOUSD(P)Department of Defense Office of General Counsel (OGC)Provide a recommendation to the Deputy Secretary of Defense on whether such a review procedure should be created, identifying any gaps in existing processes, gaps in requisite policies supplementing the DoD AI Ethical Principles, and potential in1pacts on development and fielding timelines if such review is required. ^ Explore whether or how Legal review processes can support implementation of the DoO Al Ethical Principles, including the review of the legality of weapons per DoDDs 2311.01, 5000.01, 3000.03E, and 3000.09. GuidanceUpdate or supplement DoDD 3000.09 with guidance on the DoD Al Ethical Principles._a06ed068-f759-11ec-b196-a1f52b83ea00LOE 1.4.2OUSD(P)in coordination with ...CDAOInclude any additional information requirements to fulfill the DoD Al Ethical Principles as part of the senior review package. WARFIGHTER TRUSTEnsure warfighter trust by providing education and training, establishing a test and evaluation and verification and validation framework that integrates real-time monitoring, algorithm confidence metrics, and user feedback to ensure trusted and trustworthy AI capabilities. _a06ed14e-f759-11ec-b196-a1f52b83ea00TENET 2WarfightersGoal: Achieve a standard level of technological familiarity and proficiency for system operators to achieve justified confidence io Al capabilities and AI-enabled systems. ^ Trustworthiness is bolstered by the application of TEVV frameworks that allow for the monitoring of system performance, reliability, unintended behavior, and failure modes before fielding the system and during operation. The combination of these factors contributes to a greater understanding of an Al's capabilities and limitations, which will be critical for the development of an Al-ready force. TEVVBUILD A ROBUST TEVV ECOSYSTEM AND ACCOMPANYING INFRASTRUCTURE TO DEVELOP AND FIELD AI CAPABILITIES SAFELY AND SECURELY. _a06ed22a-f759-11ec-b196-a1f52b83ea00LOE 2.1CDAOFrameworkDevelop a TEVV framework to articulate how test and evaluation (T&E) should be intertwined across an Al capability's lifecycle and pathways for continuous testing and standards for documentation and reporting._a06ed36a-f759-11ec-b196-a1f52b83ea00LOE 2.1.1CDAOin coordination with ...OUSD(R&E)Director, Operational Test and Evaluation (DOT&E)Identify synergies between AI T&E and traditional T&E (e.g. effectiveness, suitability, security, safety) to empower programs to streamline T&E efforts. Include guidance for operationalizing RAT principles into testable conjectures for common technologies, mission domains, and uses cases. T&E ToolsDevelop or acquire Al-related T&E tools to be used as a resource for Al developers and testers._fab60608-f7d7-11ec-b791-596aff82ea00LOE 2.1.2CDAOin coordination withOUSD(R&E)DOT&EThis AI T &E Toolkit shall draw upon best practices and innovative research from industry and the academic community, as well as commercially available technology where appropriate The Toolkit will be made widely available to DoD users and shall include: ^ a) Tangible, concrete guidance for PMs, testers, and other relevant T&E stakeholders about how to implement RAI T &E throughout a capability's lifecycle; ^ b) A T&E Master Plan template for A1 and a set of templates for test plans; ^ c) A library of T &E metrics for AI systems, including metrics for uustworthiness and confidence; and ^ d) Necessary tools and technologies required to detect both natural degradation of and adversarial attacks on Al, including those to detect various attacks on AI systems, and that can notify testers or operators when such attacks are occurring.Test Range & RepositoryCreate a test range environment and central repository of tools for T&E of Al, linking to existing and emerging equivalent DoD Component environments, that enables easy and continuous testing for DoD testers._fab609aa-f7d7-11ec-b791-596aff82ea00LOE 2.1.3CDAOin coordination with the ...DoD Test Resource Management CenterWhere appropriate, tools that are housed in this environment should comply with DoD Enterprise DevSecOps Reference Designs for portability across the Department. Best PracticesEstablish best practices and requirements for utilization of the T&E Master Plan and test plan templates for AI and update appropriate DoD issuances and Military Standards (MIL-STDs)._fab60bc6-f7d7-11ec-b791-596aff82ea00LOE 2.1.4OUSD(R&E)in coordination with ...CDAODOT&EHuman SystemsRegularly update the CDAO's Human Systems Integration (HSI) Framework based on user feedback and provide clear guidance on how and when the framework should be used tohelp address system design, system performance, User Experience/User Interface (UX/UI), and user training. _fab611fc-f7d7-11ec-b791-596aff82ea00LOE 2.1.5CDAOin coordination with the ...Joint HSI Steering CommitteeResearchContinue research into emerging AI topics _fab614b8-f7d7-11ec-b791-596aff82ea00LOE 2.1.6OUSD(R&E)in coordination with ...CDAOMilitary Service Labssuch as: ^ a) HSI practices to the inform the design and developments of AI capabilities that can be used consistent with the DoD AI Ethical Principles; ^ b) New methods for TEVV of Al; and ^ c) Al security and defense to protect against adversarial attacks.SecurityDevelop and distribute DoD-wide guidance for AI security, leveraging existing best practices for risk management supply chain security, and cybersecurity._fab616c0-f7d7-11ec-b791-596aff82ea00LOE 2.1.7CDAOin coordination with ...DoD Chief Information Officer (CIO)This guidance should be updated as the field matures. BEST PRACTICESDEVELOP BEST PRACTICES TO INCORPORATE OPERATOR AND SYSTEM FEEDBACK THROUGHOUT THE Al LIFECYCLE, ENSURE ADEQUATE DOCUMENTATION AND TRAINING, AND DELINEATE CLEAR ROLES AND RESPONSIBJLITIES FOR DEVELOPERS AND USERS OF THE Al CAPABILITY. _fab61a08-f7d7-11ec-b791-596aff82ea00LOE 2.2Training & DocumentationRequire Al vendors and developers to plan for, resource, and provide appropriate training and documentation, such as user manuals, to be used prior to capability fielding in order to ensure warfighters are equipped with an appropriate understanding of the capability's function, risks, performance expectations, and potential harms. _fab61c9c-f7d7-11ec-b791-596aff82ea00LOE 2.2.1DoD Component RAI LeadsAl VendorsAl DevelopersTraceable FeedbackRequire AI vendors and developers who design systems and create features with human-facing interfaces to provide traceable feedback on systems status and clear procedures to trained operators to activate and deactivate system functions._fab61f94-f7d7-11ec-b791-596aff82ea00LOE 2.2.2DoD Component RAI LeadsAI VendorsAI DevelopersThis information should be used to update training, documentation, interfaces, and/or other components as appropriate. Responsibilities & AuthoritiesPromulgate guidance in accordance with LOE 3.1.2 and 3.2.2 delineating responsibilities and authorities for Program Offices ... _fab622d2-f7d7-11ec-b791-596aff82ea00LOE 2.2.3PEOs Program Offices DoD Componentsto: ^ a) Monitor perfo1mance of their AJ systems after fielding (including guidance on metrics for monitoring and system instrumentation tools to support this); and ^ b) Establish processes for operators of Al-enabled capabilities to notify and report changes in capability performance, outcomes, emergent behavior, and/or disengagement in accordance with existing DoD processes. PRODUCTS & ACQUISITIONSDevelop tools, policies, processes, systems, and guidance to synchronize enterprise RAl implementation for the Al product throughout the acquisition lifecycle through a systems engineering and risk management approach, while enabling Al development at the pace the Department needs to meet the National Defense Strategy._fab6257a-f7d7-11ec-b791-596aff82ea00TENET 3AI PRODUCT AND ACQUISITION LIFECYCLE ~ Goal: Exercise appropriate care in the Al product and acquisition lifecycle to ensure potential Al risks are considered from the outset of an Al project, and efforts are taken to mitigate or ameliorate such risks and reduce the likelihood of unintended consequences while enabling Al development at the pace the Department needs to meet the National Defense Strategy. This includes robust documentation to understand, test, and acl on infonned risk assessments, recognizing that needs will vary based on the level of technical maturity, sensitivity, and context in which the Al capability will be used. RESOURCES & TOOLSDEVELOP RAI-RELATED ACQUISITION RESOURCES AND TOOLS, SUCH AS STANDARD LANGUAGE FOR ANNOUNCEMENTS AND CONTRACTS, AND BEST PRACTICES THAT LEVERAGE THE D0D'S ADAPTIVE ACQUISITION PATHWAYS AND ALLOW THE DOD TO IDENTIFY AND MITIGATE RISKS THROUGHOUT THE ACQUISITION AND SUSTAINMENT PROCESS. _fab6276e-f7d7-11ec-b791-596aff82ea00LOE 3.1Acquisition ToolkitDevelop an Acquisition Toolkit that draws upon best practices and innovative research from the DoD enterprise, industry, and the academic community, as well as commercially available technology where appropriate._fab62a8e-f7d7-11ec-b791-596aff82ea00LOE 3.1.1CDAOin coordination with ...OUSD(A&S)OUSD(R&E)The Toolkit will be made widely available to DoD users and shall include: ^ a) Standard language in the initial announcement, request for proposal (RFP), and request for information (RFI) for AI capabilities to provide guidance on how vendors and developers can meet the DoD Al Ethical Principles; ^ b) A set of RAI-related evaluation criteria that are testable and operationally relevant; ^ c) Standard AI contract language that provides clauses for: independent government T&E of AI capabilities, methods of immediate remediation when the vendor-provided Al capabilities cannot be used in accordance with the DoD AI Ethical Principles, requesting training and documentation from vendors, performance monitoring of AI capabilities, and appropriate data deliverables and rights; and ^ d) Any other resources as appropriate. EngagementIdentify and develop approaches for continuous engagement of RAl expertise within the DoD's six adaptive acquisition pathways (Urgent Capability Acquisition, Middle Tier of Acquisition, Major Capability Acquisition, Software Acquisition, Defense Business Systems, and Acquisition of Services) to address RAI risk considerations. _fab63132-f7d7-11ec-b791-596aff82ea00LOE 3.1.2OUSD(A&S)in coordination with ...CDAOIntellectual PropertyIdentify best practices related to strategies to preserve government intellectual property (IP) in the acquisition of Al and Al-enabled systems in order to ensure open architecture of secure data deliverables and rights that support the protection of government IP, best-value acquisition, avoidance of proprietary lock-in, and oversight of DoD use of AI capabilities and adherence to the DoD Al Ethical Principles. _fab63344-f7d7-11ec-b791-596aff82ea00LOE 3.1.3OUSD(A&S)in coordination with ...CDAORISKSADOPT INDUSTRY BEST PRACTICES FOR Al DEVELOPMENT BY DEVELOPING AND APPLYING TOOLS, TECHNOLOGIES, AND BEST PRACTICES TO IDENTIFY AND MITIGATE RISKS AS THEY RELATE TO THE Al CAPABILITY THROUGHOUT THE Al PRODUCT LIFECYCLE. _fab63696-f7d7-11ec-b791-596aff82ea00LOE 3.2Product ToolkitContinue developing a Product Toolkit that draws upon best practices and innovative research from the DoD enterprise, industry, and the academic community, as well as commercially available technology where appropriate._fab63934-f7d7-11ec-b791-596aff82ea00LOE 3.2.1CDAOOUSD(R&E)(for subtask a)The Toolkit will be made widely available to DoD users and shall include: ^ a) Defense Innovation Unit's (DIU) RAI Guidelines; ^ b) Templates for Al proj ect management with an emphasis on ensuring that developers have an understanding of user needs and operational context; ^ c) Al Data Cards and Model Cards, and corresponding catalogs, with detailed instructions; and ^ d) Any other tools, guidance for system and project documentation, or risk assessment frameworks as appropriate. Maturity, Sensitivity & RiskProvide guidance on how and when the RAI Tools in LOE 3.2.1 should be used across the AI Product Lifecycle, based on with an assessment of the AI technology's level of technical maturity, project sensitivity, and overall risk._fab63b64-f7d7-11ec-b791-596aff82ea00LOE 3.2.2CDAOin coordination with ...DoD Component RAI LeadsSuch assessments of the AI technology will be determined by DoD Component RAJ Leads.Data CatalogUse AI Data and Model Cards to publish Al data assets in the DoD federated data catalog, in accordance with the Department's Creating Data Advantage memorandum (May 5, 2021 ). _fab63f38-f7d7-11ec-b791-596aff82ea00LOE 3.2.3CDAODoD Component CDOsResources & ToolsFund the development and piloting of new resources and tools that augment the RAI Toolkit._fab64208-f7d7-11ec-b791-596aff82ea00LOE 3.2.4CDAOMilitary DepartmentsPersonal InformationPublish best practices to preserve privacy and civil liberties and to avoid unintended bias in the design and development of Al capabilities that involve the use of personal information. _fab6442e-f7d7-11ec-b791-596aff82ea00LOE 3.2.5ATSD(PCLT)Additional GuidanceDevelop additional guidance for applying the RAI Foundational Tenets to the early-stage AI research and engineering projects, such as those funded by Budget Activities 1, 2, and 3. _fab64a46-f7d7-11ec-b791-596aff82ea00LOE 3.2.6OUSD(R&E)CDAOin coordination with ...DoD Component RAI LeadsREQUIREMENTS VALIDATIONIncorporate RAI into alI applicable AI requirements, including joint performance requirements established and approved by the Joint Requirements Oversight Council, to ensure RAI inclusion in appropriate DoD AI capabilities._fab64d0c-f7d7-11ec-b791-596aff82ea00TENET 4Joint Requirements Oversight CouncilGoal: Use the requirements validation process to ensure that capabilities that leverage AI are aligned with operational needs while addressing relevant Al risks. System performance requirements validation increases the reliability and safety of systems prior to and during deployment. A formalized requirements validation process also provides for better traceability, accountability, and both internal and external oversight.RISKINTEGRATE Al RISK CONSIDERATIONS fNTO THE DOD JOINT PERFORMANCE REQUIREMENTS PROCESS BY IDENTIFYING RESPONSIBILITIES, AUTHORITIES, AND RESOURCES WHEN UPDATING OR DEVELOPING POLICIES AND PLANS. _fab64f3c-f7d7-11ec-b791-596aff82ea00LOE 4.1JROCMDraft a JROC Memorandum (JROCM) with changes that need to be made in requirement-setting processes to implement the coordination LOEs in Tenets 2 and 3. _fab652d4-f7d7-11ec-b791-596aff82ea00LOE 4.1.1Joint Staffin coordination with ...USD(A&S)CDAOJoint Requirements Oversight Council (JROC)Make recommendations to the Vice Chairman of the Joint Chiefs of Staff (VCJCS) and the Deputy Secretary of Defense (DSD), as appropriate. REQUIREMENTSDEVELOP A TAILORABLE PROCESS THAT PROGRAMS CAN FOLLOW TO WRITE Al-RELATED REQUIREMENTS THAT ARE TESTABLE AND OPERATIONALLY RELEVANT. _fab65676-f7d7-11ec-b791-596aff82ea00LOE 4.2RepositoryCreate a repository of A I-related requirements for common use cases, mission domains, and system architectures to facilitate reusability._fab658e2-f7d7-11ec-b791-596aff82ea00LOE 4.2.1CDAOin coordination with ...DoD Component RAI LeadsTEVV StrategyIntegrate and coordinate the Al requirements captured in LOE 4.2.1 with the TEVV strategy to ensure that adequate methods exist for continuous testing and validation of capabilities developed, consistent with existing policies for iterative acquisition such as the DoDI 5000.87. _fab65cca-f7d7-11ec-b791-596aff82ea00LOE 4.2.2CDAOin coordination with ...OUSD(R&E)DoD Component RAI LeadsGuidanceProduce guidance for Program Executive Offices and Program Offices to apply the resources from LOE 3.1 and write RAI requirements into future contracts, including for non-ACAT and non-MDAP systems._fab65fcc-f7d7-11ec-b791-596aff82ea00LOE 4.2.3Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD(A&S))in coordination with ...CDAODOT&EOperational Test Agencies (OTAs) RESPONSIBLE AI ECOSYSTEMBuild a robust national and global RAl ecosystem to improve intergovernmental, academic, industry, and stakeholder collaboration, including cooperation with allies and coalition partners, and to advance global norms grounded in shared values._fab661fc-f7d7-11ec-b791-596aff82ea00TENET 5Goal: Promote a shared understanding of responsible AI design, development, deployment, and use through domestic and international engagements. Such engagements will facilitate knowledge-sharing exchanges with intergovenunental stakeholders as well as partners in industry, academic institutions, and civil society. Through this, the DoD will collaborate on common challenges, advance shared interests, promote democratic norms and values, and increase interoperability with partners. COORDINATIONCOORDINATE AND COLLABORATE ACROSS DOD, THE INTELLIGENCE COMMUNITY, AND OTHER U.S. GOVERNMENT DEPARTMENTS AND AGENCIES ON THE RAI FOUNDATIONAL TENETS AS WELL AS ON THE DOD AI ETHICAL PRINCIPLES._fab66dbe-f7d7-11ec-b791-596aff82ea00LOE 5.1Intelligence CommunityU.S. Government AgenciesCongressENGAGE WITH CONGRESS ON THE TENETS AND PRINCIPLES TO RAISE AWARENESS AND SUPPORT APPROPRIATE CONGRESSIONAL OVERSIGHT OF DOD'S RESPONSIBLE Al EFFORTS. ParticipationIdentify any gaps in participation in CDAO-led governance bodies from DoD and IC Components and provide a plan to recruit and maintain new members. _fab670f2-f7d7-11ec-b791-596aff82ea00LOE 5.1.1CDAOCoordinationCoordinate regularly with Office of the Director of National Intelligence (ODNI) Al ethics team to ensure interoperability and alignment on the operationalization of the Intelligence Community (JC) AI Ethics Principles and the DoD AI Ethical Principles. _fab67336-f7d7-11ec-b791-596aff82ea00LOE 5.1.2CDAODoD Component RAI LeadsATSD(PCLT)Office of the Director of National Intelligence (ODNI)Interagency BodiesCoordinate regularly with appropriate Federal interagency bodies such as those housed in the Executive Office of the President, Office of Management and Budget, and the General Services Administration on RAI. _fab6773c-f7d7-11ec-b791-596aff82ea00LOE 5.1.3CDAODoD Component RAI LeadsATSD(PCLT)Federal Interagency BodiesExecutive Office of the PresidentOffice of Management and BudgetGeneral Services AdministrationLegislative StrategyDevelop a legislative strategy and ensure the strategy is clearly communicated across Department to ensure appropriate engagement with the CDAO and consistent messaging, technical assistance, and advocacy to Congress. _fab67af2-f7d7-11ec-b791-596aff82ea00LOE 5.1.4ASD(LA)in coordination with ... CDAOCongressENGAGEMENTSBUILD ENDURING ENGAGEMENTS AND COLLABORATION ACROSS INDUSTRY, ACADEMIA, AND CIVIL SOCIETY TO PROMOTE DEVELOPMENT, ADOPTION, AND IMPLEMENTATION OF RAI. _fab67d40-f7d7-11ec-b791-596aff82ea00LOE 5.2IndustyAcademiaCivil SocietyResearch GapsProvide a prioritized list of research gaps in RAI-related fields to the White House National Al Initiative Office to encourage funding by Department of Education, National Science Foundation, and National Institute of Standards and Technology (NIST) as authorized by the National Al Initiative Act._fab68100-f7d7-11ec-b791-596aff82ea00LOE 5.2.1OUSD(R&E)CDAOWhite House National Al Initiative OfficeDepartment of EducationNational Science FoundationNational Institute of Standards and Technology (NIST)Advisory BoardEnsure RAI expertise, including privacy, civil liberties, and data ethics expertise, is represented on the AI Advisory Board as authorized by FY21 National Defense Authorization Act, Section 233. _fab6840c-f7d7-11ec-b791-596aff82ea00LOE 5.2.2AI Advisory BoardCDAOFundingExplore funding opportunities to establish a development program on RAJ tools with industry, academia and the Department. _fab6866e-f7d7-11ec-b791-596aff82ea00LOE 5.2.3OUSD(R&E)IndustryAcademiaPublic-Affairs StrategyDevelop and execute a public-affairs strategy for DoD RAI activities that is integrated across Department._fab68a56-f7d7-11ec-b791-596aff82ea00LOE 5.2.4Assistant Secretary of Defense for Public Affairs (ASD(PA))in coordination with ...CDAOThis includes coordination with coordination of communications activities with the CDAO, regularly communicating DoD's RAJ implementation progress as appropriate to through speaking engagements, press releases and conferences, blog posts, and op-eds.SharingProduce guidance on the sharing or publication of DoD Al capabilities with entities outside the Department to preserve coordination with operational security and prevent unintended exposure._fab68d76-f7d7-11ec-b791-596aff82ea00LOE 5.2.5ASD(PA)in coordination with ... OUSD(I&S)CDAOINTERNATIONAL ENGAGEMENTSINTEGRATE RAI AS AN ELEMENT OF INTERNATIONAL ENGAGEMENTS, TO ADVANCE SHARED VALUES, LESSONS LEARNED, BEST PRACTICES, AND INTEROPERABILITY GLOBALLY. _fab6900a-f7d7-11ec-b791-596aff82ea00LOE 5.3InteroperabilityActively seek opportunities to engage allies and partners on RAI (including NATO, Five Eyes, Quadrilateral Security Dialogue, etc.) with particular emphasis on interoperability with partners and allies with data, compute, and storage systems, software, and schema. _fab693ca-f7d7-11ec-b791-596aff82ea00LOE 5.3.1OUSD(P)in coordination with ... CDAONATOFive EyesQuadrilateral Security DialoguePromotion & EducationContinue to communicate, promote, and educate the DoD's Al Ethical Principles and RAl implementation to partners and allies through the DoD AI Partnership for Defense (Al PfD) as well as bilateral and multilateral engagements. _fab696cc-f7d7-11ec-b791-596aff82ea00LOE 5.3.2CDAODoD AI Partnership for Defense (Al PfD)WorkshopOrganize a workshop with representatives from the international community (academia, industry and government) on Al ethics, safety, and trust in defense in order to exchange best practices and promote shared values. _fab69942-f7d7-11ec-b791-596aff82ea00LOE 5.3.3CDAOInternational CommunityAcademiaIndustryGovernmentAI WORKFORCEBuild, train, equip, and retain an RAI-ready workforce to ensure robust talent planning, recruitment, and capacity-building measures, including workforce education and training on RAI._fab69d20-f7d7-11ec-b791-596aff82ea00TENET 6Goal: Ensure that all DoD AI workforce members possess an appropriate understanding of the technology, its development process, and the operational methods applicable to implementjng RAI commensurate with their duties within the archetype roles outlined in the 2020 DoD Al Education Strategy. DoD AI workforce education and training should promote consistent understanding across all DoD stakeholders and build a culture within the DoD that enables RAI. Proper training and education must be accompanied with strategies torecruit and retain the personnel whom the DoD trains and educates. TALENT MANAGEMENTPARTICIPATE IN ONGOING EFFORTS TO DEVELOP THE D0D'S AI TALENT MANAGEMENT FRAMEWORK TO SUPPORT THE IDENTIFICATION, RECRUITMENT, ELEVATION, AND RETENTION OF MILITARY AND CIVILIAN PERSONNEL NEEDED FOR IMPLEMENTATION OF RAI. _fab6a036-f7d7-11ec-b791-596aff82ea00LOE 6.1ExpertiseDevelop a mechanism to identify and track AI expertise across the Department and the Military Departments and Services, including the Acquisition Workforce ..._fab6a39c-f7d7-11ec-b791-596aff82ea00LOE 6.1.1OSD(P&R)Defense Civilian Personnel Advisory Service (DCPAS)Military DepartmentsOSD(A&S)DoD CIOCDAOOffice of Personnel Management (OPM)by ^ a) leveraging existing coding efforts (e.g., DoD CIO's Cyber Workforce Framework expansion effort) and, if needed, developing new codes in coordination with the Office of Personnel Management (OPM); and ^ b) developing standardized personnel coding mechanisms, recognizable by current and future civilian and military Service department personnel systems (e.g., Defense Civilian Human Resources Management System (DCHRMS), military integrated personnel systems).KSATsConduct a gap analysis to determine whether any additional knowledge, skills, abilities, and tasks are needed for the with six archetypes captured in the 2020 DoD AI Education Strategy to successfully implement RAI. _fab6a7f2-f7d7-11ec-b791-596aff82ea00LOE 6.1.2CDAOin coordination with ...DoD Component RAI LeadsCareers & PathwaysDevelop career fields and pathways for military personnel who perform Al work as a major portion of their job, including promotion eligibility._fab6ab3a-f7d7-11ec-b791-596aff82ea00LOE 6.1.3Military Department RAl Leadsin coordination with ...CDAOOffice of the Under Secretary of Defense for Personnel and Readiness (OUSD(P&R))Civilian PersonnelRecommend to OUSD(P&R) and OPM career fields and pathways for civilian personnel who perform AI work as a major portion of their job._fab6adec-f7d7-11ec-b791-596aff82ea00LOE 6.1.4CDAOin coordination with ...DoD Component RAI LeadsOUSD(P&R)OUSD(P&R)OPMCivilian PersonnelWorkforce PlanningInitiate workforce planning processes to attract, recruit, and maintain highly skilled experts to fill the gaps identified in 6.1.2 to include (but not limited to) creating and reclassifying billets and adding AI positions in government performance of acquisition and requirements management functions. _fab6b206-f7d7-11ec-b791-596aff82ea00LOE 6.1.5DoD Component RAI LOE Leadsin coordination with ... OUSD(A&S)OUSD(P&R)TRAININGSUPPLEMENT EXISTING DOD AI TRAINING EFFORTS WITH CURRICULA THAT WILL ENABLE RAI IMPLEMENTATION. _fab6b526-f7d7-11ec-b791-596aff82ea00LOE 6.2CurriculaDevelop and update DoD-wide standardized curricula (covering topics such as AI benefits and limitations, risk factors, and with security) to be integrated into all AI education and training for the DoD Al workforce, and ensure that it is appropriate for all levels of both military and civilian seniority. _fab6b7d8-f7d7-11ec-b791-596aff82ea00LOE 6.2.1CDAOin coordination with ... DoD Component RAI LeadsProgramsIntegrate the curricula developed in LOE 6.2.1 in Components' Al education and training programs (including initial and mission qualified training programs to build RAI skills capacity of current workforce and pipeline). _fab6bc56-f7d7-11ec-b791-596aff82ea00LOE 6.2.2DoD Component RAl LeadsStandardsEstablish education, training, and experience standards relevant to respective Al-related positions based on the level of complexity of duties for DoD Components. _fab6bfb2-f7d7-11ec-b791-596aff82ea00LOE 6.2.3DoD Component RAI LeadsAcquisition WorkforceCollaborate with the Defense Acquisition Workforce Functional Managers to reshape the training, education, and experience requirements for the Defense Acquisition Workforce to include Al curricula as appropriate._fab6c246-f7d7-11ec-b791-596aff82ea00LOE 6.2.4OUSD(A&S)Defense Acquisition University (DAU)Defense Acquisition Workforce Functional ManagersThis includes: ^ a) potentially updating Foundational and Practitioner Certifications; ^ b) creating new Al credentials for RAT implementation roles; and ^ c) leveraging curricula per LOE 6.2.1 to existing training curricula available for the Defense Acquisition Workforce. EthicsExplore the need of an annual AI Ethics Awareness training (similar to the annual Cyber Awareness training) requirement for DoD Al Workforce to promote awareness and application of the DoD AI Ethical Principles, as well as consistency of understanding. _fab6c69c-f7d7-11ec-b791-596aff82ea00LOE 6.2.5CDAODoD Al WorkforceCAPACITYBUILD DOD CAPACITY FOR RAI THROUGH COMMUNITIES OF INTEREST/PRACTICE AND PROFESSIONAL DEVELOPMENT OPPORTUNITIES. _fab6caca-f7d7-11ec-b791-596aff82ea00LOE 6.3ChampionsDeliver annually the RAI Champions Program Department-wide in order to build a network of RAI advocates and experts in DoD Components. _fab6cd72-f7d7-11ec-b791-596aff82ea00LOE 6.3.1CDAORAI AdvocatesRAI ExpertsCoIs & CoPsBuild RAl DoD-wide communities of interest or practice, leveraging existing bodies to accelerate their establishment, such as the RAI Subcommittee._fab6d1d2-f7d7-11ec-b791-596aff82ea00LOE 6.3.2CDAOScale the RAI Champions Training Program through DoD (via Train the Trainer model) to create network of champions who can paiticipate in these Communities of Interest/Communities of Practice (COIs/COPs).Non-DoD TrainingIdentify funding or partnership opportunities for non-DoD training on AI ethics in other agencies, academia, and industry._fab6d54c-f7d7-11ec-b791-596aff82ea00LOE 6.3.3DoD Component RAI Leads2022-06-29https://media.defense.gov/2022/Jun/22/2003022604/-1/-1/0/Department-of-Defense-Responsible-Artificial-Intelligence-Strategy-and-Implementation-Pathway.PDFOwenAmburOwen.Ambur@verizon.net