Proposed Content for an Algorithm Change Protocol (ACP)An ACP is a description of the set of specific methods that a manufacturer has in place to achieve and appropriately control the risks of the anticipated types of modifications delineated in the SPS. The ACP provides a step-by-step delineation of the data and procedures to be followed so that the modifications achieve their goals and the device remains safe and effective after the modification. The description below is intended to highlight some components of an ACP, but is not intended to be an exhaustive list of ACP components:Food and Drug AdministrationFDA_85a3d7be-fbea-11df-a90e-29337a64ea2a_ff79d818-5a1e-11e9-81db-aaed7d73fa26To define the content of an Algorithm Change Protocol (ACP) for AI/ML-based software as a medical device_ff79d930-5a1e-11e9-81db-aaed7d73fa26Data ManagementIssue plans addressing how data will be collected, added to existing data sets, and used._ff79da2a-5a1e-11e9-81db-aaed7d73fa261Data management plan addressing how data will be collected, added to existing data sets, and used: This data management plan may include a quality assurance (QA) plan for determining which new data are appropriate for inclusion as part of an expanded training data set; an approach to the reference standard determination; a data augmentation strategy that allows for additional training and independent test data to be added; and an auditing and sequestration strategy to monitor, document test dataset independence, and control access to both the training and test datasets as additional data are being included and any revised algorithm is being retrained and tested.Quality AssuranceInclude a quality assurance (QA) plan for determining which new data are appropriate for inclusion as part of an expanded training data set_ff79dc1e-5a1e-11e9-81db-aaed7d73fa261.1Reference StandardsInclude an approach to the reference standard determination_ff79dcdc-5a1e-11e9-81db-aaed7d73fa261.2Data AugmentationInclude a data augmentation strategy that allows for additional training and independent test data to be added_ff79dd90-5a1e-11e9-81db-aaed7d73fa261.3Auditing & SequestrationInclude an auditing and sequestration strategy to monitor, document test dataset independence, and control access to both the training and test datasets as additional data are being included and any revised algorithm is being retrained and tested._ff79de4e-5a1e-11e9-81db-aaed7d73fa261.4Re-Training & OptimizationDefine protocols for re-training / optimizing medical device algorithms._ff79df02-5a1e-11e9-81db-aaed7d73fa262Protocols for re-training / optimizing the medical device algorithm: These protocols may include a retraining strategy that describes the objective of the retraining; the algorithm components that may be modified as a result of the learning process; and any criteria that must be met during the re-training process to trigger a more comprehensive performance evaluation using the test dataset.ObjectivesDescribe the objective of the retraining_ff79e10a-5a1e-11e9-81db-aaed7d73fa262.1Algorithm ComponentsDescribe the algorithm components that may be modified as a result of the learning process_ff79e1d2-5a1e-11e9-81db-aaed7d73fa262.2CriteriaDescribe any criteria that must be met during the re-training process to trigger a more comprehensive performance evaluation using the test dataset._ff79e29a-5a1e-11e9-81db-aaed7d73fa262.3Performance EvaluationEvaluate the performance of the algorithms._ff79e36c-5a1e-11e9-81db-aaed7d73fa263Performance evaluation protocols: These protocols may include a description of the intervals of when a new algorithm may be trained and evaluated to consider updating the medical device algorithm; the delineation of appropriate metrics and analysis procedures; statistical analysis plans; appropriate measures to minimize information leakage about the test data set if part of it is re-used in multiple evaluations; performance targets that the revised algorithm must achieve; and protocols for testing, which may be applicable for that device and type of change, for example, for testing “with clinicians in the loop,” as appropriate. IntervalsDescribe the intervals of when a new algorithm may be trained and evaluated to consider updating the medical device algorithm_ff79e42a-5a1e-11e9-81db-aaed7d73fa263.1Metrics & AnalysisDelineate appropriate metrics and analysis procedures_ff79e4f2-5a1e-11e9-81db-aaed7d73fa263.2Statistical AnalysesDocument statistical analysis plans_ff79e5ba-5a1e-11e9-81db-aaed7d73fa263.3Information LeakageInstitute appropriate measures to minimize information leakage about the test data set if part of it is re-used in multiple evaluations_ff79e6a0-5a1e-11e9-81db-aaed7d73fa263.4Performance TargetsSpecify performance targets that the revised algorithm must achieve_ff79e77c-5a1e-11e9-81db-aaed7d73fa263.5TestingDefine protocols for testing, which may be applicable for that device and type of change, for example, for testing “with clinicians in the loop,” as appropriate. _ff79e84e-5a1e-11e9-81db-aaed7d73fa263.6UpdatesDescribe how updated medical device algorithms will be tested, distributed, and communicated when released._ff79e916-5a1e-11e9-81db-aaed7d73fa264Update procedures that describe how updated medical device algorithms will be tested, distributed, and communicated when released: These update procedures may include a description of the update plan including expected frequency of updates and whether the updates will be global (all devices use the same version of the algorithm) or local (multiple versions of the algorithm targeted for specific subpopulations are distributed); version tracking and control; obsolescence planning; requirements for host software/hardware requirements; any plans for ‘beta’ release of the updated medical device algorithm concurrent with the previous version; and communication procedures that describe how users will be notified of updates and any information that will be conveyed to users about the update.PlanInclude a description of the update plan_ff79e9d4-5a1e-11e9-81db-aaed7d73fa264.1VersioningAddress version tracking and control_ff79ea9c-5a1e-11e9-81db-aaed7d73fa264.2ObsolescencePlan for obsolescence_ff79eb96-5a1e-11e9-81db-aaed7d73fa264.3Software & HardwareSpecify requirements for host software/hardware requirements_ff79ec5e-5a1e-11e9-81db-aaed7d73fa264.4Beta ReleasesDocument any plans for ‘beta’ release of the updated medical device algorithm concurrent with the previous version_ff79ed3a-5a1e-11e9-81db-aaed7d73fa264.5CommunicationsDocument communication procedures that describe how users will be notified of updates and any information that will be conveyed to users about the update._ff79ee34-5a1e-11e9-81db-aaed7d73fa264.62019-04-022019-04-08https://www.regulations.gov/document?D=FDA-2019-N-1185-0001OwenAmburOwen.Ambur@verizon.net