FGAI4H-N-040-A01 E-meeting, 15-17 February 2022 Source: Editors

FGAI4H-N-040-A01 E-meeting, 15-17 February 2022 Source: Editors Title: DEL0.1 Update: Common unified terms in artificial intelligence for health Att.1 Presentation Purpose: Discussion Contact: Abstract: Markus Wenzel E-mail: [email protected] This PPT summarizes the content of FGAI4H-N-040 with the deliverable DEL0.1 Update: Common unified terms in artificial intelligence for health, for presentation and discussion during the meeting.

Establish glossary with agreed AI for health terminology Motivation & objectives: Promote consistent use of important AI for health terms across ITU/WHO FG-AI4H deliverables across the different disciplines involved in this cross-disciplinary field Glossary: collection of unified terms definitions from renowned sources (journals, standards etc.) terms our definitions

Timeline of glossary developm ent May '21 Sept '21 Oct-Jan '21 Proposal by SA (WHO) to establish common terminology Shared by ITU over mailing list for feedback. Added clarification to intro. Additions by WG-RC Drafted version 0 Presentation at ITU/WHO FG-AI4H meeting M Updated version at meeting N Jun-Aug '21 28 Sept '21 Feb '22

Terms & definitions grouped in 7 categories TECHNICAL STATISTICAL CLINICAL & SCIENTIFIC PRODUCT EVALUATION POLICY ETHICS

x.1) Terms defined elsewher e 2.1.15 Semi-supervised Machine Learning [ISO/IEC 22989]: Machine learning that makes use of both labelled and unlabelled data during training. 5.1.9 Clinical Validation [IMDRF/SaMD-N41]: The ability of a SaMD to yield a clinically meaningful output associated to the target use of SaMD output in with the target health care situation or condition identified in the SaMD definition statement 6.1.15 Data triangulation [WHO AI-EG]: Techniques that can be used to reconstruct a de-identified, incomplete dataset by a third party for re-identification of an individual.

Corresponding bibliographical references [ISO/IEC 22989] ISO/IEC 22989 (2021), Information technology —Artificial intelligence —Artificial intelligence concepts and terminology. https://www.iso.org/standard/74296.html [IMDRF/SaMD-N41] IMDRF SaMD WG/N41FINAL:2017, SaMD: Clinical Evaluation (N41). http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-cl inical-evaluation 1.pdf [WHO AI-EG] World Health Organization Global (28 June 2021), WHO Guidance Ethics and governance of artificial intelligence for health. https://www.who.int/publications/i/item/9789240029200

x.2) Terms defined here 2.2.1 Test dataset: A subset of the data that is never shown to the model during training, used to verify that the model has learned what it was supposed to. NOTE –Adapted from [ISO/IEC 22989], which defines "test data: data used to assess the performance of a final machine learning model. [.] Test data is disjoint from training data and validation data. [.]. The test set is used to verify that the model has learned what it was supposed to."

Initial term collection as seed Technical terms: algorithm, application, artificial intelligence, batch learning, bias, class-activation map, continuous learning, convolutional neural networks, deep learning, fine-tuning, locked algorithm, machine learning, neural networks, reliability, semi-supervised machine learning, supervised machine learning, training, training dataset, unsupervised machine learning, training, training dataset, unsupervised machine learning, test dataset Statistical terms: area under the receiver operating characteristic curve Evaluation terms: clinical data, clinical evaluation, clinical evidence, clinical investigation, clinical outcome, clinical outcome assessment, clinical performance, clinical trials, clinical validation, development environment, effectiveness, input data, intended use, output data, performance error, post-market clinical follow-up study, post-market surveillance, safety, scientific validity (valid clinical association), verification, validation Ethics terms: anonymization, automation bias, autonomy, beneficence, bias, bio-surveillance, black-box algorithm, confidentiality, control problem, co-regulation, data altruism, data colonialism, data portability, data protection laws, data triangulation, de-identification, digital divide, digital welfare state, ethics, explainability, fairness, federated data, human rights, human warranty, impact assessment, inclusiveness, informed consent, many hands problem, non-maleficence, peer disagreement, privacy, pseudo-anonymization, responsibility, responsiveness, sustainability, transparency Product terms: software as a medical device, total product life cycle Policy terms: high income countries, low-and lower-middle income countries, focus group, sustainable development goals

New terms/definitions added by WGRC Technical terms: Artificial intelligence [IMDRF 2021]: “a branch of computer science, statistics, and engineering that uses algorithms or models to perform tasks and exhibit behaviors such as learning, making decisions and making predictions. The subset of AI known as ML allows computer algorithms to learn through data, without being explicitly programmed, to perform a task.” AI System [own definition]: “any digital solution that is developed with an AI-based component being an essential part of it as a whole“. AI Technology [own definition]: “any AI technology (e.g. machine learning, deep learning, natural language processing, computer vision, etc) used to develop an AI system.“ Ethics terms: Data integrity [FDA]: “the completeness, consistency, and accuracy of data.” Trustworthiness [own definition]: “AI systems and technologies that meet stakeholder’s expectation in terms of bias, explainability, provenance and other desirable characteristics. Therefore, stakeholders involved in the development, deployment, or operation of such AI-based systems should be held accountable for their proper functioning.”

Outline next steps to publish deliverable DEL0.1: Common unified terms in AI4H New deliverable, good progress Joint work: Pat Baird (Philips), Shada Alsalamah (WHO), Stephanie Kuku (WHO), Rohit Malpani (WHO), Andreas Reis (WHO), Simão Campos (ITU) Merge "Evaluation terms" with (empty) "Clinical & scientific terms"? Feedback about current definitions? Alternative definitions? Additional terms and definitions by WGs, TGs welcome Contribute: Term Definition Source/bibliographic ref. (from renowned journal, standardization body etc.) You'll be mentioned as contributor in this ITU/WHO FG-AI4H deliverable [email protected]