Evaluation indicators for diagnostic devices using AI

This is Kumano from the support administrative scrivener corporation.

Our company offers a wide range of services, from startup support for those looking to manufacture, sell, or import/distribute medical devices, to providing specialized services to medical device manufacturers.

ToDayAIWe will introduce the evaluation index of diagnostic equipment using .

What is a diagnostic device using AI?

A diagnostic device that uses AI refers to a medical image diagnosis support system that uses artificial intelligence technology.

In recent years, medical image diagnosis support systems that utilize various AIs have been developed. However, legislation has not caught up, and accurate evaluation indicators have not been established for diagnostic support devices whose performance changes through learning.

Current evaluation indicators

Currently, "Regarding the Publication of Next-Generation Medical Device Evaluation Indicators" (Notification No. 23 issued by Yakushokuki dated December 12, 7, Director of the Medical Device Review Management Office, Pharmaceutical and Food Safety Bureau, Ministry of Health, Labor and Welfare) Attachment 1207: "Evaluation Indicators for Computerized Diagnostic Support Devices" can be used as evaluation indicators.

①Target Among medical image diagnosis support systems, it uses artificial intelligence technology to collect clinical images obtained from various imaging devices.
・Detects only the suspected lesion area without specifying the disease name (so-called CADe)
・In addition to detecting suspected lesions, it also suggests possible disease names (so-called CADx)
・In addition to presenting disease name candidates, it also weights (ranks) them (same as above)
It is intended for.
Additionally, if the system behaves unexpectedly or malfunctions, it is a basic requirement for the user to be able to detect it.

② Matters to keep in mind during evaluation
1) Clarify the outline, basis, validity, etc. of the principles of item detection and diagnosis, learning, information security, etc.
2) Safety, quality, and performance evaluation: During the approval application process, standard values ​​are clarified and appropriate evaluations are conducted. After marketing, it is necessary to confirm that quality can be ensured and then conduct validity verification.
3) Risk management: A system is needed to avoid risks.

③Examples of things that require evidence and validity regarding learning
・Learning algorithm and program overview
·data
・Overview of database operator, organizational structure, etc.
・Business plan owned by the operator
・Contents of work that the operator has outsourced
・Types of data held in the database
・Database overview and design document
・Various procedure manuals related to data management and their operational status, etc.

Nature and challenges

①Characteristics as a black box It is difficult to clarify the ``principle of operation'' because the characteristic is that the output can be obtained if the doctor knows how to operate it. Rather than scrutinizing the diagnostic algorithm in detail, the approval review focuses on confirming whether the required output is obtained for the input, similar to the evaluation of general medical device programs. It is appropriate to evaluate its performance. The challenge is whether to notify users of unexpected behavior. Although it is extremely difficult to completely eliminate unexpected behavior and erroneous judgments, it is necessary to have an operational method that can collect such information and take countermeasures.

② Performance change While AI can learn and improve its performance, it cannot be denied that the performance may fall below what was originally specified. The following countermeasures are required.
1) Continuous performance verification method: Demonstrate validity after clarifying the obtained data.
2) Quality assurance due to performance changes...including network risks
3) Concepts regarding pharmaceutical procedures: According to regulations, an upgraded version that improves performance is considered a new product, but there is a possibility that older versions may exist on the market. Therefore, follow-up on version updates and regulatory procedures are also important.

③ Responsibility As with normal medical devices, the manufacturer is responsible for any problems such as malfunctions or breakdowns. So what about the actual situation? Although the purpose of the support system is to provide "diagnostic support," there is a possibility that the final decision will be left up to the doctor.
Therefore, it is necessary to consider necessary measures for proper use, such as clarifying the purpose and method of use to users and providing training for users, and ensure the effectiveness of implementation.

Support administrative scrivener corporation option

Given that the legal framework is not yet complete, we recommend involving a specialist in the licensing process for diagnostic devices. The field of medical devices is one of the most complex areas of licensing. If you have any problems related to medical devices, please contact us. Consultations are free of charge. You can contact us via the form by clicking "Contact Us" in the lower right corner of the building image at the top of this site. Reference: Pharmaceutical and Medical Devices Agency Notification No. 0523-2:https://www.pmda.go.jp/files/000229738.pdf