A method for evaluating diagnostic effectiveness using algorithms based on opinion obtained from artificial intelligence models

Objective: To develop a method for evaluating the diagnostic efficacy of differentiated algorithms based on artificial intelligence (AI) model output, as compared to conventional diagnostics.
Material and methods. Two routing scenarios for diagnosing malignant and benign skin neoplasms were simulated. The first scenario involves using algorithms that rely on the output from the Derma Onko Check AI model. The second scenario involves standard patient routing to general practitioners, therapists, and dermatologists/venereologists for establishing a diagnosis. The used diagnostic efficacy indicators of algorithms based on output from the Derma Onko Check AI model as well as on reports from general practitioners, therapists, and dermatologists/venereologists were obtained from previous clinical studies. The modeling was conducted using the clinical data and photographic images of 90 patients with malignant skin neoplasms (39 melanomas and 51 basal cell carcinomas) and 291 patients with benign skin neoplasms (100 non-melanocytic skin tumors and 191 melanocytic skin tumors).
Results. In order to evaluate the relative diagnostic efficacy of algorithms that rely on AI model output, calculation formulas were proposed, with visualization of the results in the form of a quadrant matrix. A mobile app called “AI-diagnostic efficiency calculator” (CalcRDAI&RNDAI) was developed for practical use to automatically compute the diagnostic diagnostic efficacy indicators of algorithms based on AI model output. Testing of the method to evaluate algorithms based on Derma Onko Check output reveals a 1.9-fold increase in the detection of skin cancer cases and a 10.5-fold decrease in missed cases. The evaluation results are in quadrant I (more cases are detected and fewer cases are missed), confirming the value of the diagnostic algorithm using algorithms relying on the Derma Onko Check AI model in the provision of medical care.
Conclusion. The proposed method for evaluating diagnostic efficacy with the use of developed formulas and with the visualization of the results in the form of a quadrant matrix enables objective efficacy evaluation of AI models in multi-stage diagnostic routing.

1. Behara K., Bhero E., Agee J.T. AI in dermatology: a comprehensive review into skin cancer detection. Peer J Comput Sci. 2024; 10: e2530. https://doi.org/10.7717/peerj-cs.2530.

2. Sangers T.E., Wakkee M., Moolenburgh F.J., et al. Towards successful implementation of artificial intelligence in skin cancer care: a qualitative study exploring the views of dermatologists and general practitioners. Arch Dermatol Res. 2023; 315 (5): 1187–95. https://doi.org/10.1007/s00403-022-02492-3.

3. Lamotkin A.I., Korabelnikov D.I., Olisova O.Yu., Lamotkin I.A. Patient routing algorithm when using the artificial intelligence program “Derma Onko Check” for differential diagnosis of skin neoplasms. Current Problems of Health Care and Medical Statistics. 2025; 5: 139–59 (in Russ.). https://doi.org/10.24412/2312-2935-2025-5-139-159.

4. Korabelnikov D.I., Lamotkin A.I. Patient routing algorithm in differential diagnosis of cutaneous neoplasms with the combined use of Derma Onko Check and Melanoma Check artificial intelligence software tools. Clinical Review for General Practice. 2025; 6 (12): 71–9 (in Russ.). https://doi.org/10.47407/kr2025.6.11.00715.

5. Lamotkin A.I., Korabelnikov D.I., Lamotkin I.A. Patient routing algorithm for differential diagnosis of skin neoplasms using the artificial intelligence program “Melanoma Check”. Medical Bulletin of the Main Military Cinical Hospital named after N.N. Burdenko. 2025; 4: 6–13 (in Russ.). https://doi.org/10.53652/2782-1730-2025-6-4-06-13.

6. Lamotkin A.I., Korabelnikov D.I., Lamotkin I.A. Preliminary differential diagnosis of benign and malignant tumors from epidermal skin tissue using an artificial intelligence program “Derma Onko Check”. Current Problems of Health Care and Medical Statistics. 2025; 2: 223–42 (in Russ.). https://doi.org/10.24412/2312-2935-2025-2-223-242.

7. Lamotkin A.I., Korabelnikov D.I., Lamotkin I.A., et al. Melanoma check the accuracy of the preliminary diagnosis of malignant melanocytic skin tumors using the artificial intelligence program “Melanoma Check”. Medical Bulletin of the Main Military Cinical Hospital named after N.N. Burdenko. 2025; 1: 42–51 (in Russ.). https://doi.org/10.53652/2782-1730-2025-6-1-42-51.

8. De Bedout V., Williams N.M., Muñoz A.M., et al. Skin cancer and dermoscopy training for primary care physicians: a pilot study. Dermatol Pract Concept. 2021; 11 (1): e2021145. https://doi.org/10.5826/dpc.1101a145.

9. Neretin E.Yu., Titov K.S., Zapirov G.M. Primary early diagnosis of skin melanoma after individual training of doctors. Russian Journal of Clinical Dermatology and Venereology. 2023; 22 (1): 99–105 (in Russ.). https://doi.org/10.17116/klinderma20232201199.

10. Chen J.Y., Fernandez K., Fadadu R.P., et al. Skin cancer diagnosis by lesion, physician, and examination type: a systematic review and meta-analysis. JAMA Dermatol. 2025; 161 (2): 135–46. https://doi.org/10.1001/jamadermatol.2024.4382.

11. Omelyanovskiy V.V., Gorkavenko F.V., Ryagina V.A., et al. Basic approaches to assessment of digital health products and services in the Russian Federation. FARMAKOEKONOMIKA. Sovremennaya farmakoekonomika i farmakoepidemiologiya / FARMAKOEKONOMIKA. Modern Pharmacoeconomics and Pharmacoepidemiology. 2025; 18 (4): 473–82 (in Russ.). https://doi.org/10.17749/2070-4909/farmakoekonomika.2025.349.

12. Babenko A.I., Pushkarev O.V. Methodological basis of the complex estimation of medical-economic efficiency of public health services. Bulletin of the Siberian Branch of the Russian Academy of Medical Sciences. 2014; 34 (2): 89–94 (in Russ.).

13. Krasnova L.S., Arkova E.S., Luchinin E.A., Kholovnya-Voloskova M.A. Methodological recommendations for organizing and conducting clinical and economic analysis of medical devices. Мoscow: Research Institute of Healthcare Organization and Medical Management; 2022: 60 pp. (in Russ.).