History of controlled trials in medicine: real priorities are little-known. Report 2. From early experiments to the present day: without alternation and randomization

The aim of the three-report review is the historical development of clinical trials, controlled trials (CT) and randomized controlled trials (RCT), and the inclusion of these approaches in health-related disciplines (Medicine and Epidemiology). Report 2 provides a description of the wellknown James Lind Library (JLL), as well as a formed database of sources on the theme. JLL was internationalized, although most of the papers belong to authors from the UK. Many studies on the history of CT and RCT are reflected in JLL publications, but remain unclaimed without changing on common milestones and priorities. Besides, the formed base of sources included 9 studies not reflected in the JLL, of which three are principled. Six of them are given in Report 2.

Half of historical milestones on the theme (168 in total) belong to the United Kingdom, 23% to the United States, and 4% to the Italy. The remaining 19 countries, ancient, medieval and modern, contribute 0.6–4% (Russia – 1.2% by the 20th century). The earliest source on the history of CT is J.P. Bull’s dissertation (1951). The formed database as of July 2020 contained more than 260 publications, and only 9 of them were Russian (2005–2018). The base includes 7 western dissertations on the history of CT.

The object of the Report 2 study was CT as such, without any attempts at randomization or even quasi-randomization by alternate allocation. The most comprehensive thematic table on non-randomized CTs has been compiled, including studies from the Chinese emperor Shen Nung (2373 BC) and the prophet Daniel (6th century BC), to BCG vaccination for children of Canadian Indians (1941–1949). PubMed search on ‘non-randomized controlled trial’ was made. For the period of 1990–2020 years, 303 publications were found (up to 32 papers in 2020). Compared to RCT, the number of such studies is small (estimated at 0.08%), but it is important to have an appropriate conjuncture in the modern period. Along with the fact that most of the drugs and therapies currently in use are developed without RCT, the revealed ‘immortality’ of CT, even without quasi-randomization, can have social significance, removing complexes and embarrassment in cases where neither RCT nor even quasi-RCT is possible, but social and public needs require the immediate receipt of at least an approximate answer to hot questions of public health (for example, in 2020).

1. Bull J.P. A study of the history and principles of clinical therapeutic trials. M.D. thesis, University of Cambridge; 1951: 80 p. Available at: https://www.jameslindlibrary.org/wp-data/uploads/2014/05/bull-19511.pdf (accessed 21.06.2020).

2. Le Clerc D. Histoire de la medecine. Geneve: J.A. Chouet, D. Ritter;1696: 694 p.

3. The James Lind Library. Building the Library. Available at: https://www.jameslindlibrary.org/about-the-library/building-the-library/ (accessed 23.06.2020).

4. Chalmers I., Milne I., Tröhler U., et al. The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments. J R Coll Physicians Edinb. 2008; 38 (3): 259–64.

5. Medical Research Council. Streptomycin treatment of pulmonary tuberculosis: a Medical Research Council investigation. Brit Med J. 1948; 2 (4582): 769–82. https://doi.org/10.1136/bmj.2.4582.769.

6. Lock S. The randomised controlled trial – a British invention. In: Lawrence G. (Ed.) Technologies of modern medicine. London: Science Museum; 1994: 81–7.

7. Jadad A.R., Enkin M.W. Randomized controlled trials. Questions, answers, and musings. 2nd ed. Malden, Oxford, Carlton: BMJ Books; 2007: 136 p.

8. Koterov A.N., Tikhonova O.A., Ushenkova L.N., Biryukov A.P. History of controlled trials in medicine: real priorities are little-known. Report 1. Basic concepts, terms, and disciplines that use medical experiment: historical and philosophical sources. FARMAKOEKONOMIKA. Sovremennaya farmakoekonomika i farmakoepidemiologiya / FARMAKOEKONOMIKA. Modern Pharmacoeconomics and Pharmacoepidemiology. 2021; 14 (1): 72–98 (in Russ.) https://doi.org/10.17749/2070-4909/farmakoekonomika.2021.059.

9. Lind J. A treatise of the scurvy. In three parts. Containing an inquiry into the nature, causes and cure, of that disease. Together with a critical and chronological view of what has been published on the subject. Edinburgh: A Kincaid and A Donaldson; 1753. Classic Reprint. Forgotten Books; 2018: 508 p.

10. Watts G. Iain Chalmers: maverick master of medical evidence. Lancet. 2006; 368 (9554): 2203. https://doi.org/10.1016/S0140-6736(06)69879-6.

11. Gaddum J.H. Discoveries in therapeutics. J Pharm Pharmacol. 1954; 6 (8): 497–512. https://doi.org/10.1111/j.2042-7158.1954.tb10979.x.

12. Hansen E.V. The Attalids of Pergamon. Ithaca, New York: Cornell University Press; 1971: 500 p.

13. McCallum R.I. President's address. Observations upon antimony. Proc R Soc Med. 1977; 70 (11): 756–63.

14. Gurunluoglu A., Gurunluoglu R., Hakobyan T. A medieval physician: Amirdovlat Amasiatsi (1420–1495). J Med Biogr. 2019; 27 (2): 76–85. https://doi.org/10.1177/0967772016682726.

15. Junod S.W. FDA and clinical drug trials: a short history. In: Davies M., Kerimani F. (Eds.) A quick quide to clinical trial. Washington: DC Bioplan Inc.; 2008: 25–5.

16. Van den Broeck J., Brestoff J.R. (Eds.) Epidemiology: principles and practical guidelines. Dordrecht: Springer; 2013: 621 p.

17. McCord J. A thirty year follow-up of treatment effects. Am Psychol. 1978; 33 (3): 284–9. https://doi.org/10.1037//0003-066x.33.3.284.

18. Paterson R., Russell M.J. Clinical trials in malignant disease. Part III. J Fac Radiol. 1959; 10 (4): 175–80. https://doi.org/10.1016/S0368-2242(59)80048-8.

19. Hemminki E. Commentary on an early placebo controlled trial in Finland. In: JLL Bulletin: Commentaries on the history of treatment evaluation; 2005. Available at: https://www.jameslindlibrary.org/articles/commentary-on-an-early-placebo-controlled-trial-in-finland/ (accessed 25.06.2020).

20. Garattini S. Italian controlled trials to assess prevention and treatment of malaria, 1900–1930s. J R Soc Med. 2019; 112 (8): 349–53. https://doi.org/10.1177/0141076819863289.

21. Hrobjartsson B., Gotzsche P.C., Gluud C. The controlled clinical trial turns 100 years: Fibiger’s trial of serum treatment of diphtheria. Brit Med J. 1998; 317 (7167): 1243–5. https://doi.org/10.1136/bmj.317.7167.1243.

22. Toth B. Pioneering controlled trials of treatments for erysipelas and pneumonia in Glasgow, 1936–1947. J R Soc Med. 2019; 112 (7): 304–12. https://doi.org/10.1177/0141076819856278.

23. Ramanna M. Nasarwanji Hormusji Choksy (1861–1939): a pioneer of controlled clinical trials. J R Soc Med. 2014; 107 (3): 120–2. https://doi.org/10.1177/0141076814523901.

24. Choksy N.H. Letter to the Editor: Professor Lustig’s plague serum. Lancet. 1900; 2: 291–92.

25. Van der Wijden C.L., Overbeke J.A. Audit of reports of randomised clinical trials published in one journal over 45 years. Brit Med J. 1995; 311 (7010): 918. https://doi.org/10.1136/bmj.311.7010.918.

26. Haffkine W.M. On preventive inoculation. Proc R Soc. 1900; 65 (1): 252–71. https://doi.org/10.1098/rspl.1899.0031.

27. Hawgood B.J. Waldemar Mordecai Haffkine, CIE (1860–1930): prophylactic vaccination against cholera in bubonic plague in British India. J Med Biogr. 2007; 15 (1): 9–19. https://doi.org/10.1258/j.jmb.2007.05-59.

28. Gachelin G., Garner P., Ferroni E., et al. Evaluating Cinchona bark and quinine for treating and preventing malaria. J R Soc Med. 2017; 110 (1): 31–40. https://doi.org/10.1177/0141076816681421.

29. Dean M.E. ‘An innocent deception’: placebo controls in the St. Petersburg homeopathy trial, 1829–1830. J R Soc Med. 2006; 99 (7): 375–6. https://doi.org/10.1258/jrsm.99.7.375.

30. Avxentyeva M.V. A controlled trial in medicine. Medical Technologies. Assessment and Choice. 2011; 3: 88–93 (in Russ.).

31. Dvoyrin V.V. Randomized clinical trials. Voprosy onkologii / Problems in Oncology. 1982; 28 (4): 70–72 (in Russ.).

32. Kiseleva E.S., Pitskhelauri V.G., Trakhtenberg A.K., et al. Results of radiotherapy and combined radiotherapy and chemotherapy of inoperable lung cancer. Neoplasma. 1983; 30 (5): 573–80.

33. Tröhler U. Quantification in British Medicine and Surgery 1750–1830, with special Reference to its Introduction into Therapeutics. PhD thesis: University of London; 1978: 552 p. Available at: https://www.jameslindlibrary.org/wp-data/uploads/2014/05/troehler-1978.pdf (accessed 28.06.2020).

34. Cox-Maximov D. The making of the clinical trial in Britain, 1910–1945: expertise, the state and the public. PhD thesis: University of Cambridge; 1997: 395 p. Available at: https://www.jameslindlibrary.org/wp-data/uploads/2014/05/cox-maksimov-1997.pdf (accessed 28.06.2020).

35. Toth B. Clinical trials in British medicine 1848–1948, with special reference to the development of the Randomised Controlled Trial. PhD thesis: University of Bristol, Department of Social Medicine; 1998: 371 p. Available at: https://www.jameslindlibrary.org/wp-data/uploads/2014/05/toth-1998.pdf (accessed 28.06.2020).

36. Edwards M.V. Control and the therapeutic trial, 1918–1948. MD thesis: University of London; 2004: 142 p. Available at: https://www.jameslindlibrary.org/wp-data/uploads/2014/05/edwards-2004.pdf (accessed 28.06.2020).

37. Williams K.J. British pharmaceutical industry, synthetic drug manufacture and the clinical testing of novel drugs 1895–1939. PhD thesis: University of Manchester; 2005: 484 p. Available at: https://www.jameslindlibrary.org/wp-data/uploads/2014/05/williams-2005.pdf (accessed 28.06.2020).

38. Bothwell L. The emergence of the randomized controlled trial: origins to 1980. PhD. thesis: New York, Columbia University Academic Commons; 2014. Available at: https://academiccommons.columbia.edu/doi/10.7916/D8K072V0 (accessed 28.06.2020).

39. Chalmers I., Dukan E., Podolsky S., Davey Smith G. The advent of fair treatment allocation schedules in clinical trials during the 19th and early 20th centuries. J R Soc Med. 2012; 105 (5): 221–7. https://doi.org/10.1258/jrsm.2012.12k029.

40. Feinstein A.R. Clinical epidemiology: the architecture of clinical research. Philadelphia: W.B. Saunders Company; 1985: 812 p.

41. Green F.H. The clinical evaluation of remedies. Lancet. 1954; 267 (6848): 1085–90. https://doi.org/10.1016/s0140-6736(54)90649-2.

42. Bull J.P. The historical development of clinical therapeutic trials. J Chronic Dis. 1959; 10 (3): 218–48. https://doi.org/10.1016/0021-9681(59)90004-9.

43. Lilienfeld A.M. The Fielding H. Garrison lecture: Ceteris paribus: the evolution of the clinical trial. Bull Hist Med. 1982; 56 (1): 1–18.

44. Morabia A. The controversial controversy of a passionate controversialist. J Clin Epidemiol. 2002; 55 (12): 1207–13. https://doi.org/10.1016/s0895-4356(02)00526-7.

45. Feinstein A.R. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol. 1995; 48 (1): 71–9. https://doi.org/10.1016/0895-4356(94)00110-c.

46. Briko N.I., Polibin R.V., Mindlina A.Ya. Clinical epidemiology: the history of becoming and development prospectivess. Medical Almanac. 2012; 3: 28–31 (in Russ.).

47. Vlasov V.V. Evidence-based medicine, demography, epidemiology and more. How epidemiology obtains knowledge. Available at: http://vechnayamolodost.ru/articles/vashe-zdorove/odokmeddem2aca/http://vechnayamolodost.ru/articles/vashe-zdorove/odokmeddem2aca/ (in Russ.) (accessed 28.06.2020).

48. Warshawskiy S.Yu. Who was the first? International Journal of Medical Practice. 2005; 4: 12–13 (in Russ.).

49. Petrenko A. Many a little makes a mickle: how the components of a clinical study are connected. Available at: https://biomolecula.ru/articles/s-miru-po-nitke-kak-soedinilis-komponenty-klinicheskogo-issledovaniia (in Russ.) (accessed 28.06.2020).

50. World history of clinical research. Old Testament studies, 6th century BC. Available at: http://clinical-trials.ru/8.html (in Russ.) (accessed 28.06.2020).

51. Spasov A.A., Chernikov M.V. The basics of evidence-based medicine. Lekarstvennyy vestnik / Medicinal Bulletin. 2007; 4 (1): 4–12 (in Russ.).

52. Sergienko V.I., Bondareva I.B. Mathematical statistics in clinical studies. Moscow: GEOTAR-Media; 2006: 304 p. (in Russ.).

53. Vasilenko V.V. The background of the dummy. Russkiy meditsinskiy zhurnal / Russian Medical Journal. 2012; 20: 1058–61 (in Russ.).

54. Talantov P. 0.05. Evidence-based medicine from magic to the quest for immortality. Moscow: AST; 2019: 769 p. (in Russ.).

55. Melikhov O.G. Clinical research. 3rd ed. Moscow: Atmosfera; 2013: 200 p. (in Russ.).

56. Sigerist H.E. A history of medicine. Vol. 1. New York: Oxford University Press; 1951: 564 p.

57. Modjarrad K. A changing paradigm for medical research: the evolution of the clinical trial. Hektoen International Journal (A Journal of Medical Humanities). 2013; 5 (4). Available at: https://hekint.org/2017/02/01/a-changing-paradigm-for-medical-research-the-evolution-of-the-clinical-trial/ (accessed 28.06.2020).

58. Nellhaus E.M., Davies T.H. Evolution of clinical trials throughout history. Marshall J Med. 2017; 3 (1): 41–8. http://dx.doi.org/10.18590/mjm.2017.vol3.iss1.9.

59. Mathews J.N. Introduction to randomized controlled clinical trials. Texts in statistical science. 2nd ed. Charman & Hall/CRC; 2006: 272 p.

60. Gallin J.I., Ognibene F.P. (Eds.) Principles and practice of clinical research. 3rd ed. Elsevier, Academic Press; 2018: 806 p. https://doi.org/10.1016/C2009-0-62909-3.

61. Fregni F., Illigens B.M. (Eds.) Critical thinking in clinical research: applied theory and practice using case studies. New York: Oxford University Press; 2018: 536 p.

62. Castiglioni A. A history of medicine. 2nd ed. New York: Alfred A. Knopf; 1947: 646 p.

63. Machin D., Day S., Green S. (Eds.) Textbook of clinical trials. 2nd ed. Chichester: John Wiley & Sons Ltd.; 2006: 763 p.

64. Ostbye T., Rochon J. An early ‘clinical trial’ as a teaching exercise: the Book of Daniel 1.1-15 (1.1-20). Med Educ. 1993; 27 (1): 97–101. https://doi.org/10.1111/j.1365-2923.1993.tb00236.x.

65. Grimes D.A. Clinical research in ancient Babylon: methodologic insights from the book of Daniel. Obstet Gynecol. 1995; 86 (6): 1031–4. https://doi.org/10.1016/0029-7844(95)00276-W.

66. Guyatt G., Sackett D., Adachi J., et al. A clinician's guide for conducting randomized trials in individual patients. CMAJ. 1988; 139 (6): 497–503.

67. Kulkarni K., Harrison J., Baguneid M., Prendergast B. (Eds.) Oxford handbook of key clinical evidence. 2nd ed. Oxford: Oxford University Press; 2016: 949 p.

68. Ibn Sina (c. 1012 CE; c. 402 AH). Kitab al-Qanun fi al-tibb (Avicenna’s The Canon of Medicine). Available at: https://www.jameslindlibrary.org/ibn-sina-c-1012-ce-c-402-ah/ (accessed 29.06.2020).

69. Cingi C., Muluk N.B. Quick guide to good clinical practice: how to meet international quality. Springer International Publishing Switzerland; 2017: 255 p. https://doi.org/10.1007/978-3-319-44344-7.

70. Diggle G.E. Thalidomide: 40 years on. Int J Clin Pract. 2001; 55 (9): 627–31.

71. Bhatt A. Evolution of clinical research: a history before and beyond James Lind. Perspect Clin Res. 2010; 1 (1): 6–10.

72. Chalmers I. Why the 1948 MRC trial of streptomycin used treatment allocation based on random numbers. J R Soc Med. 2011; 104 (9): 383–6. https://doi.org/10.1258/jrsm.2011.11k023.

73. Hasek J. Stone of life. In: Stories and feuilleton. Prague: Lidove nakladatel'stvi; 1988: 37–41 (in Russ.).

74. Read J. Prelude to chemistry; an outline of alchemy, its literature and relationships. London: G. Bell & Sons; 1936: 342 p.

75. Amasiatsi A. Useless for ignoramus. Moscow: Nauka; 1990: 880 p. (in Russ.).

76. Oberbaum M., Lysy J., Gropp C. From Nebuchadnezzar to the randomized controlled trial – milestones in the development of clinical research. Harefuah. 2011; 150 (8): 668–71, 686 (in Hebrew).

77. Gordis L. Epidemiology. 5th ed. Philadelphia: Saunders, Elsevier Inc.; 2014: 392 p.

78. Baron J.H. Scurvy, Lancaster, Lind, Scott and Almroth Wright. J R Soc Med. 1997; 90 (7): 415. https://doi.org/10.1177/014107689709000726.

79. Tickner F.J., Medvei V.C. Scurvy and the health of European crews in the Indian Ocean in the seventeenth century. Med Hist. 1958; 2 (1): 36–46. https://doi.org/10.1017/s0025727300023255.

80. Baron J.H. Sailors' scurvy before and after James Lind – a reassessment. Nutr Rev. 2009; 67 (6): 315–32. https://doi.org/10.1111/j.1753-4887.2009.00205.x.

81. Mirza R.D., Punja S., Vohra S., Guyatt G. The history and development of N-of-1 trials. J R Soc Med. 2017; 110 (8): 330–40. https://doi.org/10.1177/0141076817721131.

82. Meldrum M.L. A brief history of the randomized controlled trial. From oranges and lemons to the gold standard. Hematol Oncol Clin North Am. 2000; 14 (4): 745–60. https://doi.org/10.1016/s0889-8588(05)70309-9.

83. Huth E.J. Benjamin Franklin's place in the history of medicine. J R Coll Physicians Edinb. 2007; 37 (4): 373–8.

84. Tröhler U. The introduction of numerical methods to assess the effects of medical interventions during the 18th century: a brief history. J R Soc Med. 2011; 104 (11): 465–74. https://doi.org/10.1258/jrsm.2011.11k025.

85. Bailey R., Howick J. Did John Stuart Mill influence the design of controlled clinical trials? J R Soc Med. 2019; 112 (6): 258–60. https://doi.org/10.1177/0141076819851759.

86. Lewis H.E. Medical aspects of polar exploration: sixtieth anniversary of Scott's last expedition. State of knowledge about scurvy in 1911. Proc R Soc Med. 1972; 65 (1): 39–42.

87. Susser M., Stein Z. Eras in epidemiology: the evolution of ideas. New York: Oxford University Press; 2009: 368 p.

88. Dodgson S. The evolution of clinical trials. EMWA. 2006; 15: 20–1.

89. Collier R. Legumes, lemons and streptomycin: a short history of the clinical trial. CMAJ. 2009; 180 (1): 23–4. https://doi.org/10.1503/cmaj.081879.

90. Mayer D. Essential evidence-based medicine. 2nd ed. Cambridge University Press; 2010: 442 p.

91. Tröhler U. Lind and scurvy: 1747 to 1795. J R Soc Med. 2005; 98 (11): 519–22. https://doi.org/10.1258/jrsm.98.11.519.

92. Louis-Courvoisier M. An 18th century controlled trial prompted by a potential shortage of hospital beds. J R Soc Med. 2007; 100 (10): 481–2. https://doi.org/10.1177/014107680710001020.

93. Short B. Dr Robert Robertson (1742–1829): fever specialist and philosopher-experimenter in the treatment of fevers with Peruvian bark in the latter eighteenth-century Royal Navy. Vesalius. 2015; 21 (2): 43–55.

94. Chick H., Dalyell E.J. Observations on the influence of foods in stimulating development in backward children. Brit Med J. 1921; 2 (3182): 1061–6. https://doi.org/10.1136/bmj.2.3182.1061.

95. Chick H., Dalyell E.J., Hume E.M., et al. Observations upon the prophylaxis and cure of rickets at the University Kinderklinik, Vienna. In: Medical Research Council. Studies of rickets in Vienna, 1919–22. Special Report Series No. 77. London: Her Majesty's Stationary Office: 19–94.

96. Carpenter K.J. Harriette Chick and the problem of rickets. J Nutr. 2008; 138 (5): 827–32. https://doi.org/10.1093/jn/138.5.827.

97. Minot G.R., Buckman T.E., Isaacs R. Chronic myelogenous leukemia: age, incidence, duration and benefit derived from irradiation. J Am Med Assoc. 1924; 82 (19): 1489–94. https://doi.org/10.1001/jama.1924.02650450001001.

98. Wayne E.J. Clinical observations on two pure glycosides of digitalis, digoxin and digitalinum verum. Clin. Sci. 1933; 1: 63–76.

99. Toth B. Why the MRC therapeutic trials committee did not introduce controlled clinical trials. J R Soc Med. 2015; 108 (12): 499–511. https://doi.org/10.1177/0141076815618891.

100. Bramwell J.C., Campbell M., Evans W. Harmol hydrochloride and o-n-propylharmol lactate in angina pectoris. Lancet. 1933; 222 (5732): 69. https://doi.org/10.1016/S0140-6736(00)78542-4.

101. Spence J.C. Clinical tests of the antirachitic activity of calciferol. Lancet. 1933; 222 (5747): 911–5. https://doi.org/10.1016/S0140-6736(00)89131-X.

102. Kendrick P., Eldering G. A study in active immunization against pertussis. Am J Hyg. 1939; 29 (Sec. B): 133–53.

103. Marks H.M. The Kendrick-Eldering-(Frost) pertussis vaccine field trial. J R Soc Med. 2007; 100 (5): 242–7. https://doi.org/10.1177/014107680710000516.

104. Assinder E.W. Treatment of syphilis. Birmingham Med Rev. 1938; 13: 16–23.

105. Ferguson R.G., Simes A.B. BCG vaccination of indian Infants in Saskatchewan. Tubercle. 1949; 30 (1): 5–11. https://doi.org/10.1016/s0041-3879(49)80055-9.

106. Houston C.S. Ferguson's BCG research – Canada's first randomized clinical trial? Clin Invest Med. 1993; 16 (1): 89–91.

107. Quasi-randomised trial. European Patients' Academy (EUPATY). Glossary. Available at: https://www.eupati.eu/glossary/quasirandomised-trial/ (accessed 05.07.2020).

108. Vlasov V.V. Epidemiology. 2nd ed. Moscow: GEOTAR-Media; 2006: 464 p. (in Russ.).

109. Hill A.B. The clinical trial. Br Med Bull. 1951; 7 (4): 278–82. https:// doi.org/10.1093/oxfordjournals.bmb.a073919.

110. Chalmers I. Why transition from alternation to randomisation in clinical trials? Brit Med J. 1999; 319: (7221): 1372. https://doi.org/10.1136/bmj.319.7221.1372.

111. Davidson I., Hillier V.F. Comparison of four methods of allocation for clinical trials with small sample sizes. Physiotherapy. 2002; 88 (12): 722–9. https://doi.org/10.1016/S0031-9406(05)60715-8.

112. Papanastassiou I.D., Phillips F.M., Van Meirhaeghe J., et al. Comparing effects of kyphoplasty, vertebroplasty, and non-surgical management in a systematic review of randomized and non-randomized controlled studies. Eur Spine J. 2012; 21 (9): 1826–43. https://doi.org/10.1007/s00586-012-2314-z.

113. Moreira R.F., Foltran F.A., Albuquerque-Sendín F., et al. Comparison of randomized and non-randomized controlled trials evidence regarding the effectiveness of workplace exercise on musculoskeletal pain control. Work. 2012; 41 (Suppl. 1): 4782–9. https://doi.org/10.3233/WOR-2012-0764-4782.

114. Mueller D., Sauerland S., Neugebauer E.A., Immenroth M. Reported effects in randomized controlled trials were compared with those of nonrandomized trials in cholecystectomy. J Clin Epidemiol. 2010; 63 (10): 1082–90. https://doi.org/10.1016/j.jclinepi.2009.12.009.

115. Deeks J.J., Dinnes J., D’Amico R., et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003; 7 (27): iii–x, 1–173. https://doi.org/10.3310/hta7270.

116. Britton A., McKee M., Black N., et al. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess. 1998; 2 (13): i–iv, 1–124. https://doi.org/10.1136/bmj.319.7205.312.

117. MacLehose R.R., Reeves B.C., Harvey I.M., et al. A systematic review of comparisons of effect sizes derived from randomised and non randomised studies. Health Technol Assess. 2000; 4 (34): 1–154. https://doi.org/10.3310/hta4340.

118. Schmoor C., Caputo A., Schumacher M. Evidence from nonrandomized studies: a case study on the estimation of causal effects. Am J Epidemiol. 2008; 167 (9): 1120–9. https://doi.org/10.1093/aje/kwn010.

119. Edwards J.P., Kelly E.J., Lin Y., et al. Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgery. Can J Surg. 2012; 55 (3): 155–62. https://doi.org/10.1503/cjs.023410.

120. Weed D.L. Epidemiologic evidence and causal inference. Hematol Oncol Clin North Am. 2000; 14 (4): 797–807. https://doi.org/10.1016/S0889-8588(05)70312-9.

121. Worrall J. Why randomize? Evidence and ethics in clinical trials. In: Gonzalez W.J., Alcolea J. (Eds.) Contemporary perspectives in philosophy and methodology of science. A Coruna: Netbiblo; 2006: 65–82.

122. Vandenbroucke J.P. What is the best evidence for determining harms of medical treatment? CMAJ. 2006; 174 (5): 645–6. https://doi.org/10.1503/cmaj.051484.

123. Glasziou P., Chalmers I., Rawlins M., McCulloch P. When are randomised trials unnecessary? Picking signal from noise. Brit Med J. 2007; 334 (7589): 349–51. https://doi.org/10.1136/bmj.39070.527986.68.

124. Howick J. The philosophy of evidence-based medicine. Chichester: Wiley-Blackwell; 2011: 248 p.

125. Arditi C., Burnand B., Peytremann-Bridevaux I. Adding non randomised studies to a Cochrane review brings complementary information for healthcare stakeholders: an augmented systematic review and meta-analysis. BMC Health Serv Res. 2016; 16 (1): 598. https://doi.org/10.1186/s12913-016-1816-5.

126. Bothwell L.E., Podolsky S.H. The emergence of the randomized, controlled trial. N Engl J Med. 2016; 375 (6): 501–4. https://doi.org/10.1056/NEJMp1604635.