NEJM AI

Volume 1, No. 5 is now available! Here are the latest articles available in the May issue of NEJM AI:     Save this post to revisit later (click the 💬 button at top right of post).    📰 Editorial: Are Stronger Feature Representations All You Need for Histology Image Search? https://nejm.ai/4b7zhtl    🧠 Perspective: Who’s Training Whom? https://nejm.ai/3Un9Oq4    📱 Perspective: Patient Portal — When Patients Take AI into Their Own Hands https://nejm.ai/3xKovuO    💡 Original Article: AI-MARRVEL — A Knowledge-Driven AI System for Diagnosing Mendelian Disorders https://nejm.ai/3WfeyQ5    ⚖️ Original Article: Comparative Evaluation of LLMs in Clinical Oncology https://nejm.ai/4aJWOAY    🏛️ Policy Corner: Scaling Adoption of Medical AI — Reimbursement from Value-Based Care and Fee-for-Service Perspectives https://nejm.ai/445c5JT    🔬 Case Study: Histopathology Slide Indexing and Search — Are We There Yet? https://nejm.ai/4aSjVcs    📊 Datasets, Benchmarks, and Protocols: Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying https://nejm.ai/3w7JKGn    🏥 Datasets, Benchmarks, and Protocols: GPT versus Resident Physicians — A Benchmark Based on Official Board Scores https://nejm.ai/3Q7hB9j    Visit http://ai.nejm.org to read all the latest articles on AI and machine learning in clinical medicine. 

In attempting to break a large language model #ArtificialIntelligence system, Jonathan H. Chen, M.D., Ph.D., was inspired to consider how such human–computer interactions may not only automate many mundane paperwork tasks but actually stimulate some of the most human activities needed in medicine. With the ability to practice high-stakes conversations in a low-stakes environment, he hopes such computer systems will make us better in our next human–human interactions.   Read "Who’s Training Whom?" by Dr. Jonathan Chen of Stanford University School of Medicine: https://nejm.ai/3Un9Oq4 #AIinMedicine

#ArtificialIntelligence (AI) is a burgeoning technological advancement, with considerable promise for influencing the field of medicine. As a preliminary step toward integrating AI into medical practice, it is imperative to ascertain whether model performance is comparable with that of physicians.   The authors of a new Datasets, Benchmarks, and Protocols article present a systematic comparison of performance by a large language model (LLM) versus that of a large cohort of physicians. The cohort includes all residents who took the medical specialist license examination in Israel in 2022 across the core medical disciplines: internal medicine, general surgery, pediatrics, psychiatry, and obstetrics and gynecology (OB/GYN). The authors provide the examinations as an accessible benchmark dataset for the medical machine learning and natural language processing communities, which may be adapted for future LLM studies.   Read "GPT versus Resident Physicians — A Benchmark Based on Official Board Scores" by Uriel Katz, M.D.: https://nejm.ai/3Q7hB9j   #AIinMedicine

Sustainable reimbursement is key for medical #ArtificialIntelligence (AI) to benefit patients and populations at scale; however, achieving reimbursement is complex and requires the support of various stakeholders. The authors of a new Policy Corner article explain the roles of the different stakeholders and the extent to which reimbursement mechanisms, including fee-for-service and value-based care, align stakeholder interests and facilitate the scaling of medical AI adoption.   Read "Scaling Adoption of Medical AI — Reimbursement from Value-Based Care and Fee-for-Service Perspectives," a new Policy Corner article by Michael D Abramoff, M.D., Ph.D., Tinglong Dai, Ph.D., and James Zou, Ph.D.: https://nejm.ai/445c5JT #healthcare

Recently, several #ArtificialIntelligence (AI)-based tools have become available to the general public that will have significant implications for health care and medical practice. Many medical professionals have casually interacted with large language models (LLMs) such as ChatGPT, Bard/Gemini, and Claude, and some have begun to use these models as augmented search engines to serve as reference tools for complex medical information.   LLMs excel in natural language processing tasks and hold considerable promise for clinical use. Fields such as oncology, in which clinical decisions are highly dependent on a continuous influx of new clinical trial data and evolving guidelines, stand to gain immensely from such advancements. The primary objectives of this recently published Original Article were to conduct comprehensive evaluations of LLMs in the field of oncology and to identify and characterize strategies that medical professionals can use to bolster their confidence in a model’s response.   In the most comprehensive head-to-head comparison of modern AI-based LLMs for application in oncology, a significant heterogeneity in model accuracy was observed, with GPT-4 showing performance competitive with a human benchmark. Despite this state-of-the-art performance, all models exhibited clinically significant error rates, with many incorrect responses given confidently and consistently across repeated prompts, underscoring the current limitations of LLMs as reliable reference tools for patients and medical professionals in the domain of oncology.   Read more in "Comparative Evaluation of LLMs in Clinical Oncology" by Nicholas R. Rydzewski, M.D., and others: https://nejm.ai/4aJWOAY

In the latest episode of the AI Grand Rounds podcast, Dr. Daphne Koller charts her professional trajectory, tracing her early fascination with computers to her influential role in #ArtificialIntelligence and health care. Initially intrigued by the capacity of computers for decision-making based on theoretical principles, Koller witnessed her niche area — once considered peripheral to AI — grow to dominate the field. Her curiosity led her from abstract theory to practical #MachineLearning applications and eventually to the complex world of biomedicine. Throughout the podcast, Koller shares her shift from pure computer science to the integration of machine learning in biological and medical research. She explains the unique challenges of applying AI to biology, distinguishing it from more deterministic fields, and how these complexities feed into her work at insitro, where she is leveraging AI throughout the drug discovery and development process, from disease understanding to therapeutic application and monitoring. She advocates for the democratizing potential of AI, underscoring its capacity to enable broader participation in scientific inquiry and problem-solving.    Listen to the full episode hosted by NEJM AI Deputy Editors Arjun Manrai, PhD, and Andrew Beam, PhD: https://nejm.ai/ep17   #AIinMedicine

🌟 Our new study is now published in NEJM AI "#GPT versus Resident Physicians—A Benchmark Based on Official Board Scores" This nationwide analysis evaluates how GPT model performance compares with 849 resident physicians across various specialties in board certification exams. We provide these high-quality exams as a benchmark dataset for medical AI question-answering evaluations - https://shorturl.at/hjoLY GPT-4 has reached, and in some cases exceeded physician-level performance. Grateful to our team for their hard work and dedication in making this happen: Eran Cohen, Eliya Shachar, Jonathan Somer, Adam Fink, Eli Morse, Beki Shreiber, Ido Wolf 📄 Read the Full Study: https://lnkd.in/d8ZbgAAu

Dr. Eric Horvitz reflects on the role of technologists in responding to the urgent concerns of large-scale AI, highlighting key concerns around misuse of AI for generating synthetic media as well as the biosecurity risks associated with AI-first biological simulation. Despite challenges ahead, he asserts that “500 years from now, the next 25 years will be recognizable as a named period of time because of AI advances.”   Listen to the full episode hosted by NEJM AI Deputy Editors Arjun Manrai, PhD, and Andrew Beam, PhD: https://nejm.ai/ep16    #healthcare #artificialintelligence #ai #nejmai

The enormous potential for generative pretrained transformers (GPTs) and other #ArtificialIntelligence (AI) large language models (LLMs) to improve health care has become increasingly clear.     Software tools based on LLMs have been shown to perform as well as or better than humans on many health care–related tasks, including generation of clinical documentation, extraction of structured data from medical records, performance on a growing number of medical board examination benchmarks, and writing accurate and empathetic responses to patients’ medical questions.     However, health care and cancer care settings pose unique ethical, legal, regulatory, and technical challenges for large-scale deployment and adoption of LLMs. Such challenges include the essentiality of patient data privacy and security, the direct negative consequences of errors and biases, the need for model interpretability and supporting evidence, the necessity of safeguarding intellectual property and proprietary data, and the difficulty of modifying clinical and operational workflows. Consequently, few LLMs are in use in hospitals outside of controlled research studies or small pilot programs, and none to our knowledge is yet broadly deployed in a dedicated cancer center.     In a new case study, Renato Umeton, PhD, et al. report the challenges and lessons learned in the evaluation and deployment of LLMs at the Dana-Farber Cancer Institute for use in all business areas, including basic research, clinical research, and operations, but not in direct clinical care.     Read the Case Study “GPT-4 in a Cancer Center — Institute-Wide Deployment Challenges and Lessons Learned” by Renato Umeton, PhD, et al.: https://nejm.ai/43Gi3AM    #AIinMedicine 

Identifying a high-risk group using ML associated with decreased costs in an RCT (secondary analysis) @NEJM_AI https://lnkd.in/edkikeeE We need more studies like these if we expect healthcare to pay for ML. #AI