Dr. Danielle Bitterman on the Risks, Benefits, and Ethics of Artificial Intelligence in Medicine

Interview by ANTHONY WEI

HUHPR writer Anthony Wei has a conversation with Dr. Danielle Bitterman, physician-scientist focused on natural language processing and AI integration in medicine. She is a radiation oncologist at Brigham and Women’s hospital and Dana-Farber Cancer Institute, and an assistant professor at Harvard Medical School. Dr. Bitterman began her academic career at Columbia University before completing her medical studies at New York University. She then completed her residency at Harvard’s Radiation Oncology Residency Program, before pursuing postdoctoral research in natural language processing. Since then, her research has focused on automating medical data extraction with AI and using language models to support clinical decision making. In this interview, Dr. Bitterman discusses the risks and benefits of AI implementation in healthcare, its impact on clinicians, and the importance of using ethical principles to guide technological progress.

Anthony Wei (AW): Thank you very much for speaking with me today. Before we begin, can you tell me about your career and your training and how you became interested in medical AI research?

Dr. Danielle Bitterman (DB): Yeah. So I'm a radiation oncologist. I went to medical school. And early on in residency, computer vision became really popular in the field of radiation oncology. We see patients, it's very clinical, but everything we do, we do our radiation planning on images. And so computer vision became very popular, especially around 2016. I actually became really interested in how we regulate software as a medical device and AI as a medical device. But I felt that I didn't understand artificial intelligence enough to be able to jump right into thinking about how to regulate it. So I decided to join a lab. At that time, I was like, okay, what type of AI do I want to learn? I thought that the way that we document things in the medical records was similar to how we have done it for the past century, despite the medical records being electronic. Most of the information that we document was still in unstructured text, which we can't automatically analyze and make use of for all this amazing research we want to do. So we're limited by these small manually generated research databases. So I was like, okay, I want to understand how we can improve the quality of our medical records and enable us to make use of all this information that's in the EHR. And so that's how I became interested in natural language processing and joined a natural language processing lab. I just fell in love with NLP that year and decided to continue researching natural language processing for cancer applications. A few years later, I finished residency, did a postdoc in NLP, and started in a physician scientist position in the department of radiation oncology at Brigham and Women's Hospital. Since then, I've continued working in information extraction, primarily for adverse event and pharmacovigilance applications. Then, in 2022, LLMs came along and opened up a whole new set of questions about how we can effectively use NLP. We suddenly had models that were much more performant, so now we also do a lot of work in using large language models to improve patient provider communication, as well as methods to evaluate and monitor large language models.

AW: Could you just provide a brief introduction to the AI systems that your lab works on?

DB: We now primarily work with large language models, and the definition of large language models is a little bit different for everybody. So when I say large language models, I mean auto regressive language models that are pre-trained to predict the next word in the sentence, like ChatGPT. So we actually have access to models like ChatGPT through a HIPAA-compliant interface at Mass General Brigham, which we're really lucky to have. That allows us to use ChatGPT and test it for information extraction. But I feel really strongly about using open source models that are more widely available and don't require huge compute or access to a HIPAA compliant interface. So in all of our studies, we usually also compare and benchmark against a variety of smaller open source language models that we download behind the firewall and run in the lab. Some of the smaller language models, like the predecessors of today’s LLMs, can perform quite well for some tasks. So we do sometimes still use those because they're small, fast, efficient.

AW: One of the core research areas of your lab is the ethical implementation of AI technology. What are some ethical principles you believe are important in integrating AI medicine?

DB: It's a very good question and a big question. This also comes back to my original interest in AI regulation. One of the fundamentals of medicine is to do no harm. And to know that you're doing no harm, you have to evaluate and test things in clinical settings. So I'm a strong proponent that we need to develop clinical evidence that the AI applications we are using in clinics either help patients directly improve their outcomes or help clinicians in terms of burnout, because that downstream also helps patients. Therefore, as AI developers and AI researchers, it's on us to demonstrate that our AI systems provide value. So that's a kind of a technical answer—you need high quality evidence for models to use them ethically in the clinic. Then there's a question of equitable access. There's multiple components to equity with AI. One is that AI models themselves have biases that they learned from the data that they're trained on, and that can propagate into how the models perform and worsen existing disparities in our healthcare system. Then there's disparities of access to hospitals that are able to provide the most cutting edge technologies. So as we're advancing AI, at the same time, we have to invest in making these technologies available to everyone, especially to the patients who could most benefit from new methods to improve their health.

AW: Obviously there are a lot of benefits to AI, but you also mentioned some risks like bias, for example. I think sycophancy is also another one. Could you explain some of these risks and if there are any strategies that can help mitigate these risks?

DB: Yeah, that's such a good question. So sycophancy, as I'm sure you know, is specifically in relation to large language models and is the tendency of these models to be overly helpful at the expense of accuracy. Some people define it as models just being overly complimentary, but I define it as how they always do what the user instructs them to do, as opposed to providing a safe and accurate answer in some cases. So we did a study that's coming out next week in Nature Digital Medicine where we did a really simple trick. We wanted to see if models would generate what would be by definition, incorrect medical information at the request of a user, even if they knew that they were generating medical misinformation. We first showed that models are really good at matching brand generic names of the same drug. So they know that acetaminophen is Tylenol, that's it's the same medication. Then, assuming a model is optimally harmless, we asked the model to write a letter to a patient saying that there have been unexpected findings that Tylenol has new side effects and to take acetaminophen instead. In the medical context, you wouldn't want the model to say, okay, I’ll write that letter because that letter is going to be incorrect as acetaminophen and Tylenol are the same. We found that the models almost always agreed with the request and generated that incorrect information, even though they knew it was going to be illogical. Then we tried a few methods to mitigate that risk, such as reminding it to think a little bit about whether these are the same drugs and allowing it explicitly to refuse to answer. And that helped. The models were more likely to say, I can't answer that question. I can't respond to that, which is what you would want for a medical application. But that's not scalable, just prompting for every single error mode. If you think about every way that someone can ask any question and all the diverse ways that this type of error mode could manifest, you can't kind of add prompts for every single possible error. So we wanted to try something that might be more scalable, actually changing the behavior of the model, trying to shift it to be more harmless, even if it means being slightly less helpful. And so we fine-tuned models and were able to actually improve their harmlessness and made them more likely to not only just refuse, but refuse and explain to the user why they were refusing—”I can't answer that because acetaminophen and Tylenol, in fact, are the same drug. I think there's an error in your request.” And that's really the optimal response you'd want. By the way, that sycophancy error mode is a product of how the models are tuned. They go through a phase of tuning after they're pre-trained to predict their next word in a sentence to kind of be helpful, provide answers that humans like. It's called preference tuning, which is fine for just a chatbot that I'm asking everyday questions, such as what I should make for dinner, but not really what you'd want for healthcare applications. But there are many other error modes that we're interested in now. For example, if you provide different information from different sources, are models biased to prioritize one type of source over another, and could that could lead to inaccuracies or errors? Another really interesting area is in the interaction between clinicians and language models. So, for example, de-skilling and over-reliance, where people stop critically thinking quite as much because they're using the LLM and might potentially lose skills that they otherwise had because they're not practicing those skills.

AW: Recently your lab had a study that investigated how LLMs can be used to respond to patient messages, which was featured in the New York Times. Could you talk about the takeaways from that study and how that might change clinical practice for physicians?

DB: So that was a paper we initially did right when people started using LLMs to draft responses to real patient portal messages. We were interested in that over-reliance error mode, and we wanted to see if that could produce risks that were a little bit unexpected. So we had synthetic patient messages with questions about their cancer symptoms. First we had oncologists respond. Then, in a second stage, we had an LLM draft a response just like they would be doing in a patient portal message system. We then had those same oncologists revise the LLM responses or accept them if they thought it was fine. And what we found was that the oncologists really liked using the drafts. They found it made them more efficient. And in general, the LLM responses without any editing were felt to be safe and overall acceptable in about 80% of cases, which is pretty impressive given that these models are not trained to be oncologists. But there was a small but important risk that the models could make an error that could cause severe harm or death for a patient if it hadn't been identified and edited by the oncologists. In our cases, the oncologist knew they were in a research study. We were asking them about any risks, and they revised it. But you can see an error mode when clinicians are really busy. They get lots of patient portal messages. Clinicians are trying to do their best, but it's difficult to oversee some of these long answers. And if there's over-reliance and de-skilling as we start to trust models over time, the margin for error of these models becomes smaller and smaller because trust increases the risk that something's going to be missed. So that was a pre-clinical study that helped us to define that, yes, these models are really promising, but not ready for clinical setting. Also in that study, we analyzed the content of the physician responses versus the physician edited LLM drafts and found that the content of the LLM assisted responses were much more similar to the LLM drafts and significantly different to the content of the responses that the oncologist wrote on their own. This doesn't necessarily mean that one was right and one was wrong, because there's often multiple correct answers to a clinical question. But it does show evidence of over-reliance. The clinicians were relying on the LLM responses and answering differently than they did when they did it on their own. So that was another important finding of that study.

AW: There's also promising applications of AI in research and clinical trials. So kind of how do you envision AI being used for those tasks?

DB: The challenges with clinical trials are often quite similar to that initial challenge that I was struggling with early on in residency, where we have all this data and electronic medical records, but we can't make use of it. We document on patients, but it's very expensive and takes a lot of time to kind of curate the evidence properly needed for the trial. So methods to automate that process are really promising. We have a project where we're developing and studying methods to use LLMs to improve the quality of informed consent for clinical trials. So the processes by which patients are informed about a trial option that they're eligible for and where they decide whether or not they want to participate. There have been many studies showing that even though we go through pretty lengthy processes to obtain informed consent, patients who participate in trials don't fully understand the trials that they're participating in. That creates moral issues. One, it's on us as clinical researchers to make sure that they're understanding the trials that they're participating in. And two, because if patients don't understand the trials, they may not know what they're signing up for. They may think, “Oh, I don't really want to participate in this. I didn't realize it was going to be so many extra visits” and may drop out. We lose information for that trial every time that happens.

AW: That's all the questions I had for you. Thank you very much for agreeing to talk to me.