00:23:36 Naveen Muthu: https://www.oneusefulthing.org/ 00:30:34 Stacy Heilman: Just a reminder that Evan and Naveen invite your questions at any time during this presentation. Don't let the virtual format hold you back! Feel free to drop your questions in the chat or raise your virtual hand 00:30:55 Laura Chiang: Are the CHOA and Emory AIs roughly equivalent or do they have different strenghts and weaknesses? 00:31:22 Stacy Heilman: Replying to "Are the CHOA and Emo..." Can you be more specific about what you mean by the, Laura? 00:31:28 Stacy Heilman: Replying to "Are the CHOA and Emo..." *this 00:32:08 Naveen Muthu: Replying to "Are the CHOA and Emo..." there are some differences 00:32:24 Naveen Muthu: Replying to "Are the CHOA and Emo..." but they are still relying on the same underlying models 00:32:34 Laura Chiang: Reacted to "but they are still..." with πŸ‘ 00:32:57 Seyma Katrinli: Replying to "Are the CHOA and Emo..." In my personal experience, even Emory’s co-pilot is bad compared to new Claude, ChatGPT models 00:33:10 Naveen Muthu: Replying to "Are the CHOA and Emo..." we'll talk about that when i get to one of my slides 00:33:20 JaMor Hairston: Reacted to "but they are still r..." with πŸ‘ 00:34:04 Naveen Muthu: Replying to "Are the CHOA and Emo..." the primary differences on the public tools will not be in terms of the underlying large language model but the other optimization of other capabilities they've done for individual products 00:34:19 Naveen Muthu: Replying to "Are the CHOA and Emo..." you'll see Evan talk more about Claude's connectors as an example in a second 00:34:20 JaMor Hairston: Reacted to "the primary differen..." with ❀️ 00:35:07 Naveen Muthu: mid-May is the target 00:35:28 Seyma Katrinli: Many AI models have enterprise/business models. Do you think academic institutions (and Emory) will invest in these tools? For instance, Emory purchases a licence for SPSS. Is it likely that they will ever purchase an AI business license? 00:35:48 Megan Urbanski: This is already so helpful! Is there a link to all approved-Emory tools with descriptions, security protections, etc.? Thanks! 00:36:47 Stacy Heilman: Replying to "This is already so h..." If anyone on this Zoom knows the answer to this, please respond! (I don't.) 00:36:58 Naveen Muthu: Replying to "Many AI models have ..." Yes - both Qualified Health and Microsoft Copilot are licensed products Emory pays for currently. Historically OpenAI and Anthropic have not had healthcare enterprise products but that has changed in the past year and many organizations are reviewing how they will incorporate those options. 00:37:06 Stacy Heilman: Replying to "This is already so h..." But, seems like a very useful thing to pull together 00:37:24 Seyma Katrinli: Reacted to "Yes - both Qualified Health and Microsoft Copilot are licensed products Emory pays for currently. Historically OpenAI and Anthropic have not had healthcare enterprise products but that has changed in the past year and many organizations are reviewing how they will incorporate those options." with πŸ‘ 00:37:45 Seyma Katrinli: Replying to "Many AI models have ..." I wish they pay for OpenAI or Anthropic instead of Copilot. 00:37:54 Stacy Heilman: Reacted to "I wish they pay for ..." with πŸ’― 00:37:57 Marta Rowh MD, PhD, MPH (she series): Reacted to "I wish they pay for ..." with πŸ’― 00:38:23 Naveen Muthu: Replying to "This is already so h..." this is the only site I am aware of but as you can see, it is incomplete: https://it.emory.edu/catalog/ai_services/index.html 00:38:32 Lauren Nolan: Replying to "This is already so h..." This may not have all of the detail, but it is a good overview (in beta): Emory Approved Secure AI Technology (EASAT) Registry Beta 00:39:12 Megan Urbanski: Reacted to "This may not have al..." with πŸ‘ 00:39:15 Megan Urbanski: Reacted to "this is the only sit..." with πŸ‘ 00:39:30 Naveen Muthu: Replying to "This is already so h..." @Lauren Nolan Thank you for sharing!! This is why we love chat! Can all learn from each other! 00:39:45 Ravi Mangal Patel: Does Emory/CHOA allow for the use of Claude Cowork or any plans for that in the future? 00:40:02 Lauren Nolan: Reacted to "@Lauren Nolan Thank ..." with ❀️ 00:40:07 Stacy Heilman: Reacted to "This may not have al..." with πŸ‘ 00:40:15 Seyma Katrinli: Are there any resources that we can learn how to β€œcreate” these multi-agent tools? 00:41:41 Naveen Muthu: Replying to "Does Emory/CHOA allo..." While there are ways to access Claude models being developed / available now, the specific "Cowork" product is an information security (slash nightmare) because of the extent of access it receives to the user device 00:42:14 Naveen Muthu: Replying to "Does Emory/CHOA allo..." *information security challenge 00:42:49 Naveen Muthu: Replying to "Are there any resour..." I have enjoyed all the Anthropic training materials: https://www.anthropic.com/learn/build-with-claude 00:43:02 Seyma Katrinli: Reacted to "I have enjoyed all the Anthropic training materials: https://www.anthropic.com/learn/build-with-claude" with πŸ‘ 00:44:49 Rebecca Lewis: Is the research coming to CHOA AI? 00:46:00 Daniel Herrera Rodriguez: Are there examples of agents or agentic approaches that function as or are intended as clinical decision support tools? 00:46:28 Naveen Muthu: Replying to "Is the research comi..." not in this next version, part of the challenge is that it is quite challenging to work around the PHI compliance portion when LLMs have access to both PHI and the internet 00:46:52 Naveen Muthu: Replying to "Is the research comi..." the newer OpenAI for Healthcare product for example handles this by making their own private copy of the internet periodically 00:47:17 Naveen Muthu: Replying to "Is the research comi..." but that's obviously a huge lift, so unless needing confidential info, we would suggest using the public tools on this one 00:47:44 Daniel Herrera Rodriguez: Thank you! 00:47:44 Stacy Heilman: I've heard that Claude is the most aligned with academics. Do you agree with that? 00:47:46 Ashley Awet: Reacted to "I wish they pay for OpenAI or Anthropic instead of Copilot." with πŸ’― 00:47:49 Parmi S Suchdev: Are pubmed searches now obsolete? 00:47:51 Rebecca Lewis: Replying to "Is the research comi..." Thank you! 00:47:56 Naveen Muthu: Replying to "Are there examples o..." yes, lots of vendors exploring this space including Abridge, OpenEvidence and UpToDate 00:48:05 Kaprice Welsh: yes 00:48:08 Ravi Mangal Patel: I've found Consensus useful at times: Consensus: AI for Research 00:48:10 Naveen Muthu: Replying to "I've heard that Clau..." align in which sense? we use them all 00:48:13 Maneesha Agarwal MD: I have found for less well described topics (like the intersection of social media and injury prevention), STORM isn’t all that helpful. 00:49:16 Kaprice Welsh: How are you acknowledging their use, journals, grants ect. 00:49:20 Naveen Muthu: Replying to "Are pubmed searches ..." i'd argue no in the sense that if you want structured retrieval 00:49:27 Naveen Muthu: Replying to "I've found Consensus..." Thanks for sharing! 00:49:54 Naveen Muthu: Replying to "Are pubmed searches ..." multiple search strategies are probably your best bet 00:50:02 Naveen Muthu: Replying to "How are you acknowle..." we'll address this in a bit 00:50:52 Maneesha Agarwal MD: Replying to "How are you acknowle..." I’m on the Pediatrics editorial board. We are updating our guidance (hopefully out in the next few weeks). In looking at the landscape, a lot of journals are struggling with this. There is no consensus on approach. 00:51:06 Naveen Muthu: Replying to "I have found for les..." yeah, many of those clinical lit review tools are clearly built around trying to answer medical intervention PICO questions and struggle with more broad health services questions 00:51:15 Naveen Muthu: Reacted to "I’m on the Pediatric..." with βž• 00:52:45 Nikolaos papadantonakis: Qualified Ai in Emory altbough offers selection of models, the quality is much less from same paid models. What would you suggest? 00:53:52 Stacy Heilman: Reacted to "I’m on the Pediatric..." with πŸ‘ 00:54:27 Evan Orenstein: Yes this often happens - the primary reason (but not the only reason) is the public models have access to the internet - they take your prompt, formulate questions, search the internet, pull that data in, and incorporate all that into the response. The HIPAA-compliant versions do not do this for fear that PHI in your prompt might get into the search terms. 00:55:01 Evan Orenstein: Replying to "Qualified Ai in Emo..." There is work happening to improve this by setting up screens of what gets searched on the internet, but it's not yet strong enough for most health systems to bless. 00:55:43 Nikolaos papadantonakis: Reacted to Yes this often happe... with "πŸ‘" 00:55:44 Evan Orenstein: Replying to "Qualified Ai in Emo..." So for example - if you use your HIPAA-compliant versions and ask a fact that has happened since the model was trained (e.g. a news event from last few weeks), it will not know. By contrast, if you ask public models, it will often search the internet and "know" the answer. 00:55:59 JaMor Hairston: Reacted to "Yes this often happe..." with πŸ‘ 00:58:25 Evan Orenstein: Replying to "Qualified Ai in Emo..." So an approach to dealing with this: 1. If no PHI or proprietary information, use a public model. 2. If you do have PHI or proprietary information, you can ask a public model how you should prompt an internal model. We have done this a lot where we ask Claude what prompt we should use with CoPilot when interacting with our files. Or if it's a data thing (like what Naveen is showing), you can try to make a synthetic version with fake data and pass that to the public model - then ask it for the analysis code and pull that code back internally and run it on your real data. 00:59:31 Laura Chiang: Hermes rather than Louis Vuitton? Interesting. 00:59:38 JaMor Hairston: Reacted to "Hermes rather than L..." with πŸ˜‚ 00:59:43 Stacy Heilman: Reacted to "Hermes rather than L..." with πŸ˜€ 00:59:55 Evan Orenstein: Replying to "Hermes rather than L..." If you can come up with a better backronym, we are all ears 01:00:01 Neil Lall: Reacted to "Hermes rather than L..." with πŸ‘› 01:00:12 Laura Chiang: I will ask Claude 01:00:27 JaMor Hairston: Replying to "Hermes rather than L..." Anyone have that HERMES link handy? 01:00:40 Evan Orenstein: Reacted to "I will ask Claude" with πŸ˜„ 01:00:51 Ravi Mangal Patel: Are you aware of whether there are opportunities to do this with research CRFs using previously completed CRFs from coordinator-abstracted data as training or tools that could allow for something like this? 01:00:52 Evan Orenstein: Replying to "Hermes rather than L..." Streamlit - you will need to be internal to Children's environment 01:01:22 JaMor Hairston: Reacted to "Streamlit - you will..." with ❀️ 01:01:45 Evan Orenstein: Replying to "Are you aware of whe..." Not sure I totally follow - are you saying like passing the source data to HERMES and then asking it to spit out the discrete fields in the CRF? 01:01:57 Ravi Mangal Patel: Replying to "Are you aware of whe..." yes 01:02:23 Gerry Lee: Could you please review 1 more time the workflow for using AI for data visualization and analysis that’s allowable by CHOA/Emory IRB? 01:02:24 Evan Orenstein: Replying to "Are you aware of whe..." We could try it! If interested, Nikolay.braykov@choa.org is the manager leading that work and collecting use cases 01:02:31 Ravi Mangal Patel: Replying to "Are you aware of whe..." like you pass discharge summary text into HERMES and then see how much of a CRF could be completed 01:03:13 Evan Orenstein: Create a synthetic version of your data (not just de-identified real data, but truly fake data) that has the same structure as your real data. Note: You can use AI to help create code for how to do this (and then run it internally)! Pass that to a public model and tell it how you want to analyze - ask it do that analysis and create the code. Pull that code into an internal environment where you can use it on your real data. 01:03:46 Marco Aguilar: Sorry to interrupt the wonderful discussions here, but when you all have chance, the link to the Post-event survey is here: https://survey.qualtrics.emory.edu/jfe/form/SV_cYEBbdjzbGZPetw We encourage all to respond to ensure that we can prepare future sessions that are beneficial to everyone. Thank you! 01:04:37 Gerry Lee: Replying to "Could you please rev..." thanks 01:04:40 Annalyse Kohley: Can you clarify if hermes is available now or when it may be available? 01:05:50 Evan Orenstein: Replying to "Can you clarify if h..." At Children's we are collecting use cases (again Nikolay.braykov@choa.org is right person to contact and you can cc Naveen and me) and in the "intermediate pilot" stage - i.e. we are still helping guide folks to use it, but hoping to get to where we just release it this year. 01:06:20 Evan Orenstein: Replying to "Can you clarify if h..." Only available at Children's at the moment. 01:06:54 Lauren Orenstein (she/her): Amazing talk, Evan and Naveen! Thank you so much! 01:07:03 Ravi Mangal Patel: Reacted to "At Children's we are..." with πŸ‘ 01:07:25 Stacy Heilman: Please be sure to click on the evaluation survey before we conclude! https://survey.qualtrics.emory.edu/jfe/form/SV_cYEBbdjzbGZPetw 01:07:31 JaMor Hairston: Reacted to "Amazing talk, Evan a..." with ❗ 01:07:41 Evan Orenstein: Reacted to "Amazing talk, Evan a..." with πŸ™ 01:08:03 Parmi S Suchdev: Amazing. We need a 2.0 version of this talk in the near future! 01:08:08 Stacy Heilman: Reacted to "Amazing. We need a 2..." with πŸ’― 01:09:26 Kaprice Welsh: Reacted to "Amazing. We need a 2..." with πŸ’― 01:09:35 Ravi Mangal Patel: Great talk 01:09:39 Stacy Heilman: NEJM article: https://www.nejm.org/doi/full/10.1056/NEJMra2503232 01:09:53 Evan Orenstein: Reacted to "Amazing. We need a 2..." with πŸ™ 01:09:56 Evan Orenstein: Reacted to "Great talk" with πŸ™ 01:09:59 Marisa Young: Thank you for this talk! 01:10:04 Evan Orenstein: Reacted to "Thank you for this t..." with πŸ™ 01:10:06 Kartik Reddy: Thanks! Nice talks 01:10:32 Katie Krause: Thank you 01:10:33 Zhulin He: Thank you! 01:10:44 Naveen Muthu: Reacted to "I will ask Claude" with πŸ˜„ 01:10:46 Naveen Muthu: Reacted to "Hermes rather than L..." with πŸ˜‚ 01:10:51 Joshua Loiten: Thank you! 01:11:00 Aladdin Kashlan: can you please send presentation?