Skeptical Journal Club: How To Read A Medical Study

we look at Journal articles there's probably going to be a lot of overwhelming things in these articles so the goal is I want to sort of use this as an experimental exercise National

Science Foundation experiment to see how this goes with you guys in terms of showing you how you do a road map to go through a journal article no matter what

your level of Medical Training might may be and so this is designed for everyone here in terms of folks who are intelligent and interested but may not have a background in any of the things

that the journal articles are talking about so my first disclaimer has to be that this is my cya statement all of my statements represent my own personal

opinions and do not necessarily reflect those of the institutions I employed by or am academically affiliated with namely John Hopkins for my teaching and Sinai Hospital in Baltimore which who

gives me money in the form of a paycheck um it's also worthwhile to note that I have no conflicts of interest I am the head for our continuing medical education committee at Sinai and so

we're always grilling folks who come to give lectures with regards to do you have any ties to the pharmaceutical industry or do own stock in any companies and it means that relatively

speaking I am financially poor but I have no conflicts of interest and hopefully that means that the things that I say are uh possibly less biased than you might get from some places so I

want to go through a full things before we get to the journal articles don't worry they'll coming around um but the first thing I I think is to sort of reinforce the description of what evidence-based medicine is just from the

standpoint of of how I view it um there is a official definition and this is sort of a a cannibalized version of that but it's pointing out that Diagnostic and therapeutic data meaning that when

you when a doctor or a nurse or someone sees a patient and gathers information about what's going along that's never sufficient enough of course to make a clinical decision and what

evidence-based medic medicine really is it's not just that we have studies and they show things and we pick one of those studies and do what it tells us but it's really trying to integrate the

best research evidence that you can find with the person's own clinical expertise and also taking it back to the patient and I think this is great this was the established definition for

evidence-based medicine back in 1991 when it was originally coined and if you read these words what you'll see is essentially this is the same thing I learned on the first day of medical school which is that all of us doctors

we practice holistically the honest definition of that word and that's why it's frustrating when you see folks who are complimentary and alternative practitioners who try to co-op that word

and say that it doesn't have anything to do with Western medicine or it doesn't have anything to do with what your regular doctor does because your regular doctor is just interested in treating

you like a disease we treat you holistically but I can give you firsthand experience that from day one in medical school this is how we've always been trained as Physicians and this is thee of evidence-based medicine

because it doesn't matter what the best thing is for the patient if the own patient's opinion or circumstances make them not want to engage with that thing and so you're the the whole point of

individualizing what you're doing for someone and so the fancy way to say that is it results in an alliance between clinicians and patients that optimizes

the clinical outcomes and the quality of life so this is a somewhat complex pyramid that was developed by folks associated with something called the

American College of Physicians Journal Club to sort of describe how do you approach this massive mound of stuff that is the scientific evidence back

when I was a resident there were only four s's in this model and then people think too much about this there's actually a journal this is from the Journal of well this is from ACP Journal Club but there are journals about

evidence-based medicine how meta is that so you have evidence-based medicine about evidence-based medicine and so the point of this is to sort of illust for a practicing clinician like myself if I

want to make a smart decision about a patient and engage the wealth of scientific evidence I should start start at the top the easiest things to use

because the most frequent stuff the stuff that is all out there is the studies and that makes the basis of this pyramid so what's at the top are things called systems and these are

computerized decision support systems there are very few of these but the idea behind these is that you have some sort of a program maybe even tied into your electronic medical record that you have

a particular question about how to treat a patient for a certain uh disorder you put in the patient's information into this program and then it spits out an answer there are actually very few of

these and so uh for that reason of course we still even though we all carry around smartphones we all still have to use our brains and so then it goes down in terms of these other things which are

uh summaries synopsis synthesis Lots sord that pretty much describe that you start out with a bunch of studies and you have different ways to sort of make them more manageable and I'll point out

why this this whole mess is kind of important to practicing clinicians in just a moment so when it comes to thinking about Journal articles and to the lay

public you guys reading Journal articles there are a few barriers that are important to know now I I I'm talking about this because it's very important to me especially as any of us like

Steven or any of us who work with the media um in terms of you hear reporting about say a specific study but what does that mean when you're actually looking at the study itself and it's really

interesting because sometimes uh Dan Rodricks on the radio show will say hey you're coming on tomorrow and I saw this article in the New York Times about this study could you uh comment on it and so

you go to the article and it turns out it's very difficult to find the actual study and sometimes I have to click through different news articles to actually find the published study

sometimes I'm never able to and most of the time it's something that for someone who is not a doctor and who does not have access to say a medical Library like the Welsh library at Johns Hopkins which fortunately has great electronic

access and you pretty much can get most to every Journal article that is out there U it can be challenging to actually even get the study yourself if you're interested to read it and figure out what is the veracity of how the

media is reporting this so there are other barriers of course in addition to this which is access to full studies you know if it's a detailed study about a thing just in terms of whatever your

profession is whatever your interests are what is your level of understanding of the underlying science behind a study and the second thing and this is uh particularly challenging is that statistics is very intimidating and

statistics is very intimidating actually for most doctors because like most doctors I am not a statistician we all get a basic statistics course but to be an applied scientist like a physician

out treating patients we are not unless we are researchers or unless we have degrees in statistics having intimate understanding of a lot of the different statistical tests and for someone who

doesn't even have that context to go by seeing anything involving statistics can be very uncertain and confusing so this is why I was mentioning that 6s uh uh thing before so

barriers to clinicians so that was talking a little bit about the general a public but what this is is this is essentially every Journal article article that if I had an infinite amount

of time I would like to read every week so there are certain journals I run The Residency program at Sinai for internal medicine so obviously I need to keep on top of my General Internal

Medicine and so the New England Journal of Medicine that's put out by the Massachusetts Medical Society the anals of internal medicine put on the by the American College of Physicians um the Journal of the American Medical

Association the bmj British medical journal and the Lancet both of the last two of course in the United Kingdom are generally depending on the year considered the top five most impactful Internal Medicine journals and so if you

want a week- toe basis want to follow what is the latest and most important articles that are being published or comments that are being made these would make sense to read another good one is the American Journal of Medicine which

is uh put on by the alliance for Academic Internal Medicine it's actually a journal that I'm a peer reviewer for some of their articles and then also being an infectious diseases guy I want to follow the infectious diseases world

as well and probably three of the best journals for doing that are clinical infectious diseases the Journal of infectious diseases and the Lancet infectious diseases any one of these journal A journals on a given week

separate from commentaries separate from reviews will have anywhere from four to 10 good primary research articles that may or may not have any particular

interest to me in the midst of everything else that I do and for any other clinician who's who's very busy doing other things sort of looking at this and thinking about well how do I

even start to keep up with the volume of this material it's very very challenging and that's why we want to sort of narrow things down as much as possible in our

practice what can get a good accurate sound bite for things in the current digital age what a lot of Physicians do is they will get the table of contents for this week's Journal emailed to them

the E talk or maybe like I do I've got everything set up in my RSS feeds rest in peace Google Reader along Feedly and that way but the thing is that it's not like most weeks I'm having

enough time to read every single article so you skim what you can and usually that means you skim the abstracts and usually what that means is you just read the conclusion and one of the things I

want to point out in a little bit is that sometimes what the especially what the conclusion of the abstract says does not necessarily accurately reflect what was done in the study and so I'm just

pointing this out as a problem as an issue and why it's something that's that's very challenging for a lot of clinicians to keep up with so in terms of barriers to everyone when you're

looking at studies the first thing and this is very important is the influence of industry and so one of the things if you're looking at a particular research study is to figure out who was who actually gave the money for the people

to do the thing if it's something like a survey well that's not costly at all that's more man or woman power that you're using to generate this and ask patients but if you're looking at a

particular test if you're looking at a particular drug somebody has to Pony up the money to be able to supply that and so often times when you're looking things that say a new antibiotic the

person who's going to be doing the study is the company that makes the antibiotic of course because they want to fund that study and furthermore they're doing it possibly less for the pure agenda of

science but for other potential reasons publication bias is something that we've heard a lot about and a lot of times this again links up with the first one with regards to influence of Industry

because it turns out of course if somebody say a company is invested in bringing a drug to Market and they need studies to prove to the FDA that this is a safe and effective drug and they put

time and money into a study that doesn't show that they may change something not tell anyone about the results of that study and do another study and so what you can get is studies that are not

published because the people who did them were not Happ with what the results were and so that's a very important thing to keep in mind that what you are seeing in terms of a single study that

you're looking at may not be representative of what this same group or this same uh entity has done before as I knowed before science is not always

the prime motive a lot of times uh companies are aiming for if they have a a drug for instance uh their goal is to do something that will allow the FDA to

approve that they can sell it to people which then will allow Physicians to prescribe it to people and hopefully there will be enough evidence there that insurance companies will then pay for it

when that drug is prescribed this has led to the rise of something that is the bane of my existence as an infectious diseases person who's trying to assess things like different antibiotics and

that's something called a non-inferiority trial and what that is is you have say a brand new sexy antibotic and you want to bring it to Market well we're in a realm now where

if you have someone who with an infection it's not ethical to do it against a placebo because that is not the best practice of medicine so what you'll do is you'll take good antibiotic

one yours and then you want to compare it to say old established antibiotic number two and it turns out that the FDA doesn't have to see that new antibiotic

works better than old antibiotic they just want proof that new antibiotic does not work worse and so you do statistical magic you say old anti will treat this

infection and say prevent death in X number of patients and you figure out what the standard deviation is around that number and you say that's my target I want new antibotic to get within that

standard deviation and if it is the FDA will then say that's great it's non- inferior it has a non-inferiority complex which is not a real diagnosis and then you can bring it to

trial and it's really interesting for things like even common infections like methylin resistant staff arus MRSA or as everyone other than me says Mera

infections a interesting question is of say the last seven new antibiotics that were brought to Market when you compare them with the gold standard vamis and which of them works better the answer is

I have no idea because they were all brought to Market on the basis of non-inferiority Trials there are some studies that can guide you one way or the other but it's really interesting to think about that and so when people talk

about things like the idea of comparative effectiveness research the potential beauty of doing that is maybe we can actually get money from unbiased sources to try to answer these questions

to say okay we've got these eight different antibiotics and it's probably true that some of them work better than the others but the studies that they were looked at were never designed to

show that and so that's interesting a big thing of course when you're thinking about complimentary and alternative medicine is this whole question of Prior probability which is

something that our standard statistics is ill equipped to handle and most of the statistics that are used in terms of a framework for studies really don't address so as the two articles will be

talking about today one of them is looking at preventing malaria in people and we have a very excellent wealth of knowledge with regards to knowing that what malaria is that there are certain

drugs that will prevent it or will treat it and that is good the other one is sort of talking about the use of Homeopathy to treat premenstrual syndrome symptoms except of course

homeopathy aathy as I'm sure most of you in the audience if not all of you are aware uh for it to be real as the folks who use it and study it say that it's real would have to essentially break

every law of physics and chemistry and physiology so the prior probability the probability that this thing should work is zero and when you run regular statistics on something like that you

can't trust the results because anytime you do something we don't live in a world with black and white we live in a world with lots of uncertainty statistics are trying to describe that

uncertainty and so if a if something becomes statistically significant on something where the prior probability is very low it is much more likely that

that was a false positive result than a proof that this thing actually worked the last piece and this is probably the most challenging and is the hardest to sort of quantify and it's the fact that

we're all human when people do studies we make mistakes sadly people will want to achieve say professional success and they may make decisions such as

falsifying something in their paper that is completely not apparent to you when you're the reader looking at that paper unless something seems grossly abnormal

compared with your prior experience or the prior uh prior literature that exists and so that's why I love websites like retraction watch which I say everybody who has any interest in

following what's going on in science news should be following because it's not just it started off as a Blog that was sort of listing what retractions were happening and it is evolved into

this great resource that often tackles sort of the philosophy behind retractions and they will engage authors who directly have retracted or the papers to get sort of what is their

opinion on that or why did they choose to do that and so that is that is a very challenging thing especially because in this age we do studies and studies will cite other studies and it's interesting

to see when a study that had a major flaw was retracted and has something like 1,200 other studies that cite it and so was the supposition that the later science was done on by different

people erroneous in the sense that it was based off of something that may not have been valid there are a few different types of studies so what I want to do is generally tell you what

they are and point out what the good points are that you would look at in those studies so the first type is therapy so you're looking at a study of a medication to do a thing the type of

things that are good when these studies are well done random allocation of patients to comparison groups so you're this is getting into that word randomized um the outcome measure of a

known or probable clinical importance this would seem kind of obvious but you want to do studies on things that actually matter and when evidence-based medicine is taught to Physicians the

focus is actually on back to this whole fire hose of studies that we come out every week you should only really be bothering reading studies that matter to you and so the basic algorithm when you

take a basic evidence-based medicine course is if you have a study question number number one is do you does this matter to you and if the answer to that is no then they say okay why are you reading this move on to the next thing

because it's a waste of your time um and then the other thing you want to see for studies of therapy is you want to see hopefully as few people as possible during the course of the study be lost

to followup compared with the number of bad outcomes or people who had adverse re events or the medication did not prevent the thing that was supposed to prevent and so often times people use

the number of 80% or greater people retained in the study during the course of it the second type is of diagnosis and when I say this I mean of a diagnostic test sometimes we will do

studies on a new diagnostic test to diagnose a diabetes um patient obviously some good things that you would want to look for is did the patient who would

want this test applied to them is that something that you're actually Tak care of is this someone that you're going to see um are the is is there an objective or reproducible diagnostic standard and

is that applied to everybody or is it only applied to certain groups of people in the study which doesn't make any sense or can there be a blind assessment

of the test and diagnostic standard involved uh there's a type of study that looks at prognostic factors so if you have certain risk factors will you

develop X disease so good things in these studies can be looking at an Inception cohort meaning a group of patients who are early on in the course of their disorder and are initially free

of the outcome that you're looking at so you can follow them along and see Who develops who has a prognostic factor for developing a thing and Who develops it versus who does not is there an

objective or reproducible assessment of clinically important outcomes um and again when you look at some of these things that seem to be stating the obvious it is amazing when you do a lot of reading of the literature how much

out there seems to be that someone published a study that seems like it was for the sake of them publishing a study but did not actually answer a clinically important question and so that it's it's awful when you finish reading one of

those and you realize you possibly just wasted the last 10 or 15 minutes of your otherwise very busy day and again few people lost to followup is also

important causation uh showing that something definitely causes something else as opposed to of course correlation so for doing this you want to see a clearly identified comparison group for

those who are at the risk of or having the outcome you're looking at and also you'd like to see blinding of observers to the outcome or exposure and blinding observers of the exposure to the outcome

because sometimes you have two different groups of people who are dealing with seeing the exposure and diagnosing the outcome and of course the last is things

like review articles or metaanalyses these are if if it's done well it's a comprehensive search for relative art relevant articles explicit criteria for selecting articles and rating them

validly and then including the ones that only the most relevant there of course is ends up being a Continuum of different quality of studies and so individual studies can

range from everything like a simple survey where you're asking people questions but when you do that it's it's it it can give you some good data but it's not the same as going all the way

up to doing a randomized double blind Placebo controlled trial of a thing which is trying of course to control for all the variables that could be affecting your question of whether or

not something that you're doing to someone is actually effective of course there are things you can do to look at groups of studies you can do review articles which of course really just are

about searching for studies and talking about them or a metaanalysis which is taking different studies getting their actual data sets and then smashing them

on together and running statistics on that whole big group of data and as uh many other smarter people than I often say metaanalysis is garbage in garbage

out and the quality of any conclusion that you can draw from a meta analysis is really based off of how good are the studies that you included are they similar enough or if any differences or

any poor quality is going to negatively affect what you see so of course the basic anatomy of a research article you have the title The Abstract the introduction methods results discussion

and references and each of these when you're reading a journal article has its own relative importance uh so this is a general road map for how you walk through a medical study I'm going to run through this quickly and then we're

actually going to run through the two journals that I passed out to you again this is an exercise and especially the one article that I picked is somewhat has a lot of depth and complexity which

we will not necessarily touch on but I also want to show you that especially when you see the title of the first article I don't want you flee and Terror from such things in a very real way it

is easy to read a journal article even when some of the statistics may be foreign and even when some of the science may not be something that you are immediately aware of so when you

look at the title is it effective at communicating the study design and content you'd be shocked at how many titles of Journal articles do not seem to logically flow from the study that

they actually did is it accurate and is it free of sensationalism I will have an example or two at the end of ones that are not so free of sensationalism authors and funding is

number two so we haven't even made it to the other parts yet uh it's always good to look at where they are from and how many if you're looking at someone who's talking about an important topic as as

if you don't already know this it will not surprise you to know that folks who are researchers tend to do a lot of research on the same topic and so if you're delving into something say uh

claustrum defal infections I'll pull randomly from my own experience there are of course lots of different people doing those research but it turns out that there are different groups or different authors who are very well published there and so sometimes

especially if you've read other articles on the topic you can see okay so this is this group and they've done this previous work and so you can link that back to what is their pedigree of other

articles that they've done looking at where did the money uh to do the study came from so is this something that's coming from a far pharmaceutical company is this coming from an NIH Grant uh do the authors have any conflicts of

interest things we've already mentioned uh in terms of the introduction so now I'm going to pause and say I haven't talked about the abstract yet because if you've already decided to read a study I suggest that

you actually read the abstract last you've already looked at it but there's a reason for that so the introduction the whole point of it is to thoroughly Define the background for the question you're looking at does describe the

specific rationale for why this study is being done and does it tell you what the null hypothesis is that basic statistical thing that's saying that I'm doing this intervention and my null

hypothesis is that this intervention will do nothing it will not change an outcome whereas you're looking to prove that yes if you valid invalidate the null hypothesis you've proven that this

intervention does change the outcome methods this is where you get in all the crunchy bits and so this is do they give relevant details about the patients who did they include who did they choose not

to include some sometimes there are very interesting decisions made that you as the reader when you read it may not make logical sense but it did make sense to them in the context of when they did the trial who the researchers were were they

blinded to the outcomes or did they know whether or not they were giving the study drug or stud or the certain tests to different patients because that could affect what the ultimate results that

you're getting how long of a time did the study take to be completed data things like what are the primary Endo what is the thing that ultimately the big question that they wanted to answer

or look at are there secondary endpoints um and it doesn't doesn't matter who collected the data versus who analyze the data a good good thing about this is a lot of earlier pharmaceutical trials

it turns out that you had study people who did it and then the data was actually then analyzed by someone at the drug company now you of course you cannot say that immediately invalidates

the data but in many cases that makes it kind of suspect and so a good study especially one that done by a drug company will often very blatantly say

that we had absolutely nothing to do with data analysis or writing this paper and then of course the statistics test used why did they use it the software

they used things like that um and also looking is there a specific description in the methods for what type of trial it is those words that we've already talked about some of them randomized

perspective blinded is there allocation concealment this is one that I until I took an evidence-based medicine course myself about seven years ago I somehow made it all the way through through

residency without even really knowing what this term meant and relatively speaking this is not a term that has come into General use until about 10 years ago so if you're doing a

randomized double blinded trial the idea is that you're randomly picking people and let's say you're testing them for to a drug versus placebo you're randomly picking these people and you're having a

way that you are randomly saying this person gets Placebo then this person gets the drug but it turns out that the way that that's done actually matters

because a lot of times what it is is you're in a hospital let's say and you have patients who are coming in and you are saying okay I want to enroll you in the trial and what they've done is

they've generated maybe a card in a envelope that has the whether or not they're going to get the drug or the placebo there was a relatively famous

trial of children with who had diarrhea a number of years ago and so it was looking at using new drug to treat diarrhea versus the standard of therapy the people who were actually dealing

with the patients and assigning them to the different groups and again this is all randomized they have in theory the person is blinded so they don't know inside this envelope whether or not the patient is going to be assigned to a

treatment or to a placebo they became convinced that for the sickest patients the new drug actually worked and and so what they

would do is that when you had a new patient come in they figured out that if you held up the envelope to the light you could actually read what was on the card and So to avoid breaking study

protocol but also do the right thing in their opinion if they had a very sick child who came in they would take the next randomization envelope they would hold it up and if it was Placebo they

would choose not to enroll them in the trial so that they could then go ahead and give them the drug they wanted to give and they would hold that envelope until the next kid who came in wasn't

that sick and then they would go ahead and do that and that's what allocation concealment is is there a way you've you've tried to Blind it but is there a way that in that original allocation

piece when people are being assigned that people it's concealed as to what group they're going to belong to another concept has come something called intention to treat analysis what this is

it's very easy it means that if you're randomizing people into two different groups at the very end whatever group they started in when you do the statistics at the end you count them all

as being in that group and that's very important because as you go through any trial you get people who may miss doses they may not make the protocol they may completely disappear off the face of the

Earth they may die they may switch from one thing to the other and so if you if you sort of eliminate all of those other things and you only do statistics on the

people left at the end you're not going to get the result you originally designed the study for and so even if you have people who are leaving who are following out who are not actually following up with a study you have to

sort of include them in that statistical analysis because it's the purest way to actually look at your results the last thing is something called a power estimate and what this is is there is a

statistical way that you can come up with uh a description of based off of what you know about a disease you're studying its prevalence in the community

you can come up with okay I need this many human beings in my study for my results to be valid that's the simplest way to describe it so it's always good if the methods actually touches on that

and explains where they got their power estimate or did they even do one because a lot of times studies may not even do that so a note on statistics and this is always hard when you're talking even

physician to physician because sometimes people feel like you know if I'm really interested in this study I have to understand the logi mic regression that was done on this and again most of us are not statisticians and even

practicing Physicians we've got a Bas the basic level of Statistics that we need to know in order to interpret studies but in terms of even we can't do I'm not going to sit down and do a Kai

squared test for instance so it's relevant to learn what the names of the standard tests are and when you would use them because that way if you have an article and you see okay this is a

pretty basic scenario and they chose to do a wi coxian rank sum okay that makes sense I don't need to think about it anymore I don't need to understand how it was done I just need to know that they said they use the

right test for the situation why that's important is sometimes you will see an article that is looking at a situation that seems simple and they use some wacky statistical test you've never

heard of and so that raises the possibility of why did they do that maybe they did that because they ran the other test and it gave them a result a they didn't like as much and so again it's just a relative thing you can sort

of gauge what is the veracity of what they did the most the most challenging and dreaded thing and whenever I think about the P value I get very very depressed and feel like I want to quit

doing anything science related is understanding what the concept of the P value is and I'm sure everyone in this room has has heard or read or seen

people who have talked at nauseum about what a P value really means the important thing is that it does not mean that it is if you have a statistically significant P value it does not mean

that there is a 95% chance that what you did was real that is not true what A P value means is that if you do an experiment and you get a result and you

have a let's say a p value attached to that result it says that if you assume the null hypothesis was true so if you're assuming the thing that you did

did not actually do anything did not make an outcome happen and you then repeat did this study again that is the chance that you would get just as

extreme of a result or an even more extreme result than uh otherwise and so it's it's kind of hard to sort of get

that into your mind but what it's it's sort of it's sort of an indirect marker of how good we did we do but it is not fair to flip it around and say this is

sort of proof with a statistical level of certainty that what we did was effective because the P value relies on

the assumption that the null hypothesis was met and a big piece of the null hypothesis is assuming that it ignores the idea of Prior probability

and so for instance that's where you get into issues where people do studies say on Homeopathy where the prior probability is amazingly low and then maybe that study comes up with a very

significant P value and they say ha proves that this actually worked but it doesn't actually prove anything because again the prior probability is very low

um there's also things you'll see called a 95% confidence interval and this is the idea that you may get results for certain things that then have a standard deviation range around them and that is

saying that if you were to repeat this study again there's a 95% chance that this value you got that the true value is not maybe that but it's in this

numerical range around a thing so for instance let's say I'm doing a study to see whether or not exposure to Bruce press who is very sweet induces

diabetes and my statistical results come up with that people who are exposed to Bruce have a yes and you're pointing but it's not on camera but that's good um

that have an increased rate of developing diabetes and let's say that rate that ratio ends up being 1.7 so they're seems to be a positive Association but if the confidence inter again the confidence interval is not

saying that data point is absolutely correct it's saying really if you do the statistics and you repeat it you're going to get a range and the True Value 95% Chance is going to lie in that range

and let's say the lower limit of this range which is 1.7 for exposure to Bruce is actually 0.95 so it crosses number one and this is a ratio and so one is

just 50/50 odds so if you see a confidence interval that crosses over that in this particular case you would say I'm actually suspicious that Bruce does not induce

diabetes the other thing is knowing the difference between what is statistically significant and what is clinically significant because the things that look statistically significant when you do the math may not actually matter when

you apply them to patients there was a uh really interesting study that was done on residency programs a few years back that was looked at the concept of um I don't know if any of you Especial

of you who are clinicians in the audience have ever heard the term of a black cloud meaning when you have a physician who is on call this person always seems to have more patients that

are being admitted and having more work and this is something that's sort of legendary in residency Circle certain residents get this label that H if I'm on call with him that means I'm not going to be able to have any free time because I'm going to be admitting all

these patients but these other people are more lucky and so they're white clouds well somebody actually did a study looking at whether or not people who were perceived as black clouds had

more admissions than people who were perceived as white clouds and the results were statistically significant that the people who were black clouds were having more admissions than those

who had white clouds but when you look at the actual numbers it was 0.25 and so it was not clinically significant because you are not admitting one quarter of a patient so and of course it was a small

study and had other problems as well but that's an example of what it means to be statistically significant versus clinically significant so when you're looking at the results questions are the results

presented in a clear manner are the statistics appropriate are the primary and secondary endpoints emphasized and did they do post Haw or other analyses meaning you designed a study to answer a

question and sometimes studies will then when they're done be like well I actually want to go back and just look at this other little subset with this particular risk factor after the fact or

post talk and whenever you do that the whole question is that that puts to some degree the results in question because you didn't design the study to look at that thing to begin with and so that's

something else to capture in terms of discussion uh is any discussion points are they reasonable or objective conclusions made on the basis of the results are weaknesses of the study

appropriately noted and reflected on were there comparisons made to similar studies and did they focus on the primary end points or did they really focus on this post Haw analysis that

they did references are actually it's it's that's the list of things that you skip over right you don't have to read those but it turns out that if you are really really interested in something it

is very instructive to go and see if a statement is made in the article and then you go back to follow the reference does the reference actually back up the statement that was made there was a

study a few years ago that came out there was a very small thing but it was looking at a supposed outbreak of M RSA related infections in homeless people

who were exposed to bed bugs now bed bugs are not known to carry bloodborn illnesses like hepatitis but the person was trying to make the association that maybe the B bugs Were Somehow

transmitting the MRSA to these particular patients and they made a statement that essentially said this has been proven before citation and I looked at that statement while I was looking at

the article and I thought well that's odd I didn't know that that's really interesting let me go find what that was and again this was about 2011 2012-ish so I actually go and look at

the reference and is a reference to a paper from 1970 little old okay so I went I was able to find it in PubMed pull up the paper searching through the entire paper

there is one line in this paper a throwaway line it's not even what the paper was about that essentially said bed bugs have been proven to transmit

staff bacteria before with another citation so then I went and I tried to find the other citation I couldn't because it was a paper from the 1930s

from a German Journal that despite all the Google Foo I could bring to bear I could not I did not I determined it did no longer existed and so that's really interesting but that is a citation in

that paper that was the focus for their main contention it's an extreme example but if someone makes a particularly uh strong Claim about a thing or if you're really interested to this is something

the study is talking about it is worthwhile to go and see what references did they use do they come from credible journals um go and find those journals and see if you can see what they're basing it on because sometimes people

will cite things and they will sort of cite it in the spirit of what the of what the things said and not actually what the things said I think that's probably an experience that anyone has when you're writing any sort of

scientific paper especially growing up in high school and college and then of course looking for attractions is also important important if it's something that's very important to you and then I argue you can then go back to the

abstract because then you can say okay was this an accurate representation of the study I just read and did the conclusion follow directly from the findings of the study because again A lot of people are only reading the

studies by skimming the ab the conclusion of the abstract so I passed out two Journal articles to you and I'm going to explain to you why I'm giving these to you so

and this gets into another thing that I want to take a moment to rant about which has to do with with both lay folk access as well as sometimes physician access to journal articles I originally wanted to do something from the New

England Journal of Medicine and from the Journal of alternative and complimentary medicine except like most journals they are completely copyrighted and they are behind a pay wall and it turns out that

I have access of course to the New England Journal as a subscriber but I do not have access to the other one so to get a copy of this article would have costed me $51 which did not seem like a worthwhile

use of my money but that does show the barrier of if someone does not have access to a medical library or has access to a particular subscription when you're

trying to if there's an interesting article you hear about in the media there's a pretty good chance you actually won't be able to access it because it's locked behind a pay wall and that's a whole copyright issue and it's and it also has to do with sort of

financially a lot of journals for instance get their funding a lot of it comes from article reprints that people people want to get say drug reps who their study was published in the New

England Journal and they want to be able to pass this out to doctors while they're whining them and dying them and trying to convince them to buy their drug so they will order a 100 copies of

this article at a very lucrative type of uh money and then that goes directly to the journal and helps fund them so for instance I did the calculation I I had asked beforehand how many people uh

maximum usually would come to an event like this and I was said it could be up to 50 so that's great so I went on the New England Journal and they have this very uh they have this very streamlined digital way that you can figure out how

to you can digitally request permission to photocopy their articles so the copyright gives you uh for the New England Journal gives you the right to use it for personal use and most

journals do not extend that right to include for educational use which is what I would say this is or say a journal Club in a hospital or something like that so it turns out that if I had wanted to make 50 copies of that article

for you I would have had to pay $200 I just think that's interesting so in the interest of illustrating access to journal articles I have chosen two that

are both from Open Access journals so they have a Creative Commons copyright which means that you can make any copies of them you want you can use them for anything you want just as long as you

attribute the S attribute the article back to the people who made them and so that's why I chose these two articles because the one comes from Public Library of science medicine PL Lo

medicine which is probably the number one Open Access medicine Journal that's easy to find and that publishes relevant trials that is currently out there and then the other one actually comes from

an Open Access complimentary and alternative medicine Journal uh the editor is someone at Georgetown University and the editor-in chief is someone in Italy and I don't know what

that means but that was very interesting to me to see just getting back to previous discussions of complimentary and alternative medicine in mainstream medical pra in mainstream medical

schools this is one that at least has a very close tie to Georgetown the other restriction I put myself under is I wanted to pick something recent and so the two articles I picked were

relatively recent I'm actually very pleased that no one has fled the room yet after seeing the title of this one so this is the experimental part what I would like to do and of course you guys

are at a disadvantage because you actually haven't read this but I want to sort of walk through using the road map thing I just did to hit what I think are

the high points from both of these articles to illustrate some things and just to give you the experience of what it's like going through one of these

articles so title intermittent preventive treatment of malaria in pregnancy with methin and HIV infected women receiving cotoxil prophylaxis a

multicenter randomized placebo control trial that's really long but what do you think about the title and what I would say say is that this title is very descriptive it is ridiculously wory but

it actually tells you in a very objective way what this trial is doing it's doing a very specific thing and what the trial is about so I think in my mind it is effective at communicating

the study design and content it's accurate and no one would accuse this of being sensationalistic so number two is who are the authors and where did their funding come from so helpfully the

author names are of course right underneath the title of the article you can immediately see that from both the names and from their institutional affiliations this was done primarily with a bunch of folks from Spain

Barcelona and then also some ties into Africa as well and then at the bottom of the first page you actually it has a perfect disclosure of funding so where did the money come from to do this trial

that in that uh is involving drugs so you have to have money to be able to do it and the first line says it all no funding bodies had any role in the study design data collection analysis decision

to publish or Preparation of the manuscript and the money actually came from the European developing countries clinical trials partnership etc etc bottom line is that these are public not

for-profit entities that fund science it is not coming from a someone from industry who has a vested interest in convincing you to buy a thing or convincing Physicians to do a particular

thing so number three is Introduction and this is this introduction actually does a great job of laying out what all the issues are so

the issue is that of course this is looking at pregnant women who are infected with HIV HIV is a virus when you get it it can be latent or

asymptomatic in your body for a while before it slowly eats away at your immune system this is the easy way I describe it to patients more specifically it infects certain immune

cells in your body certain types of T lymphocytes and when it causes them to go down below a certain threshold that makes you susceptible to certain wacky infections that people with a normal

immune system usually don't have to worry about and that's what AIDS is AIDS describes when your immune system is in really bad shape and the goal of treating folks for HIV even though we

can't cure it is that it can essentially suppress the virus activity so that it's stalled out and your body can slowly recoup its immune system and generate more of these te- cells so that

hopefully you can get back up to a um more effective state in terms of fighting off these wacky infections people who are pregnant who have HIV are at potentially higher risk for many

things and one thing that of course we're worried about is transmission of virus from mother to baby because when you start out during gestation the baby's blood supply is completely

separate from the mother's and so the body of evidence shows it's really important to treat pregnant mothers who have HIV for their HIV because that's the best thing to do for the babies

why this article jumped out at me obviously when you're looking at this and it's clear that this has something to do with Africa this seems not to be a patient population that is necessarily

uh reflective of most if not anyone in this room or possibly even our practice in the US but that's not necessarily true Sinai actually we one of our big things that we have is we have a lot of

folks who specifically medically immigrate from Africa to have their babies and this has actually been our biggest concern with ebola because we have a lot of folks who come from Nigeria and Liberia and Sierra Leon

specifically to come to Sinai to have their baby and then they go back and there are interesting factors as to why that is and so separate from the Ebola question there's a lot of malaria there

and so then this makes me interested I have several of my patients actually are sort of doing this interesting medical reverse tourism thing and they're very affluent folks who choose to come to the US to have their babies and then they go

back and then they have more babies and come back again and so there is a sign ific chance that maybe they will have exposure to or actually have a malaria infection and other things they point

out in the introduction are just talking about the fact that people who have HIV when you get malaria especially if your immune system is in a bad State this could lead to worse issues with your HIV

and potential increased severity with the malaria so it's very important to if as all possible to prophylax people or give them medication to prevent them from getting malaria so anyone of course

who's traveled to a malaria endemic region that's one of the things we talk about is getting a drug that you can take on a regular basis when you go to one of those areas to prevent yourself from getting malaria while you were

there they also talk about the fact that many people who are have HIV when you're in that aids range it's very important that we give you other antibiotics to

prevent you from getting some of these wacky infections and the one that we use the most of they refer to here as cotoxil you guys will know it better possibly as Baum because that's our

brand name in the United States is essentially the same thing but the idea is that if someone is in the AIDS range you give them this medicine they take it every day and it's trying to prevent

these other infections from taking hold the problem as they talk about is that if you're doing this the standard medications that they use to prevent people from getting malaria often will

have very bad interactions with Baum because it's a has a similar metabolic pathway Baum in and of itself has a propensity for I would say between penicillin and Baum that's something

like 90% of drug allergies that you run into for anyone and so this can actually increase the severity of a potential allergic reaction and so they're identifying there is a need to come up

with a different drug for these folks to prevent them from getting malaria and they talk about how the candidate and a malarial would hopefully would be something you can give once it has a

long halflife and hopefully will be pretty safe because we're talking about pregnant women and many drugs of course are a concern with regards to how they might affect the baby so they came up

with this one drug called mequin which some of you if you've ever had to take malaria prophylaxis for any reason that is a drug that can be commonly used for that and so they wanted to study this

and so they decided to do a multi-center double blind Placebo controlled randomized trial and they decided to do this in three of the subsaharan African countries so going on to the methods

you'll see that the first thing under methods is a very nice ethics statement because especially and this is something that should be in any piece of research but especially when you have folks who are going into another country or

they're in not in their own country it's important to make sure that you are respecting the population of people that you're working with under study area and population they talk about the three

countries Tanzania mosm Beek and Kenya and they note that the enrollment for the trial actually went over about two years from March of 2010 to April 2012 and follow-up ended in January of

2013 why that's important is sometimes you can see very very long time courses for studies and it's not entirely clear why those studies took five years to do

this thing that maybe should have been done in six months and sometimes that will raise a red flag as to say why the study was done in the first place so

they talk about under study design that their primary endpoint the thing that they wanted to see was the pre prence of one of peripheral maternal malaria infection at delivery so when the

pregnant mother gave birth to the child did they have malaria because that's what we're trying to prevent they talk a little bit about the power calculation which is not necessarily worth going into right

now um then they go through and talk about the inclusion criteria this is the top of Page Three up on the left and essentially they wanted folks who were coming into the clinic for the first

time who had not received any propy ACC for malaria but who were pregnant had the opportunity of being included they had to be permanent residents of the area so not migratory uh they had to be

at a relatively young gestational age they had of course to be HIV positive and not have any allergies to any of the medications that we were worried about uh later on in that paragraph they

note that they recruited women with regardless of their level of immunosuppression and whether or not they were already on treatment for HIV and in my mind at the and this ends up being a weakness because they end up not

talking about this again because someone who is very sick on multiple medications May react very differently to getting a new medication than someone who is relatively healthy it's a very logical

thing in the middle on the left side they talk that the patients were randomized by block randomization I.E

chopping them UPS into groups of six and then these were stratified to getting this cooil to prevent any infections along with the h hi as well as the

mequin to prevent malaria or CL cotoxil with a placebo pill and it makes a note that the pharmacy department at the hospital clinic in Barcelona actually made all of the pills so that the

placebo pill looked exactly the same as the mefon pill so that way you're hopefully blinding everyone as to what's going on um they made the point and this goes back to the allocation concealment

they say that study number allocation was concealed in opaque so therefore could not hold it up to the light sealed envelopes that were sequentially numbered and opened only after

recruitment by study health personnel and they make the point that essentially everyone involved was blinded uh so then they talk about

essentially that they the first administration of mequin the antimalarial was observed and then one month later they got number two and one month later they got number three they

were getting a total of three doses one important point they make at the bottom of the left hand column on page three was that the biggest side effect was apparently immediately upon taking this at this dose was

vomiting and so if you vomited within the first first 30 minutes they went ahead and gave you another dose because the assumption is you didn't actually absorb it if you vomited it between 30

to 60 minutes later you only got a half dose and their goal was to try to get enough of the dose into you but obviously this is potentially very problematic um when you go up to the on

the upper right hand column they're talking under followup they assessed they actually had people who were going to the homes two days later to sort of assess how people were doing and then they had them come back to Clinic

essentially every month and if they got sick of course they were going to come back to the clinic uh before that time if someone clinically had malaria they went ahead and treated them with the

appropriate treatment which was a different drug than mequin and then once the uh mother delivered they took samples from the blood test for different things for HIV as well as

malaria and then 6 weeks later they also checked the mother's blood again again 6 weeks after delivery for malaria not worth it chatting about the laboratory

methods but if you go to page four uh figure one this in a lot of studies this is sort of the nice encapsulation of how many people were involved and what

happened to them and so you'll often get a flowchart like this where you can see that 10,000 people were screened they ended up uh they list all the reasons why they excluded most of those folks

they ended up randomizing a little bit over a thousand half to Placebo and half to mequin and then the next thing down you see init that means intention to

treat that's the this means this is the intention to treat groups so no M matter what happens to these people these people are going to be included in the final statistical analysis and as you

can see the next thing down is received first and of course they decided as scien like to do to come up with Arcane abbreviations and what iptp essentially means is they got their first dose of

the uh either Placebo or mequin then they got the second and the third and you can see that people are sort of slowly falling out from each group as you go down for various reasons till you get to the

end um they talk up a little bit on the statistics at the bottom of page four and they just note the intention to treat I won't go into the statistics at all but I will note that all the tests

here seemed to make obvious sense to me so there was no funny business in terms of what was chosen on page five this is kind of the legendary table one and I say legendary because usually in any

trial there is always a table one and that table one tells you what are the Baseline characteristics of all the patients that you looked at and so they have them divided up by control group

mequin group and they go through things like age weight etc etc and the most important thing to look at here is that there's no difference between those two

groups when you run statistics on them and the whole reason you would do that is to prove that things were well randomize because if you do end up with one of two groups where something is way off in terms of the age or something

else not only might it affect the results but it might throw a little bit of Suspicion onto how well those two groups of people ended up being randomized so if you go to page six is

where they start talking about the result results and the thing that is uh uh great and I mean that somewhat sarcastically about this study is they have an amazing amount of tables going

into a lot of detail on the results most of them aren't necessarily worth it for us to look at um but if you go to the uh on the right hand column on page six it

says primary endpoint so again the thing that this study was number one designed to do was to see if compared to Placebo we gave these ladies mequin did it

prevent them or did it decrease their chances of getting malaria um by the time they delivered their baby baby and the answer to that was yes and so they

ended up saying that 7.6% of people in the placebo group versus 3.5% in the methan group uh ended up getting malaria

and the risk ratio the ratio was 047 which means again it was about half of the people and the confidence interval you'll see there was. 27

through .82 so again what that means is that if you repeated this trial again the statistics are saying that the true number of how many people of that effect

was between 0. 27 and 082 it does not cross over one so that means that no matter what this looks like it had the effect that they were attending for they

sort of proved that with this trial giving these folks in methin did significantly by statistics reduce the number of folks who ended up having malaria

I will skip ahead uh if you go to page eight they have a comment on safety and they say there were no differences in the prevalence of miscarriages still births premature births and congenital

malformations that is great and that was another thing they were trying to pick this drug so that it had no harmful fetal effects however the other important piece about this is the next paragraph which is

tolerability um and this was noting the fact that the Toler the overall immediate Toler abity meaning as soon as you took the mequin of the drug was much

poorer in the mequin group so if you go to page 12 they've listed out in terms of the patients I appreciate that um table seven at the top what the most

important thing to look at is that in the control group only one person vomited in the meth group a significant number of people vomited it was not everybody but so you can see that in

terms of the ends uh During certain times we're talking like 6 6 3 those type of numbers but when you compare it to nobody else this is a very significant

issue so we'll stay on page 12 just to point out a couple of things about the discussion uh it found and essentially what they're saying is that they found that there was this significant risk

reduction for getting malaria but there was also this increased issue with vomiting on the part of the patients immediately when they got the mequin and this actually raises a number of

potential problem so one it's hard to say how much of the drug were those people absorbing into their system the other thing is if this is something that they're going to continuously take HIV

medications we live in a world now where uh a lot of many of my patients are on one pill one time a day to treat their HIV because we need three medicines to

treat HIV and we're at a point where we have medicines that have relatively speaking few toxicities and can be combined together and are very easy to

take um but the flip side is that back say uh 15 years ago patients would tell you horror stories of their three medicines they had to take hi to take to

suppress their HIV they had three jars of medicine that they would have to take during the course of the day it was something like 40 or 50 pills it would make them incredibly nauseous and vomit and have horrible diarrhea and it was

very challenging to take these medications back then I point this out because if you go to page uh 14 table 11 they list the HIV medicines that people

were on in the trial some of which are actually medicines that we haven't used in the United States for quite a number of years due to the fact that they have

high toxicities things like the fourth one there stavudine is very much associated with major issues with pancreatitis um major major cholesterol

issues and so it's even though it works it's something that we have stopped using and I have seen maybe one patient who's been on that drug in the last 10

years but in a resource poor area like some areas of Africa these are still effective drugs and this gets into that whole thing about pharmaceutical companies and Drug pricing and Drug

availability and things like that so bottom line is that they're using drugs that possibly some of them have higher toxicities and so if you're asking them to take an additional medication that's

going to make you puke then that could give you problems with regards to your HIV are you going to stay on your HIV medicine so that when you give birth is there going to be a higher chance that

the baby might end up HIV positive and that's something else they talk about in the discussion the the women who delivered who were in the methin group actually had a higher chance of that and

had higher overall viral loads or the amount of HIV that was in their blood by the end and so that was one of the reasons that they were speculating about

that so there conclusion on page 15 was that uh these results show an effective antimalarial drug given as prophylaxis can provide additional protection against malaria and I would say that

that is a very consistent with what their results were however because of the poor tolerability at the dose they gave which was a standard dose they're not recommending it so that's a very sober thing they went through this whole

exercise and they're saying in terms of recommending that we give this to all patients it looks like this would be a problem in context of their HIV and other things but they also do point out

at the very end finally these results may also have implications regarding the antimalarial drug combinations containing mequin that is currently recommended for malaria treatment for

everybody um I will note that that weakness I had mentioned before they nowhere in this study or any of the supplementary material do they talk about do they sort of look at the different groups of folks those who were

doing very well with their HIV versus those who are sick those who are on medications and those who are not and to someone like myself that would be very interesting to look at and so I'm

disappointed that that wasn't in there but overall I think this was a very well done if possibly complex study for you guys to look at so the next set is

references I won't talk about them beyond the fact that they all kind of make sense and then I will go to abstract so we're back to the abstract so the conclusion of the abstract was an

effective antimalarial which was mequin added to uh cotoxil in these patients and I didn't even talk about this but there are the

litn is their abbreviation for a uh anti- mosquito net that has been injected with a long acting insecticide because that's one of the population

control methods has been very effective in Africa for controlling malaria is if you can't get anyone to take a medication you can at least get them to take a net that they can put up when they go to sleep so when you add that in

HIV pregnant women can improve malaria prevention however it was not well tolerating tolerated limiting its use and it was associated with an increased risk of mother to child transmission of

HIV which warrants a better understanding of the pharmacological interactions between antimalarials and anti- retroviral drugs I want to pause there because I

want to see if if the energy levels flagging a little bit because this is kind of a this was kind of a a big thing for me to tackle

um any questions comments Bruce I don't want diabetes don't say no I'm kidding um so I I don't know I mean I'm not somebody who looks at Journal articles all the time but I

have referenced them at various times and this is the first time I've seen a very clear I totally forgot to mention that editors s I've never seen that before is that so this is this is in so

what Bruce is talking about look at the very last page and in my opinion the best Journal articles will actually have something like this that sort of puts everything in context so what's on the

back is essentially a better better more readable description of why this stud is important what did they do by the editor and some other journals like an the anals of internal medicine does have

that as a bullet pointed thing that's included with the journal so that that way if you are skimming and you just see the conclusion in the abstract you've got something else really quickly you can look at that's going to help you get

a better gauge of whether or not it's relevant and how much did you trust that they're human they could be lying that's well I mean I meant from a standpoint of

like do does the journal editor have any kind of bias or I mean so is it something you should look at is or is something you should it's absolutely something you should look at and you always and and my sarcastic comment of

that's I mean that's what we say oftentimes is everybody lies especially when you're talking about the elements of of taking a patient history even from every human being we're all subject we're all prone to error but yes it's

something that typically is a very useful thing to look at and I would go ahead and Trust in terms of I would hope the editor is maybe reviewing this with the author or one of the authors right

right sanity check type thing exactly and typically that's how the peer review process ends up working is that the editor is the one who coordinates that they send out uh to the individual peer

reviewers we're able to read it over and express our opinion but at the end of the day it's the editor's decision and they are directly communicating with the author because often times they are suggesting maybe even grammatical

changes has nothing to do it's really like a editor of any sort of written piece so absolutely my question is actually about peer review what you get to see in the

journal is the end product with no insight as to how many iterations there were who the reviewers were what the reviewing process was yes so that's

behind the curtain and it sounds like it stays behind the curtain is that another aspect of how to evaluate these articles that is but the problem is because it's

behind the curtain a lot of times even for experienced clinicians reading a journal article it's hard to get a sense of that and we've all seen stories over the last few years on uh some of the

skeptic blogs and as reported in the media about how something that says that it's peer-reviewed it turns out it's not it may be peer-reviewed but the author is also the editor of the journal or

there are other major issues there that clearly the reason why you have a peerreview process it didn't work because it wasn't given a chance to work the review process are the reviewers

unblinded does the do the authors find out who the reviewers were does that then become they do they do keep that completely blinded because they don't the other thing too is that you know if

if especially for a major journal publication could have implications for your career and if some PE and if a peer reviewer has major issues in the journal based off of that peer rreview chooses

not to publish that is not without the realm of possibility that uh that you know we live in a world where maybe the author would then want to come after the peer reviewer I would hate to think that

would ever happen but there there are reasons why that stays completely blinded um to follow up on that is there

a sopes equivalent of here's here's roughly how to find out what kind of peer review is done on these studies or even at a high level if you're reading a journal article from Source X our our

experience tells us that for the most part their peer review process is like this you know you might not know is there an equivalent source for that so unfortunately no although I will especially for the former because again

it's all sort of in the magical black box but whenever that breaks down and people find out about it I have no association at all with retraction watch I just love it but they will point out

things like that on that blog and so for me that is a resource where you will occasionally see commentary on that type of an issue when there's been funny business that then comes to light so the

second study which I will obviously let you peruse on your own it's actually far easier in terms of accessibility to read and probably far more entertaining this

is from the Open Access gr uh Journal alternative and Integrative Medicine what do you think about the title I will at least ask you this evaluation of semi-standardized individualized

homeopathic treatment of 77 women with premenstrual disorders observational study with 9 months of follow follow up I read this title and I thought well

that's interesting maybe I'm going to get an article that's looking at using Homeopathy to treat women with premenstrual disorders and see what happens that's actually not what this article is

about and it's interesting because when you go through all this stuff I'm going to forward through the slides but if you look at this article what it is about is

it is very meta it is an article about how to do a research study on Homeopathy for PMS it is not a research study on

Homeopathy for PMS because keep in mind the main contention that a lot of folks who deal with Homeopathy claim is that it's not possible to study Homeopathy

because everything is individualized and you're using it's it's symptom based and what may work for you for your symptom may not work for you for the exact same

symptom so what is essentially its goal is to come up with a framework a semi-standardized framework to look at and examine uh how could how is it

possible to study Homeopathy and so for instance if you look on uh page four and page five they have these amazingly

colorful charts and these are just the charts that are sort of looking at the patients who were Ines they did look at Patients but these are they put far too much effort into these very very pretty

charts that's sort of how you display your results because that was the point they want this to be a blueprint for how to do a study like this and then when you look at the results they pretty much

don't they say we can't comment on anything clinically because that wasn't the point of us doing this study the interesting thing is they included something like 77 people and the study

itself ran for something like four years and it was at 20 clinics in the Netherlands which seems incredibly odd that you have that little amount of people for a study that took that amount of time that's interesting they also

point out that they called a halt to the study for one year in the middle of it because essentially they needed to figure out how to be reimbursed I'm not kidding it's so it's actually it's actually legitimately very

interesting to read so I promised you two two fun things at the end so I I wanted to I didn't want to go for as they say the low hanging fruit which is

a which is a phrase that I detest so I prefer slow moving meat for you Carnival so when I was looking for a study from the Open Access alternative medicine Journal there was one that I

loved that I didn't want to be the one that I handed out to you and that was actually this one uh but you should look this up vtic vibrational healing first of its kind unconventional healing

system non-intrusive highly efficacious I should point out that this journal makes itself essentially it's peer reviewed they actually never use the words peer review any anywhere I

actually did a word hunt for the phrase peer review and it's nowhere in the journal but they do describe the review process an by other people so this study passed PE review and essentially it is a

commercial for vtic vibrational healing it talks about how they treated three patients uh who had diabetes and it lists what their blood sugar was before and then it shows a much lower value for

after of course it doesn't give you any details we don't know if maybe they also treated their diabetes at the same time um it talks about treating someone for psoriasis and just described their feet were about to fall off and now their

feet are totally fine we win and that's the level of sort of data that they included in here and when they talk about in the middle of this abstract atic device they're talking about wrist Brands they actually say it could be a

wristband it could be a necklace it could be something worn on the head and this is a very interesting because the entire is completely a biased commercial for this thing that is published in what

if I were someone who was trying to promote the evidence-based sort of Viewpoint of complimentary and alternative medicine I would be frankly embarrassed that this was in this journal which at least according to

Google is the number one hit for Open Access complimentary and alternative medicine journals that's interesting one last F thought for you uh in a different Open Access alternative Journal I found

this article spirituality and religiosity and the cardiovascular system this is also worthwhile at some point for you to look up is a review article and essentially is trying to

make the case that if you are very spiritual or if you have a high degree of religiosity you have less heart attacks lower blood pressure and the like and the conclusion is possibly the

greatest conclusion that I have ever seen in a journal article I'm about to show you the entire in conclusion this is the entire discussion section this review showed that spirituality religiosity presents a

significant association with the cardiovascular system the reference and cited in our review suggests the effect on prevention or treatment of cardiovascular diseases and indicate that is also relevant for cardiac

surgery recovery an important message in our review is that spirituality and religiosity may be indicated as a treatment a compliment and

non-pharmacological therapy for cardiovascular disorders so that's interesting I hadn't thought about that I will leave that without f f

further comment so in the last few minutes let me go back to take some questions but that's sort of what I wanted to go through SC you mentioned about the um issue of uh industrial

vested interests being called out isn't it conceivable there could be U institutional vested interests if you're getting grant money through uh an IH is

complimentary and alternative medicine you know Tom Harkin I guess he's retiring in a month or to you know back there it's like hey let's get those

beting therapies going and then my second question would be I I not sure if I heard you correctly but it sounded like youve you're dismissive of

metanalysis I'm dismissive of bad metaanalyses because a long and and my point is that a lot of folks will they will see a meta analysis and by virtue

that it exists they will take its they will take its conclusion as being very very important without actually going the extra step of it being important to

them to analyze it to see just read it were did they include good trials was it well done so just because a thing is a metaanalysis doesn't mean that it it is

it is a excellent source of information uh back to your first point that gets into sort of the issue with us all being humans because some people especially on

the other side sometimes of these skeptical conversations will say well aren't you don't you have a conflict of interest because you're biased towards

this this standard medicine or science type of thing we're always all of us have our innate biases and that's the whole point of us identifying with the word skeptic is we're trying to be

objective we're trying to uh not be cred credulous about everything we run into in our lives but to some degree yes that's absolutely true so it's the the we always talk about industry because

that's the easiest thing to talk about but if you're getting money from anything or if you're getting if you have a vested interest in anything then that's something that could sort of bias

even in a small way whatever it is you do sir I was curious Len uh people got

headaches from placebos and from the medicine in roughly equal numbers if this was put out would would headaches be on the label as one of the side

effects it would be on the label and one of those side effects I said really fast in the bottom of the commercial that's showing it to you on TV I mean people got headaches from the people just get headaches apparently and

where side effects come from is essentially whatever trial or trials that's used to bring a drug to Market if any one human being had any of those side effects they put it on the list and

that's why you have that very fast talker on those commercials the placebo people got the right even if the placebo people got the same stuff it happened when you got the drug and so that means

who knows maybe it was from yeah but the other flip I I should also note that I'm violently opposed to any of the drug the commercials for pharmaceutical things to the lay public because it's hard enough

for practicing clinicians to understand them and deal with them and then when you put that information out and people either uh buy into oh this is the perfect drug for me or they get completely terrified of the side effects

and maybe they won't then because they heard that commercial be willing to take a drug that otherwise I think would be very important for them that's an awfully scary Dr yes that's exactly

right yeah it's interesting in the conclusion in the article they said the first article they say they don't recommend it and the conclusion in the abstract they

kind of they don't say we don't recommend it they they just say it's not well tolerated limit limiting its potential and indicating the need to

find alternative isn't that sort of unethical I I don't know that I would say it's unethical but it is certainly a difference of tone and it's the question of so I would say that which of those

two statements I prefer I would actually prefer them not recommending things in the not talking about recommending things because in the purest sense what you want to do with the article is

you're trying to put it in context you're stating your opinion at the end of the day you want to objectively put out the data and so then if you start as soon as you start making certain

recommendations then that starts getting a little bit too speculative and especially in studies that aren't so good unfortunately in a lot of complimentary and alternative medicine studies uh for instance when you read

some of the other article you'll see that they make very Grand sweeping statements about how uh well Homeopathy has clearly been proven to be superior to this without really any corroborating

data and we recommend that this has done that and that's that shouldn't in my mind be the role of what you're trying to state so I think the abstract is actually in that sense worded more appropriately at least to my mind than

the conclusion is but that's a very good point yeah did the people who uh participated in the studies find out the conclusion typically no having been a

sub study participant myself it's not it's not like they push that out to people if you ask for it I mean oh absolutely there would be no reason it's all over but usually and

it's interesting because it get this gets into a very good last Quick point that I will make and it again gets into where study money comes from because a lot of times even if you ask I'm trying to ask someone who's a good friend of

mine who's a colleague who's finishing up a study and they'll even say I've got an NDA I can't even tell you even though it's about to be published it's weird embargoed information it has to do with

data funding it has to do with copyright but if you get money from a public source and you registered from with clinicaltrials.gov which by the way if you ever want to find out some oftentimes you can find Details in

clinical trials.gov about a the data behind a published study that is not actually in the publication because if you get money from the ni you have to disclose everything and so

those type of things the information is freely made available I hope all of you found this to be a useful exercise please give me feedback if it was not otherwise thank you so much for your

attention I appreciate it