Computer generated transcript

Warning!
The following transcript was generated automatically from the content and has not been checked or corrected manually.

All righty. Let's get started. Welcome everyone. Um It's been a while since we've held a research webinar and I'm sorry about that. But today's topic is the critical analysis of lab research. And I'm gonna keep interchanging between the words analysis and appraisal. So don't freak out about that. I mean them in the same way. Um And I know this topic is not everyone's favorite. It's not very spicy, but it's an essential skill to have to be in the healthcare profession. So, and that's because you're going to read hundreds of studies and you need to know if they're real or not because I think I read recently about 60 50 or 60% of papers out there are actually fake. So it's really important to be able to distinguish between them and be wary of dodgy studies. So the learning objectives for today are listing the sections of the lab research study, which I'm sure most of you can already do. And the, the bulk of it will be understanding how to critically appraise the different sections. What that means is just reading through a section in a paper, not necessarily in full detail, but trying to get, trying to find any inaccuracies or any downfalls that might make you question their conclusions and we're gonna see how to do that. And I have an actual paper here as an example. So we'll go through that a little bit as well. So, like I said, why bother you have to know if a study is reliable with so much fake news out there. Um Specifically in research studies, you must be able to distinguish between what's good and what's bad. So before we actually get started, what we're gonna have is a pre webinar quiz just so I can gauge um your knowledge right now. So let me just get that started and share that screen. So I'm um the code is up there. I'm also gonna put it in the chat. So there's a link in the chat and the code is on your screens right now. So if you could just join up, we'll give it a couple of minutes. Um And then we'll get started with the quiz. It's not too long, it's quite short and it might be a lot of new stuff that you haven't seen before, but don't let that frighten you. Um because we're gonna learn all about all about what the quiz is talking about. Ok, give it one more minute. So it hits 7 20 then we'll get started. Excellent. Everyone's getting on really quickly, which is always a good sign. Um And if you have any questions throughout, just put them through the chat. We'll have a Q and A section at the end. All right, let's get started. Everyone if you're ready, do you have a bit of time? You have 20 seconds because there's a quite a bit of text. But how did it go? Perfect. Everyone's voted. Let's see. Yeah, the majority got it right. And we'll talk about why that's important and why maybe the most up to date method is not always the best option. Um Repeating is important, but it's not something you would scrutinize cause it's not something you can scrutinize. Um OK. Question two. Oh OK. I had a feeling people would go for skimming the abstract because that's what everyone does. Um It's what I used used to do as well, but the first section of the talk is going to hopefully show you why abstracts are generally misleading and you might actually get the incorrect information if you just read the abstract. So let me get back to the presentation now. Uh Perfect. And just again, if you have any questions, put them in the chat and I'll go through them at the end of the talk. OK. Moving on to the abstract section. So it's just the 1st 250 to 3 50 word summary of the entire study that's been taking place. But that, that's been carried out. And sometimes these days actually, you can also find an image abstract or graphic abstract, which is actually quite useful to summarize what's going on. But the problem is it's so short that it only tends to answer the research question or the title of the paper as a yes or no, like it did work or it didn't work. Um And it only gives a very quick summary of the methods used. You, you, you want to know the details just to make sure they've done it right. And they've used the right methods and it doesn't provide although they can have graphics in them, that's usually just an overview. They don't actually provide images of the results in the abstract to judge their quality. Um And there can be bad quality data which um throughout the study, it's a lot easier. Uh If you want to look later on, I would refer back to a previous presentation I did for mind of blip about lab techniques and you'll see some images of bad data um which is very poor quality. You can't get any information out of that. So, and you can't tell that from the abstract, you need to read all the way through. So I'm just gonna, this is the study I've chosen to talk about because it has pros and it has cons and it's quite a good basic one to go through. Don't, there's a lot of text, don't read the abstract, that's not what this is about. Listen to me and I'm going to explain what's going on and it's mainly gonna be us talking about the underlying bits. OK. So, what they've done is they have this green protein here called EGFP. I'm sure quite a few if you've heard of GFP, it glows green and that protein is connected to a couple of other proteins which can by connect or bind to MRN. And what they've got here is that they've almost targeted it to this beta actin, Mrna. So what they're trying to do is use this GFP and RNA binding protein combination and use it to locate the M RNA in the cell. That's the gist of it. We're gonna, the words I'm gonna be using is GFP for the green fluorescent protein and Puff, puff for the RN A binding proteins that are on its side. And just M RNA for the M RNA that is being connected to OK. So looking at the underline sections, they've given you a little bit about the methods, they've used fluorescence microscopy, but they've told you nothing more than that. Um And they've given you a bit of result as well that the probe was labeled precisely uh with the MRN in the cytoplasm. So they're saying that every MRN matched up perfectly with the protein and we'll actually go on to look at the results and you can decide for yourself whether you think that's true or not. Um And then the final concluding sentence very strong in this abstract is the ignore Pum HD. The, the, the GFP conjugate here can enable visualization of the MRN localization and dynamics in cells. Like I said, it gives you a yes or a no here it's saying yes, we can use our protein to locate MRN in the cell. Sounds almost too good to be true. Um And actually, this paper is one I used in my phd as well and I discovered very quickly that early on I should have probably scrutinized it in a lot more detail. And you'll see uh what appraising it properly can uh can tell you about the study. So moving on to how you talk about the method section, the things you would look for in a me the method section is a, a little bit tricky to analyze or appraise because you have to have a little bit of background on the lab technique that's being used. So again, this is a point where I'd ask you to refer back to the webinar I did earlier about basic lab techniques uh because that knowledge will take you through most ST most lab research studies out there. So the most important point is are the most appropriate methods used to answer the research question. Yes, it's great to use the fanciest type of microscopy. So here in the previous study, we mentioned, they just use simple fluorescence microscopy. Why didn't they use super resolution microscopy in a two photon microscope which quantitate everything perfectly? Uh That might all sound good on paper. But the more complicated or the more advanced a method gets, the more s usually the more sensitive it tends to get. And that can introduce issues into the actual measurement and it's extremely expensive. It's just not practical for everyone to do it. So you will never see a paper always using the most advanced techniques. But as long as you use the most appropriate techniques, it's all right. That's what you're looking for. The second point is have they used an orthogonal method to verify the findings. Orthogonal is a weird word. What it means is have they used more than one method to prove the same result? OK. I'll give you an example here, the puff proteins we mentioned in the previous slide in the abstract, connect to RN A. There's different ways of measuring that connection. Have they used only one way of measuring the connection or have they tried a variety of methods and re reached the same answer? Obviously, it's a stronger result if they've used multiple methods and come to the same conclusion, which is why you're looking for that. But again, it's not an essential in studies, it just highlights which studies might be a slightly slightly better than others. Now, you might have noticed that I'm talking about a few positive things here. Have they use an orthogonal method because not doing that is not necessarily a negative. But another important thing to know about critical appraisal is, it's not only about negatives, it's also about positives or good features of the study. Um And then, yeah, like I said, if you, if you're familiar with the technique, you can look at the details of the actual procedure, they've used to see whether it makes sense because a lot of studies will do things. So like fluorescence microscopy or PC R and use the strangest settings you've ever seen. And that might make you a bit suspicious about whether they actually did the study or not or whether it's realistic. Or you also want to compare between papers in the field to see whether a lot of people are using the same method because that makes it a bit more reliable rather than a rogue study, which has just done whatever they wanted. Um Keep ro studies can be good if they're done well. But it, if you're familiar with the technique, you'll be able to differentiate between the two. So like I said, our trickier section, but let's go on to the next. Uh We're gonna talk about the paper again a little bit. Ignore all the text at the bottom. I'm gonna do slump, summarize it down for you. So like I was mentioning earlier, they had to see if the puff protein was actually connecting to the RN A. So they use something called immunoprecipitation. That's great. But if you look at the methods section, they only used one type of primer So they found out whether the protein was binding to the RN A, they thought it would bind to, but they didn't look at whether it bound to any other RNA S in the cell. They have no idea in this paper about nons nonspecific binding, whether it was binding this that and the other because they only looked for what they wanted to find. That's a kind of thing that you would critically appraise. that makes me doubt the results of this paper and we'll keep building on this avenue in the few in the upcoming slides. Um Yeah, we won't talk about much of the others, but I, what I've highlighted in the text taken from the paper is that they've only used one type of primer. OK? We won't go too much into the fish in microscopy right now because I don't want to bore you. And I'm not sure if everyone is familiar with the details of these techniques, but it they will be there for you to look at later on if you're interested to go and Google and have a look at what I'm talking about there. So moving on to the results section. Now this is where all the meat of the critical appraisal is going to lie when you result sections can look really scary in lab research papers because these days they tend to have 56 different subsections in the results. But if you kind of how a result section is normally structured is it tells a story. The first subsection will give you a little bit of information. The second subsection will build on that information and so on to eventually give you a full picture, which is why again, it's quite important to read through all of them. Maybe not in the the greatest detail in the world, but at least briefly. So you understand the story because otherwise it's like reading a book without and skipping the climax. You know, it's it's important to look through all of the subsections, but how would you critique them then? So here you have graphs, you have things to look at. You have visual displays of data and you have the associated comments uh from the authors as well. So there's five points and we're gonna go through the five points in more detail as well and also refer back to the paper we've been looking at. So one of the very important points is have they used the correct control experiments? I'm not sure how familiar all of you are with control experiments, but it is, it's well, we'll go into it in a bit more detail then you have, have they commented on unexpected findings? It's biology, you're always going to have unexpected findings, but it's important to acknowledge them and provide some sort of a potential explanation for them because otherwise it's kind of like you're ignoring it. And as a reader, I'd be like, oh why have they not talked about it? Probably because it doesn't make sense, which might undermine the rest of the experiment. So it's important to look out for that. Uh They're not always going to comment on every tiny little detail, but if it seems important to you and they haven't talked about it, that's, that's suspicious. Um, and this, the way I'm, I think about this myself, which I'll summarize at the end as well is you always want to ask yourself, could there be some other reason for the results I'm seeing in this paper, some, some reason other than what they've suggested in the paper, if the answer is yes, you have to look around elsewhere to see if other papers have thought of this reason if they haven't and you're still very suspicious. I would be wary of the paper because if they haven't thought about it and it's an important factor that might affect their results, then maybe they haven't done their due diligence to do the experiments uh to the best standards. Um Then moving on to number three, have they overinflated the results? So like we saw in the abstract, people love to claim yes and yes or no. Something works. Something doesn't. That's, that's a red flag right there because in research, you, you never have a yes or no, there's always more experiments to be done. It's very difficult to conclusively prove something, at least in a single paper with only one study group doing something. So it's important to look for over inflation of the results. Another place where you might have heard about this in a bit more detail is the Wakefield study, the old undermined MM R autism study where Wakefield just overinflated the link between the MM R vaccine and autism and look at the problems that created. So it's very important to look out for that. Then another important thing to do is compare the results to other papers in the field. And I've been actually mentioning this um throughout the talk and it's because it's more reliable if a lot of people have done it because not everyone is going to fake their data. Um And not everyone's gonna end up with the same answer, but it's important to see the differences that people have obtained and why, which might help you explain why one single study has obtained the results they have by looking for differences compared to other similar studies and the field in general. Now, of course, if you're just doing this for a lab report or an essay project for school or for university, then you're not gonna have that much time to read through all the papers in the field or at least a few papers. But if you do it, you're pushing yourself to the top of the, to the top of the mark there. Uh That's what people look for in very effective critical appraisal and very high level critical appraisal because you've taken the time to actually uh look through the field. And then lastly are the results Generali. So again, this relates to comparing to other papers in the field. But for example, if, if I have a study and it said this drug in this cell line seems to reduce the expansion of cancer cells and then you read in their conclusion or the results, something like this drug is, if an effective treatment for cancer and should be moved on to randomized clinical controlled trials, then you'll be like, oh, but they only tried it in one cell line. They only tried it in vitro. Why have they come to this final conclusion? You don't know if it's Generali to other types of cancer. There are other types of cell lines. It's quite important to see how narrow their question was. Um and then interpret it accordingly. So now moving on to the fun part, looking at the examples. So have they used the correct controls? Why are correct controls important? Let's do this as an example first. And then we'll talk about a bit more here. I have a graph that I have created myself and it's looking at the effect of drug M TB on activating the insulin receptor. OK. On the Y axis, we have biological response and on the x axis, we have two bars referring to the drug and an inactive version of the drug. It's been heated to two hun uh, 100 degrees which is deactivated it. So you can see that there's no significant difference between the biological responses of the, uh, two conditions. So you might think, oh, the drug is not working. And here, actually it, the inactive drug is also control. A control is just something you compare your actual results to, to see if it's working or not. Um, and according to these results right here, the drug doesn't seem to work because it has the same amount of activity as the inactive drug. Quite interesting. Uh But they've not done all the correct controls. So here again, you're trying to think of what could be another possible reason that my drug shows no more activity than the inactive drug control. And again, this comes with practice and googling and reading up a bit more and well coming to webinars like this. But one idea I would think of is OK, you don't just add a drug as a powder to some cells. You add it as a liquid, maybe the, the buffer, you've put the, put the drug in before adding it to the cells affected the experiment somehow. So in research or medical terms, you'd call that the vehicle. So has the vehicle of the drug affected its biological response. And so then I'd be like, ok, I need to do a control with only the vehicle and see what happens. So I do that experiment and I found, oh my God, how interesting the vehicle control by itself, no drug, no inactive drug. Just the buffer itself seems to have a massive biological response on these cells. So, using that vehicle control with the drug doesn't allow me to really see the true effects of the drug. It's almost like I came up with a good analogy for this the other day. So I'm gonna share it with all of you paint. Let's say you have, let's say the vehicle control is red paint and the effect of the drug is a blue, is blue paint. Ok? If I have a bad vehicle control that activates my receptor, imagine it as a massive blob of red paint where whereas a good vehicle control will be a smaller blob of red paint and then the effect of the drug is the same in both. So it's much easier to visualize the blue paint in a small patch of red paint compared to a massive ocean of red paint. If that makes sense, if you mix it in together, one will look purple, but one will still look red because it's a lot more red than the blue. So I ho I hope that made sense. But essentially the way to solve this is to get a better w better vehicle. So then I'd repeat the experiment with the, with my drug mixed into a different buffer. And I end up with this, my new vehicle control does absolutely nothing to the self and that reduction in interference allowed the effect of my drug to shine through. So now you can see there is actually a significant difference between the biological response created by my drug compared to the inactive control. And hopefully, this explains to you why having the correct controls is so important because otherwise you might see negative results in a paper, but they might be false negatives because they haven't done the correct control or there might be false positives because they haven't done the correct control. So it's important to think back to yourself. Like, OK, could there be another reason that there's no effect there or could there be another, another reason for the effect there that they haven't thought about? So hopefully that's something. Now you'll let's go back something you'll keep thinking about in the back of your mind while reading research papers and looking at it in a real paper. What we're looking at here is forget the reading again. I'm gonna talk everyone through it. What we're looking at here is again, they've like we've been saying the Puff protein would connect up to the RN, but they need to see if it's connecting up. So what they do is they isolate that GFP Puff protein combination using antibodies. So here they've used a GFP antibody to isolate their protein and then they've used PC R to see if the RN A was present or not. OK. And you can see in the blue rounded rectangle that I've highlighted that when the protein was present, there was also RN A present along with it, the white band in the middle represents the RN A. Whereas when the probe wasn't present, the probe being the GFP Puff protein, there was no RN A present and that might look really positive at the start, something like OK, there's only RN A when my RN A binding proteins are actually there, which makes sense. But we're thinking to ourselves, could there be another reason for that? And another potential reason could be because maybe my protein is just sticking to the tube that I'm using M maybe. And that would result in a false positive because you have RN in your tube even though it's not being attracted by the antibody. So the perfect control for this type of experiment would be using a random antibody and hoping that no RN A shows up with that and that'll really show much better that it's your protein that's causing the RN A to appear in this PCR, not just the protein sticking to the tube if that makes sense, it's this one's a little bit tricky. And if you come back to it and read it again later on with the text I've written, hopefully it'll make a bit more sense um commenting on expe unexpected findings. So there were a lot of unexpected findings in the paper. If you look at the two images that are on there. The green dots like we've been talking about so far is the GFP, the red dots is, are the true locations of the RN A which have been found out by another technique. We're not gonna worry too much about that right now. But essentially, if this is, if this is working, what you'd expect is all the green dots should have a tiny red dot next to them because the green dots are meant to connect to the RNA that is represented by the red dots. But like I've circled or pointed with arrows, you have green dots which are nowhere near red dots and you have red dots which are nowhere near green dots. So what's going on in their abstract? They said very conclusively that our probe, our protein binds perfectly to these RNA S and can be used to localize them in cells. But that's not the case because these green floating probes all around the place are not bound to any RN A or uh the correct RN A because there's no red dot next to them. So, and they haven't commented on that through the paper at all. They've just constantly written that this binds perfectly. This will allow us to locate the RN A perfectly. Whereas that might not be the case just looking at this one image. Um And I I put the references uh paper in one of the earlier slides. If you actually go and look this is a tiny little image in the corner of the paper almost as if it's hidden or tucked away, which is why it's important to just scrutinize through and be really careful. Because this clear what this tells me is that my protein is not binding really well to the RN A or there's something else blocking it from binding to the RN A, which might be a flaw of the experiment itself. OK. So I might need to modify my experiment slightly. I might need to modify my tactic for, for binding to the RN A. There's a lot of things that can be modified here because this is clearly not working uh as well as they claim it to be working. So you can see you can sort of build up and see all the plot holes in this study and how we've arrived to them by scrutinizing the different sections and then the last three are much quicker. So have they overinflated the results? Um again, here, what I've highlighted says this resulting indicates that the proteins or the probes precisely represent the localization of the MRNA S not true from the images we saw earlier. They don't because they are green dots floating around without any red dots around them. So yes, in fact, they have overinflated their results which makes me very suspicious comparing their results to other papers in the field. Um So again, what I would do is just read up on other papers that have similar proteins or similar ways of locating RNA S and see if they get uh similar results. Um And then lastly, are there results Generali what cells have they done these experiments in? Have they tried this experiment to connect to other RNA S? So the one they've used, which was, which is on this slide as well, they've looked at it connecting to beta Ain MRN. But have they tried any other ones? They haven't? So all of this is a very narrow focused study to really uh prove what they're trying to do. They have to try a lot of different rnas. Um And they have to have better explanations and better controls to show why you have empty green without any red around it. So that was heavy and I'm really sorry for that, but hopefully you can watch this back later or it helped you in some way or the other and you'll be thinking in the back of your mind now, when you read an experimental paper, um so let's summarize quickly. So we'll go through it in steps. You want to go systematically through the study to identify its strengths and weaknesses. You want both. Um If you don't understand the method, Google it honestly, even now, I can't really understand, I would say a majority of papers that I read because they use, there's always new techniques, there's always papers get longer as time passes because our expectations are so high. So I'm always Googling stuff and I recommend all of you to do it as well or even to supplement this uh this webinar. I would go and Google how to critically analyze lab research studies as well or how to analyze a specific methodology or what controls should I have. If I'm doing this specific technique, for example, here you could Google up um what controls should I have for an immunoprecipitation experiment. And it will clearly tell you that you need to have an antibody which doesn't act a random antibody which doesn't bind to your protein. So it's all of these other ways as well. You can enhance your knowledge around this subject and then be as cynical as possible. Like I've been trying to reiterate throughout this webinar. Does this make sense? Just que just questions you can ask yourself while you're reading, which will help you sort of see things with a bit more clarity? Um But the most important, one of those questions I would say is, are there any other explanations for the results that they have obtained? And in a very good paper, they've talked about all the major ones. So if you look at things like nature, not, not that I'm saying all nature papers are excellent. Some of them are really bad. But if you look at things like nature, you'll have a general idea of what extremely high standard is. Um And no research is perfect. So you will always find flaws and weaknesses in the study because some things just can't be done in a certain way or some things have to be done in a certain way. That doesn't mean that the conclusions are invalid. Ok. And that's where again, it brings in the importance of looking at other papers in the field because they would have experienced similar limitations if you see a paper, if 10 papers in a field have talked about this limitation or have found this method difficult to do. And there's one paper which has done it perfectly, be suspicious again, something that caught me out during my phd. Um And uh these things are obviously more relevant if you want to do something like an SFP or academic research or academic clinical medicine. Uh But everyone will also be reading papers generally as well. So it's quite important to have a basic understanding of what we've talked about in this um webinar. All righty. So we won't do questions yet. What we'll do is go through a quick post webinar quiz to see how much, how much you've understood. And I know it might be daunting right now because there's a lot we went through in a short amount of time. But hopefully you find it a bit helpful cause a little quiz at the end is always um helpful I think. Let me just get it ready. All righty. So you have the code up there. I'm gonna also put it in the chat. I'll give you a couple of minutes to join up and then we'll get started. Don't be sho, I promise this one will feel easier than the previous one because you had no background in what we're gonna talk about. So, a couple more minutes for a few more people to join in and we get started. All right. Why don't we get going? And people can join in between if they would like. Um So a little bit of a trick question there, I guess that I put, you can never remove all the confounding variables. Like I said, there's always going to be something, some limitation, some downfall to the experimental method and to the paper itself, no control experiment can remove all the confounding variables. If you think about it, the air temperature in the room, you can't control that every day. The, the rate at which you do it, how quickly you add things to different test tubes. Not, not everything is controllable. So you can't control experiments, can't remove all confounding variables. They remove the primary ones, the main ones, what they're at, what the, what their true, what their true purpose is is to make sure that your drug is actually doing what the results show like I had in that little example that I made, you wanna make sure that your drug is actually not working and it's not just because of something else. That's why you have a control experiment. OK. Next one. All right. Let's get started. Perfect. I mean, all of them were OK. Answers. Um, while you're reading papers, I'm definitely upset or irritated when reading a long unnecessary paper. Sometimes intrigued. But yes, you want to be cynical to be appraising papers because you want to question everything that they've done with their life. OK? You wanna, you wanna to essentially feel like you're tearing their paper to shreds because that's when you start finding the true flaws in experiments. OK. So we'll just go back to the powerpoint and I don t see any questions in the chat at the moment, but please keep posting questions in the chat um as you have them. Um And while you're doing that, here's the feedback QR code and that'll be really helpful to me if you fill that in just because that's also how you get your certificate. And it'll be great if you sign up to the next webinar we're doing next week on Tuesday about the critical analysis of randomized control trials, which might a lot of people might find it more interesting or more relevant. I will also put the feedback link in the chat if you just give me a second, right? So the feedback link is also in the chat, please do sign up for the next webinar um and fill in the feedback form for your certificate. I'll be here for a few more minutes if you have any questions. But otherwise, thank you everyone for attending and I hope you took something away from this talk. All right, let me answer this question. Great que question. There. Does all research require a control group? Yes. All research has to have at least at the minimum one control group because otherwise you have nothing to compare your experiment to. For example, if I wanna see if a drug is working, I can add the drug to my cells and measure a response. But I don't know whether that response means it's working or not because I have nothing to compare it to. So I would want an inactive version of the drug or just the vehicle control the vehicle by itself just to have something to compare to, to see if the drug is giving more of a response than the control. So yeah, all research requires a control group. OK. So if there's, if there's no more questions, we'll finish off a little bit early um after quite a heavy webinar. So do take a look at it after it'll be up on metal and youtube. So I hope you enjoy this webinar and please do join us for the next one as well. And if you have any questions, email mind the bleep and we'll get back to you um for any suggestions at all.