Computer generated transcript

Warning!
The following transcript was generated automatically from the content and has not been checked or corrected manually.

Right. Hello everyone. Uh Thank you for coming along to tonight's session. It'll hopefully be a very brief and kind of beginner's introduction to kind of critical appraisal. So I'm James and I've got Ryan with me. Hello, everyone. Uh So we're the academic reps for clubs this year. So we're hopefully gonna try and do a few events and really this session is to kind of set us up. Uh or there'll be a few societies wanting to run kind of journal clubs on different papers. So we thought we'd kind of introduce you guys to uh a Costco. Um So I'm gonna post a few links under the chart here. So this is the cost pill that we'll kinda be working through tonight and this is the brief paper that we um wanted to go through. Um So last year I doctor Calvert and uh Steven did a session like this where they did a bit of critical appraisal and kind of went to different studies. So this is gonna kind of hopefully mimic this. Um So yeah, I'll just get started. Uh So kind of brief is what is critical appraisal to begin with? It's a word or phrase it always thrown about. People don't really understand what it is, what's its purpose and its importance. Then uh then Ryan's gonna go through AP O framework and that's role and evaluating research. Then we're going to have a look at the difference between internal, external holiday and their significance. And then we're then going to introduce the cost pill and essentially just reach different study type. Um It kind of takes you through a checklist that allows you to critically appraise a paper and we're going to apply that uh to a sample research paper to massive disclaimer. Uh This session is designed for beginners um in critical appraisal to kind of have a brief structured approach and it does involve various elements. And unfortunately, we can't really cover everything in this session because one, we don't have enough time and two or not qualified to. Um So, uh essentially, this is just to try and get you guys uh a bit of an introduction to it and then you can go away and uh work out of yourselves. Oh What is critical appraisal? And essentially, it's really just a process and it's whenever you kind of examine a paper uh or research in a careful and a systematic way and this is to judge uh kind of three aspects of it. So the validity then um and this will be specifically related to the methods. So you want to look at the study design and the research question and does the is does the methods like with the research question and therefore answer that research question? Next, we're having a look at the trustworthiness of the paper and this will be linked to the results. Just looking at um the data analysis being performed, is it appropriate? And does it give the results that you want and need? So it's determining uh what the research is telling us, whereas the first part is telling us how trustworthy the past of research is and then we're looking at the value and the relevance of the research. So it doesn't align with kind of studies that have already been published in the relevant literature. And I can these findings then be applied to um what you're interested in and then to the broader context of the field of research that that paper is kind of looking at. So really it's an essential skill for what the phrase it's really so about is evidence based medicine. So in order for you to practice evidence based medicine, you need to look at papers and also just accept them at face value, you need to have a bit of interrogation of them. Why is critical appraisal important? So essentially you need to critical praise everything that you read can't just take it at face on you and you need to have a look at a paper and try and dissect it. So you're trying to see if there are any flaws in it, any limitations or any biases. And this is so that you can appreciate the research within the context that has been done. So you also want to look and see whether the data that is being presented is robust and trustworthy. And this will will allow you to make a basis for making your own conclusions and inferring conclusions. And through doing this, it will be promote evidence based practice. And really this is using high quality research studies to inform your clinical decision making. And whenever you do that, you will hopefully improve uh patient occur, which will make it safer. But importantly, you shouldn't just judge a kind of paper based upon the journal that is published in and also the author has published it because this isn't really critical at president, maybe that's what people have thought about before really interrogating the paper and not looking at it just from a standpoint of it has such and such an impact factor. So this kind of directly relates to the Costales. So um it's taking you through, there are different types of studies that you can read about. And what we're trying to do here is just highlight a few of the different types of study. And therefore after you'll be able to have a look at the C bills that are available for that. So it's just kind of appreciating what studies there are and then what tools you can use to kind of critically appraise them. So these first two are very good. So you have your randomized controlled trial, whereby you will have um a quarter of patients that will be eligible for a study and will then be randomized to receive um or it will be randomized to receive an allocation, either a treatment or a control or a standard of care. And really this is looking at the efficacy between two treatments, then we have systematic reviews and these are kind of structured questions that we will try and encompass all the available studies that kind of fit those parameters. And we'll look at the evidence base that is present and evaluate the interventions um that are um have been documented next time we have cohort studies and these are really observational studies. So it isn't randomized normally looking at two groups of individuals sometimes more. And it's looking at patients where I kind of a period of time comparing different outcomes between these two different groups whenever then have a look at the case control studies. Um It's looking once again at observational research, looking between two different groups. However, um it's looking at differing and an outcome and this is identified uh based upon a supposed causal attribute. So something um isn't heard that it is causing the disease or not. So now we've moved on to cross sectional studies again, another observational study and it's looking at the relationship between health outcomes and this is for a specific defined population at a certain point in time. And it's looking then at the prevalence of the disease or the risk factors for that disease. Whenever you then consider qualitative research, um it's kind of looking at a study, not using numeric factors. So it would be using um unstructured data such as your interviews or your um questionnaires. And essentially um quite a lot of that is good for um this recent uh thing done in Queens where by they had a anatomy uh teaching interme uh fraud sound and it was gauging then the reaction from that and the positives and negatives. So qualitative research does have it does have its place. Then importantly, now, especially with the health service, you have your economic evaluations and looking at two different uh alternatives it's looking at and the costs and consequences. So looking at your qualities, then then kind of lastly for me, then you have your diagnostic studies. So really this is looking at the performance of diagnostic test, this is looking at whether or not it is good at predicting the presence or the absence of disease. And then lastly, then it's looking at your case series or your case studies. Really, this is looking at uh the characteristics of patient groups or an individual patient um in a disease or they've undergone a specific procedure, was looking at the observations. Then from that and from these kind of smaller studies, this can then inform larger studies So while they might not hold as much weight in the evidence base, they are important to inform later studies to develop into court studies and then potentially trials, you know, I'll pass the wig on. Thank you James. So I'm gonna continue now by just talking a little bit about how we would go about actually critically appraising the paper. So when critically appraising the paper, you can sort of consider the paper in five areas. Uh So firstly, relating more to like the validity side of things, you would want to consider the research question. Uh Is it clear, is it appropriate and well defined? And I'll talk a little bit more about um the research question in a moment. And then secondly, the methods they should be examined for the rigor. Is the study design appropriate. Is it valid and easily reproduced? Next, relating to the trustworthiness of the paper, uh We will need to critically appraise the results. Um Are the findings clearly presented uh precise and are the statistical analyses robust and then to identify if the study is relevant, the conclusion should be checked. Um And it should be checked for alignment with other studies and relevance. Um As James said to see how relevant it is and the broader context and among other patient populations, including the one that you're interested in or your local population. And then in a similar way, the value of the study can be assessed by identifying the limitations of the study and um the implications. And if the informed practice or policy and some background considerations you would want to have while less important uh would be to consider the title. Um for instance, is the title uh sort of claiming something and this claim is not appropriately proven. So the best type of titles would be sort of objective titles uh rather than those that are sort of promoting a claim. And then similarly, you want to look briefly at the authors, maybe some of their previous work. And if they have any affiliations that might be influencing the writing, uh quickly looking at maybe the quality of the journal and the impact factor, the setting for context and also the ethical standards and any funding or conflicts of interest. So to talk a little bit more about the research questions, so there's t two real areas that you need to look at when you're examining the research question. So number one would be, does the research question actually align with the study design? So different research questions, um different types of questions are going to require different study designs. Um So the paper should have the right type of design to answer the question. And secondly, when this would become obvious when you're using the CASP to, so different designs will be prone to different biases and they're therefore critically appraised slightly differently. And there's a specific tool for each type of study. Um So we're just included as well as some examples of different types of questions that could be asked and then some appropriate designs for these type of questions. So for a study that is asking a question about etiology, cohort studies would generally be used um to explore a cause and effect relationship. Uh And then for risk factors, case control studies would be more appropriate. Um So it can compare individuals with one particular condition, for example. Um And then those that don't have it and then retrospectively look at any risk factors or exposures that they had and then for incident cohort study again, would be ideal. Um since you're following the groups over time and tracking their currents um of new icons and then for prevalence, cross control cross sectional studies er would be better um because they can sort of measure uh the condition, the patients with a condition at a specific point in time. Um that for harm questions, uh case control or cohort studies again, are gonna be used there. Then for prognosis questions, you, you could again also do a core study, it would be appropriate. Um But if the goal would be to sort of understand the factors that are influencing your prognosis, you would probably want to use a survival study. Then if you're looking to evaluate value for money, uh you would do a study like um an economic evaluation or cost effectiveness study. Um So you're comparing the cost and the outcomes of different interventions. And then as James said, for effectiveness, traditionally, you'd use uh RCT and this can also be used for diagnosis questions or also a diagno diagnostic test study um is appropriate to evaluate the accuracy of a test to diagnose on that. Again, as James said, patient per patient experiences, quality of studies would be best um using interviewing or surveys or questionnaires. So the second question you'd want to ask would be is the research question clearly defined and we would use this pea ot framework to do that. So that would stand for population. So what is your target population intervention? What is intervention is a clearly defined treatment test procedure? For example, um the comparing or comparator is a control group or is it a comparison for the intervention? Uh then the outcome, what are the outcomes and what is the timeframe? So an example of an appropriately defined research question would be in symptomatic patients with a meniscal tear and knee osteoarthritis. So there's a clear population being investigated, does arthroscopic partial a mastectomy? So that's your intervention and then compared to non operative therapy as your comparison result in better functional outcomes. So that's the outcome that you're wanting to assess and then there's a clear time frame. So over a 12 month period, so once you've sort of established, if your research question is appropriate and clearly defined, um this is when you would use the C to. So the CS tool provides a systematic framework to evaluate your validity of methods, trustworthiness of the results and then the relevance of your conclusions. And then, as we said before, um it offers tailored frameworks for different study designs. So then you're focusing on the most relevant aspects um of the study based on its design. So we've just picked out an example paper that we're gonna go through using the CA to uh I think James should have linked it in the chart and the cast to that we're going to use. Um So this was titled Long Term Recurrence and Complications Associated with elective incisional hernia Repair and just a bit of context about the paper. Um since we've literally just put it in the chart, so it was a prospective observational cohort study um with a large scale population of patients with incisional hernia er from a Danish National Patient Registry. And the timeline was between January 1st 2007 and December 31st 2010 with follow up until the first of November 2014. So their aim was to evaluate the five year risk of recurrence firstly and complications after elective incisional hernia repair um between those who've had a hernia repaired with a mesh. Um and that could be either open or laparoscopic uh and those who had open repair without a mesh. So I'm gonna pass it back over to James who's gonna go through the first section of the ca two relating to the validity of the methods. Uh Yeah. So essentially for the cost to them uh for this is for specifically a cohort study, uh we're gonna go through uh different sections. So there's A B and C and the first section A considers are the results followed for the first question. Then it did the study address a clearly focused issue. So as you can see here, I just want you to look at basically eco almost. So your population studied the risk factors that were studied, is it clear whether they tried to uh tried to detect a be beneficial or harmful effect and then the outcomes considered. So for this study, then it was investigating the risk of long term recurrence and mesh related complications following elective abdominal wall hernia repair in a population with complete follow up. So it was essentially looking at open repair, uh which is done with mesh or non mesh and then laparoscopic repair that was done with mesh. And the question then was what are the long term consequences in patients undergoing an incisional hernia repair? So this paper did find its study population it wanted and it did look at the uh risks of long term recurrence and the complications and we'll talk about it about the results kind of later on, but it essentially looked at a whole host of complications from major to um kind of clinically important ones. And we'll just walk through the rest of it. Now, the, the next question considers then was the cohort recruited in an acceptable way? This is a cohort study. And essentially, it's looking at the uh patients from January the 1st 2007 to December the 31st 2010. So it's looking at consecutive patients undergoing this elective additional hernia repair. It's kind of based upon the danish hernia database. So it's wanting us to look for selection bias which might compromise the generalisability of the findings. So as a court representative of the population, was there something special about the cohort or was everybody included as you should have been? So essentially because it was consecutive patients, everyone was included, but because it was a cohort study and not a randomized one, there may be an element of selection bias introduced. Mm However, they will adjust for this later on which we'll kind of consider in a bit. The next question is then considering was the exposure accurately measured to minimize bias. So we're looking for the measurement or classification of bias so that they use subjective or objective measures. So in this case, they used objective measures and do the measures truly reflect what you want them to and have they been validated? Um Well, yes, they've looked at a whole host of um kind of preoperative patient demographics and characteristics, looking at age hernia defect of this primary or current and then looking at at kind of parameters to the mesh. And then thereafter looking at the um complications such as bowel obstruction or perforation or bleeding. So, these are validated outcomes that they're looking at. We're all subjects classified into exposure groups using the same procedure. So yes. So essentially they used a priori objective measurements and they were defined beforehand. So it was set out what each kind of uh patient would fulfill if they met that um objective or not. It was looking at the risk of reoperation for recurrence and then mesh related complications. So was the outcome accurately measured to minimize bias. So it's looking for the measurement or classification of bars. So again, it's looking at, was it subjective or objective measurements? In this case, objective? Do the measurements truly reflect what you want them to have been validated? Yes. So it's using validated outcomes that have been used in previous studies to lie for comparison. Um has a reliable system been established for dete detecting all of these cases from the disease. So, because this is a registry and for um this danish population, everything has been measured properly and there's been total follow up which will come to you in a bit with the measurement methods similar in different groups. So they did use the same measurements between the different groups. And then this allowed for a comparison between um were the subjects that come especially blinded to exposure. In this case. They weren't as it was an observational study. So it was non blinded. So this next section is then looking how the authors identified all important confounding factors. And it's asking you to, to list the ones you think might be important and then the ones that authors might have missed. So in this case, we're really just looking at the method section here. And secondly, it's saying have they taken account of confounding factors in the design or analysis? So because this was an observational study, there was a risk for selection virus. However, they have accounted for this ba propensity score adjustment and this was done through a logistic regression. And then thereafter, a propensity adjusted cox regression analysis was performed with the mesh complications and all patients with a mesh repair. And that kind of looked at different variables including propensity score operation pipe. So was it open mesh a laparoscopic mesh and then looking at the different mesh parameters and age, do they have kind of accounted for any potential confounding factors honestly then? And what's quite good about this study then is, was the follow up of subjects complete enough. And in this case, it was as it achieved 100% follow up rate, which you won't find in many studies. But in this case, you do, so you're considering the persons that are lost to follow up may have different outcomes than those available for the assessment. In this case, that wasn't, didn't occur and in an open or a dynamic cohort was anything special about the outcome of people leaving or the exposure of people entering the cohort. So in this case, because the uh all up rate has been 100% we don't need to worry about that last. And then for my part, it's looking at was the follow up of the subjects long enough and it's looking at the good or the bad effects did have had long enough to reveal themselves. Essentially. In this case, the follow up period was long enough and it extended from the index of the hernia repair until they had a reoperation. So for recurrence till they um death occurred, emigration or mesh removal or then reached the end of the study period in 2014. In this case, the paper was quite good in that sense that it had a complete follow up and for a long enough duration. So next han around he's done quite well and trying to dissect the paper in the next actions. OK. So now that we've talked a little bit about the validity uh on the matter section, we're gonna go on to talk a wee bit about the results. So I'm just gonna talk through the the bottom line results like the main outcomes um because we'd be here all night if we're going to dissect every single thing that was done. So, in terms of reoperation for recurrence rates, um they found that non mass repair had the highest percentage of this 17.1 with 95% confidence intervals, 13.2 to 20.9. And then for open mesh repair, this was 12.3. Um with confidence intervals, 10.4 to 14.3. And then laparoscopic mesh repair had the lowest uh reoperation rates 10.6 with confidence intervals of 9.2 to 12.1. Um I've sort of highlighted here that we can see that the confidence intervals do overlap. And this is not as important for the open mesh and the laparoscopic mesh since they're really aiming to compare non mesh and mesh repair, but we can see that the non mesh open repair compares or overlaps with the open mesh repair. And then the risk reduction compared to null mesh for open mesh was minus 4.8% and for laparoscopic mesh was minus 6.5%. You can see the confidence intervals for those and then looking at the long term complications at five year five years, the cumulative incidence of mesh repair was 4.5%. And this can be further broken down into 5.6% for open mesh repairs and 3.7% for laparoscopic mesh repairs. And then this compares to only 0.8% of complications long term for those who had non mesh repair. And then the risk reduction compared to non mesh for open mesh repair was 5.3%. Um for laparoscopic mesh repair was 3.4%. And they found that open mesh repair was actually an independent risk factor for long term complications versus laparoscopic repair. And this was a significant finding. So to move on to section B of the hospital, what are the results? Question? Seven, sort of blatantly asking what are the results? Are they clear to the reader? Are they obvious? Are they presented in a clear way? And I would say yes, the bottom line results are very clearly stated and you can deduce from the start, recurrence rates were highest in non mesh. Then the open mesh and then the laparoscopic mesh and then for long term complications, open mesh had the highest rate, then laparoscopic and then non mesh had the lowest rate. Then question eight is asking how precise the results I would say the results are precise. Um Due to the narrow confidence intervals, there was an overlap of confidence intervals for recurrence rates between non mesh and both laparoscopic mesh and non mesh, which does reduce the sta statistical strength of the comparisons. However, I would say the risk reductions will still suggest that there's meaningful differences between mesh and non mesh, which is what they were aiming to demonstrate. And there was statistically significant differences for all their key comparisons that they wanted to demonstrate and then just putting everything that we've done so far together, question nine is asking, do you believe the results and I would say that based on what we've just went through in questions 1 to 8, we can be quite confident that the results and the methods are valid and trustworthy and that we can believe the results and the conclusions drawn from the study. Um just to really briefly in one second and summarize what they were concluding. So they stated that was long term follow up the benefits attributable, attributable to mesh, which was that there was a lower risk of reoperation for recurrence that these benefits are offset in part due to the higher complication rates with the mesh repair compared to the non mesh. And then quickly question C is just asking a wee bit about the generalisability, the external validity of the paper and will the results help locally? Um Which is what question tan is asking. So the paper discussed the issues regarding the lack of mandates for meshes that, that um they didn't require randomized controlled trials for their widespread use in the USA. And this was a similar issue was, was also prevalent in the UK and in Denmark where the study is sat, although they don't discuss this explicitly. OK. The UK and Denmark, they will have similar patient demographics and both are currently using mesh and non non mesh hernia repair. So I would say yes, it could be applied to the local population. Question 11. Do the results of the study fit with other available evidence. So this one is a bit of a mixed bag. So there was a single R TT which looked into the rates of complications with mesh. And this, this wasn't the best RCT. It had quite limited patient follow up. And also the inclusion and exclusion criteria were not very robust. This found that there were higher rates of complications with mesh than was found in this study. Um about 20%. And there have also been some other case studies and retrospective studies here and there that have also demonstrated that that the complications with mesh will be more than the 4.5% reported in this study. Hi Albert, a subsequent prospective study was done in 2021 and it was like in at non mesh versus mesh for complications and it had sort of similar findings and conclusions in this study had. However, generally there is a lack of research in this area. So I've said, can't tell for that question in the hospital question 12 is asking what are the implications for practice. So the study quite explicitly states that the implications from their perspective would be that they need better and more long term safety assessments and better documentation for mesh products that are being produced before they go um to be used widespread among the patient population. So just to finish with the appraisal summary that's included in the C A too. So some of the positives from the paper would be that they had a clear research question with um an appropriate study design and 100% follow up was achieved. They had clear and precise results and appropriate analysis. They did critically appraise the relevant literature in the discussion and they didn't overstate their conclusions uh than some of the negatives. They defined recurrence specifically as needing a reoperation um only so this might overestimate or underestimate the rates for occurrence. And then again, the overlap of confidence intervals, disagreement with some of the prior literature and the the study may not conclusively answer the research question which they do admit to in the study and then just to repeat the unknowns. So the only real unknown that I could identify was sort of the lack of comparison coming from the lack of comparison to other studies. So there was a lack of studies that are using high quality large scale registry such as this one. So that would be sort of questioning the generalisability of the study. So that was just a quick whiz through a critical appraisal. Um Thank you all for listening. I'm going to just open up the floor to any questions. And we also wanted to just sample some uh of the societies in queens that are planning on running journal clubs this year. So um if you thought that tonight was useful, these might be good events to attend, to sort of put your skills into practice and to sort of further develop your critical appraisal knowledge and skills. Um I think there's a question in the chart there, James for you. So in this case, I didn't even like to have as long a follow up period as possible, but I guess it just depends on the, it, it will be proceeded upon them and it will basically dedicated by how back here, how often your kind of complication would occur. So to be honest, I'm not entirely sure it will be procedure dependent. And ideally you want as long as follow up as possible, that is feasible. Um So I will have a look up and get back to you because there's no appointment. I think it's something that I'm not too sure on. Sorry about that. Yeah, I think that's fair enough. I mean, typically you'd want at least five years I would say for a prospective study, but again, it is gonna depend. Mhm Yeah. So uh thank you very much for attending guys. Um That maybe wasn't as interactive as hopefully future sessions will be, but hopefully it's giving you a brief introduction to critical appraisal skills program tools. Um It's basically just signposting you to the fact that these things exist and allow you to have a kind of a better interrogation of papers to critically appraise them. Um So as Ryan said, um there will be the Student Health Care Society and cardio so will be leading a few journal clubs and then hopefully with specialty challenges as Well, we'll be having a look at a few papers, um especially it just to try and get people a bit more comfortable with um scientific and medical, lit, medical literature and hopefully, uh this was kind of short enough and pain free. Um just to kind of signpost of things. But uh thank you very much for attending. Hope you got something out of it. Thank you very much to Ryan. He did well here. Um He did a lot of the presentations. So thanks to him and hope you guys have a nice night. Bye.