Computer generated transcript

Warning!
The following transcript was generated automatically from the content and has not been checked or corrected manually.

Yeah. Hello, everyone. And welcome to the National Research Collaborative meeting 2020. Thank you for joining us. We're excited to present our third webinar efficient clinical trials. An introduction to pragmatic trial design with discussion of some of the key considerations surrounding designing trials. Now, I'm so pleased to introduce our chair for the evening. Professor Maron Campbell is vice principal, the University of Aberdeen on Professor of Health Services Research in the Health Services Research Unit. She's a medical statistician and clinical trial list, and her main research interests are in the design conduct on analysis of clinical trials, especially complex trials on the design and conduct of surgical and device trials. Well, compressor Campbell. Well, thanks very much. I'm absolutely delighted to be here tonight, Um, called an exciting light up with an exciting set of speakers and to speak with you tonight. We're going across the pawn, Teo Canada's to Ottawa. We're picking up from McMaster and we're coming to Aberdeen as well. So I'm just going to let you know who the speakers are on then, without further ado will get into the excitement science. But before you introduce our speaker's, I do want to encourage you to us. Questions on this has been to be a chair at an interaction. And if the fantastic opportunity to ask all those questions, you want to find the answers to s So it's it's a beautiful your interaction. Otherwise I'm gonna hold all the questions, and that would be very dull. Eso at this thing will go four speakers tonight and they'll just like you know who they are, eh? So we've got marked climbers. We've got Gregory Pond. We've got sharing the key, and we've got Shawn two week, and so I'm eating them. We're gonna have 10 minutes or so to give a presentation. But first of all, we're going to turn to the first speaker who's professor more Clements. And now let me introduce you to him. So Mark is a clinical investigators at the Auto Health Search Institute. Onda, staff medical oncologist of the Ottawa Hospital Center and a professor of medicine on at the university of all to them. And he's involved in a large number of studies for patients with early stage, local, locally advanced and metastatic breast cancer. He created the Rethinking Clinical Trials program. They react which place is the patient on their family at the court of designing and implementing trials that actually direct and improve patient care. So Mark is gonna talk, just say, the title of his Tokars, an introduction to pragmatic trials on integrated or a look incent from their hands over to your mark, looking forward to it. Thank you very much for the invitation to speak to an audience who's parked at home for me. So my mandate is essentially a trial for every patient in a patient for every trial, on going to talk about the real world experiences of really world trials. My talk we followed by Greg Pond was going to talk more about statistics and then showing degree. Who's going to talk about a V, a world practical child that integrated all the consent? And it's methodology as a working example of how the's kind of child should be done more on how all of us can be involved. I'm going to talk about what the state of cancer trials is at the moment. A zone oncologist. I'm going to talk about the react program. That's the rethinking single charge program that we've been running here in Canada for about six years now on a mortar going to talk about integrated or consent for standard of care options, which is particularly important for discussing surgical kind device child, where many different options are considered standard. So we're calling today for, um, Ontario, Canada, largest population by province on. But we can see from this graph that despite the fact that the incidence of cancer has risen, however recent years, the proportion of patient being involved in clinical trials it's now less than 3% of adults on. Many of these are for phase one and face to child, where the benefits for the individual patient often marginal at best. Why is a core so low on? Do you see many of the slight presented where it talks about position engagements? The efforts of the consent process? My to spiraling yellow arms are to show how sometimes contacts and I will be take a significant length of time. That's cost the superfluous data collection. Some people say patient engagement is one of the challenges on. I disagree. I think the elephant in the room is the biggest barrier of clinical trial. A cool is physician engagement. If a physician does not approach the patient about the study. They will not be enrolled covert. It's affecting all of us across the world in this paper came out in JAMA last week on Essentially, what it's saying is, um, patients in the rival of covert, irrespective off that age, sacks income are less unless interested in being involved in clinical trials. At the moment on the graph on the right times, their shows that the biggest concern for patients at the moment is fear off exposure to covert. And I'm sure you've all found that your clinics are becoming more, more virtual. Um, so we're not seeing the patients in person. We need to find a better way of doing our trial. So the power doctors I suit is we have more, more constipation's. We know more more about cancer. We have more treatments on our healthcare budgets are growing for most exponentially, however, we have little guidance in many years off care that actually impact on how really patients do in the real world. How frequent affect their quality of life. A personal interest of mine is healthcare system sustainability. How do we avoid burn out on? Also, we have more and more reasonable treatment options be the counter therapy device, a surgery to offer patients. A simple example that I like to use is the factor. If I get the results of the midst of your urine and all of us get these without most days, it shows that you can rely. Yeah, have a choice of antibiotics to give, but which is the best on we almost never know. The answer on this is what we term. The eccentricities. Opposition decision making process is the physician time to use the one that used to using one? Sometimes, though, based our decision on what's funded. Yeah, really it based on actual evidence comparing these different antibiotics. What a path Matic child. You're going to hear more about them. The evening goes on. Explanatory trials are where we use a well defined and controlled setting. These child to traditionally, um, select for patient to really don't on the whole, represent the border population of patients. We treat their younger, the richer. They have less comorbidities, and they take less. Other medications, which pragmatic trial are where your testing and effectiveness intervention in broad routine clinical practice. So the react program has been going to set for about six years now on these the elements of the program. But for tonight discussion, it's really around performing surveys on systematic reviews to demonstrate equity poise on, then the integrated or consent model. So before planning any study, we were asked patients and healthcare providers, What are you doing today? On in particular, we will ask patients what treatment you are getting on. Also, we ask in our survey what trial endpoint are the most important to you? It we're going to do, child. What do you want the endpoint to show? Because often we find what? How care providers. In all their wisdom, one is not always what patients want. The other advantage of doing these studies is the survey It it get publications for our trainees on more data for playing for grants, the traditional consent process. This is supposed to be me talking to the patient. I leave the room than the research stuff. Come and talk to the patient, often with some delay. The patient is then faced with a very long consent form that then allows various blood tests and scans to be performed on. Then the patients back with me to talk about the trial again. It is not surprising that every step leads to lost a patient interest in trials on also significant cost for actually during studies. So the integrated consent model is what we've seen developing, based on UK data on de sensually at me, talking to the patient, a one page consent template, all concern on vandalization there and then in the clinic. So, for example, one of the child's we did and Shaun, we'll talk about this more in a few minutes time. There's a drug that we give us an injection to boost the white cells. During chemotherapy. We can give 57 or 10 days of injections, all of them of usable. All of them are standard of care. We want to do a trial comparing them. Are you interested? That is integrated consent. I then will dictate that these injections are given for reductions. Risk of infections. 57 days. A 10 day two standard I've given the patient the concern template on randomization had been done, and there and then this is very different than the 20 page consent model that it's often used in other types of child I think men, if you and your gyn to a very like this is exactly this or thing that we can do in our clinic and at the bad side with the up that we use. We have very few eligibility criteria because we want this population to represent really patient that we see that there's a brief checklist on. Once we've entered the patient's name and hospital number, it will email us in the clinic when the patient do bark to capture data. So it's very practical. I thought, What we actually do in real world practice The other thing is we do not being patient back for child defined visits. Visits are order in Our spoke our standard of care with now involved coach to 3000 patients into these react trials. With the surgical ordinance, I put those two the left hand side, but we've done supportive care, hard given trial device trials, imaging study, hard palate to care studies, the surgical trial that we've done the first ones while looking at two different types of mash for breast reconstruction On, we have a large study running at the moment of preparing patients. The colorectal studies what is the best form of preparation of the patient to the surgery. Is it MCG out mechanical bowel preparation? Is it all about it? Simple practical questions. So what are my conclusions? I would say to everybody, Ask questions in the clinic. Dogma is just after dark. Ask yourself, Why do I do such and such for this intervention? Why do my colleagues do something else? Who's right? Who's wrong? You probably all right. React is not perfect. We still need phase 12 and three child. With any clinical trials, the more you asked how catheterize to do the last day will do so. Asking for more Get you less. This is important for your message in your child design. We use integrated all consent for comparing standard of care options. This is not or consent for phase 12, child. This is where we have multiple standard to care. Covert 19 has made all of our lives challenging, but I'd like to say thought of potentially exciting because we have to design our child's. Now, based on the fact the patient have been becoming less and less to the clinic, we need to be able to consent and collect data by telephone on email. With that, that's my talking pleated, and we'll go back to marry. And who will introduce the next speaker? Excellent. And that's already given as a lot of food for thought. And the idea of being able to randomize in the clinic setting is fantastic, and I'm sure that's going to generate a lot of questions. Um, as we go and three the session. So let's move on. And I thought, We'll do all the talk together because they're interrelated on be it'll be important. But some talks all these other questions for us. So the second speaker tonight is Gregory Pond on, and he is the by statistician specializing in cancer research. He's on associate officer in the Department of Oncology at McMaster University in Hamilton area and a member off the Interior Clinical Oncology Group. He's also on investigator of Theis Car Payment Cancer Research Institute, an affiliation with McMaster and his main research interests as the statistical analysis of cancer clinical trials with the aim of translating scientific discoveries into the clinic. So Greg's talk tonight is the title is statistical considerations. My end points and sample fives are so important So it's over to you, Greg. Thank you. Um um So thinking, married for the introduction. So the title of my talk, the importance of endpoints and sample size effects Um, I think the importance I think most people here would be readily understand the importance of them. So I changed it a little bit and want to talk about how different end points for a very standard or is potentially a simple clinical trial scenario. Congrat really affect the sample size in the interpretation. So I'm going to just briefly go through my disclosures and what I am going to do and talk for the next 10 minutes just present you with the very a relatively straightforward scenario. And then, as I said, discuss how the different outcomes and in the sample size implications as well as the interpretations of the results on bacon, be make different inferences based on what outcomes you end up choosing. So the scenario that I chose is an actual trial that I was working on a little while ago. And so we're looking at Gleason Grade two, which is formally known as Precinct Gleason three plus four prostate cancer patients. This is a population, generally elderly men. There's often some mild comorbidities associated with these men and at least in Canada, from my colleagues who have been told the standard of care is one of two options, the first one is an active surveillance. Basically, there's no really therapeutic benefit at the time. They just all of them up in, and we'll have potentially repeat biopsies while the PS A, um again, because the men are generally elderly in this population, a lot of these patients will die, not a prostate cancer. So active surveillance is believed to be a good potential option for these patients. The other standard of care option is radical radiotherapy, a radical prostatectomy. Now there's a definite therapeutic benefit, but there's a lot of side effects greatly affects the patient. Quality of life is kind of on the opposite. So while there's definitely an improvement in terms of the prostate cancer in the efficacy related to the prostate cancer, it affects the other avenues of the patients in terms of quality of life. So there's these two options that we presently have now. In this particular scenario, there's a new treatment called focal ablation therapy, which has been proposed for investigation. Now if this was successful, the investigator, the lead investigators, wanted to apply this for regulatory approval, whether through Health Canada in Canada, the FDA in the US and so they used the FDA guidance and in looking to see at what they needed for regulatory approval. And they're they're basically coming to the conclusion that you need to do that demonstrate clinical birth. That's what the FDA guidances see. And so the question really comes out to How do you define clinical benefit, particularly in this population? So the easiest and I should say, the easiest. The most objective measure in terms of clinical benefit is overall survival. There's a definite defined endpoint. Everybody eventually will die. So there's Ah find endpoint for every single patient. But again in this particular population, the prostate cancer. Because it doesn't necessarily kill a lot of the patients, they will die of other things. So the 10 year overall survival, even in the active surveillance arm on given one particular reference is very high, potentially 92%. So it's just to see a a small improvement over the active surveillance arm say, from 92% to 94%. You could have a very large study of over 16,000 patients, which are required, and it can take a very long time. Get five years plus follow up, plus time to prepare and and develop the study. This could be a +789 year study. And for the study investigators, they needed something much quicker than this because again they're looking for to get FDA now Canada approval within a matter of a couple of years. So this was something that, although is a good up, good long term option, is really not beneficial for what the investigators were looking for it this time. Now there's other similar types of endpoints survival type endpoints that we're also being need a vote by the investigators at different times, um, potential options that you could look at so prostate cancer specific mortality. Now this does have fewer events than overall mortality, but is actually specific because again, overall survival, a lot of these patients do die of other things. Prostate cancer specific. We know that it's it's related directly to the prostate cancer, so if you can, if the treatment was effective, you would greatly reduced the prostate cancer mortality. Of course, some of the difficulties are How do you actually confirm the cause of death in most of the men again? Um, if with many of these elderly men, when they die and do not undergo an autopsy and the cause of death as listed on, um, the death certificate is sometimes not thoroughly thought through, I would say, um and just, you know, they have cancer. These rate it down. What he died of cancer or something like that. So it's not necessarily prostate cancer specific mortality. It's kind of a difficult thing to measure. Another possible outcome that people way talked about was time to progression, while in this case, if you have a non prostate cancer death, it's a competing risk. But again, this relies on having a good cause of death associated with the death certificate, which doesn't always happen. In addition to that, in this particular population, there's some questions on how do you determine progressive disease for many of these men after the undergo a biopsy once they tend not to want to go under biopsy a second time so you can't use a biopsy as a Z, a measure of progressive disease. Um, some of the other outcomes that they use presently for progressive disease may not be absolutely perfectly objective. Like cross, um, ps a progression or something along those lines. So the determination of what is progression may also be very difficult. So again, these are all of the different considerations that you have to consider now, going back to the actual scenario that we're talking. What are some of the inferences that you can get when you start thinking about what are what are you comparing and what What's the vocal ablation versus active surveillance or a radical treatment? Well, theoretically, focal ablation should be better than active surveillance because you're giving some sort of treat regardless of the input. So just having some measure of improvement in terms of efficacy may not be enough, because again, a lot of these patients don't go on radical treatment right now because of the side effects. So part of the question that you have to answer is, um, is the quality of life of these patients appropriate? But then that goes against the radical prostatectomy arm radical radiotherapy arm. When Focalin is a some, um, a less harsh treatment compared to these radical treatments. So you should have improved quality of life again. If you're looking at these long term outcomes thinking about the non prostate cancer deaths, that happens a lot of these particular patients. So how presented is overall survival or some of these other markers of efficacy? I'm given that many of these patients die of other things. Um, And as I mentioned, when I when you compare the ablation versus of radical treatment, if you're looking at the quality life, um, you you were expecting to see an efficacy are quality of life improvement or a toxicity improvement. And so what you really want to look at if you're looking at these efficacy, um, comes is a noninferiority question, as opposed to a superiority question. If you're comparing focal ablation with active surveillance so again, depending on what the Comparator is in, this particular setting may change the design from a superiority to a noninferiority question or vice versa. Now, just to give you an idea, another possible thing that we talked about was time to more aggressive treatment, which was kind of a short term, um, potentially surrogate outcome compared to overall survival or time to progression that sort of thing. And so the thought was that a delayed two more aggressive treatment gives an improvement in the patient all the life again, just in terms of how the L come May may change the sample size. If we use seven year treatment free survival of 42% which I found quoted in a different um publication. If you had were looking for improvement of wrote 10% from 42% to 52% your sample size would change to about 1500 patients again, depending on the length of all that you have. So you can see how a small change in your Elkem may have a big change in your sample size calculations. But then again, going into the inferences here. By definition, focal ablation will be better than any of the radical treatments because all the radical treatments have a more aggressive treatment on day zero. That's by definition, so they're automatically getting it, so you can't really have a comparison in that sense, so you can only compare focal ablation with active surveillance. And so the question is, is it delaying? Um, hum time, too radical treatment time to more aggressive treatment compared to active surveillance. But you have to really find a very strong tree benefit in this particular case because you are giving some sort of treatment. And but, you know, theoretically, we should expect some treatment is going to have an increased cost. You're going to have some side effects, even if they're small or or minor and compared to an active surveillance. So you need to see a very strong, superior difference again. One of the things in terms of a treatment choice. One of the problems with this is the patients actually have a choice, whether or not they're going to wonder, go this future treatment And a lot of the patients will offer the treatment just because they have a fear of the cancer returning or the progressive disease. Some patients will just refuse the treatment because they don't like the side effects. So there's other options that are affecting the outcome, not just the efficacy of the treatment itself. The patient has a great decision to make on a great influence on whether or not um the income is going to occur, so it may not necessarily be the best outcome again. So another possibility that we looked at is, let's just look at the toxicity. How does the Focalin blazing compared to the other treatments in terms of the toxicities toxicity? It's so I gave you of the toxicity rains here, the pain on a It's great one Grade two or grade three. But now you have to start thinking about If you're planning into a trial based on the toxicity rates, which one is the important? Is it any toxicity grade one or above? Or doesn't have to be a Grade three toxicity? Are you only focused on G U toxicities? Or do you need to have GI or G I toxicities? You have to get all these questions and decide what is actually clinically important to the patient in terms of the outcomes. But again, just to give you an idea, how does this change the outcomes? If you were looking at saying great two and above gee, you toxicities, the rate was 54% so if you want to reduce it to 44% you only need about 800 patients. Or, if you want to look at the Grade three toxicities, your your sample size goes down from from to about 1374 patients. Based on reducing the same are the toxicity rate from 9% down to 5% now. Alternatively, if you're lonely looking at toxicity, protect, perhaps you don't need a randomized trial. You only need a single arm try again. This could vastly affect the sample size that you're looking at using the same estimates in terms of the toxicity rates. You may only need 200 patients or 300 patients, depending on the toxicity rate that you're actually interested. So again you could have vast difference is in terms of what this sample size would be, depending on what you're actually looking at. Now, toxicity isn't the only measure of quality of life or how things were going. So quality of life. You have a number of choices that you could also alternatively use and not just a straight toxicity. Really. Um, but again, you have to decide what's the quality of life measured that a you to use the S F 36 epic. There's all kinds of different, um, call your life question your surveys that are out there, and they all have actually a bunch of subsections, which maybe it's one particular subsection that you're interested in, not necessarily the entire global quality life score. In addition, you have to look at Are you interested in the absolute score of the change in the score? Are you looking at adjusted scores based on other factors that may occur? Are you looking at a time complete? And what I mean by that is, what's the time after treatment that's really important. Is it at three months after treatment? Is it six months is a two years, or use it across the entire spectrum of time after treatment that you're really interested in again, depending on the score that you can use? Um, you may have vast difference is So, for instance, there's one paper here which showed that the epic urinary function had a 15 point change compared to cost a prostatectomy where, as active surveillance basically had no changes, you would kind of expect. But if you looked at those particular numbers, the sample size that you would need for using this particular score as a measure quality life, you'd only need about 30 patients so again, vastly different than 16,000 that you need for another overall survival. But again, you have to consider what are the outcomes that you're going to order the inferences you're going to have at the end of the result of the end of your study. Again, active surveillance should theoretically have 0% toxicity because you're not actively treating the patient at that particular, you're just following the patient over time. It should also have no change to call the life. So, by definition, active surveillance should be better than Focalin leash. So if you want to use a quality life measure, maybe you really only can compare to the radical treatments. Radical prostatectomy, by definition or theoretically, focal a shin should have fewer call a better quality of life in a better treatment than radical treatment. But again, if you think back well, the active surveillance was 0% toxicity. You know, change quality Life is going to have better outcomes compared to the radical treatment. So fewer treatments we'll just give you less adverse events, and therefore it doesn't necessarily mean that it's better. It's just it's giving you a different okay, and again, as I said, what's really important is that the quality of life now, or is it called the of life five years from now, or 10 years from now? Maybe when you do need treatment if you have active surveillance. So these are all questions that you would have to consider. So ultimately, um, the sample size, whatever trial designed that you end up using single arm randomized to arm trial in fear, superiority designed, non inferior or design, they can vary widely in the same scenario in the same city, depending on the outcomes in the objectives that you're really interested. Um, there is no one size fits all approach and really the best thing that everybody should do. All investigators is really think through all the possible results and all the interpretations or inferences, not necessarily from your point of view, but from the point of view of a not a mist and, from the point of view, a pessimist in terms of how it would be interpreted and what could be the results as well. At the end of the state, Once you start really thinking through all these alternatives, then you can come up with what is the best design in the best outcome for my particular question that I'm going to ask, Um, thank you. And I'm going to pass it back, uh, to Doctor Campbell at this point for the next week. Super. That's really interesting that trade off between which I come, you choose and what the sample sizes. It's, um what if the big issues that we have to think about when we're designing of trials and I think that will be is quite a number of questions when we come to the question, but at the end? So let's move all men s. So next up his doctor, Shannon he on. But she's a medical oncology fellow and assistant professor of medicine of University of Water. She's a clinician investigator. And at the Auto Possible Research Institute on a Principle investigator for the React 12, there are specialist interests are breast cancer and uncle fertility and the management of safe side effects and toxicities related cancer treatments. So Sharon is going to speak tonight, and the title of her talk is the react Red trial. A practical example off a pragmatic trial. So over to you, Sharon, Thank you so much. Good evening, everyone. Hopefully you can all hear me Eso. My name is Sharon McGee, and I'm a medical oncologist at the Ottawa Hospital Cancer Center in Canada. On I'd like to talk to you this evening and to give a practical example off what a pragmatic clinical trial looks like. This is a trial that I am principal investigator for, and it's looking at the timing off radiation on endocrine therapy in patients with really stage breast cancer. So again, just hoping to give you a brief overview on a quick presentation today to really highlight some of the key steps in developing and a pragmatic clinical trial to give you an idea of what's involved and also then finishing with a few points on on why I and and we think the react pragmatic and clinical trial program works and why you should certainly, you know, consider it, um, establishing a similar set up in your in your own institution. So the first step in designing a clinical, pragmatic clinical trial is to identify your research question on Is Mark alluded to before these questions are really all around you, and if you take a look around in your patient clinics or in the theater, you know there are several things that we do on a day to day basis, that we really don't have a lot of evidence to guide us on as a non college ist and particularly focusing on breast cancer, which is one of the most common or the most common cancer diagnosed in women. The majority of my pact us and will present with early stage disease. And that is usually, ah, hormone receptive disease or basically cancer breast cancer that's growing in response to Easter gyn on your progesterone. The main part of treatment for these patients is going to be endocrine therapy, so things like tamoxifen as well as radiation. Although we've been using these treatments really for several years now, we actually don't have a definitive answer as to what the optimal timing of these therapies is on. As such. Practising in the clinic is highly variable, with some patients starting the treatments alongside one another on other patients may be completing radiation first and then going on to start their endocrine therapy. So the question arose as to, you know, is there an optimal sequence or timing for these therapies? Should they be given concurrently or should it be something that's done sequentially. The second step that we follow them is to demonstrate clinical equity goes on. That essentially means that you're showing or demonstrating. That's your clinical question, you know, is something that is as yet on answered by the available data on. This is a really important part in the process because it allows you to review the data to help drive the future design of your clinical trial. And it's also a step towards starting to generate research papers and publications, which we all know isn't is important in academic medicine towards our research study and on this clinical trial we conducted on published a systematic review on the available days A regarding the timing off radiation on Under Concerta pee in early stage breast cancer. From this, we weren't able to answer whether there was an optimal treatment sequence on. It was also unclear if there was any impact on the efficacy or toxicity of treatments. With the timing, we also conducted unpublished a survey off Canadian physicians that likewise demonstrated that there was highly variable practice with respect to the timing of these treatments on, we identified that some of the concerns that physicians had regarding the timing of these therapies were related to the potential for toxicity with endocrine therapy or radiation impacts, with compliance to endocrine therapy, as well as potential impacts on the effectiveness of efficacy of radiotherapy. In this survey, we were also able to determine that physicians were very much interested in conducting or designing a pragmatic clinical trial to try and answer this question on. We were also able to identify from them watch, key or meaningful, and points would be for them. Step three in the process is then, you know, to design your trial on the big thing. I will always say, here and on which we kind of try to adhere to a smart has said, is to keep it simple and keys to success that we have found in this regard and from very various clinical private. A clinical studies that we've developed here in Ottawa have the number one toe have a defined clinical question on on hypothesis. It's also important to make sure that your patient population is focused. The endpoints you choose fear study are also very important, and they should be simple and clinically relevant, and they should also have the ability to give you results on both the short and long term basis in oncology were often driven by end points like overall survival on disease free survival. However, these can take many, many years for final results to come. So if you do want to have some results before potentially you're retiring, it's important to have a ah more shorter term and point. It's also critical is part of your team that you have someone with, you know, a sound background. In statistics. Such a doctor pond, who really is is essential and guiding us, making sure that are, are statistical plans air sound on that? Whatever our endpoint is that we're choosing, we have the most appropriate sample size now. This can always change is the trial is being conducted, but these are key points at the outset. When you're designing your study, you should always also make sure that the process for data collection is simple. This more simpler. It is for the patient, for the physician, for your clinical research assistants, the more successful your study is going to be, and having lots of people review the study is a protocol is very important, too, because it make sure that, you know the conditions are on board, and they feel that the clinical question is relevant and meaningful. The statistics, this sound, the cancer, um, your and see our Azor Research Administration style feel that this is something buck a confuse, a bleed do within the clinic. And it's also helpful to know your ethics board within your institution, you know, so to help make sure that you don't get the bottle neck of getting your trials true through approval and open. In the case of our studies, the React Rhett, which was looking at the timing of endocrine and radiation therapy in early stage breast cancer, are central hypothesis. Here was that concurrent endocrine and radiation therapy would be non inferior to sequential treatment. The primary outcome we chose was endocrine therapy toxicity. This essentially came from this survey that we conducted off Canadian physicians, as this was one of their key concerns on one of the endpoints that they identified as being most meaningful to them. It's also, um, or shorter term and point. That readout could be available within the course of a few months to a year on there also exists a validated and established quality of life assessment score that can be used to assess this. Additional secondary outcomes that we, um, chose again, guided by the results from our survey of physicians were related to radiation toxicity, endocrine therapy, compliance, overall quality of life as well as cost effectiveness. Step for in the approach is then, you know, once your trial has been designed and hopefully gotten through approval, that you're able to approach patients, and this is really the the fun part of the process. He's to success here are making sure that you have physicians who are aware of your study and who are engaged and open to approaching patients on what really helps here is that, you know, again this factor of making things simple for everybody. Whilst we did say that you need to to find population, it's important that that's a relatively large population, so that you have a good chance of accruing patients quickly to the study. As Mark mentioned, you know, really central to the success of these problematic clinical trials is this verbal consent model, where physicians can approach patients right there in the clinic. Explain the study, obtained verbal consent quickly randomized them, using a Web based tool to have answer there and then as to which arm they're going to be participating in. I'm just simply documenting that in the medical record for us. Essentially, it's in their clinic note for the day so that they can begin, you know, participating in the trial. Keeping it simple and easy for everyone is really a key to success throughout this whole process. Well, for this study, you know, again, we had a defined but broad population with early stage breast cancer patients having hormone responsive disease who were who were planned to receive treatment with the endocrine on radiation. They had to be over the age of 18 and able to give herbal consent. There were really very few exclusion criteria, so this is quite a large population that we see in the clinic. Those eligible patients were able to be approached by either their medical, a radiation oncologist, as I said, obtaining sort of verbal consent there and then, as well as randomizing there and then within the clinic, so that patients can begin the process of collecting data for the endpoints, which for us the primary one being the impact of the timing of treatment on toxicities with respect to endocrine therapy. Particularly, this is just a recent, um, slide showing the patient too cruel to this trial. So arts, our sample size here waas 218. The study opened in September of last year, and the latest figures from this month have shown. Actually, it's an update. As of today, rather than 214 we have 215 out of the goal of 218 patients recruited. So this really represents one of our most successful and fastest accruing problem other clinical trials. I'm primarily because it hits many of those factors that we reviewed before in terms of being a large but defined patient population. Physicians who are engaged both within our medical oncology department as well as our radiation oncology department. And I'd also like to point out that you know this a cruel at this excellent, a cruel, really continued right throughout the covert pandemic, which we and you have seen has really changed clinical practice that the rapid switch to virtual care and we were able to continue offering and enrolling paper patients in clinical trials like this throughout that time and you know, just by making some some obviously different changes and compromises, the biggest impact for us was really the ability to assess radiation toxicity, which is something that's performed by the physician during a physical exam. However, as that was a really a secondary endpoint for us, we felt that it was possible to continue with the trial stuff. Five or the final step is really, you know, to review the days on plan ahead on this is really critical to help you identify problems early on, so we would definitely recommend and from our experience, you know, reviewing the days and cleaning it as the trial proceeds is really important. Having regular oversight meeting so that you can identify and act upon issues. I'm really sitting down or early to try and plan your research outputs, be they abstracts, publications or talks. It's often not until you sit down with the days own. Try to see what the story may be that you identify gaps, which you can then really make changes or amendments to your protocol to address those on really again ensure the success and relevance of your study on the final point, I will say it's it's always good to start thinking about what your next pragmatic clinical trial. It's so finally, I just wanted to mention a couple of points on why we think or I certainly feel the reactor prognostic Clinical trial works. Firstly, this is a modality of research that is really accessible to everyone. Trainees, fellows early in later career physician. So the last speaker is Professor shown three week and he is professor off health services research in the health services, researching it of the university off Aberdeen. This research interests are mainly inefficient and pragmatic while designs, and he's leading the trial for it initiative that seems to be much more systematic about how we generate and use research, evidence, making trial design, conduct analysis and reporting decisions. And tonight shown is going to talk and the title of his talkers. Why not thinking about Hae we do our trials leads to research waste I'm gonna handle, which is sure. Thank you. So hopefully you dizzy straightaway that's disappeared were eight. Okay, hopefully you can see that looks for me. If it doesn't work, I'm sure the technical people will tell me. So I'm gonna give a big picture. Open you off. Why not? Thinking about how you're asleep. Research waste on their first thing I'm going to do. Let's talk about a bowl of paper. So many of you will recognize the person picture here. This is a drug called manana. A little giant of medical statistics. You died a few years ago, 25 years ago. He wrote the paper that we're looking at right now with the scandal of poor medical research where he talked about it. Okay, problems that are routine in much medical research. On the only regret that drug ever had about this paper was that he chose the word poor rather than bad. I wished it being called the scandal off bad medical research. And we're talking about efficient trials today. Please. Bad medical research. Bad trials are the ultimate in poor efficiency. It not article was this text. This is the front page off a notebook. In fact, that takes to give two new research students a Zar age memoir. So what we ought to be about, we do not need endlessly more four research. I'm going to show you some examples that way, we don't need more and more and more what we read. Dooney is better research on research them for the right reasons. And we've heard from all of our speak so far about the sorts of things that make the right reasons, meaning that research is going to be meaningful to those individuals who are trying to make decisions about but the health care. Now, this little task I'm going to demonstrated to, you know, if something everybody can do in the privacy off the Oh no, you could do this at any time. I've done it several times and that is Let's look at the Cochran Live. Let's go online. Look at a couple library on. Look at the reviews that are flagged under highlights and reviews, and I did this last week it before here or five. I don't know anything about these reviews, just the ones that were high right, And I looked in this this week, and the purpose of this exercise that YouTube could do if you wanted to, is let's look at each of these reviews on Look at one part of it, and that is the risk of by its assessment for the studies, the trials that are included in the systematic reviews. So these are all big, systematic abuse. They searched the global literature and pulled out trials in quite diverse areas of restart. So you have something that's out there looking at diet preterm infants. We have a surgical one there, the second the right of the bomb. We have something that mechanically assisted walking for people that are Children with cerebral palsy and the task. It's let's look at those risk of bio success mints on see what they throw up. So that risk of biased assessment in those in the reviews, for each of probably is telling us something about Should we believe these results? Have they done things which suggest that there are real doubt as to whether we should believe these results? Whatever those results out on, I'm going to break it down into essentially a couple of categories, which is good about good means. No, there are no good reasons to death about the way they put together that study, so that would be low risk by some bad means. High risk of by tell is plenty of reasons to be concerned, so I did this last week, and this is what I found for the first one in that systematic view, seven studies, one of them good, three of them bad and the other ones are uncertain, so we're not quite sure. Write a lot of them. So the things to look at, what we really want to see, numbers of gifted. They look very similar to the total number of studies. And if you put your I down there, what you'll see is actually, there are quite a lot of students, and they're very few that good. In other words, they're very few that come out as being at low risk of bias. There are plenty of reasons to doubt thumb. And if you look at the number of studies and compare that to the number of bad or the number of good, what you quickly conclude is that their their entire bodies of evidence that have bean, uh, elected linked a particular systematic review question where the entire body or almost the entire body, could be classified as a rubbish. Highly inefficient, clearly. But just like that, it doesn't matter when you do this, I've done it before. You do it next week, you'll get a second result. It doesn't matter. This should be plea shockers. Uh, hundreds and hundreds, hundreds and hundreds of miles come out of this sort of setting poor that bothers. So the billion dollar question is used. Drugs work. Why is so much health research bad and obviously focusing here on trials? So dog himself in 1994 actually go of a bigot? Answer. There's a lot of research that it done. Biggest restrictions. Feel compelled to do something to do tomorrow? Search on. They do not have the skills to do it. Nobody stops them on. We're forgetting proliferation. Off Search that answers nobody's questions, is weak in many, many ways and represents gold plated researched waste. If you remember the last slide, there was a bit of the bottom for the cerebral palsy, where the entire body of evidence it's expressed your viewers by risk of bias. You just didn't have confidence in those results. Nobody stop them doing those trials, and why it that why did they do it? That body of research, not just a single study, is not this one bad apple, everything in the entire area. So what I want to do is give a few what I think our top tips. And actually this is merely summarizing many of the point that we've already heard in this session already from my previous history speakers. So there's no rocket science. This is Some arising repackaging much of what we've already have it. The first thing is, do something that's important. Find some priority question We've heard already over the importance of the question. There are organizations and groups to do this well, have the resources that with the skills and go out there. They ask people who are the people who were trying to support on they stick. These are the important distance guidelines groups, for example. Often I like gaps. They move it up. Research evidence for a particular clinical guideline highlight gap in there evidence. We need to know something about Vest because that makes it difficult for us to make recommendations. Vickie's on the source of things that we should be doing with our research. It's an important question on it, a priority setting process that has been done well. So if if these exist well, is there a great place to do an important piece of work rather than scratching back, uh, once in the afternoon in your own home, thinking of something that might be quite need These give you a head start with your question on. Of course, here in the UK in excellence, you have Elvis. But does a lot of the Clobex or their commission researches these known important questions top way, avoiding wasting everybody's time, Know what's been done already? I think it was starring. He mentioned this systematic reviews. If you're working up any research program, protests protocol, particular trial, if you have not talked about systematic review, we really need to look on, start thinking about systematic reviews. What does that body of evidence see that Cochran. Something where I was talking about good and bad studies. You could go to any of those system. Ask reviews that there will be a very big pointers in that toe. How your city cannot be as bad as all of the previous studies that are in that body of evidence. Don't repeat what they did because they rub it a lot of them. So you need to know what the systematic reviews this big so they can build on what we know and this time generate some evidence that is really, really useful on funders again, the HIV in the UK are increasingly thinking about Well, what is this two math you have you ever since to back up rational for your research question is not there. We should really be questioning. How do we know that this is not research waste? This one designed for you is definitely be mentioned. Who are you trying to help with your picture of research and what did that? I need not know what you need or what do you think is neat but might be quite interesting. But what does that mean? What about a N port? That they the outcomes answer their questions? Make sure you know what your intended. Use it on dying your study around them so they don't have to throw it away later on has been completely irrelevant. So he uses a writer. This protest is important. Let's not think that we know how to do a fabulously efficient trial, even if the question is really spot about this person, Might Clark. He came up with a term called SWAT Studies within a trial on It's a great way off vibrating trial process alternatives so we could have one recruitment method versus another recruitment method. What's best way of recruiting our particular trial population? Well, if we're not certain that's doing a valuation, so that's what's what's her about. Ask what we doing this make It is gripping. Is that why we're doing this swatter and able us to start providing evidence for trial process? Decisions have to recruit how to retain out, collector date out, present our data. Those sorts of things think about process that can avoid an efficiency on this point has definitely be made. Research is a TV in exercise. This is not something which one individual is gonna do alone, or even if it's a couple where we will have our own expertise. But few of us are experts and everything, and members of the public and patients with particular conditions are absolutely experts in that we need their help. They should be there from the very beginning and throughout on find ah, statistician, find Greg become friends on involve them from start to finish so that you don't at the end, try and present your half baked set of data to a statistician and and ask more. Or that too Mad Cup, some great piece of works too late shot. So that's that station needs to be there from the very beginning. So these sorts of things, I think, take cute. What do I is into encourages to do so? Think about what makes our research that we don't need more, more, more, better respects that we don't repeat what we saw on that summary on researched on the right reasons, thinking about our users and bring that it's a priority question knowing that symptomatic of uses put a systematic reviews appointments in that direction. Onda, I'll do that with you and happened back to marry. Thank you. Excellent. Thank you Shown lots of people for thought there. So And we've had four excellent talks that I'm sure have generated lots off questions for you. And I'm delighted to see there are lots of questions coming in in the chat bar. So I think they're going to start the question and answer session night. Um, and I think that seems developing years, so I'm going to start. So the first ones were obviously stimulated by marks. Talk started by or a look incent. So there's the question here from Natalie Blank. Cool. I was asking, Do you ever run into difficulties using shorter consent forms with the ethics committee? So I think we'll start the mark. And do you want to take that one first? No, thanks. Not leave for the question. Um, I think the inner surgeon in me is based last one. Yep. Religion, personality on Dad. Been able to keep going at the r e B about the So if you're the sort of person is gonna give up at the first rejection, then pragmatic trials are not for you. Yes, we have faced motor challenges with the six board. Um, as I said in my tour, it is not a lack of consent and what I often say These meeting to a 20 of courage form that being signed with not consent. It's just a 20 page form that be in scientific means. Nothing on the beauty are the integrated concern Post is that it's part of your normal conversation with patients. It should be that we're offering multiple, different standard care on accepting that we don't know what's best now. It's very difficult to tradition. We always think we know what's best reality is. We often don't on Brebes have been extremely receptive on what I normally do it. Every time I submit a protocol we send, as in with the journal papers on whole consent process is with the application because our our baby has regular, um, turnover of staff. And if you would like having the same conversation over and over again, So, yes, uh, it's a challenge, but don't give up. Excellent. Thanks for that response, Mark. Um, and and so I have another question. Probably. State back to you. Mark, This is from Alison Fielding. Um, So do you have any issues with patient drop? Oh, after the clinic, when the patients go home and then they consult with family, Do you then find that patients will drop by having already signed up and to the trial in clinic again over to you, Mark for the response to that one. Thanks, Alice. In another great question. So when I'm auditing somebody else's trial, if I find that there are no dropouts, it kind of raises a red flag to me that stopped into going wrong. So we actually published. This is part of our consort diagram will say how many people live and, um, I've started. He agreed in our me What? Um, the acceptance rate of the randomization arm is not that great at you. Incredibly low, I'm pleased to say, but yes, we collect it on. But it's not that again. This is a voiding concerned. It's having those honest conversations with patient on. They don't get vandalized until they've made that decision. So if a patient need to go home and discuss it with the family, absolutely fine. Excellent. Thanks for that response so that there's no a couple of questions about It comes. So I think Greg will be in the firing line for these ones, and it's It's a lymph. There's the normal in In in statistics old sample size samba bitches. The dance between the clinician and the statistician and the staff station saves you need 2000 patients and conditions is I can only get away with 200 please lenders, this sort of slide dance around where we get to a common decision point between the two extremes. But the question probably for a great big blind from him caressing. Do you have any advantage, advice and composite I'd comes because clearly they may be more efficient and they help you with a sample size calculations. But do you have any advice on that on on how to construct and then test complicit measure to inform that sample size Greg a response from you? Sure. I mean, complicit and points They can be a very good thing, but they can also be problematic. So one of the, uh, with the big concerns with, um constant points is if each end point measures slightly, the, uh, the direction of the outcomes are different. So you can have one, which is wildly in favor, say the experimental arm one, which is moderately in favor of the control arm. When you combine them, you're gonna get a neutral result and a negative result as results of one of the key things when you're talking to compensate and points is you really have to understand or have to believe that the outcome is or the direction of outcome is going to be the same for both outcomes that make up for what the all the outcomes that make up the complicit endpoint. Um, you see that I've seen that a number of times, I don't do cardiovascular, but I've seen it in cardiovascular all the time. They because the event rate is so small that the the make a copy of that point about four or five different outcomes. And unfortunately, sometimes one goes in one direction and one goes the other and it ends up in a neutral. So you have to really understand that the other thing is you really just getting into the understanding of it. You really have to make sure that one outcome doesn't overwhelm the other one. So if if, um say one outcome happens in 15% of patients and you got two other outcomes with chaps, it happens in one or 2% of patients. There's not really much benefit in having a cop said Don't come. In that case, you can really power it based on the single large outcome that again, hopefully that's clinically important. But you have to really focus on that. So you really love it just comes down to understanding what is really important and what's going to happen. The direction of the outcomes Um Pa with standard for the experimental arm, making sure that they're consistent on both, thinking it through again at the end. One of the things you really have to do is think through at the end of the study. If I get a confident point, is that important? Or is it more important than one of the outcomes? Was more really what's going to be the driver behind everything? Should I focus on that? One will come out supposed to have a copy. Thanks. Excellent advice there and just keeping on a theme. Off it comes. I think we'll probably ask Sharon this one. So the Natalie Blanco has said, You know, given the tensions between different possible primary I comes, how do you optimally select between plainly it comes. Which one do you use in the end of high? Do you reach that decision, Sharon, Do you want to comment? Yeah, thanks so much for the question. It's a great question, really. Our approach in the pragmatic trial program is to really involve the people who were hoping this this research will impact, which is the physicians and the patients. So, you know, from our, you know, data review, we might have certain in point and points or outcomes that we think are interesting or relevant to look at on. Then usually our practice, you know, is to go to the physicians and also ask patients what is meaningful to them, what is going to change their mind or their approach about how things are done on Do you know, obviously taking into consideration whether it's some things that a feasible that the weed out would be relatively quick or impossible to do? We would usually focus on that. So that's our usual approach to to, um, to picking an endpoint. You know, seeing all the relevant ones. I'm putting it to the patients and physicians who are really going to be using your data. Yes, I just had something just follows abusive. But Sharon is just That's okay. So in exactly the same way you just probed Turn. Uh, we were looking at a bunch of trials published. I think we're in the New England Journal. The Lantus that we got all of that becomes that they described the primaries and secondaries, mix them up on, then presented them to a panel off a healthcare professionals and patients or patient representatives to ask that to rank them so I didn't know what was the primary. What was the second year? They asked them if you're making a decision about this treatment, What was the you put them in on? Interesting. It was 22 trials involved, and in only six of them was the prime of the number one outcome. Sense of the whole team had not made the right choice. Uh, in most cases, the primary was highly wrecked in the top by in most of those troubles. Kind of the number would, you know, appeared top five. But we're a couple at that 22 where the primary was the where to be seen. So it is in is very important. You discuss with your future. You just your gums because you messed royally. You know I need that chair. It'll possible? Clearly, we want to avoid that situation. Excellent. Thank you for that. Um, sure on. I suppose they're just building on that team. I was gonna ask. Sure. And then And probably you, Shawn as well. Where does this sit? With core. Oh, come set. So that happen? A number? Of course. I couldn't say it's developed for certain disease. 80 is where that piece of work has been done, Um, whereby the different stakeholders have played to address that the questions on the oak is that important? In what situation should one use the core right? Come say it as as the first port of coal unduly exist in the trials that you've been involved in Sharon Or is it very much the clinician that leads on this? The Shaina last year first, Yeah, I think it's more so. The clinician and the patients that lead, you know, the core outcomes, you know, are certainly important. And but sometimes that that might also might already have been addressed in a certain clinical question on that might not be watched. You know, you know, stopping physicians or patients from making a decision about treatment. So you know they are there, and we will offer they will often be included in what we present to patients and physicians as possible outcomes of interest. And if they, you know, reiterate you're select thumb was as of interest to them then, then definitely we would look at them or really were more driven by you know, what is what is helping physicians in the clinic and what is guiding patients. Excellent. So thanks for that, Sharon. So maybe, actually, I'll just pick up another question and I'm going to go to you. Sure, this time when? So this is from Kristen Hoffman on D, and she's been commenting on your talk and saying, What? What would you recommend as the most efficient and most important change that could be implemented to increase the quality of health research as a big question, find the methodologies endorsed well and statistician and possibly the same person. So in, um, my experience and I have Benadryl just a small, um, ventures, but not that decision. But if you look at many research proposals, where you on funding committee of colleagues looking at a a piece of work, it's It's possible often quite quickly, to find really substantial holes in it, and you look across the team and you find that there there are no individuals within that team. If you have expertise in method or stand or stats, there's lots of expertise in other areas, but not those areas on day that I think if there were there was much more acceptance that that methods, research method is and expertise on If you do not possess it yourself, you need to go out and find it, because without it you're in danger off doing more research that no one has the opposite one called Mom was saying so. Statisticians methods. People find that embrace thumb and involves them in all of your work. So we have our full team of expertise. Super thanks show a little pain. You later being a status down. I think you're finding friends for me. So that's coming. Oh, so another question of those turned back to to mark again for this one. So something Pathak helping like the court. If you ask for more, you will get less. And certainly in selling the UK and maybe in Canada, too. There's a lot of time spent filling in screening logs. So you log every possible patients whether they're eligible or not, where they ended up getting randomized. No on. But it takes a north a lot of time and detail and commitment. So the question would be what are your views on the collecting of details reading logs and and are the important. So Mark, you want to commend and and given opinion on this? Yeah, Um, absolutely. So let me just move to camera. So you see a pile of boxes there that Ah, much smaller than you were last week. This is dated husband kept for 25 years. As per our ethics. Onda what horrendous waste of time and money on. So we collect superfluous data all the time. Um, one of our ward of thumb when developing out of the act protocols is with every situation. The protocol must get shorter, very simple to stop having too many cooks in the kitchen and everybody adding their thing. Logs of fascinating. So one of our institutional qualities have bean that every potential position that might see a patient with that condition should be on the logs. We change that. We ask people stocks before we start to study. Are you Are you not going to approach patients for the study on If there's no reply, we don't put them on the logs. This irritates are people are essentially to a poor stuck in previous models. But I think the matador logical point of view is we collect far too much data of reading. I thought you might ask me about basic science because um, if everything was example, it, um that really hasn't added much. The clinical part is ah, yet you're unlikely to get a gold. Good. Greg won't treat me now on, you know, um, not a bit of basic science. And they're on you don't score you more points on. I know that I work in a lab. It it a lot easier to vital about your grant in the clinical ground. Because as conditions, we live in shades of grey. We used to the fact that nothing is precise. No end point. I could it apart and death. Um, so I'm going off topic here that it is a huge issue that I I find very interesting that that's the grants canes and ship another session for another day. Super excellent. Can I just had I just wanted that in a couple cards because we did. We did a study in our local clinical trials group a number of years ago, and we found something about 30% of the data that's collected is never used at all. Um, particularly if you look at baseline comorbidity ease a baseline medical history. These are things that are routinely collected and just rarely ever looked at after the fact. So these are areas where you can really shorten the protocol. Any protocol, If you're not really interested in it, you don't need to have to collect it of the labs is another area. Um, and one of the other things that we looked at is, um, this was in Canada a number of years ago, But for every data point collected week, we estimate it was about $50.50 Canadian dollars about 25 £30 per data collected point. And that's per patient. So if you have 1000 patients, that's, you know, you just multiply that data point by 1000 again it just explodes exponentially so you can see how it it increases cost, increase his workload. And there's not a lot of benefit. If if you're not gonna use it to really think through even the CR efs, what data points do I want to collect at the beginning of the trial? See, pair of um and so we're getting near the end of our session, so I'm going to finish over there. Really excellent question from Emily Kirk. Um and you know her commenters do? Do you think too much emphasis is made of getting your work for published for the sake of the CVS folder rather than getting the trial design right? So it it's answering the right question. So this one, I'm going to, um, ask shoulder to come in for us. Still probably ask you all pretty of use of that. Yes, I think so, yes. I think it's easy for me because I have no old to say yes. And the problem is is that the academic, um, career progression process has perverse incentives for publishing nonsense on a. Supposed to be spending plenty of time doing that really great design and get me a really quality piece of work on, then spending plenty of time doing other forms of dissemination that single acting acre and do it. Yes, I think I have a big job. I'm changing it. And it is much easier for people like me to say, Well, it's just above than it is for every career researchers because of the way academic career progression is setup, It's bad. Yeah, and I'm gonna ask Marcus Well, from the surgeons perspective on this, I hate to be thought of It's a surgeon. I said. Have you get in your inner surgeon or No, no, my personality disorder is very surgical in. It sort of might benefit you in the personal person that wants to get things done in those of magnificent traits in a surgeon. Absolutely. Research for the sake of doing research is extremely white to spread. Are funding on the UK is the same as Canada? Um, it's very, very difficult to get grants on. There's a disincentive performing teams big to include people are competing with each other on Do we know that on the grand panels, same old, same old tends to get the grant money. It's perverse incentives by great Don't know what we're going to do about it on day. I agree. Don't want you on the are the newer to death end of the live spectrum, um, have led to be concerned about. But I do. We published a load of trash. Okay, so I think it's probably time Teo wrap up the sessions that my learning points that I've taken from this is that you need to think about your child design from each point. You need to think clearly about what it comes matter which, once you're going to choose, How you gonna measure it, how you think your sample size needs to be too into you. Get that. Keep it simple, minimize all the surrounding, um, that trust, as we've been heard on, keep the surrounding excess April work simple. Keep it fundamentally focused on the trial question at hand. And if we do that, we will fulfill the Goldman's even to do less right research with better. And I think that's an excellent point on which to wrap up profession. So I just want to thank all their speaker's for an excellent set of presentations and excellent set of questions and answers. And I want to thank everybody who's participated in the women are you had a great questions coming in, keeping the chat fluing. I hope you've learned a lot from this and I'm sure are in our CM. Colleagues will be inviting you very much to sign up to the future Webinars on that are coming on the stream over the next week or so. So I'm hoping to see you there. I'm over the next couple of weeks of the next Webinar, so thank you very much for your contribution to night