#NephJC Chat
Tuesday January 29th, 2019 at 9 pm Eastern
Wednesday January 30th, 2019 at 9 pm Indian Standard Time (New!)
Wednesday January 30th, 2019 at 8 pm GMT, 12 noon Pacific
JASN January 2019, 30 (1) 13-22
Quantity and Reporting Quality of Kidney Research
Markos Kyriakos Tomidis Chatzimanouil, Louise Wilkens, and Hans-Joachim Anders
PMID : 30545982 Full text at JASN (free thanks to JASN till Feb 27th 2019!)
Introduction
Anyone who has attended a renal course/ series of lectures will undoubtedly have seen this slide:
Figure from Strippoli et al, JASN, 2004
It shows the number of RCTs over time with nephrology in last place, well behind cardiology, oncology and digestive disease. GI?! Gastroenterology is ahead of us. Oh this is so humiliating.
This is a serious issue and leads to the obvious question of “why is this the case?” Nephrologists are interested in doing what’s best for their patients. We are interested in science and evidence-based medicine. So why was so little research being done?
A few themes come up as possible explanations
There’s no money in it for industry - the perception is that big trials in cardiology and oncology are initiated and funded by industry and that happens much less in nephrology
CKD/ESKD patients are a unique group and difficult cohort to treat as the trail of negative RCTs in dialysis attests to
It’s hard to recruit nephrology patients- they’re often patients with large treatment burdens. Getting them to participate in research is a big ask
There is plenty of research but it’s all observational
As a trainee at these events it always seemed like the opposite was happening. There seemed to be a never-ending stream of new research on every conceivable topic – each report with a slightly nuanced question, an answer only applicable to a certain population and often a big problem with the study that only came to light after it was finished. There definitely didn’t seem to be a dearth of kidney research. Added to this, most of us either are, or know, other people actively involved in nephrology research.
So, have we gotten better? Did we caught up to the other specialties? Do we do as much (or more research!) as our colleagues? Are we doing better at constructing and reporting our research?
The Study
Markos Kyriakos Tomidis Chatzimanouil, Louise Wilkens and Hans-Joachim Anders aimed to relook at this topic. The original paper, mentioned above, by Strippoli et al from 2004, reported that not only were kidney related trials low in number, they also had poor design or poor reporting. Low in quantity and low in quality.
The authors suspected that increased abstract submissions for conferences, more nephrology journals, and more research from developing countries, as well as with improved guidelines for trial design and reporting, that this may have translated into more kidney research of higher quality. They also reviewed preclinical studies which hadn’t previously been examined.
Methods
First part: Quantitative analysis of:
Randomized clinical trials
Preclinical (animal) studies
Second part Qualitative analysis of:
Randomized clinical trials
Preclinical (animal) studies
The investigators mined Cochrane library and MeSH terms to uncover all of the RCTs in nephrology and other specialities between 1945 and 2016. When looking at quality they focused on five journals: The New England Journal of Medicine (NEJM) The Lancet, Kidney International, Journal of American Society of Nephrology (JASN) and American Journal of Kidney Disease (AJKD). The methodological quality was assessed by one investigator using the criteria of the original CONSORT 2010 checklist for RCTs. For pre-clinical studies the investigators looked at JASN, Kidney International, Nephrology Dialysis Transplantation (NDT). These journals were selected by impact factor and tendency to publish preclinical studies in the kidney domain. They were assessed using the ARRIVE guidelines.
Results
The annual number of reports of clinical kidney-related trials more than doubled between 2004 and 2014 along with reports in other medical disciplines so the amount of kidney research being done is definitely going up.
In 2004, the average slope of RCTs reported each year from 1966 -2003 was 27.7 for all disciplines, and slopes were 7.5 for kidney diseases (p < 0.001)
Since then the slope has increased to 68.4 for all other disciplines, (meaning a big increase in number of RCTs/year) but only increased to 23.3 for kidney diseases (p < 0.001). Still playing catch up? See below for graphical display of what this means:
Figure 1A from Chatzimanouil et al, JASN 2018
Within kidney related trials, hypertension continued to dominate:
Figure 1B from Chatzimanouil et al, JASN 2018
The number of preclinical studies lags behind other specialities:
Figure 2A from Chatzimanouil et al, JASN 2018
118 RCTs were analysed for appropriate design and reporting. The reporting quality analysis revealed improvements compared to 2004, but deficits in reporting of clinical trial design, mode of randomization, and intention-to treat analysis remain e.g. An intention-to-treat analysis regarding the primary end point was performed in less than half (45.8%) of studies.
Figure 3 from Chatzimanouil et al, JASN 2018
Quality assessment of reporting clinical trials in the main final paper according to the Consolidated Standards of Reporting Trials (CONSORT) criteria reveals improvements with time. Each of the CONSORT criteria was assessed as nonreported, unclear/insufficiently reported, or sufficiently reported in representative samples selected from The New England Journal of Medicine (NEJM), The Lancet (Lancet), Journal of the American Society of Nephrology (JASN), American Journal of Kidney Disease (AJKD), and Kidney International (Kidney Int.) of the years 1996, 2006, and 2016. Shown are the percentages of papers fulfilling the criterion “sufficiently reported” for (A) all journals in each of the 3 years to detect changes over time or (B) each of the journals across all time points. The selected number of papers was too small to also analyze trends over time for each journal.
135 preclinical studies were analysed revealing substantial reporting deficits across all leading journals, with little improvement over the last 20 years.
Figure 4 from Chatzimanouil et al, JASN 2018
Quality assessment of reporting preclinical studies in the main paper according to the Animal Research: Reporting In Vivo Experiments (ARRIVE) criteria does not reveal improvements with time. Each of the ARRIVE criteria (except for title and abstract criteria) was assessed as nonreported, unclear/insufficiently reported, or sufficiently reported in representative samples selected from Journal of the American Society of Nephrology (JASN), Kidney International (Kidney Int.), and Nephrology Dialysis and Transplantation (NDT) of the years 1996, 2006, and 2016. Shown are the percentages of papers fulfilling the criterion “sufficiently reported” for (A) all journals in each of the 3 years to detect changes over time or (B) each of the journals across all time points. The selected number of papers was too small to also analyze trends over time for each journal.
Discussion
One might say, ‘res ipsa loquitur’ : after all that, what is there needed to discuss?
Although the total number of nephrology trials has increased since 2004, as postulated by the authors, this number has remained low in comparison to other specialties. Preclinical studies in nephrology are also low in proportion to other studies in other specialties. It is important to note that the majority of kidney related trials have been done in hypertension, an area which is of interest to a large number of specialties (cardiology, endocrine, geriatrics, general practice) and may not even be seen by many as a kidney specific area.
The quality of data reporting in papers presenting RCTs keeps improving but is still suboptimal in many ways. It is noticeable that this suboptimal reporting is present even in the high impact factor, high quality journals selected by the authors. The quality of data reporting of preclinical studies is still in its infancy and may contribute to reproducibility problems.
The authors note that efforts at all levels are needed to overcome these deficits in the future. Given the central role of kidney disease– related morbidity and mortality, as well as health care costs, greater investments in kidney research are needed.
Perhaps even more important is to ensure that the money invested in kidney research well spent - reporting guidelines exist, what can we do in the nephrology community, as researchers, editors, reviewers, and readers, to enforce these standards? To increase not just the quantity, but also the quality of published research?
Summary by Sarah Gleeson
Nephrology Registrar, London, UK
NSMC Faculty Mentor