The Journal of the American Medical Informatics Association (JAMIA) published recent work by Enrico Coiera and colleagues from Macquarie University and the University of Health Sciences (Austria) of a narrative review of literature on research replication challenges. The review concluded that the cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics.
Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies; and the field of biomedical and health informatics is no different.
While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting.
The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the non-reproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.
Jan 29, 2020