Issue Cover

  • Previous Issue
  • Previous Article
  • Next Article

Clarifying the Research Purpose

Methodology, measurement, data analysis and interpretation, tools for evaluating the quality of medical education research, research support, competing interests, quantitative research methods in medical education.

Submitted for publication January 8, 2018. Accepted for publication November 29, 2018.

  • Split-Screen
  • Article contents
  • Figures & tables
  • Supplementary Data
  • Peer Review
  • Open the PDF for in another window
  • Cite Icon Cite
  • Get Permissions
  • Search Site

John T. Ratelle , Adam P. Sawatsky , Thomas J. Beckman; Quantitative Research Methods in Medical Education. Anesthesiology 2019; 131:23–35 doi:

Download citation file:

  • Ris (Zotero)
  • Reference Manager

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Image: J. P. Rathmell and Terri Navarette.

Image: J. P. Rathmell and Terri Navarette.

There has been an explosion of research in the field of medical education. A search of PubMed demonstrates that more than 40,000 articles have been indexed under the medical subject heading “Medical Education” since 2010, which is more than the total number of articles indexed under this heading in the 1980s and 1990s combined. Keeping up to date requires that practicing clinicians have the skills to interpret and appraise the quality of research articles, especially when serving as editors, reviewers, and consumers of the literature.

While medical education shares many characteristics with other biomedical fields, substantial particularities exist. We recognize that practicing clinicians may not be familiar with the nuances of education research and how to assess its quality. Therefore, our purpose is to provide a review of quantitative research methodologies in medical education. Specifically, we describe a structure that can be used when conducting or evaluating medical education research articles.

Clarifying the research purpose is an essential first step when reading or conducting scholarship in medical education. 1   Medical education research can serve a variety of purposes, from advancing the science of learning to improving the outcomes of medical trainees and the patients they care for. However, a well-designed study has limited value if it addresses vague, redundant, or unimportant medical education research questions.

What is the research topic and why is it important? What is unknown about the research topic? Why is further research necessary?

What is the conceptual framework being used to approach the study?

What is the statement of study intent?

What are the research methodology and study design? Are they appropriate for the study objective(s)?

Which threats to internal validity are most relevant for the study?

What is the outcome and how was it measured?

Can the results be trusted? What is the validity and reliability of the measurements?

How were research subjects selected? Is the research sample representative of the target population?

Was the data analysis appropriate for the study design and type of data?

What is the effect size? Do the results have educational significance?

Fortunately, there are steps to ensure that the purpose of a research study is clear and logical. Table 1   2–5   outlines these steps, which will be described in detail in the following sections. We describe these elements not as a simple “checklist,” but as an advanced organizer that can be used to understand a medical education research study. These steps can also be used by clinician educators who are new to the field of education research and who wish to conduct scholarship in medical education.

Steps in Clarifying the Purpose of a Research Study in Medical Education

Steps in Clarifying the Purpose of a Research Study in Medical Education

Literature Review and Problem Statement

A literature review is the first step in clarifying the purpose of a medical education research article. 2 , 5 , 6   When conducting scholarship in medical education, a literature review helps researchers develop an understanding of their topic of interest. This understanding includes both existing knowledge about the topic as well as key gaps in the literature, which aids the researcher in refining their study question. Additionally, a literature review helps researchers identify conceptual frameworks that have been used to approach the research topic. 2  

When reading scholarship in medical education, a successful literature review provides background information so that even someone unfamiliar with the research topic can understand the rationale for the study. Located in the introduction of the manuscript, the literature review guides the reader through what is already known in a manner that highlights the importance of the research topic. The literature review should also identify key gaps in the literature so the reader can understand the need for further research. This gap description includes an explicit problem statement that summarizes the important issues and provides a reason for the study. 2 , 4   The following is one example of a problem statement:

“Identifying gaps in the competency of anesthesia residents in time for intervention is critical to patient safety and an effective learning system… [However], few available instruments relate to complex behavioral performance or provide descriptors…that could inform subsequent feedback, individualized teaching, remediation, and curriculum revision.” 7  

This problem statement articulates the research topic (identifying resident performance gaps), why it is important (to intervene for the sake of learning and patient safety), and current gaps in the literature (few tools are available to assess resident performance). The researchers have now underscored why further research is needed and have helped readers anticipate the overarching goals of their study (to develop an instrument to measure anesthesiology resident performance). 4  

The Conceptual Framework

Following the literature review and articulation of the problem statement, the next step in clarifying the research purpose is to select a conceptual framework that can be applied to the research topic. Conceptual frameworks are “ways of thinking about a problem or a study, or ways of representing how complex things work.” 3   Just as clinical trials are informed by basic science research in the laboratory, conceptual frameworks often serve as the “basic science” that informs scholarship in medical education. At a fundamental level, conceptual frameworks provide a structured approach to solving the problem identified in the problem statement.

Conceptual frameworks may take the form of theories, principles, or models that help to explain the research problem by identifying its essential variables or elements. Alternatively, conceptual frameworks may represent evidence-based best practices that researchers can apply to an issue identified in the problem statement. 3   Importantly, there is no single best conceptual framework for a particular research topic, although the choice of a conceptual framework is often informed by the literature review and knowing which conceptual frameworks have been used in similar research. 8   For further information on selecting a conceptual framework for research in medical education, we direct readers to the work of Bordage 3   and Irby et al. 9  

To illustrate how different conceptual frameworks can be applied to a research problem, suppose you encounter a study to reduce the frequency of communication errors among anesthesiology residents during day-to-night handoff. Table 2 10 , 11   identifies two different conceptual frameworks researchers might use to approach the task. The first framework, cognitive load theory, has been proposed as a conceptual framework to identify potential variables that may lead to handoff errors. 12   Specifically, cognitive load theory identifies the three factors that affect short-term memory and thus may lead to communication errors:

Conceptual Frameworks to Address the Issue of Handoff Errors in the Intensive Care Unit

Conceptual Frameworks to Address the Issue of Handoff Errors in the Intensive Care Unit

Intrinsic load: Inherent complexity or difficulty of the information the resident is trying to learn ( e.g. , complex patients).

Extraneous load: Distractions or demands on short-term memory that are not related to the information the resident is trying to learn ( e.g. , background noise, interruptions).

Germane load: Effort or mental strategies used by the resident to organize and understand the information he/she is trying to learn ( e.g. , teach back, note taking).

Using cognitive load theory as a conceptual framework, researchers may design an intervention to reduce extraneous load and help the resident remember the overnight to-do’s. An example might be dedicated, pager-free handoff times where distractions are minimized.

The second framework identified in table 2 , the I-PASS (Illness severity, Patient summary, Action list, Situational awareness and contingency planning, and Synthesis by receiver) handoff mnemonic, 11   is an evidence-based best practice that, when incorporated as part of a handoff bundle, has been shown to reduce handoff errors on pediatric wards. 13   Researchers choosing this conceptual framework may adapt some or all of the I-PASS elements for resident handoffs in the intensive care unit.

Note that both of the conceptual frameworks outlined above provide researchers with a structured approach to addressing the issue of handoff errors; one is not necessarily better than the other. Indeed, it is possible for researchers to use both frameworks when designing their study. Ultimately, we provide this example to demonstrate the necessity of selecting conceptual frameworks to clarify the research purpose. 3 , 8   Readers should look for conceptual frameworks in the introduction section and should be wary of their omission, as commonly seen in less well-developed medical education research articles. 14  

Statement of Study Intent

After reviewing the literature, articulating the problem statement, and selecting a conceptual framework to address the research topic, the final step in clarifying the research purpose is the statement of study intent. The statement of study intent is arguably the most important element of framing the study because it makes the research purpose explicit. 2   Consider the following example:

This study aimed to test the hypothesis that the introduction of the BASIC Examination was associated with an accelerated knowledge acquisition during residency training, as measured by increments in annual ITE scores. 15  

This statement of study intent succinctly identifies several key study elements including the population (anesthesiology residents), the intervention/independent variable (introduction of the BASIC Examination), the outcome/dependent variable (knowledge acquisition, as measure by in In-training Examination [ITE] scores), and the hypothesized relationship between the independent and dependent variable (the authors hypothesize a positive correlation between the BASIC examination and the speed of knowledge acquisition). 6 , 14  

The statement of study intent will sometimes manifest as a research objective, rather than hypothesis or question. In such instances there may not be explicit independent and dependent variables, but the study population and research aim should be clearly identified. The following is an example:

“In this report, we present the results of 3 [years] of course data with respect to the practice improvements proposed by participating anesthesiologists and their success in implementing those plans. Specifically, our primary aim is to assess the frequency and type of improvements that were completed and any factors that influence completion.” 16  

The statement of study intent is the logical culmination of the literature review, problem statement, and conceptual framework, and is a transition point between the Introduction and Methods sections of a medical education research report. Nonetheless, a systematic review of experimental research in medical education demonstrated that statements of study intent are absent in the majority of articles. 14   When reading a medical education research article where the statement of study intent is absent, it may be necessary to infer the research aim by gathering information from the Introduction and Methods sections. In these cases, it can be useful to identify the following key elements 6 , 14 , 17   :

Population of interest/type of learner ( e.g. , pain medicine fellow or anesthesiology residents)

Independent/predictor variable ( e.g. , educational intervention or characteristic of the learners)

Dependent/outcome variable ( e.g. , intubation skills or knowledge of anesthetic agents)

Relationship between the variables ( e.g. , “improve” or “mitigate”)

Occasionally, it may be difficult to differentiate the independent study variable from the dependent study variable. 17   For example, consider a study aiming to measure the relationship between burnout and personal debt among anesthesiology residents. Do the researchers believe burnout might lead to high personal debt, or that high personal debt may lead to burnout? This “chicken or egg” conundrum reinforces the importance of the conceptual framework which, if present, should serve as an explanation or rationale for the predicted relationship between study variables.

Research methodology is the “…design or plan that shapes the methods to be used in a study.” 1   Essentially, methodology is the general strategy for answering a research question, whereas methods are the specific steps and techniques that are used to collect data and implement the strategy. Our objective here is to provide an overview of quantitative methodologies ( i.e. , approaches) in medical education research.

The choice of research methodology is made by balancing the approach that best answers the research question against the feasibility of completing the study. There is no perfect methodology because each has its own potential caveats, flaws and/or sources of bias. Before delving into an overview of the methodologies, it is important to highlight common sources of bias in education research. We use the term internal validity to describe the degree to which the findings of a research study represent “the truth,” as opposed to some alternative hypothesis or variables. 18   Table 3   18–20   provides a list of common threats to internal validity in medical education research, along with tactics to mitigate these threats.

Threats to Internal Validity and Strategies to Mitigate Their Effects

Threats to Internal Validity and Strategies to Mitigate Their Effects

Experimental Research

The fundamental tenet of experimental research is the manipulation of an independent or experimental variable to measure its effect on a dependent or outcome variable.

True Experiment

True experimental study designs minimize threats to internal validity by randomizing study subjects to experimental and control groups. Through ensuring that differences between groups are—beyond the intervention/variable of interest—purely due to chance, researchers reduce the internal validity threats related to subject characteristics, time-related maturation, and regression to the mean. 18 , 19  


There are many instances in medical education where randomization may not be feasible or ethical. For instance, researchers wanting to test the effect of a new curriculum among medical students may not be able to randomize learners due to competing curricular obligations and schedules. In these cases, researchers may be forced to assign subjects to experimental and control groups based upon some other criterion beyond randomization, such as different classrooms or different sections of the same course. This process, called quasi-randomization, does not inherently lead to internal validity threats, as long as research investigators are mindful of measuring and controlling for extraneous variables between study groups. 19  

Single-group Methodologies

All experimental study designs compare two or more groups: experimental and control. A common experimental study design in medical education research is the single-group pretest–posttest design, which compares a group of learners before and after the implementation of an intervention. 21   In essence, a single-group pre–post design compares an experimental group ( i.e. , postintervention) to a “no-intervention” control group ( i.e. , preintervention). 19   This study design is problematic for several reasons. Consider the following hypothetical example: A research article reports the effects of a year-long intubation curriculum for first-year anesthesiology residents. All residents participate in monthly, half-day workshops over the course of an academic year. The article reports a positive effect on residents’ skills as demonstrated by a significant improvement in intubation success rates at the end of the year when compared to the beginning.

This study does little to advance the science of learning among anesthesiology residents. While this hypothetical report demonstrates an improvement in residents’ intubation success before versus after the intervention, it does not tell why the workshop worked, how it compares to other educational interventions, or how it fits in to the broader picture of anesthesia training.

Single-group pre–post study designs open themselves to a myriad of threats to internal validity. 20   In our hypothetical example, the improvement in residents’ intubation skills may have been due to other educational experience(s) ( i.e. , implementation threat) and/or improvement in manual dexterity that occurred naturally with time ( i.e. , maturation threat), rather than the airway curriculum. Consequently, single-group pre–post studies should be interpreted with caution. 18  

Repeated testing, before and after the intervention, is one strategy that can be used to reduce the some of the inherent limitations of the single-group study design. Repeated pretesting can mitigate the effect of regression toward the mean, a statistical phenomenon whereby low pretest scores tend to move closer to the mean on subsequent testing (regardless of intervention). 20   Likewise, repeated posttesting at multiple time intervals can provide potentially useful information about the short- and long-term effects of an intervention ( e.g. , the “durability” of the gain in knowledge, skill, or attitude).

Observational Research

Unlike experimental studies, observational research does not involve manipulation of any variables. These studies often involve measuring associations, developing psychometric instruments, or conducting surveys.

Association Research

Association research seeks to identify relationships between two or more variables within a group or groups (correlational research), or similarities/differences between two or more existing groups (causal–comparative research). For example, correlational research might seek to measure the relationship between burnout and educational debt among anesthesiology residents, while causal–comparative research may seek to measure differences in educational debt and/or burnout between anesthesiology and surgery residents. Notably, association research may identify relationships between variables, but does not necessarily support a causal relationship between them.

Psychometric and Survey Research

Psychometric instruments measure a psychologic or cognitive construct such as knowledge, satisfaction, beliefs, and symptoms. Surveys are one type of psychometric instrument, but many other types exist, such as evaluations of direct observation, written examinations, or screening tools. 22   Psychometric instruments are ubiquitous in medical education research and can be used to describe a trait within a study population ( e.g. , rates of depression among medical students) or to measure associations between study variables ( e.g. , association between depression and board scores among medical students).

Psychometric and survey research studies are prone to the internal validity threats listed in table 3 , particularly those relating to mortality, location, and instrumentation. 18   Additionally, readers must ensure that the instrument scores can be trusted to truly represent the construct being measured. For example, suppose you encounter a research article demonstrating a positive association between attending physician teaching effectiveness as measured by a survey of medical students, and the frequency with which the attending physician provides coffee and doughnuts on rounds. Can we be confident that this survey administered to medical students is truly measuring teaching effectiveness? Or is it simply measuring the attending physician’s “likability”? Issues related to measurement and the trustworthiness of data are described in detail in the following section on measurement and the related issues of validity and reliability.

Measurement refers to “the assigning of numbers to individuals in a systematic way as a means of representing properties of the individuals.” 23   Research data can only be trusted insofar as we trust the measurement used to obtain the data. Measurement is of particular importance in medical education research because many of the constructs being measured ( e.g. , knowledge, skill, attitudes) are abstract and subject to measurement error. 24   This section highlights two specific issues related to the trustworthiness of data: the validity and reliability of measurements.

Validity regarding the scores of a measurement instrument “refers to the degree to which evidence and theory support the interpretations of the [instrument’s results] for the proposed use of the [instrument].” 25   In essence, do we believe the results obtained from a measurement really represent what we were trying to measure? Note that validity evidence for the scores of a measurement instrument is separate from the internal validity of a research study. Several frameworks for validity evidence exist. Table 4 2 , 22 , 26   represents the most commonly used framework, developed by Messick, 27   which identifies sources of validity evidence—to support the target construct—from five main categories: content, response process, internal structure, relations to other variables, and consequences.

Sources of Validity Evidence for Measurement Instruments

Sources of Validity Evidence for Measurement Instruments


Reliability refers to the consistency of scores for a measurement instrument. 22 , 25 , 28   For an instrument to be reliable, we would anticipate that two individuals rating the same object of measurement in a specific context would provide the same scores. 25   Further, if the scores for an instrument are reliable between raters of the same object of measurement, then we can extrapolate that any difference in scores between two objects represents a true difference across the sample, and is not due to random variation in measurement. 29   Reliability can be demonstrated through a variety of methods such as internal consistency ( e.g. , Cronbach’s alpha), temporal stability ( e.g. , test–retest reliability), interrater agreement ( e.g. , intraclass correlation coefficient), and generalizability theory (generalizability coefficient). 22 , 29  

Example of a Validity and Reliability Argument

This section provides an illustration of validity and reliability in medical education. We use the signaling questions outlined in table 4 to make a validity and reliability argument for the Harvard Assessment of Anesthesia Resident Performance (HARP) instrument. 7   The HARP was developed by Blum et al. to measure the performance of anesthesia trainees that is required to provide safe anesthetic care to patients. According to the authors, the HARP is designed to be used “…as part of a multiscenario, simulation-based assessment” of resident performance. 7  

Content Validity: Does the Instrument’s Content Represent the Construct Being Measured?

To demonstrate content validity, instrument developers should describe the construct being measured and how the instrument was developed, and justify their approach. 25   The HARP is intended to measure resident performance in the critical domains required to provide safe anesthetic care. As such, investigators note that the HARP items were created through a two-step process. First, the instrument’s developers interviewed anesthesiologists with experience in resident education to identify the key traits needed for successful completion of anesthesia residency training. Second, the authors used a modified Delphi process to synthesize the responses into five key behaviors: (1) formulate a clear anesthetic plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize one’s limits. 7 , 30  

Response Process Validity: Are Raters Interpreting the Instrument Items as Intended?

In the case of the HARP, the developers included a scoring rubric with behavioral anchors to ensure that faculty raters could clearly identify how resident performance in each domain should be scored. 7  

Internal Structure Validity: Do Instrument Items Measuring Similar Constructs Yield Homogenous Results? Do Instrument Items Measuring Different Constructs Yield Heterogeneous Results?

Item-correlation for the HARP demonstrated a high degree of correlation between some items ( e.g. , formulating a plan and modifying the plan under changing conditions) and a lower degree of correlation between other items ( e.g. , formulating a plan and identifying performance improvement opportunities). 30   This finding is expected since the items within the HARP are designed to assess separate performance domains, and we would expect residents’ functioning to vary across domains.

Relationship to Other Variables’ Validity: Do Instrument Scores Correlate with Other Measures of Similar or Different Constructs as Expected?

As it applies to the HARP, one would expect that the performance of anesthesia residents will improve over the course of training. Indeed, HARP scores were found to be generally higher among third-year residents compared to first-year residents. 30  

Consequence Validity: Are Instrument Results Being Used as Intended? Are There Unintended or Negative Uses of the Instrument Results?

While investigators did not intentionally seek out consequence validity evidence for the HARP, unanticipated consequences of HARP scores were identified by the authors as follows:

“Data indicated that CA-3s had a lower percentage of worrisome scores (rating 2 or lower) than CA-1s… However, it is concerning that any CA-3s had any worrisome scores…low performance of some CA-3 residents, albeit in the simulated environment, suggests opportunities for training improvement.” 30  

That is, using the HARP to measure the performance of CA-3 anesthesia residents had the unintended consequence of identifying the need for improvement in resident training.

Reliability: Are the Instrument’s Scores Reproducible and Consistent between Raters?

The HARP was applied by two raters for every resident in the study across seven different simulation scenarios. The investigators conducted a generalizability study of HARP scores to estimate the variance in assessment scores that was due to the resident, the rater, and the scenario. They found little variance was due to the rater ( i.e. , scores were consistent between raters), indicating a high level of reliability. 7  

Sampling refers to the selection of research subjects ( i.e. , the sample) from a larger group of eligible individuals ( i.e. , the population). 31   Effective sampling leads to the inclusion of research subjects who represent the larger population of interest. Alternatively, ineffective sampling may lead to the selection of research subjects who are significantly different from the target population. Imagine that researchers want to explore the relationship between burnout and educational debt among pain medicine specialists. The researchers distribute a survey to 1,000 pain medicine specialists (the population), but only 300 individuals complete the survey (the sample). This result is problematic because the characteristics of those individuals who completed the survey and the entire population of pain medicine specialists may be fundamentally different. It is possible that the 300 study subjects may be experiencing more burnout and/or debt, and thus, were more motivated to complete the survey. Alternatively, the 700 nonresponders might have been too busy to respond and even more burned out than the 300 responders, which would suggest that the study findings were even more amplified than actually observed.

When evaluating a medical education research article, it is important to identify the sampling technique the researchers employed, how it might have influenced the results, and whether the results apply to the target population. 24  

Sampling Techniques

Sampling techniques generally fall into two categories: probability- or nonprobability-based. Probability-based sampling ensures that each individual within the target population has an equal opportunity of being selected as a research subject. Most commonly, this is done through random sampling, which should lead to a sample of research subjects that is similar to the target population. If significant differences between sample and population exist, those differences should be due to random chance, rather than systematic bias. The difference between data from a random sample and that from the population is referred to as sampling error. 24  

Nonprobability-based sampling involves selecting research participants such that inclusion of some individuals may be more likely than the inclusion of others. 31   Convenience sampling is one such example and involves selection of research subjects based upon ease or opportuneness. Convenience sampling is common in medical education research, but, as outlined in the example at the beginning of this section, it can lead to sampling bias. 24   When evaluating an article that uses nonprobability-based sampling, it is important to look for participation/response rate. In general, a participation rate of less than 75% should be viewed with skepticism. 21   Additionally, it is important to determine whether characteristics of participants and nonparticipants were reported and if significant differences between the two groups exist.

Interpreting medical education research requires a basic understanding of common ways in which quantitative data are analyzed and displayed. In this section, we highlight two broad topics that are of particular importance when evaluating research articles.

The Nature of the Measurement Variable

Measurement variables in quantitative research generally fall into three categories: nominal, ordinal, or interval. 24   Nominal variables (sometimes called categorical variables) involve data that can be placed into discrete categories without a specific order or structure. Examples include sex (male or female) and professional degree (M.D., D.O., M.B.B.S., etc .) where there is no clear hierarchical order to the categories. Ordinal variables can be ranked according to some criterion, but the spacing between categories may not be equal. Examples of ordinal variables may include measurements of satisfaction (satisfied vs . unsatisfied), agreement (disagree vs . agree), and educational experience (medical student, resident, fellow). As it applies to educational experience, it is noteworthy that even though education can be quantified in years, the spacing between years ( i.e. , educational “growth”) remains unequal. For instance, the difference in performance between second- and third-year medical students is dramatically different than third- and fourth-year medical students. Interval variables can also be ranked according to some criteria, but, unlike ordinal variables, the spacing between variable categories is equal. Examples of interval variables include test scores and salary. However, the conceptual boundaries between these measurement variables are not always clear, as in the case where ordinal scales can be assumed to have the properties of an interval scale, so long as the data’s distribution is not substantially skewed. 32  

Understanding the nature of the measurement variable is important when evaluating how the data are analyzed and reported. Medical education research commonly uses measurement instruments with items that are rated on Likert-type scales, whereby the respondent is asked to assess their level of agreement with a given statement. The response is often translated into a corresponding number ( e.g. , 1 = strongly disagree, 3 = neutral, 5 = strongly agree). It is remarkable that scores from Likert-type scales are sometimes not normally distributed ( i.e. , are skewed toward one end of the scale), indicating that the spacing between scores is unequal and the variable is ordinal in nature. In these cases, it is recommended to report results as frequencies or medians, rather than means and SDs. 33  

Consider an article evaluating medical students’ satisfaction with a new curriculum. Researchers measure satisfaction using a Likert-type scale (1 = very unsatisfied, 2 = unsatisfied, 3 = neutral, 4 = satisfied, 5 = very satisfied). A total of 20 medical students evaluate the curriculum, 10 of whom rate their satisfaction as “satisfied,” and 10 of whom rate it as “very satisfied.” In this case, it does not make much sense to report an average score of 4.5; it makes more sense to report results in terms of frequency ( e.g. , half of the students were “very satisfied” with the curriculum, and half were not).

Effect Size and CIs

In medical education, as in other research disciplines, it is common to report statistically significant results ( i.e. , small P values) in order to increase the likelihood of publication. 34 , 35   However, a significant P value in itself does necessarily represent the educational impact of the study results. A statement like “Intervention x was associated with a significant improvement in learners’ intubation skill compared to education intervention y ( P < 0.05)” tells us that there was a less than 5% chance that the difference in improvement between interventions x and y was due to chance. Yet that does not mean that the study intervention necessarily caused the nonchance results, or indicate whether the between-group difference is educationally significant. Therefore, readers should consider looking beyond the P value to effect size and/or CI when interpreting the study results. 36 , 37  

Effect size is “the magnitude of the difference between two groups,” which helps to quantify the educational significance of the research results. 37   Common measures of effect size include Cohen’s d (standardized difference between two means), risk ratio (compares binary outcomes between two groups), and Pearson’s r correlation (linear relationship between two continuous variables). 37   CIs represent “a range of values around a sample mean or proportion” and are a measure of precision. 31   While effect size and CI give more useful information than simple statistical significance, they are commonly omitted from medical education research articles. 35   In such instances, readers should be wary of overinterpreting a P value in isolation. For further information effect size and CI, we direct readers the work of Sullivan and Feinn 37   and Hulley et al. 31  

In this final section, we identify instruments that can be used to evaluate the quality of quantitative medical education research articles. To this point, we have focused on framing the study and research methodologies and identifying potential pitfalls to consider when appraising a specific article. This is important because how a study is framed and the choice of methodology require some subjective interpretation. Fortunately, there are several instruments available for evaluating medical education research methods and providing a structured approach to the evaluation process.

The Medical Education Research Study Quality Instrument (MERSQI) 21   and the Newcastle Ottawa Scale-Education (NOS-E) 38   are two commonly used instruments, both of which have an extensive body of validity evidence to support the interpretation of their scores. Table 5 21 , 39   provides more detail regarding the MERSQI, which includes evaluation of study design, sampling, data type, validity, data analysis, and outcomes. We have found that applying the MERSQI to manuscripts, articles, and protocols has intrinsic educational value, because this practice of application familiarizes MERSQI users with fundamental principles of medical education research. One aspect of the MERSQI that deserves special mention is the section on evaluating outcomes based on Kirkpatrick’s widely recognized hierarchy of reaction, learning, behavior, and results ( table 5 ; fig .). 40   Validity evidence for the scores of the MERSQI include its operational definitions to improve response process, excellent reliability, and internal consistency, as well as high correlation with other measures of study quality, likelihood of publication, citation rate, and an association between MERSQI score and the likelihood of study funding. 21 , 41   Additionally, consequence validity for the MERSQI scores has been demonstrated by its utility for identifying and disseminating high-quality research in medical education. 42  

Fig. Kirkpatrick’s hierarchy of outcomes as applied to education research. Reaction = Level 1, Learning = Level 2, Behavior = Level 3, Results = Level 4. Outcomes become more meaningful, yet more difficult to achieve, when progressing from Level 1 through Level 4. Adapted with permission from Beckman and Cook, 2007.2

Kirkpatrick’s hierarchy of outcomes as applied to education research. Reaction = Level 1, Learning = Level 2, Behavior = Level 3, Results = Level 4. Outcomes become more meaningful, yet more difficult to achieve, when progressing from Level 1 through Level 4. Adapted with permission from Beckman and Cook, 2007. 2  

The Medical Education Research Study Quality Instrument for Evaluating the Quality of Medical Education Research

The Medical Education Research Study Quality Instrument for Evaluating the Quality of Medical Education Research

The NOS-E is a newer tool to evaluate the quality of medication education research. It was developed as a modification of the Newcastle-Ottawa Scale 43   for appraising the quality of nonrandomized studies. The NOS-E includes items focusing on the representativeness of the experimental group, selection and compatibility of the control group, missing data/study retention, and blinding of outcome assessors. 38 , 39   Additional validity evidence for NOS-E scores includes operational definitions to improve response process, excellent reliability and internal consistency, and its correlation with other measures of study quality. 39   Notably, the complete NOS-E, along with its scoring rubric, can found in the article by Cook and Reed. 39  

A recent comparison of the MERSQI and NOS-E found acceptable interrater reliability and good correlation between the two instruments 39   However, noted differences exist between the MERSQI and NOS-E. Specifically, the MERSQI may be applied to a broad range of study designs, including experimental and cross-sectional research. Additionally, the MERSQI addresses issues related to measurement validity and data analysis, and places emphasis on educational outcomes. On the other hand, the NOS-E focuses specifically on experimental study designs, and on issues related to sampling techniques and outcome assessment. 39   Ultimately, the MERSQI and NOS-E are complementary tools that may be used together when evaluating the quality of medical education research.


This article provides an overview of quantitative research in medical education, underscores the main components of education research, and provides a general framework for evaluating research quality. We highlighted the importance of framing a study with respect to purpose, conceptual framework, and statement of study intent. We reviewed the most common research methodologies, along with threats to the validity of a study and its measurement instruments. Finally, we identified two complementary instruments, the MERSQI and NOS-E, for evaluating the quality of a medical education research study.

Bordage G: Conceptual frameworks to illuminate and magnify. Medical education. 2009; 43(4):312–9.

Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: Theory and application. The American journal of medicine. 2006; 119(2):166. e7–166. e116.

Franenkel JR, Wallen NE, Hyun HH: How to Design and Evaluate Research in Education. 9th edition. New York, McGraw-Hill Education, 2015.

Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB: Designing clinical research. 4th edition. Philadelphia, Lippincott Williams & Wilkins, 2011.

Irby BJ, Brown G, Lara-Alecio R, Jackson S: The Handbook of Educational Theories. Charlotte, NC, Information Age Publishing, Inc., 2015

Standards for Educational and Psychological Testing (American Educational Research Association & American Psychological Association, 2014)

Swanwick T: Understanding medical education: Evidence, theory and practice, 2nd edition. Wiley-Blackwell, 2013.

Sullivan GM, Artino Jr AR: Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education. 2013; 5(4):541–2.

Sullivan GM, Feinn R: Using effect size—or why the P value is not enough. Journal of graduate medical education. 2012; 4(3):279–82.

Tavakol M, Sandars J: Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical teacher. 2014; 36(10):838–48.

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

  • ASA Practice Parameters
  • Online First
  • Author Resource Center
  • About the Journal
  • Editorial Board
  • Rights & Permissions
  • Online ISSN 1528-1175
  • Print ISSN 0003-3022
  • Anesthesiology
  • ASA Monitor

Silverchair Information Systems

  • Terms & Conditions Privacy Policy
  • Manage Cookie Preferences
  • © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access


Research Article

Recent quantitative research on determinants of health in high income countries: A scoping review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Centre for Health Economics Research and Modelling Infectious Diseases, Vaccine and Infectious Disease Institute, University of Antwerp, Antwerp, Belgium

ORCID logo

Roles Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

  • Vladimira Varbanova, 
  • Philippe Beutels


  • Published: September 17, 2020
  • Peer Review
  • Reader Comments

Fig 1

Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature.

We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019). We searched for studies that performed cross-national statistical analyses aiming to evaluate the impact of one or more aggregate level determinants on one or more general population health outcomes in high-income countries. To assess in which combinations and to what extent individual (or thematically linked) determinants had been studied together, we performed multidimensional scaling and cluster analysis.

Sixty studies were selected, out of an original yield of 3686. Life-expectancy and overall mortality were the most widely used population health indicators, while determinants came from the areas of healthcare, culture, politics, socio-economics, environment, labor, fertility, demographics, life-style, and psychology. The family of regression models was the predominant statistical approach. Results from our multidimensional scaling showed that a relatively tight core of determinants have received much attention, as main covariates of interest or controls, whereas the majority of other determinants were studied in very limited contexts. We consider findings from these studies regarding the importance of any given health determinant inconclusive at present. Across a multitude of model specifications, different country samples, and varying time periods, effects fluctuated between statistically significant and not significant, and between beneficial and detrimental to health.


We conclude that efforts to understand the underlying mechanisms of population health are far from settled, and the present state of research on the topic leaves much to be desired. It is essential that future research considers multiple factors simultaneously and takes advantage of more sophisticated methodology with regards to quantifying health as well as analyzing determinants’ influence.

Citation: Varbanova V, Beutels P (2020) Recent quantitative research on determinants of health in high income countries: A scoping review. PLoS ONE 15(9): e0239031.

Editor: Amir Radfar, University of Central Florida, UNITED STATES

Received: November 14, 2019; Accepted: August 28, 2020; Published: September 17, 2020

Copyright: © 2020 Varbanova, Beutels. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: This study (and VV) is funded by the Research Foundation Flanders ( ), FWO project number G0D5917N, award obtained by PB. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.


Identifying the key drivers of population health is a core subject in public health and health economics research. Between-country comparative research on the topic is challenging. In order to be relevant for policy, it requires disentangling different interrelated drivers of “good health”, each having different degrees of importance in different contexts.

“Good health”–physical and psychological, subjective and objective–can be defined and measured using a variety of approaches, depending on which aspect of health is the focus. A major distinction can be made between health measurements at the individual level or some aggregate level, such as a neighborhood, a region or a country. In view of this, a great diversity of specific research topics exists on the drivers of what constitutes individual or aggregate “good health”, including those focusing on health inequalities, the gender gap in longevity, and regional mortality and longevity differences.

The current scoping review focuses on determinants of population health. Stated as such, this topic is quite broad. Indeed, we are interested in the very general question of what methods have been used to make the most of increasingly available region or country-specific databases to understand the drivers of population health through inter-country comparisons. Existing reviews indicate that researchers thus far tend to adopt a narrower focus. Usually, attention is given to only one health outcome at a time, with further geographical and/or population [ 1 , 2 ] restrictions. In some cases, the impact of one or more interventions is at the core of the review [ 3 – 7 ], while in others it is the relationship between health and just one particular predictor, e.g., income inequality, access to healthcare, government mechanisms [ 8 – 13 ]. Some relatively recent reviews on the subject of social determinants of health [ 4 – 6 , 14 – 17 ] have considered a number of indicators potentially influencing health as opposed to a single one. One review defines “social determinants” as “the social, economic, and political conditions that influence the health of individuals and populations” [ 17 ] while another refers even more broadly to “the factors apart from medical care” [ 15 ].

In the present work, we aimed to be more inclusive, setting no limitations on the nature of possible health correlates, as well as making use of a multitude of commonly accepted measures of general population health. The goal of this scoping review was to document the state of the art in the recent published literature on determinants of population health, with a particular focus on the types of determinants selected and the methodology used. In doing so, we also report the main characteristics of the results these studies found. The materials collected in this review are intended to inform our (and potentially other researchers’) future analyses on this topic. Since the production of health is subject to the law of diminishing marginal returns, we focused our review on those studies that included countries where a high standard of wealth has been achieved for some time, i.e., high-income countries belonging to the Organisation for Economic Co-operation and Development (OECD) or Europe. Adding similar reviews for other country income groups is of limited interest to the research we plan to do in this area.

In view of its focus on data and methods, rather than results, a formal protocol was not registered prior to undertaking this review, but the procedure followed the guidelines of the PRISMA statement for scoping reviews [ 18 ].

We focused on multi-country studies investigating the potential associations between any aggregate level (region/city/country) determinant and general measures of population health (e.g., life expectancy, mortality rate).

Within the query itself, we listed well-established population health indicators as well as the six world regions, as defined by the World Health Organization (WHO). We searched only in the publications’ titles in order to keep the number of hits manageable, and the ratio of broadly relevant abstracts over all abstracts in the order of magnitude of 10% (based on a series of time-focused trial runs). The search strategy was developed iteratively between the two authors and is presented in S1 Appendix . The search was performed by VV in PubMed and Web of Science on the 16 th of July, 2019, without any language restrictions, and with a start date set to the 1 st of January, 2013, as we were interested in the latest developments in this area of research.

Eligibility criteria

Records obtained via the search methods described above were screened independently by the two authors. Consistency between inclusion/exclusion decisions was approximately 90% and the 43 instances where uncertainty existed were judged through discussion. Articles were included subject to meeting the following requirements: (a) the paper was a full published report of an original empirical study investigating the impact of at least one aggregate level (city/region/country) factor on at least one health indicator (or self-reported health) of the general population (the only admissible “sub-populations” were those based on gender and/or age); (b) the study employed statistical techniques (calculating correlations, at the very least) and was not purely descriptive or theoretical in nature; (c) the analysis involved at least two countries or at least two regions or cities (or another aggregate level) in at least two different countries; (d) the health outcome was not differentiated according to some socio-economic factor and thus studied in terms of inequality (with the exception of gender and age differentiations); (e) mortality, in case it was one of the health indicators under investigation, was strictly “total” or “all-cause” (no cause-specific or determinant-attributable mortality).

Data extraction

The following pieces of information were extracted in an Excel table from the full text of each eligible study (primarily by VV, consulting with PB in case of doubt): health outcome(s), determinants, statistical methodology, level of analysis, results, type of data, data sources, time period, countries. The evidence is synthesized according to these extracted data (often directly reflected in the section headings), using a narrative form accompanied by a “summary-of-findings” table and a graph.

Search and selection

The initial yield contained 4583 records, reduced to 3686 after removal of duplicates ( Fig 1 ). Based on title and abstract screening, 3271 records were excluded because they focused on specific medical condition(s) or specific populations (based on morbidity or some other factor), dealt with intervention effectiveness, with theoretical or non-health related issues, or with animals or plants. Of the remaining 415 papers, roughly half were disqualified upon full-text consideration, mostly due to using an outcome not of interest to us (e.g., health inequality), measuring and analyzing determinants and outcomes exclusively at the individual level, performing analyses one country at a time, employing indices that are a mixture of both health indicators and health determinants, or not utilizing potential health determinants at all. After this second stage of the screening process, 202 papers were deemed eligible for inclusion. This group was further dichotomized according to level of economic development of the countries or regions under study, using membership of the OECD or Europe as a reference “cut-off” point. Sixty papers were judged to include high-income countries, and the remaining 142 included either low- or middle-income countries or a mix of both these levels of development. The rest of this report outlines findings in relation to high-income countries only, reflecting our own primary research interests. Nonetheless, we chose to report our search yield for the other income groups for two reasons. First, to gauge the relative interest in applied published research for these different income levels; and second, to enable other researchers with a focus on determinants of health in other countries to use the extraction we made here.


  • PPT PowerPoint slide
  • PNG larger image
  • TIFF original image

Health outcomes

The most frequent population health indicator, life expectancy (LE), was present in 24 of the 60 studies. Apart from “life expectancy at birth” (representing the average life-span a newborn is expected to have if current mortality rates remain constant), also called “period LE” by some [ 19 , 20 ], we encountered as well LE at 40 years of age [ 21 ], at 60 [ 22 ], and at 65 [ 21 , 23 , 24 ]. In two papers, the age-specificity of life expectancy (be it at birth or another age) was not stated [ 25 , 26 ].

Some studies considered male and female LE separately [ 21 , 24 , 25 , 27 – 33 ]. This consideration was also often observed with the second most commonly used health index [ 28 – 30 , 34 – 38 ]–termed “total”, or “overall”, or “all-cause”, mortality rate (MR)–included in 22 of the 60 studies. In addition to gender, this index was also sometimes broken down according to age group [ 30 , 39 , 40 ], as well as gender-age group [ 38 ].

While the majority of studies under review here focused on a single health indicator, 23 out of the 60 studies made use of multiple outcomes, although these outcomes were always considered one at a time, and sometimes not all of them fell within the scope of our review. An easily discernable group of indices that typically went together [ 25 , 37 , 41 ] was that of neonatal (deaths occurring within 28 days postpartum), perinatal (fetal or early neonatal / first-7-days deaths), and post-neonatal (deaths between the 29 th day and completion of one year of life) mortality. More often than not, these indices were also accompanied by “stand-alone” indicators, such as infant mortality (deaths within the first year of life; our third most common index found in 16 of the 60 studies), maternal mortality (deaths during pregnancy or within 42 days of termination of pregnancy), and child mortality rates. Child mortality has conventionally been defined as mortality within the first 5 years of life, thus often also called “under-5 mortality”. Nonetheless, Pritchard & Wallace used the term “child mortality” to denote deaths of children younger than 14 years [ 42 ].

As previously stated, inclusion criteria did allow for self-reported health status to be used as a general measure of population health. Within our final selection of studies, seven utilized some form of subjective health as an outcome variable [ 25 , 43 – 48 ]. Additionally, the Health Human Development Index [ 49 ], healthy life expectancy [ 50 ], old-age survival [ 51 ], potential years of life lost [ 52 ], and disability-adjusted life expectancy [ 25 ] were also used.

We note that while in most cases the indicators mentioned above (and/or the covariates considered, see below) were taken in their absolute or logarithmic form, as a—typically annual—number, sometimes they were used in the form of differences, change rates, averages over a given time period, or even z-scores of rankings [ 19 , 22 , 40 , 42 , 44 , 53 – 57 ].

Regions, countries, and populations

Despite our decision to confine this review to high-income countries, some variation in the countries and regions studied was still present. Selection seemed to be most often conditioned on the European Union, or the European continent more generally, and the Organisation of Economic Co-operation and Development (OECD), though, typically, not all member nations–based on the instances where these were also explicitly listed—were included in a given study. Some of the stated reasons for omitting certain nations included data unavailability [ 30 , 45 , 54 ] or inconsistency [ 20 , 58 ], Gross Domestic Product (GDP) too low [ 40 ], differences in economic development and political stability with the rest of the sampled countries [ 59 ], and national population too small [ 24 , 40 ]. On the other hand, the rationales for selecting a group of countries included having similar above-average infant mortality [ 60 ], similar healthcare systems [ 23 ], and being randomly drawn from a social spending category [ 61 ]. Some researchers were interested explicitly in a specific geographical region, such as Eastern Europe [ 50 ], Central and Eastern Europe [ 48 , 60 ], the Visegrad (V4) group [ 62 ], or the Asia/Pacific area [ 32 ]. In certain instances, national regions or cities, rather than countries, constituted the units of investigation instead [ 31 , 51 , 56 , 62 – 66 ]. In two particular cases, a mix of countries and cities was used [ 35 , 57 ]. In another two [ 28 , 29 ], due to the long time periods under study, some of the included countries no longer exist. Finally, besides “European” and “OECD”, the terms “developed”, “Western”, and “industrialized” were also used to describe the group of selected nations [ 30 , 42 , 52 , 53 , 67 ].

As stated above, it was the health status of the general population that we were interested in, and during screening we made a concerted effort to exclude research using data based on a more narrowly defined group of individuals. All studies included in this review adhere to this general rule, albeit with two caveats. First, as cities (even neighborhoods) were the unit of analysis in three of the studies that made the selection [ 56 , 64 , 65 ], the populations under investigation there can be more accurately described as general urban , instead of just general. Second, oftentimes health indicators were stratified based on gender and/or age, therefore we also admitted one study that, due to its specific research question, focused on men and women of early retirement age [ 35 ] and another that considered adult males only [ 68 ].

Data types and sources

A great diversity of sources was utilized for data collection purposes. The accessible reference databases of the OECD ( ), WHO ( ), World Bank ( ), United Nations ( ), and Eurostat ( ) were among the top choices. The other international databases included Human Mortality [ 30 , 39 , 50 ], Transparency International [ 40 , 48 , 50 ], Quality of Government [ 28 , 69 ], World Income Inequality [ 30 ], International Labor Organization [ 41 ], International Monetary Fund [ 70 ]. A number of national databases were referred to as well, for example the US Bureau of Statistics [ 42 , 53 ], Korean Statistical Information Services [ 67 ], Statistics Canada [ 67 ], Australian Bureau of Statistics [ 67 ], and Health New Zealand Tobacco control and Health New Zealand Food and Nutrition [ 19 ]. Well-known surveys, such as the World Values Survey [ 25 , 55 ], the European Social Survey [ 25 , 39 , 44 ], the Eurobarometer [ 46 , 56 ], the European Value Survey [ 25 ], and the European Statistics of Income and Living Condition Survey [ 43 , 47 , 70 ] were used as data sources, too. Finally, in some cases [ 25 , 28 , 29 , 35 , 36 , 41 , 69 ], built-for-purpose datasets from previous studies were re-used.

In most of the studies, the level of the data (and analysis) was national. The exceptions were six papers that dealt with Nomenclature of Territorial Units of Statistics (NUTS2) regions [ 31 , 62 , 63 , 66 ], otherwise defined areas [ 51 ] or cities [ 56 ], and seven others that were multilevel designs and utilized both country- and region-level data [ 57 ], individual- and city- or country-level [ 35 ], individual- and country-level [ 44 , 45 , 48 ], individual- and neighborhood-level [ 64 ], and city-region- (NUTS3) and country-level data [ 65 ]. Parallel to that, the data type was predominantly longitudinal, with only a few studies using purely cross-sectional data [ 25 , 33 , 43 , 45 – 48 , 50 , 62 , 67 , 68 , 71 , 72 ], albeit in four of those [ 43 , 48 , 68 , 72 ] two separate points in time were taken (thus resulting in a kind of “double cross-section”), while in another the averages across survey waves were used [ 56 ].

In studies using longitudinal data, the length of the covered time periods varied greatly. Although this was almost always less than 40 years, in one study it covered the entire 20 th century [ 29 ]. Longitudinal data, typically in the form of annual records, was sometimes transformed before usage. For example, some researchers considered data points at 5- [ 34 , 36 , 49 ] or 10-year [ 27 , 29 , 35 ] intervals instead of the traditional 1, or took averages over 3-year periods [ 42 , 53 , 73 ]. In one study concerned with the effect of the Great Recession all data were in a “recession minus expansion change in trends”-form [ 57 ]. Furthermore, there were a few instances where two different time periods were compared to each other [ 42 , 53 ] or when data was divided into 2 to 4 (possibly overlapping) periods which were then analyzed separately [ 24 , 26 , 28 , 29 , 31 , 65 ]. Lastly, owing to data availability issues, discrepancies between the time points or periods of data on the different variables were occasionally observed [ 22 , 35 , 42 , 53 – 55 , 63 ].

Health determinants

Together with other essential details, Table 1 lists the health correlates considered in the selected studies. Several general categories for these correlates can be discerned, including health care, political stability, socio-economics, demographics, psychology, environment, fertility, life-style, culture, labor. All of these, directly or implicitly, have been recognized as holding importance for population health by existing theoretical models of (social) determinants of health [ 74 – 77 ].


It is worth noting that in a few studies there was just a single aggregate-level covariate investigated in relation to a health outcome of interest to us. In one instance, this was life satisfaction [ 44 ], in another–welfare system typology [ 45 ], but also gender inequality [ 33 ], austerity level [ 70 , 78 ], and deprivation [ 51 ]. Most often though, attention went exclusively to GDP [ 27 , 29 , 46 , 57 , 65 , 71 ]. It was often the case that research had a more particular focus. Among others, minimum wages [ 79 ], hospital payment schemes [ 23 ], cigarette prices [ 63 ], social expenditure [ 20 ], residents’ dissatisfaction [ 56 ], income inequality [ 30 , 69 ], and work leave [ 41 , 58 ] took center stage. Whenever variables outside of these specific areas were also included, they were usually identified as confounders or controls, moderators or mediators.

We visualized the combinations in which the different determinants have been studied in Fig 2 , which was obtained via multidimensional scaling and a subsequent cluster analysis (details outlined in S2 Appendix ). It depicts the spatial positioning of each determinant relative to all others, based on the number of times the effects of each pair of determinants have been studied simultaneously. When interpreting Fig 2 , one should keep in mind that determinants marked with an asterisk represent, in fact, collectives of variables.


Groups of determinants are marked by asterisks (see S1 Table in S1 Appendix ). Diminishing color intensity reflects a decrease in the total number of “connections” for a given determinant. Noteworthy pairwise “connections” are emphasized via lines (solid-dashed-dotted indicates decreasing frequency). Grey contour lines encircle groups of variables that were identified via cluster analysis. Abbreviations: age = population age distribution, associations = membership in associations, AT-index = atherogenic-thrombogenic index, BR = birth rate, CAPB = Cyclically Adjusted Primary Balance, civilian-labor = civilian labor force, C-section = Cesarean delivery rate, credit-info = depth of credit information, dissatisf = residents’ dissatisfaction, distrib.orient = distributional orientation, EDU = education, eHealth = eHealth index at GP-level, exch.rate = exchange rate, fat = fat consumption, GDP = gross domestic product, GFCF = Gross Fixed Capital Formation/Creation, GH-gas = greenhouse gas, GII = gender inequality index, gov = governance index, gov.revenue = government revenues, HC-coverage = healthcare coverage, HE = health(care) expenditure, HHconsump = household consumption, hosp.beds = hospital beds, hosp.payment = hospital payment scheme, hosp.stay = length of hospital stay, IDI = ICT development index, inc.ineq = income inequality, industry-labor = industrial labor force, infant-sex = infant sex ratio, labor-product = labor production, LBW = low birth weight, leave = work leave, life-satisf = life satisfaction, M-age = maternal age, marginal-tax = marginal tax rate, MDs = physicians, mult.preg = multiple pregnancy, NHS = Nation Health System, NO = nitrous oxide emissions, PM10 = particulate matter (PM10) emissions, pop = population size, pop.density = population density, pre-term = pre-term birth rate, prison = prison population, researchE = research&development expenditure, school.ref = compulsory schooling reform, smoke-free = smoke-free places, SO = sulfur oxide emissions, soc.E = social expenditure, soc.workers = social workers, sugar = sugar consumption, terror = terrorism, union = union density, UR = unemployment rate, urban = urbanization, veg-fr = vegetable-and-fruit consumption, welfare = welfare regime, Wwater = wastewater treatment.

Distances between determinants in Fig 2 are indicative of determinants’ “connectedness” with each other. While the statistical procedure called for higher dimensionality of the model, for demonstration purposes we show here a two-dimensional solution. This simplification unfortunately comes with a caveat. To use the factor smoking as an example, it would appear it stands at a much greater distance from GDP than it does from alcohol. In reality however, smoking was considered together with alcohol consumption [ 21 , 25 , 26 , 52 , 68 ] in just as many studies as it was with GDP [ 21 , 25 , 26 , 52 , 59 ], five. To aid with respect to this apparent shortcoming, we have emphasized the strongest pairwise links. Solid lines connect GDP with health expenditure (HE), unemployment rate (UR), and education (EDU), indicating that the effect of GDP on health, taking into account the effects of the other three determinants as well, was evaluated in between 12 to 16 studies of the 60 included in this review. Tracing the dashed lines, we can also tell that GDP appeared jointly with income inequality, and HE together with either EDU or UR, in anywhere between 8 to 10 of our selected studies. Finally, some weaker but still worth-mentioning “connections” between variables are displayed as well via the dotted lines.

The fact that all notable pairwise “connections” are concentrated within a relatively small region of the plot may be interpreted as low overall “connectedness” among the health indicators studied. GDP is the most widely investigated determinant in relation to general population health. Its total number of “connections” is disproportionately high (159) compared to its runner-up–HE (with 113 “connections”), and then subsequently EDU (with 90) and UR (with 86). In fact, all of these determinants could be thought of as outliers, given that none of the remaining factors have a total count of pairings above 52. This decrease in individual determinants’ overall “connectedness” can be tracked on the graph via the change of color intensity as we move outwards from the symbolic center of GDP and its closest “co-determinants”, to finally reach the other extreme of the ten indicators (welfare regime, household consumption, compulsory school reform, life satisfaction, government revenues, literacy, research expenditure, multiple pregnancy, Cyclically Adjusted Primary Balance, and residents’ dissatisfaction; in white) the effects on health of which were only studied in isolation.

Lastly, we point to the few small but stable clusters of covariates encircled by the grey bubbles on Fig 2 . These groups of determinants were identified as “close” by both statistical procedures used for the production of the graph (see details in S2 Appendix ).

Statistical methodology

There was great variation in the level of statistical detail reported. Some authors provided too vague a description of their analytical approach, necessitating some inference in this section.

The issue of missing data is a challenging reality in this field of research, but few of the studies under review (12/60) explain how they dealt with it. Among the ones that do, three general approaches to handling missingness can be identified, listed in increasing level of sophistication: case-wise deletion, i.e., removal of countries from the sample [ 20 , 45 , 48 , 58 , 59 ], (linear) interpolation [ 28 , 30 , 34 , 58 , 59 , 63 ], and multiple imputation [ 26 , 41 , 52 ].

Correlations, Pearson, Spearman, or unspecified, were the only technique applied with respect to the health outcomes of interest in eight analyses [ 33 , 42 – 44 , 46 , 53 , 57 , 61 ]. Among the more advanced statistical methods, the family of regression models proved to be, by and large, predominant. Before examining this closer, we note the techniques that were, in a way, “unique” within this selection of studies: meta-analyses were performed (random and fixed effects, respectively) on the reduced form and 2-sample two stage least squares (2SLS) estimations done within countries [ 39 ]; difference-in-difference (DiD) analysis was applied in one case [ 23 ]; dynamic time-series methods, among which co-integration, impulse-response function (IRF), and panel vector autoregressive (VAR) modeling, were utilized in one study [ 80 ]; longitudinal generalized estimating equation (GEE) models were developed on two occasions [ 70 , 78 ]; hierarchical Bayesian spatial models [ 51 ] and special autoregressive regression [ 62 ] were also implemented.

Purely cross-sectional data analyses were performed in eight studies [ 25 , 45 , 47 , 50 , 55 , 56 , 67 , 71 ]. These consisted of linear regression (assumed ordinary least squares (OLS)), generalized least squares (GLS) regression, and multilevel analyses. However, six other studies that used longitudinal data in fact had a cross-sectional design, through which they applied regression at multiple time-points separately [ 27 , 29 , 36 , 48 , 68 , 72 ].

Apart from these “multi-point cross-sectional studies”, some other simplistic approaches to longitudinal data analysis were found, involving calculating and regressing 3-year averages of both the response and the predictor variables [ 54 ], taking the average of a few data-points (i.e., survey waves) [ 56 ] or using difference scores over 10-year [ 19 , 29 ] or unspecified time intervals [ 40 , 55 ].

Moving further in the direction of more sensible longitudinal data usage, we turn to the methods widely known among (health) economists as “panel data analysis” or “panel regression”. Most often seen were models with fixed effects for country/region and sometimes also time-point (occasionally including a country-specific trend as well), with robust standard errors for the parameter estimates to take into account correlations among clustered observations [ 20 , 21 , 24 , 28 , 30 , 32 , 34 , 37 , 38 , 41 , 52 , 59 , 60 , 63 , 66 , 69 , 73 , 79 , 81 , 82 ]. The Hausman test [ 83 ] was sometimes mentioned as the tool used to decide between fixed and random effects [ 26 , 49 , 63 , 66 , 73 , 82 ]. A few studies considered the latter more appropriate for their particular analyses, with some further specifying that (feasible) GLS estimation was employed [ 26 , 34 , 49 , 58 , 60 , 73 ]. Apart from these two types of models, the first differences method was encountered once as well [ 31 ]. Across all, the error terms were sometimes assumed to come from a first-order autoregressive process (AR(1)), i.e., they were allowed to be serially correlated [ 20 , 30 , 38 , 58 – 60 , 73 ], and lags of (typically) predictor variables were included in the model specification, too [ 20 , 21 , 37 , 38 , 48 , 69 , 81 ]. Lastly, a somewhat different approach to longitudinal data analysis was undertaken in four studies [ 22 , 35 , 48 , 65 ] in which multilevel–linear or Poisson–models were developed.

Regardless of the exact techniques used, most studies included in this review presented multiple model applications within their main analysis. None attempted to formally compare models in order to identify the “best”, even if goodness-of-fit statistics were occasionally reported. As indicated above, many studies investigated women’s and men’s health separately [ 19 , 21 , 22 , 27 – 29 , 31 , 33 , 35 , 36 , 38 , 39 , 45 , 50 , 51 , 64 , 65 , 69 , 82 ], and covariates were often tested one at a time, including other covariates only incrementally [ 20 , 25 , 28 , 36 , 40 , 50 , 55 , 67 , 73 ]. Furthermore, there were a few instances where analyses within countries were performed as well [ 32 , 39 , 51 ] or where the full time period of interest was divided into a few sub-periods [ 24 , 26 , 28 , 31 ]. There were also cases where different statistical techniques were applied in parallel [ 29 , 55 , 60 , 66 , 69 , 73 , 82 ], sometimes as a form of sensitivity analysis [ 24 , 26 , 30 , 58 , 73 ]. However, the most common approach to sensitivity analysis was to re-run models with somewhat different samples [ 39 , 50 , 59 , 67 , 69 , 80 , 82 ]. Other strategies included different categorization of variables or adding (more/other) controls [ 21 , 23 , 25 , 28 , 37 , 50 , 63 , 69 ], using an alternative main covariate measure [ 59 , 82 ], including lags for predictors or outcomes [ 28 , 30 , 58 , 63 , 65 , 79 ], using weights [ 24 , 67 ] or alternative data sources [ 37 , 69 ], or using non-imputed data [ 41 ].

As the methods and not the findings are the main focus of the current review, and because generic checklists cannot discern the underlying quality in this application field (see also below), we opted to pool all reported findings together, regardless of individual study characteristics or particular outcome(s) used, and speak generally of positive and negative effects on health. For this summary we have adopted the 0.05-significance level and only considered results from multivariate analyses. Strictly birth-related factors are omitted since these potentially only relate to the group of infant mortality indicators and not to any of the other general population health measures.

Starting with the determinants most often studied, higher GDP levels [ 21 , 26 , 27 , 29 , 30 , 32 , 43 , 48 , 52 , 58 , 60 , 66 , 67 , 73 , 79 , 81 , 82 ], higher health [ 21 , 37 , 47 , 49 , 52 , 58 , 59 , 68 , 72 , 82 ] and social [ 20 , 21 , 26 , 38 , 79 ] expenditures, higher education [ 26 , 39 , 52 , 62 , 72 , 73 ], lower unemployment [ 60 , 61 , 66 ], and lower income inequality [ 30 , 42 , 53 , 55 , 73 ] were found to be significantly associated with better population health on a number of occasions. In addition to that, there was also some evidence that democracy [ 36 ] and freedom [ 50 ], higher work compensation [ 43 , 79 ], distributional orientation [ 54 ], cigarette prices [ 63 ], gross national income [ 22 , 72 ], labor productivity [ 26 ], exchange rates [ 32 ], marginal tax rates [ 79 ], vaccination rates [ 52 ], total fertility [ 59 , 66 ], fruit and vegetable [ 68 ], fat [ 52 ] and sugar consumption [ 52 ], as well as bigger depth of credit information [ 22 ] and percentage of civilian labor force [ 79 ], longer work leaves [ 41 , 58 ], more physicians [ 37 , 52 , 72 ], nurses [ 72 ], and hospital beds [ 79 , 82 ], and also membership in associations, perceived corruption and societal trust [ 48 ] were beneficial to health. Higher nitrous oxide (NO) levels [ 52 ], longer average hospital stay [ 48 ], deprivation [ 51 ], dissatisfaction with healthcare and the social environment [ 56 ], corruption [ 40 , 50 ], smoking [ 19 , 26 , 52 , 68 ], alcohol consumption [ 26 , 52 , 68 ] and illegal drug use [ 68 ], poverty [ 64 ], higher percentage of industrial workers [ 26 ], Gross Fixed Capital creation [ 66 ] and older population [ 38 , 66 , 79 ], gender inequality [ 22 ], and fertility [ 26 , 66 ] were detrimental.

It is important to point out that the above-mentioned effects could not be considered stable either across or within studies. Very often, statistical significance of a given covariate fluctuated between the different model specifications tried out within the same study [ 20 , 49 , 59 , 66 , 68 , 69 , 73 , 80 , 82 ], testifying to the importance of control variables and multivariate research (i.e., analyzing multiple independent variables simultaneously) in general. Furthermore, conflicting results were observed even with regards to the “core” determinants given special attention, so to speak, throughout this text. Thus, some studies reported negative effects of health expenditure [ 32 , 82 ], social expenditure [ 58 ], GDP [ 49 , 66 ], and education [ 82 ], and positive effects of income inequality [ 82 ] and unemployment [ 24 , 31 , 32 , 52 , 66 , 68 ]. Interestingly, one study [ 34 ] differentiated between temporary and long-term effects of GDP and unemployment, alluding to possibly much greater complexity of the association with health. It is also worth noting that some gender differences were found, with determinants being more influential for males than for females, or only having statistically significant effects for male health [ 19 , 21 , 28 , 34 , 36 , 37 , 39 , 64 , 65 , 69 ].

The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries.

Measuring population health via relatively simple mortality-based indicators still seems to be the state of the art. What is more, these indicators are routinely considered one at a time, instead of, for example, employing existing statistical procedures to devise a more general, composite, index of population health, or using some of the established indices, such as disability-adjusted life expectancy (DALE) or quality-adjusted life expectancy (QALE). Although strong arguments for their wider use were already voiced decades ago [ 84 ], such summary measures surface only rarely in this research field.

On a related note, the greater data availability and accessibility that we enjoy today does not automatically equate to data quality. Nonetheless, this is routinely assumed in aggregate level studies. We almost never encountered a discussion on the topic. The non-mundane issue of data missingness, too, goes largely underappreciated. With all recent methodological advancements in this area [ 85 – 88 ], there is no excuse for ignorance; and still, too few of the reviewed studies tackled the matter in any adequate fashion.

Much optimism can be gained considering the abundance of different determinants that have attracted researchers’ attention in relation to population health. We took on a visual approach with regards to these determinants and presented a graph that links spatial distances between determinants with frequencies of being studies together. To facilitate interpretation, we grouped some variables, which resulted in some loss of finer detail. Nevertheless, the graph is helpful in exemplifying how many effects continue to be studied in a very limited context, if any. Since in reality no factor acts in isolation, this oversimplification practice threatens to render the whole exercise meaningless from the outset. The importance of multivariate analysis cannot be stressed enough. While there is no “best method” to be recommended and appropriate techniques vary according to the specifics of the research question and the characteristics of the data at hand [ 89 – 93 ], in the future, in addition to abandoning simplistic univariate approaches, we hope to see a shift from the currently dominating fixed effects to the more flexible random/mixed effects models [ 94 ], as well as wider application of more sophisticated methods, such as principle component regression, partial least squares, covariance structure models (e.g., structural equations), canonical correlations, time-series, and generalized estimating equations.

Finally, there are some limitations of the current scoping review. We searched the two main databases for published research in medical and non-medical sciences (PubMed and Web of Science) since 2013, thus potentially excluding publications and reports that are not indexed in these databases, as well as older indexed publications. These choices were guided by our interest in the most recent (i.e., the current state-of-the-art) and arguably the highest-quality research (i.e., peer-reviewed articles, primarily in indexed non-predatory journals). Furthermore, despite holding a critical stance with regards to some aspects of how determinants-of-health research is currently conducted, we opted out of formally assessing the quality of the individual studies included. The reason for that is two-fold. On the one hand, we are unaware of the existence of a formal and standard tool for quality assessment of ecological designs. And on the other, we consider trying to score the quality of these diverse studies (in terms of regional setting, specific topic, outcome indices, and methodology) undesirable and misleading, particularly since we would sometimes have been rating the quality of only a (small) part of the original studies—the part that was relevant to our review’s goal.

Our aim was to investigate the current state of research on the very broad and general topic of population health, specifically, the way it has been examined in a multi-country context. We learned that data treatment and analytical approach were, in the majority of these recent studies, ill-equipped or insufficiently transparent to provide clarity regarding the underlying mechanisms of population health in high-income countries. Whether due to methodological shortcomings or the inherent complexity of the topic, research so far fails to provide any definitive answers. It is our sincere belief that with the application of more advanced analytical techniques this continuous quest could come to fruition sooner.

Supporting information

S1 checklist. preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (prisma-scr) checklist..

S1 Appendix.

S2 Appendix.

  • View Article
  • Google Scholar
  • PubMed/NCBI
  • 75. Dahlgren G, Whitehead M. Policies and Strategies to Promote Equity in Health. Stockholm, Sweden: Institute for Future Studies; 1991.
  • 76. Brunner E, Marmot M. Social Organization, Stress, and Health. In: Marmot M, Wilkinson RG, editors. Social Determinants of Health. Oxford, England: Oxford University Press; 1999.
  • 77. Najman JM. A General Model of the Social Origins of Health and Well-being. In: Eckersley R, Dixon J, Douglas B, editors. The Social Origins of Health and Well-being. Cambridge, England: Cambridge University Press; 2001.
  • 85. Carpenter JR, Kenward MG. Multiple Imputation and its Application. New York: John Wiley & Sons; 2013.
  • 86. Molenberghs G, Fitzmaurice G, Kenward MG, Verbeke G, Tsiatis AA. Handbook of Missing Data Methodology. Boca Raton: Chapman & Hall/CRC; 2014.
  • 87. van Buuren S. Flexible Imputation of Missing Data. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2018.
  • 88. Enders CK. Applied Missing Data Analysis. New York: Guilford; 2010.
  • 89. Shayle R. Searle GC, Charles E. McCulloch. Variance Components: John Wiley & Sons, Inc.; 1992.
  • 90. Agresti A. Foundations of Linear and Generalized Linear Models. Hoboken, New Jersey: John Wiley & Sons Inc.; 2015.
  • 91. Leyland A. H. (Editor) HGE. Multilevel Modelling of Health Statistics: John Wiley & Sons Inc; 2001.
  • 92. Garrett Fitzmaurice MD, Geert Verbeke, Geert Molenberghs. Longitudinal Data Analysis. New York: Chapman and Hall/CRC; 2008.
  • 93. Wolfgang Karl Härdle LS. Applied Multivariate Statistical Analysis. Berlin, Heidelberg: Springer; 2015.

Quantitative medicine: Tracing the transition from holistic to reductionist approaches. A new "quantitative holism" is possible?


  • 1 University of Cagliari, Cagliari, Italy.
  • PMID: 37361238
  • PMCID: PMC10286173
  • DOI: 10.1177/22799036231182271

The practice of medicine has evolved significantly over time, from a more holistic to a reductionist or mechanistic approach. This paper briefly traces the history of medicine and the transition to quantitative medicine, which has enabled more personalized and targeted treatments, and improved understanding of the underlying biological mechanisms of disease. However, this shift has also presented some challenges and criticisms, including the danger of losing sight of the patient as a unique, whole individual. This paper explores the underlying principles and key contributions of quantitative medicine, as well as the context for its rise, including the development of new technologies and the influence of reductionist philosophies. The challenges and criticisms of this approach, and the need to balance reductionist and holistic approaches in order to achieve a comprehensive understanding of human health will be discussed. Ultimately, by integrating insights from philosophy, physics, and other fields, we may be able to develop new and innovative approaches that bridge the gap between reductionism and holism and improve patient outcomes with the new "quantitative holism."

Keywords: Medicine; evolution; holism; philosophy; quantitative medicine.

© The Author(s) 2023.

Quantitative Methods in Global Health Research

  • Living reference work entry
  • First Online: 24 November 2020
  • Cite this living reference work entry

Book cover

  • Jamalludin Ab Rahman 5  

78 Accesses

Quantitative research is the foundation for evidence-based global health practice and interventions. Preparing health research starts with a clear research question to initiate the study, careful planning using sound methodology as well as the development and management of the capacity and resources to complete the whole research cycle. Good planning will also ensure valid research outcomes. Quantitative research emphasizes a clear target population, proper sampling techniques, adequate sample size, detailed planning for data collection, and proper statistical analysis. This chapter provides an overview of quantitative research methods, explains relevant study designs, and presents considerations on all aspect of the research cycle along four phases: initiation, planning, data collection, and reporting phase.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Bourque LB, Clark VA (1992) Quantitative applications in the social sciences: processing data. Hist Soc Res.

Ford I, Blauw GJ, Murphy MB, Shepherd J, Cobbe SM, Bollen ELEM et al (2002) A prospective study of pravastatin in the elderly at risk (PROSPER): screening experience and baseline characteristics. Curr Control Trials Cardiovasc Med 3(1):8–8.

Article   PubMed   PubMed Central   Google Scholar  

Fricker R (2017) Sampling methods for online surveys. In: The SAGE handbook of online research methods, pp 162–183.

Chapter   Google Scholar  

Hennessy S, Bilker WB, Berlin JA, Strom BL (1999) Factors influencing the optimal control-to-case ratio in matched case-control studies. Am J Epidemiol 149(2):195–197

Article   CAS   Google Scholar  

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (2019) Cochrane handbook for systematic reviews of interventions. Wiley, Hoboken

Book   Google Scholar  

Hill AB (1965) The environment and disease: association or causation? Proc R Soc Med 58(5):295–300. Retrieved from .

CAS   PubMed   PubMed Central   Google Scholar  

Hu FB, Willett WC, Manson JE, Colditz GA, Rimm EB, Speizer FE et al (2000) Snoring and risk of cardiovascular disease in women. J Am Coll Cardiol 35(2):308–313.

Article   CAS   PubMed   Google Scholar  

Koplan JP, Bond TC, Merson MH, Reddy KS, Rodriguez MH, Sewankambo NK, Wasserheit JN (2009) Towards a common definition of global health. Lancet 373(9679):1993–1995.

Article   PubMed   Google Scholar  

Luk JW, Wang J, Simons-Morton BG (2010) Bullying victimisation and substance use among U.S. adolescents: mediation by depression. Prev Sci: The Off J Soc Prev Res 11(4):355–359.

Article   Google Scholar  

O’Donnell MJ, Chin SL, Rangarajan S, Xavier D, Liu L, Zhang H et al (2016) Global and regional effects of potentially modifiable risk factors associated with acute stroke in 32 countries (INTERSTROKE): a case-control study. Lancet 388(10046):761–775.

Palipudi KM, Morton J, Hsia J, Andes L, Asma S, Talley B et al (2016) Methodology of the global adult tobacco survey – 2008–2010. Global Heal Prom 23(2_suppl):3–23.

Preece DA (1987) Good statistical practice. J Roy Stat Soc Ser D (The Statistician) 36(4):397–408.

Sajedinejad S, Majdzadeh R, Vedadhir A, Tabatabaei MG, Mohammad K (2015) Maternal mortality: a cross-sectional study in global health. Glob Health 11:4.

Scandinavian Simvastatin Survival Study, G (1994) Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S). Lancet 344(8934):1383–1389.

Shepherd J, Blauw GJ, Murphy MB, Bollen EL, Buckley BM, Cobbe SM et al (2002) Pravastatin in elderly individuals at risk of vascular disease (PROSPER): a randomised controlled trial. Lancet 360(9346):1623–1630.

Souza JP, Betran AP, Dumont A, de Mucio B, Gibbs Pickens CM, Deneux-Tharaux C et al (2016) A global reference for caesarean section rates (C-Model): a multicountry cross-sectional study. BJOG 123(3):427–436.

Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A et al (2016) The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3(1):1–9

Worthington R (2016) Best practices for designing data collection forms. Retrieved from

Google Scholar  

Download references

Author information

Authors and affiliations.

Kulliyyah of Medicine, International Islamic University Malaysia, Kuantan, Pahang, Malaysia

Jamalludin Ab Rahman

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jamalludin Ab Rahman .

Editor information

Editors and affiliations.

Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC, Australia

Robin Haring

Global Health Programme, Graduate Institute of International and, Geneva, Switzerland

Ilona Kickbusch

Charite-Univ.-Med. Berlin, Berlin, Berlin, Germany

Detlev Ganten

Africa, Brazzaville, Rep of the Congo, World Health Organization Regional Offic, Brazzaville, Congo, Demographic Republic (Zaire)

Matshidiso Moeti

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editors and the World Health Organization

About this entry

Cite this entry.

Ab Rahman, J. (2021). Quantitative Methods in Global Health Research. In: Haring, R., Kickbusch, I., Ganten, D., Moeti, M. (eds) Handbook of Global Health. Springer, Cham.

Download citation


Received : 24 August 2020

Accepted : 24 August 2020

Published : 24 November 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-05325-3

Online ISBN : 978-3-030-05325-3

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research
  • Open access
  • Published: 01 April 2022

Assessing changes in the quality of quantitative health educations research: a perspective from communities of practice

  • Katherine M. Wright   ORCID: 1 ,
  • Larry D. Gruppen   ORCID: 2 ,
  • Kevin W. Kuo 3 ,
  • Andrew Muzyk   ORCID: 4 ,
  • Jeffry Nahmias 5 ,
  • Darcy A. Reed 6 ,
  • Gurjit Sandhu 7 ,
  • Anita V. Shelgikar   ORCID: 8 ,
  • Jennifer N. Stojan 9 ,
  • Toshiko L. Uchida   ORCID: 10 ,
  • Rebecca Wallihan 11 &
  • Larry Hurtubise   ORCID: 12  

BMC Medical Education volume  22 , Article number:  227 ( 2022 ) Cite this article

2258 Accesses

9 Altmetric

Metrics details

As a community of practice (CoP), medical education depends on its research literature to communicate new knowledge, examine alternative perspectives, and share methodological innovations. As a key route of communication, the medical education CoP must be concerned about the rigor and validity of its research literature, but prior studies have suggested the need to improve medical education research quality. Of concern in the present study is the question of how responsive the medical education research literature is to changes in the CoP. We examine the nature and extent of changes in the quality of medical education research over a decade, using a widely cited study of research quality in the medical education research literature as a benchmark to compare more recent quality indicators.

A bibliometric analysis was conducted to examine the methodologic quality of quantitative medical education research studies published in 13 selected journals from September 2013 to December 2014. Quality scores were calculated for 482 medical education studies using a 10-item Medical Education Research Study Quality Instrument (MERSQI) that has demonstrated strong validity evidence. These data were compared with data from the original study for the same journals in the period September 2002 to December 2003. Eleven investigators representing 6 academic medical centers reviewed and scored the research studies that met inclusion and exclusion criteria. Primary outcome measures include MERSQI quality indicators for 6 domains: study design, sampling, type of data, validity, data analysis, and outcomes.

There were statistically significant improvements in four sub-domain measures: study design, type of data, validity and outcomes. There were no changes in sampling quality or the appropriateness of data analysis methods. There was a small but significant increase in the use of patient outcomes in these studies.


Overall, we judge this as equivocal evidence for the responsiveness of the research literature to changes in the medical education CoP. This study identified areas of strength as well as opportunities for continued development of medical education research.

Peer Review reports

Health professions education is a landscape of practice made up of multiple Communities of Practice (CoP) [ 1 , 2 , 3 , 4 ]. CoP are groups of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis. Barab, Barnett, and Squire stress that CoPs are persistent and develop mutual professional values and shared history [ 5 ].

Published research literature is clearly a critical component of an academic CoP. The scholarly literature reflects the three components of a CoP [ 2 ]. First, the published literature reflects the domain of a CoP. The domain is the common ground of relevant problems, topics of interest, knowledge, and practice that define the contributions and participation of members of the community. The domain has boundaries that help define the community as well as ‘leading edges’ for expanding or redirecting the domain.

Secondly, the literature reflects the community and social fabric of the CoP. As a vehicle for communication, the literature enables shared ideas, knowledge, and priorities. It also reflects the social networks within the community through collaborations and citations [ 6 ]. Thirdly, scholarly publications serve as a repository and resource of community practice. The literature is particularly important for identifying new techniques and methods, theoretical perspectives, findings, and language for the community.

CoPs change over time as new members enter into the core of the community and older members leave. They change as the domain of the community shifts and grows or shrinks (becomes more specialized). Changes in practice also changes the CoP. Many changes in the medical education CoP can be identified: the recent emphasis on competency-based education [ 7 , 8 ], newer models of faculty development [ 9 ], the comings and goings of different curricular models (systems-based, problem-based, team-based), the shift in a predominantly male community in the 1970s to an increasingly gender diverse community in the early twenty-first century, and the movement from a preponderance of quantitative research methods to a breadth of quantitative, qualitative, and mixed methods.

Although change is inevitable in a CoP and the associated scholarly literature that is part of it, we know little about the dynamics of those changes. Of particular interest in the present paper is how and how quickly the characteristics of the scholarly literature change over time. Changes in the scholarly literature may be both the result of change as well as the agent of change in the CoP. Understanding the dynamics of change in the research literature informs appropriate selection and design of interventions to improve that communication stream within the medical education CoP.

Our research question for this study is “How much and what kinds of change take place in the quality of research literature for medical education over a (11 year) period of time?” The question of change in a CoP can be challenging. One must identify a specific outcome to evaluate over some period of time but neither outcome nor time period are obvious. Gathering outcomes data over a period of time is also difficult, given the paucity of databases that preserve these kinds of data. Literature databases (e.g., MEDLINE) often serve as the data source for such studies, either through an analysis of outcomes that can be assessed over a period of time, such as the academic disciplines represented in research topics [ 10 ], or a longitudinal examination of specific topics or themes, like clinical reasoning [ 11 ].

Another methodological approach is to identify an historic study and seek to replicate it sometime later. By comparing results before and after some intervening period, investigators can make observations about changes and their potential implications. One example of this approach examined eight units of medical education research, comparing individual reports in a special issue of Academic Medicine with new interviews of the original unit directors 14 years later [ 12 ]. The investigators analyzed transitions in community characteristics such as research productivity, community membership, and goals of the community.

For the present study, we have elected to follow a similar method to this last example. We identified a major study by Reed et al. [ 13 ], which examined the methodological strengths and weaknesses of the concurrent medical education literature by analyzing studies published in 13 medical and medical education journals between September 2002 and December 2003.

Since this initial work, there has been continued growth in the numbers of medical education research journals and conferences, the number of advanced degree programs in medical education scholarship) [ 14 ], as well as the number of individuals engaged in medical education research. Regulatory agencies increasingly mandate more rigor in educational assessment and innovation [ 15 , 16 ], and the research and publication environment has become more competitive. However, it is unknown how medical education research quality has changed in tandem with these changes in the CoP.

We sought to investigate the nature and magnitude of potential changes in medical education research quality by replicating Reed, et al.’s study 11 years after the original analysis. We explored the question of whether the quality of medical education research studies would have increased, decreased or remained constant when reassessed after a period of time, using the same measures of study quality and the same journals to gauge how changes in the scholarly literature may relate to evolution of the medical education CoP.

Literature search and retrieval

An informationist with expertise in conducting literature searches guided the development of the search strategy with the goal of replicating Reed, et al. [13] using the same 13 peer-reviewed journals included in the initial study. These journals represent broad multidisciplinary medical research (JAMA, New England Journal of Medicine), seven core medical specialties (Academic Emergency Medicine, American Journal of Obstetrics and Gynecology, American Journal of Surgery, Annals of Internal Medicine, Family Medicine, Journal of General Internal Medicine, Pediatrics), as well as medical education-specific journals (Academic Medicine, Medical Education, Medical Teacher, Teaching and Learning in Medicine). The search was conducted on MEDLINE for research studies published from 9/01/2013 to 12/31/2014 to match the timeframe of the original study and included the keywords medical education and medical education research; MeSH term: Education, Medical (see appendix for full search syntax). The interval between the first and subsequent sample of the literature (11 years) reflects the time period in which the authors established their collaboration and began the time-consuming work of literature screening and abstraction and then data analysis, writing and publication. While this is not intended primarily as an indicator of current literature quality, it does provide insight into the evolution of communities of practice in medical education.

Eligibility screening

Consistent with the previous study, medical education research was operationally defined as “any original research study pertaining to medical students, residents, fellows, faculty development, or continuing medical education for physicians” [ 13 ]. Studies focusing on patient education and/or non-physician clinicians were excluded. As in the original study, additional exclusion criteria were: qualitative studies (because the MERSQI does not assess the quality indicators of qualitative studies), meta-analyses and systematic reviews, clinical reviews, letters, editorials, and reports of educational interventions without any evaluation or outcomes.

Eleven of the authors participated in the screening and review process. As an initial calibration exercise, the research team reviewed articles outside the review sample for inclusion-exclusion decision agreement. Each of the 9286 articles in the review sample was then screened by arbitrary pairs of reviewers for inclusion-exclusion decisions. Disagreements between raters were arbitrated through group discussion until consensus was achieved. A kappa coefficient was calculated to estimate rater agreement in selection screening using a sub-sample of 10% (928 papers) and demonstrated moderate agreement between raters (Cohen kappa = 0.43).

After the title and abstract screening, the full-text of all articles meeting inclusion criteria were retrieved. The same inclusion and exclusion criteria as the title/abstract screen were then applied to these full-text articles. The full-text articles that met inclusion-exclusion criteria were abstracted for the study variables.

Data abstraction

We used the Medical Education Research Study Quality Instrument (MERSQI) [ 13 ] to measure the methodological quality of medical education research studies. The MERSQI was designed to measure methodologic quality rather than the quality of reporting (but it is still dependent on the information provided in the written manuscript [ 17 ]). This instrument includes 10 items grouped into 6 domains of study quality including: study design (with options of single group cross-sectional or single group post-test only; single group pre and post-test; non-randomized, 2 group; and randomized controlled experiment), sampling (number of institutions (1, 2, or more) and response rate (< 50%, 50–74%; ≥ 75%), type of data (assessment by study subject; or objective measurement), validity evidence (internal structure, content, and relationships to other variables), data analysis (appropriateness and complexity), and outcomes (satisfaction, attitudes, perceptions, opinions, general facts; knowledge, skills; behaviors; patient/health care outcome). Each MERSQI domain has a maximum possible score of 3. Prior work documents an intraclass correlation coefficient for interrater reliability ranging from 0.72 to 0.98 for scoring the 6 domains [ 13 ].

The MERSQI has excellent inter- and intra-rater reliability in addition to strong validity evidence related to construct, content, and internal structure. The original MERSQI report has been widely cited in the medical education literature (86 citations in PubMed as of 19 January 2021). It is frequently used as a quality measure in systematic and other reviews in a wide range of medical fields [ 18 , 19 , 20 ] Validity evidence for assessing methodological and research characteristics has been reported [ 17 , 21 ].

Descriptive statistics were calculated to explore indicators of study quality. Current data were compared to the Reed, et al., [ 13 ] results using chi-square tests for relative frequency data and t-tests for comparison of mean scores. The primary outcomes were the six mean MERSQI scores for the individual categories of study quality. These were calculated by standardizing the percentage of total achievable points after accounting for “not applicable” responses. A total score was not computed for the MERSQI, following recommendations of the original authors [ 17 ] .For all analyses, a two-tailed alpha level of 0.05 was used to determine statistical significance. Effect sizes are reported for all comparisons; Cohen’s d for t-tests and h for tests of two proportions [ 22 ] (Table  1 ). Data were analyzed using SPSS version 24 for Mac (IBM Corp., Armonk, New York) and R version 3.4.3 for Mac (R Foundation for Statistical Computing, Vienna, Austria).

The Northwestern University Institutional Review Board deemed this study exempt from review (STU00205046).

Identification of studies

A total of 9286 articles were initially identified by the search. After inclusion and exclusion screening, 877 (9.4%) articles remained. Full text articles were retrieved for these 877 articles and screened again, using the same inclusion and exclusion criteria. This resulted in 482 (55.0%) articles that went on to be coded for quality using the MERSQI tool. A summary of the eligibility screening process is presented in Fig.  1 . Overall, 482 (5.2%) articles met eligibility criteria from the 2013–14 sample, compared to 210 articles (2.5%) out of 8505 total publications in the original Reed et al. study from 2002 to 03.

figure 1

PRISMA flow diagram [ 23 ] showing the process for identifying and screening articles for inclusion in the study. Data were obtained from Ovid MEDLINE and included citation data for research studies published in 13 medical education journals between 2013 and 2014

Comparisons of study quality measures between 2002 and 03 vs. 2013–14

Consistent with the prior study, the highest mean domain quality score in the replication review was for the data analysis domain (mean = 2.6, SD 2.6, Table 1 ). The overall MERSQI score increased from 9.9 (SD 2.3) to 10.7 (SD 2.6) between 2002 and 03 and 2013–14 ( p  < 0.001). Of the six domains of study quality measured by the MERSQI, there were statistically significant improvements in four measures: study design, type of data, validity and outcomes. Scores that did not change significantly in the time between the two analyses were in the domains of data analysis and sampling.

The mean score on the study design domain improved from 1.3 to 1.4 ( p  < 0.01), but there were no statistically significant changes for any specific type of study design. The majority (64.1%) of designs continued to be single group cross-sectional or post-test only. Randomized control designs were still infrequent, although their relative proportion among published studies increased almost four-fold over this time period, from 2.9 to 11.0% of included studies.

For the sampling domain, the proportion of studies that were multi-institutional was stable over this period. Despite calls for more collaborative, multi-institutional research, there was little change over the intervening decade, with the majority of papers (62.2%) continuing to be single-institution studies.

The 2013–14 set of studies had a significantly greater use of objective measurements than the prior cohort of studies (45.7% of articles in 2002–03 and 54.4% of articles in 2013–14, p  < 0.001).

The reporting of validity evidence for medical education research studies was a frequent deficiency in the literature from 2002 to 03 and, although there was a statistically significant improvement (from 0.69 in 2002–03 to 1.06 in 2013–14, p  < 0.001), this was still the lowest scoring domain among all of the MERSQI dimensions (mean = 1.06 out of a possible maximum score of 3.00). The 2013–2014 analysis showed increased reporting of all three forms of validity evidence (internal structure, content, relationship to other variables) analyzed using the MERSQI, compared to the 2002–03 analysis.

The highest scores were in the domain of data analysis (2.6 and 2.6 out of 3.0, in 2002–03 and 2013–14, respectively), and these did not change significantly over the study time period ( p  = 0.22). The large majority of studies in both time periods were considered to have appropriate data analysis procedures for the data reported and most went beyond simple descriptive statistics to reflect the number and relationship among variables in the study.

Study outcome scores showed a small but statistically significant increase from the 2002–03 sample to the 2013–14 sample (means = 1.4 vs. 1.6, p  = 0.01). The most common outcomes in both samples were attitudes, satisfaction, perceptions, opinions and general facts (48.6% of articles in 2002–03 and 41.3% of articles in 2013–14, p  < .0.01)). Patient and health care outcomes were reported almost four times more frequently in the 2013–14 sample compared to the previous sample (2.4% of articles in 2002–03 and 9.1% of articles in 2013–14), yet these important outcomes were still reported in only a small fraction of medical education research studies.

The larger CoP for medical education research has changed over the past couple of decades in ways that these results may reflect. There has been an increase in the number of medical education research journals. This may have acted to decrease the number of submissions to the journals included in this study by spreading potential publications across a greater number of outlets. On the other hand, the percentage of articles meeting study inclusion criteria more than doubled from 2.5% in 2002–03 to 5.2% in 2013–14, which may indicate that these high-impact journals are attracting more high-quality submissions while less rigorous work has other outlets.

Similarly, the proliferation of professional societies and academic conferences related to medical education globally has grown significantly, which suggests that there are many more investigators producing research articles. This increased demand for journal space may have driven the increase in the number of journals, but the causal relationship is not clear.

These findings suggest that the methodological quality of quantitative medical education research improved from 2002 to 03 to 2013–14. This is encouraging, given the established need for increased methodological rigor, efforts to increase faculty skills in education research, and the recognized importance of a robust evidence base in medical education [ 24 ]. The improvement in methodologic quality reflects growth in both the domain of the CoP as well as the practice of medical education research itself.

Some of the most challenging components of study quality within medical education had notable gains between the two time periods. In particular, the inclusion of and attention to validity evidence for the measures used in the studies increased significantly from 2002 to 03 to 2013–14. The medical education research community has called for an emphasis on validity evidence for more than 20 years [ 25 , 26 , 27 , 28 ]. This, therefore, is a welcome improvement in medical education research quality as defined by the accuracy and relevance of the measurement methods used to acquire data. Reporting of patient and healthcare outcomes also increased nearly four-fold. Although only 9.1% of studies assessed patient outcomes in the 2013–14 cohort, this is an important step towards the ultimate goal of medical education—to improve health. At the same time, there was a comparable decrease in reliance on learner self-reported data such as satisfaction, opinions and self-assessments as primary outcome measures.

Our analysis also reveals that randomized controlled trials (RCTs) were being used more frequently in medical education in our analysis compared to the 2002–03, although RCTs still comprised only 11% of education studies. While RCTs are viewed as the gold standard in the clinical world, that is not necessarily the case in education. RCTs can be costly and time consuming to conduct and, in medical education, they may violate ethical principles related to withholding a potentially beneficial educational intervention from the learners who are randomized to the control arm. A well-designed quasi-experiment may generate more meaningful evidence than a poorly designed RCT. Methodological and ethical limitations unique to medical education warrant ongoing discussion around best practices in research design.

In 2013–14, nearly two thirds of education research studies still used single group designs. Single group studies are more convenient to conduct and often reflect the natural environment of education, which tends to provide curricular and teaching innovations for the entire learner group rather than segregate them into comparison conditions. Nonetheless, reliance on single-group designs hinders interpretation of the effects of the studied educational interventions.

Similarly, almost two-thirds of the studies in both samples were conducted at single institutions. This limits the generalizability of these studies to other settings, learners, and contexts. The lack of growth in multi-institutional studies over the period of this study is a concern and may partially reflect limited funding for medical education research. Indeed, in the 2002–03 cohort there was a much greater proportion of multi-institutional studies among studies with higher levels of funding, as multi-institutional collaboration facilitates rigorous, generalizable research but requires additional resources [ 29 ].

This study has several limitations. While the follow-up time period of 2013–14 is not current, the goal of this study was to examine the change in methodologic quality of medical education research and how the CoP is evolving, not to give a current snapshot of the medical education literature.

We also note that the MERSQI assesses aspects of study design, not study hypotheses or research questions. Study design needs to match the research question and single group, post study assessment may be a perfectly appropriate design for some research questions. In other words, our analyses implicitly assume that the content, focus, and questions are more or less consistent from the initial to the comparison time period. If that is not the case, changes in study design quality become more difficult to interpret. Another limitation of MERSQI is its lack of assessment of quality indicators of qualitative studies. The evolving interest in use of qualitative studies in medical education research demonstrates a shift in the CoP’s priorities, as qualitative studies have become foundational in medical education and other health professions education research.

Reviewers were not blind to the study authors or journals. We attempted to mitigate this issue by asking reviewers to recuse themselves from the review if a potential conflict of interest was noted. Additionally, inter-rater agreement on the screening decisions was only moderate (Cohen kappa = 0.43), which attenuates the ability to make statistically significant distinctions between our results and those of Reed et al. [ 13 ]. We acknowledge that our quality ratings were derived from published reports only, and publication requirements and practices (e.g., electronic appendices and other supplemental information) may limit the data that are included in publications, thereby impacting MERSQI scores. However, this was necessary to provide comparable data to Reed et al. [ 13 ].

In addition, in order to compare our data to Reed et al., our study focused solely on the journals that were included in the 2002–03 cohort. In contrast, an examination of all published education studies (across a wider array of journals) would provide useful data on the full body of medical education research. There has been a proliferation of journals that accept or are devoted to medical education research, but these new journals were excluded from this analysis to maintain consistency with the original study.

It is also very important to note that the original study and this replication only examined quantitative research. Any changes to the number and rigor of qualitative studies was not addressed in this study. To the extent that qualitative studies emphasize exploratory investigations and deeper understanding of mechanisms and phenomena, it may be that the inclusion of qualitative studies would increase the preponderance of outcomes in the attitudes, perceptions and opinions category over patient and health care outcomes.

Despite these limitations, our study may serve as a data point to chart the evolution of medical education research quality and its impact on the medical education research CoP. We found that quality improved from 2002 to 03 to 2013–14 as measured by the MERSQI. By 2013–14, a greater proportion of studies reported validity evidence and used patient-centered endpoints and more rigorous study designs. With continued attention to these areas, medical education research quality could continue to rise in coming years. Medical education research quality is positively associated with research funding [ 30 ] and this characteristic of the CoP may drive increases in resources dedicated to medical education resources. Engagement of the medical education research CoP with professional organizations, governmental and non-governmental groups may further support development of a high quality evidence base to guide medical education practice and further improve patient outcomes.

In terms of the larger question of how the research literature serves as a means of communication for the medical education CoP, these results may be a glass half full or half empty. Indeed, some characteristics of the literature show improvement over an 11-year period, yet others do not. The pace of change might also be disappointing to some who hope to see a more rapid transformation of the CoP toward an evidence base in education that supports adoption of new models of medical care, greater access to care, and a responsive educational system. Although the interpretation of these findings are open to discussion, we believe it does provide some encouragement for efforts to map the changes in the CoP with changes in one of its primary means of communicating information, values, and perspectives.

Availability of data and materials

Data were obtained from Ovid MEDLINE.


Community of Practice

Medical Education Research Study Quality Instrument

Wenger EC, Snyder WM. Communities of practice: the organizational frontier. Harv Bus Rev. 2000;78:139–45.

Article   Google Scholar  

Wenger E, McDermott RA, Snyder W. Cultivating communities of practice: a guide to managing knowledge. Boston: Harvard Business Review Press; 2002.

Google Scholar  

van der Vleuten CPM. Medical education research: a vibrant community of research and education practice. Med Educ. 2014;48(8):761–7. Accessed 8 Oct 2014. .

Wenger E, Trayner-Wenger B. Communities of practice: a brief introduction. :1–8. 2015. Accessed 2018 Dec 8. .

Barab SA, Barnett M, Squire K. Developing an empirical account of a community of practice: characterizing the essential tensions. J Learn Sci. 2002;11(4):489–542. .

Newman MEJ. The structure of scientific collaboration networks. PNAS. 2001;98(2):404–9.

Frank JR, Snell L, Englander R, Holmboe ES. Implementing competency-based medical education: moving forward. Med Teach. 2017;39(6):568–73. .

Frank JR, Snell LS, Cate O ten, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: theory to practice. Med Teach 32(8):638–645. .

Steinert Y. Faculty development in the new millennium: key challenges and future directions. Med Teach. 2000;22(1):44–50.

Norman G. Fifty years of medical education research: waves of migration. Med Educ. 2011;45(8):785–91. .

Norman G. Research in clinical reasoning: past history and current trends. Med Educ. 2005;39:418–27.

Humphrey-Murto S, O’Brien B, Irby DM, van der Vleuten C, ten Cate O, Durning S, et al. 14 years later: a follow-up case-study analysis of 8 health professions education scholarship units. Acad Med. 2019;95(4):629–36. .

Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–9.

Tekian A, Harris I. Preparing health professions education leaders worldwide: a description of masters-level programs. Med Teach. 2012;34:52–8.

Swanson DB, Roberts TE. Trends in national licensing examinations in medicine. Med Educ. 2016;50(1):101–14. .

Edgar L, Roberts S, Holmboe E. Milestones 2 . 0 : A Step Forward. J Grad Med Educ. 2018:367–9.

Cook DA, Reed DA. Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle-Ottawa scale-education. Acad Med. 2015;90(8):1067–76. .

Wasson LT, Cusmano A, Meli L, Louh I, Falzon L, Hampsey M, et al. Association between learning environment interventions and medical student well-being a systematic review. JAMA. 2016;316(21):2237–52. .

Timberlake MD, Mayo HG, Scott L, Weis J, Gardner AK. What do we know about intraoperative teaching? Ann Surg. 2017;266(2):251–9. .

Geerts JM, Goodall AH, Agius S. Evidence-based leadership development for physicians: a systematic literature review. Soc Sci Med. 2020;246:112709. .

Sawatsky AP, Beckman TJ, Edakkanambeth Varayil J, Mandrekar JN, Reed DA, Wang AT. Association between study quality and publication rates of medical education abstracts presented at the Society of General Internal Medicine annual meeting. J Gen Intern Med. 2015;30(8):1172–7. .

Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates, Inc; 1988.

Moher D, Liberati A, Tetzlaff J, Altman DG, Altman D, Antes G, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7). .

Gruppen LD, Yoder E, Frye A, Perkowski LC, Mavis B. Supporting medical education research quality: the Association of American Medical Colleges’ medical education research certificate program. Acad Med. 2011;86(1):122–6. .

Chen FM, Burstin H, Huntington J. The importance of clinical outcomes in medical education research. Med Educ. 2005;39(4):350–1. .

Kalet AL, Gillespie CC, Schwartz MD, Holmboe ES, Ark TK, Jay M, et al. New measures to establish the evidence base for medical education: identifying educationally sensitive patient outcomes. Acad Med. 2010;85(5):844–51. .

Chen C, Petterson S, Phillips R, Bazemore A, Mullan F. Spending patterns in region of residency training and subsequent expenditures for care provided by practicing physicians for Medicare beneficiaries. JAMA. 2014;312(22):2385–93.

Chahine S, K Mahan K, Wright S, Monteiro S, LEM G, Barber C, et al. A call to investigate the relationship between education and health outcomes using big data. Acad Med. 2018;93(6):829–32. .

Gruppen LD, Durning SJ. Needles and haystacks: finding funding for medical education research. Acad Med. 2016;91(4):480–4. .

Reed DA, Kern DE, Levine RB, Wright SM. Costs and funding for published medical education research. JAMA. 2005;294:1052–7.

Download references


The authors wish to thank Emily C. Ginier, MLIS for her assistance retrieving articles.

This project was partially supported by a Central Group on Educational Affairs (CGEA) mini-grant.

Author information

Authors and affiliations.

Department of Family & Community Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA

Katherine M. Wright

Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI, USA

Larry D. Gruppen

Stanford University, Palo Alto, CA, USA

Kevin W. Kuo

Department of Medical Education, Duke University, Durham, NC, USA

Andrew Muzyk

University of California, Irvine, Orange, CA, USA

Jeffry Nahmias

Mayo Clinic College of Medicine and Science, Rochester, MN, USA

Darcy A. Reed

Department of Surgery, University of Michigan, Ann Arbor, MI, USA

Gurjit Sandhu

Department of Neurology, University of Michigan, Ann Arbor, MI, USA

Anita V. Shelgikar

Departments of Internal Medicine and Pediatrics, University of Michigan, Ann Arbor, MI, USA

Jennifer N. Stojan

Northwestern University Feinberg School of Medicine, Chicago, IL, USA

Toshiko L. Uchida

General Pediatrics Residency, Nationwide Children’s Hospital & The Ohio State College of Medicine, Columbus, OH, USA

Rebecca Wallihan

The Michael V. Drake Institute for Teaching and Learning, The Ohio State University, Columbus, OH, USA

Larry Hurtubise

You can also search for this author in PubMed   Google Scholar


All authors wrote and edited the main manuscript text. K.W. analyzed data and prepared the figures. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Katherine M. Wright .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors have nothing to disclose.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit . The Creative Commons Public Domain Dedication waiver ( ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wright, K.M., Gruppen, L.D., Kuo, K.W. et al. Assessing changes in the quality of quantitative health educations research: a perspective from communities of practice. BMC Med Educ 22 , 227 (2022).

Download citation

Received : 15 November 2021

Accepted : 28 March 2022

Published : 01 April 2022


Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Medical education research
  • Research quality
  • Research methods
  • Methodological quality

BMC Medical Education

ISSN: 1472-6920

quantitative research about medicine and its contribution

  • Research article
  • Open access
  • Published: 03 May 2019

Qualitative and quantitative research of medication review and drug-related problems in Hungarian community pharmacies: a pilot study

  • András Szilvay 1 ,
  • Orsolya Somogyi 1 ,
  • Attiláné Meskó 1 ,
  • Romána Zelkó 1 &
  • Balázs Hankó 1  

BMC Health Services Research volume  19 , Article number:  282 ( 2019 ) Cite this article

7785 Accesses

8 Citations

Metrics details

Pharmaceutical care is the pharmacist’s contribution to the care of individuals to optimize medicines use and improve health outcomes. The primary tool of pharmaceutical care is medication review. Defining and classifying Drug-Related Problems (DRPs) is an essential pillar of the medication review. Our objectives were to perform a pilot of medication review in Hungarian community pharmacies, a DRP classification was applied for the first time. Also, our goal was the qualitative and quantitative description of the discovered DRPs, and of the interventions for their solution in order to prove the safety relevance of the service and to map out the competence limits of GPs and community pharmacists to drug therapy.

The project took place in Hungarian community pharmacies. The study was performed with patients taking vitamin K antagonist (VKA) and/or ACE inhibitor and NSAID simultaneously (ACEI-NSAID). 61 pharmacists and 606 patients participated in the project. Pharmacists reviewed the medication for 3 months and the classification of DRPs was performed (category of DRP1 – DRP6). Patient data were statistically analyzed.

Patients consumed on average 7.9 ± 3.1 medications and other products. 571 DRPs were detected in 540 patients, averaging 1.06 DRPs per patient (SD = 1.07). The highest frequency category was DRP5 (non-quantitative safety problem; 51.0%). The most common root cause was an interaction (42.0%) and non-adherence (19.4%.). The most commonly used intervention was education (25.4%) and medication replacement by the pharmacist (20.1%). The changing of the frequency and dosage in any direction were negligible.


Patients are struggling with many DRPs that can be assessed and categorized by this system and which remain unrecognizable without pharmacists. Further projects need to be developed to assist in the development of physician-pharmacist cooperation and the widespread dissemination of pharmaceutical care.

Peer Review reports

According to the definition of Pharmaceutical Care Network Europe “Pharmaceutical Care is the pharmacist’s contribution to the care of individuals to optimize medicines use and improve health outcomes”. [ 1 ] The goal of the pharmacists is to collect the patient’s medications (Rx, OTC) and other products (e.g. dietary supplements) to ensure their necessity, efficacy and safety. [ 2 ]

The main tool of pharmaceutical care is medication review, “a structured, critical examination of a patient’s medicines with the objective of reaching an agreement with the patient about treatment, optimizing the impact of medicines, minimizing the number of medication-related problems and reducing waste”. [ 3 ] The best way to make medication review is in collaboration with the patients and their general practitioners. [ 4 , 5 ] To demonstrate the benefits of medication review, several but controversial articles have published. It reduces the number of cases requiring emergency care [ 6 , 7 ], the number of (re) hospitalizations, but its beneficial effects on quality of life, adverse drug reactions and mortality are non-significant in high-risk groups. [ 7 , 8 ] However according to other articles it reduces the number of (unnecessary) drugs [ 9 , 10 , 11 , 12 ], it helps to detect and solve drug-related problems (DRPs) [ 9 , 13 , 14 , 15 , 16 , 17 ], especially in collaboration with hospital pharmacists [ 18 ], it increases the patients’ trust in the therapy [ 19 ] and the cost-effectiveness of the treatment [ 20 ].

Defining and classifying drug-related problems is an essential pillar of the medication review. The drug-related problems are “situations in which in the process of use of medicines cause or may cause the appearance of a negative outcome associated with the medication.” [ 21 ] There are many reasons for the drug-related problems, which may result that drug therapy is not achieving its purpose or even becoming harmful. There are more than 20 types of DRP classification system in the literature, which differ in, e.g. DRP groups and methodology. [ 22 ] It is also important to involve patients in the process of detecting drug-related problems. [ 23 ]

The aim of our research was to perform a pilot of medication review in Hungarian community pharmacies as part of basic pharmaceutical care, using a drug-related problem classification for the first time to lay the foundation for wider adoption of this service in Hungary. Also, our goal was the qualitative and quantitative description of the discovered drug-related problems, and of the interventions for their solution in order to prove the safety relevance of the service and to map out the competence limits of GPs and community pharmacists to drug therapy. The latter can contribute to the development of a future “target model” of doctor-patient-pharmacist cooperation.

Description of the project

The project took place between December 2015 and August 2016. The data were collected by pharmacists (they have not received monetary compensation) participating in specialist training at Semmelweis University. The participation of pharmacists was obligatory to complete the second year of the training, and the cooperative pharmacies were their own workplaces. All participating pharmacies were accredited at the Semmelweis University. All the participating pharmacists from all around the country went to Budapest and participated in one-day training at Semmelweis University, which included the description and requirements of the project, and the presentation of the drug-related problem classification.

The pharmacists at the beginning of the project visited family practitioners working near the pharmacy, practically those whose patients often go to the pharmacy. They were invited to participate in the project. After that, the patients were involved in the pharmacy. The study was performed with patients taking vitamin K antagonist (VKA) and/or ACE inhibitor and NSAID simultaneously (ACEI-NSAID). The latter is considered as a high-risk group because of the increased chance of renal failure [ 24 ] and the potential inadequacy of the therapy [ 25 ]. A patient could have been in both categories. Patients had to be at least 18 years old, had to buy medicines themselves and had to be the patients of the general practitioner involved. All pharmacists tried to have around 10 patients. The process of the first and further occasions is described in Fig.  1 .

figure 1

The process of the occasions

Characteristics of participating pharmacies and patients

Data for patients and pharmacies in the study are shown in Table  1 . 61 pharmacists took part in 61 pharmacies. The survey was close to nationwide coverage (16 of 20 counties). Most of the pharmacies were in the capital (35.6%). 606 patients participated in the project (9.9 patients/pharmacy; SD = 3.0), 57.3% were women and 42.7% were males. 497 patients (mean = 8.1; SD = 2.2) took part in every requested meeting with the pharmacist (traced patients), 18.0% of the patients left the project. However, we used data from all the patients involved, not only from traced patients. The average age of patients was 65.0 years (SD = 11.9). 55.6% of the participants took ACE inhibitor and NSAID simultaneously, 39.8% of them took vitamin K antagonists, and 4.6% of them were included in both categories.

  • Medication review

Pharmacists have been conducting medication review at each consultation; they looked at medication taken from the point of necessity, effectiveness and safety. The medication review and the classification of drug-related problems were performed according to the Third Consensus of Granada on Drug-Related Problems classification system [ 21 ] and to the Hungarian National Committee of Pharmaceutical Care Metabolic Syndrome Pharmaceutical Care Programme. [ 26 ] In the process of assessing drug-related problems, the pharmacist classified the DRPs into six classes and identified the root cause. (Table  2 ).

The number and cause of the drug-related problem were recorded by the pharmacist. In addition, the intervention was also recorded. The medication review was performed at all pharmacist meetings. Anonymous patient data were statistically analyzed.

Statistical analysis

Statistical calculations were performed using SPSS 20.0. After the descriptive statistical analysis, two sample t test, paired sample t test, and a variance analysis were performed on continuous data to detect differences and correlations. When calculating the Pearson correlation coefficient, the p value for the correlation coefficient was < 0.005. For discrete data, the Kruskal-Wallis test and chi-square test were used. Control of normality was performed with Kolmogorov Smirnov test. The significance level was p = 0.05.

Ethics approval and consent to participate

The project was implemented with the support and cooperation of the National Health Development Institute’s Primary Care Directorate [ 27 ]. The unified professional protocol made available in the course of the co-operation is a document agreed with the Primary Service Directorate. We have not received a waiver of ethics approval since the participation in the questionnaire survey, and the pharmaceutical service was not linked to one Institute (University) and was absolutely free and undoubtedly noninvasive, so IRB deemed unnecessary according to the similar national regulations. In Hungary according to Regulation No 44/2004 MoHSFA and Act XLVII of 1997, pharmacies did not need to be individually ethically licensed, because the service complies with statutory regulations, and pharmacies are legally entitled to perform such activities [ 28 , 29 , 30 , 31 ]. Verbal informed consent was obtained from all participants in the pharmacies; no written consent was required according to the Act CLIV of 1997 on Health (noninvasive pharmaceutical service and questionnaire survey) [ 32 ].

The investigation was a free service of pharmacies with operating licenses. The patients involved voluntarily participated in the process. Patients participating in the project received verbal information in accordance with the national regulations mentioned above. Qualified pharmacists conducted the project. The data were handled by pharmacy and health data management according to Act XLVII of 1997. Data were transmitted without personal information to process the results. The personal and health data of the patients included in the study were not damaged.

Descriptive results

In the assessment of drug-related problems, 540 patients from 606 patients were collected and analyzed. On average, patients consumed 7.9 ± 3.1 medications and other products: 6.3 were prescription drug (SD = 2.8), 1.1 OTC (SD = 1.1) and 0.4 other product, for example dietary supplements (SD = 0.8). (Table 1 ).

During the study, 571 drug-related problems were detected in these 540 patients, averaging 1.06 DRP per patient (SD = 1.07). The highest frequency category was DRP5 (non-quantitative safety problem: 51.0%). Approximately one-fourth of cases (24.0%) belonged to DRP3 (non-quantitative ineffectiveness) and 10% to DRP1 (untreated health problem). DRP2 (Effect of unnecessary medicine), DRP4 (Quantitative ineffectiveness) and DRP6 (Quantitative safety problem) were less frequent (8.2, 4.6, 2.3%). (Fig.  2 -All patients).

figure 2

The relative proportion of different drug-related problem categories per patient group (All patients: all the participating patients ( n  = 571); ACEI-NSAID: patients taking ACE inhibitor and NSAID simultaneously ( n  = 330); VKA: patients taking vitamin K antagonist ( n  = 212); Both: patients included in both categories ( n  = 29); DRP: drug-related problem (see Table 2 ))

Analyzing the root causes of drug-related problems, the most common was the interaction (42.0%), the second was non-adherence (19.4%). The Quantitative safety problem caused by improper dosage was the rarest (2.3%). (Fig.  3 -All patients ).

figure 3

The relative proportion of the underlying cause of drug-related problems per patient group (All patients: all the participating patients ( n  = 571); ACEI-NSAID: patients taking ACE inhibitor and NSAID simultaneously ( n  = 330); VKA: patients taking vitamin K antagonist ( n  = 212); Both: patients included in both categories ( n  = 29); DRP: drug-related problem (see Table 2 ))

In the case of ACEI-NSAID patients, the DRP1 category appears to be higher (13.9%) than in the case of VKA patients (4.7%). The ratio was reversed in the case of DRP3 (22.1 and 27.8%) and DRP5 (48.2% or 52.4%), the latter is due to a higher rate of interactions. (Figs.  2 and 3 ) The ratio of interaction was extremely high for those patients who were in both categories (65.5%). (Fig. 3 ) However, these differences are not significant either in the number of drug problems or in the occurrence of the individual categories and causes. There was no “other” problem that cannot be categorized elsewhere.

Results of statistical analysis

There are no differences in the prevalence of drug-related problems between men and women (p = 0.070) and between the patients over and under 65 years. (p = 0.552).

There is a significant difference between the types of settlement in the occurrence of the drug-related problem. In the capital city, the pharmacists have found two DRPs per patient in a significantly higher ratio, while in other settlements it was markedly higher that the pharmacist found no mistake in the medication. There is a correlation between the number of DRPs and the total number of used medications, but the correlation is weak (Pearson correlation coefficient = 0.214 (p < 0.005)). The relationship between the number of prescription drugs and the number of drug-related problems is similar, somewhat lower (Pearson correlation coefficient = 0.152 (p < 0.005)).

Table  3 summarizes the rates of interventions used to eliminate drug-related problems. The most common intervention for the elimination of each underlying cause was indicated with bold number, while underlined number indicates the interventions with an incidence higher than 10%. Overall, the most commonly used intervention was education (25.4%) and medication replacement by the pharmacist (20.1%). More than 10% of the problems the intervention was not necessary (10.9%), or the pharmacist sends the patient to a physician (14.5%) or the pharmacist warned the GP (11.7%). The changing of the frequency and dosage in any direction were negligible.

Due to a large number of patients involved and the low drop-out rate, patients are interested and find the service provided by pharmacists useful. The fact that a large number of patients who had NSAIDs with an ACE inhibitor were included in the study underlined the relevance of this problem. Such a problem frequently does not show up at the doctor but at the pharmacy. The project involved a large number of patients with more than 5 medicines (also known as polypharmacy patients [ 33 ]).

It is noteworthy that patients use an average of 1 OTC drug on a regular basis, and that 4 out of 10 patients also use some other formulations (e.g. dietary supplements).

The use of these two product categories can only be supervised by the pharmacist. The patient’s medication is fully matched at expedition at the pharmacy only (Rx, OTC, other products) so the pharmacy service presented in the project plays an essential role in the assessment and resolution of drug-related problems. It is supported by a large number of DRPs that have been identified and classified in this project based on a drug-related problem classification system that has been used for the first time in Hungary. In addition to the reasons mentioned above, the overload of general practitioner services can also contribute. Among the DRP categories, there is a high amount of non-quantitative safety problems in all patient groups, which are mainly drug-drug or drug-other product interactions. The latter is also influenced by the patient’s involvement in the ACEI-NSAID group. However, we cannot talk about such a factor in the VKA group. This phenomenon is due to the uncontrolled use of the vast amounts of prescription and OTC medicines mentioned earlier and the other medicines are taken by 4 out of 10 patients. The problem may be solved by pharmacists who have resolved the situation in our research with education, medication replacement (especially OTC-OTC drug switching) and by sending the patient to the GP. In the case of interactions, “stop medication” has hardly occurred, and pharmacists seem to be hesitant to take this step, as they think that the physician is the one who competent to make this decision. Another major problem is the Non-quantitative ineffectiveness of the medication of a quarter of patients due primarily to their deliberate or unintended non-adherence.

Non-adherence is a widespread problem with chronic diseases for example in the case of conditions treated with ACE inhibitor and vitamin K antagonist. The pharmacist can help by detecting the problem and education. It is also important to mention that every tenth ACEI-NSAID patient is suffering from an untreated health problem. The research has shown that in many cases the pharmacist has noticed such a health problem, which has been solved by drug recommendation and by sending the patient to the GP. Based on these results, a medication review in the framework of basic pharmaceutical care can be a solution beyond the problems mentioned above in preventing the risks of self-medication.

In the case of medication review, it is also necessary to address the issue of competency conflict between the pharmacist and the general practitioner. By looking at the pharmacists’ interventions to resolve drug-related problems, we can see that 59.7% of the problems have been solved by the pharmacist without the involvement of a physician, primarily through education and the exchange of a patient’s drug with an OTC drug. The pharmacist in his/her own solved only about 5–6% of the cases by recommending a new drug or stopping a therapy, while the changing of frequency and dosage were as rare as possible without consultation with the physician. Pharmacists sent patients more to the physician without indicating the problem being diagnosed to them. Pharmacists preferred to send patients to doctors without consulting the GP, suggesting low levels of co-operation between two professions and pharmacists’ fears of doctors. Analyzing the effect of certain population factors on drug-related problems, it can be stated that among the examined factors, the number of DRPs is only influenced by the geographical location of the pharmacies. This, assuming that patients seek indirectly the pharmacy closest to their home first, refers indirectly to the influence of the type of residential township. Based on the results, it is assumed that a more extensive settlement poses a higher risk for patients, due to the less personal physician-patient and pharmacist-patient relationship, the more likely to be accessed by more accessible medical services. So the development of a regular pharmacist-patient relationship is of the utmost importance in this area.

Based on the results of the 540 patients surveyed in the 61 Hungarian pharmacies we can conclude that patients are struggling with many drug-related problems that can be assessed and categorized by this system and which remain unrecognizable without pharmacists. To achieve this, further projects need to be developed to assist in the development of physician-pharmacist cooperation and the widespread dissemination of pharmaceutical care. Our results provide a reasonable basis for the widespread use of medication review. In the future, it would be worthwhile extending the study to other patient groups, such as elderly patients with polypharmacy.


Patients taking ACE inhibitor and NSAID simultaneously

  • Drug-related problem

Patients taking vitamin K antagonist

Allemann SS, van Mil JW, Botermann L, Berger K, Griese N, Hersberger KE. Pharmaceutical care: the PCNE definition 2013. Int J Clin Pharm. 2014;36(3):544–55.

Article   PubMed   Google Scholar  

Cipolle RJ, Strand LM, Morley PC. Pharmaceutical care practice: the patient-centered approach to medication management. 3rd ed. McGraw Hill Professional; 2012.

Google Scholar  

Clyne W, Blenkinsopp A, Seal R. A guide to medication review 2008. 2008. . Accessed: 15.11. 2018.

Geurts MM, Talsma J, Brouwers JR, de Gier JJ. Medication review and reconciliation with cooperation between pharmacist and general practitioner and the benefit for the patient: a systematic review. Br J Clin Pharmacol. 2012;74(1):16–33.

Article   PubMed   PubMed Central   Google Scholar  

Kwint HF, Bermingham L, Faber A, Gussekloo J, Bouvy ML. The relationship between the extent of collaboration of general practitioners and pharmacists and the implementation of recommendations arising from medication review: a systematic review. Drugs Aging. 2013;30(2):91–102.

Christensen M, Lundh A. Medication review in hospitalised patients to reduce morbidity and mortality. Cochrane Database Syst Rev. 2016;(2).

Roth MT, Ivey JL, Esserman DA, Crisp G, Kurz J, Weinberger M. Individualized medication assessment and planning: optimizing medication use in older adults in the primary care setting. Pharmacotherapy. 2013;33(8):787–97.

Leendertse AJ, de Koning GH, Goudswaard AN, Belitser SV, Verhoef M, de Gier HJ, et al. Preventing hospital admissions by reviewing medication (PHARM) in primary care: an open controlled study in an elderly population. J Clin Pharm Ther. 2013;38(5):379–87.

Article   CAS   PubMed   Google Scholar  

Lenaghan E, Holland R, Brooks A. Home-based medication review in a high risk elderly population in primary care--the POLYMED randomised controlled trial. Age Ageing. 2007;36(3):292–7.

Frank C, Weir E. Deprescribing for older patients. Cmaj. 2014;186(18):1369–76.

Hurmuz MZM, Janus SIM, van Manen JG. Changes in medicine prescription following a medication review in older high-risk patients with polypharmacy. Int J Clin Pharm. 2018;40(2):480–7.

Mudge A, Radnedge K, Kasper K, Mullins R, Adsett J, Rofail S, et al. Effects of a pilot multidisciplinary clinic for frequent attending elderly patients on deprescribing. Aust Health Rev. 2016;40(1):86–91.

Cooper JA, Cadogan CA, Patterson SM, Kerse N, Bradley MC, Ryan C, et al. Interventions to improve the appropriate use of polypharmacy in older people: a Cochrane systematic review. BMJ Open. 2015;5(12).

Vinks TH, Egberts TC, de Lange TM, de Koning FH. Pharmacist-based medication review reduces potential drug-related problems in the elderly: the SMOG controlled trial. Drugs Aging. 2009;26(2):123–33.

Messerli M, Blozik E, Vriends N, Hersberger KE. Impact of a community pharmacist-led medication review on medicines use in patients on polypharmacy - a prospective randomised controlled trial. BMC Health Serv Res. 2016;16(1):145.

Lenander C, Bondesson Å, Viberg N, Beckman A, Midlöv P. Effects of medication reviews on use of potentially inappropriate medications in elderly patients; a cross-sectional study in Swedish primary care. BMC Health Serv Res. 2018;18(1):616.

Chua SS, Kok LC, Yusof FAM, Tang GH, Lee SWH, Efendie B, et al. Pharmaceutical care issues identified by pharmacists in patients with diabetes, hypertension or hyperlipidaemia in primary care settings. BMC Health Serv Res. 2012;12(1):388.

Ensing HT, Koster ES, Dubero DJ, van Dooren AA, Bouvy ML. Collaboration between hospital and community pharmacists to address drug-related problems: the HomeCoMe-program: Res Social Adm Pharm; 2018.

Geurts MM, Stewart RE, Brouwers JR, de Graeff PA, de Gier JJ. Patient beliefs about medicines and quality of life after a clinical medication review and follow-up by a pharmaceutical care plan: a study in elderly polypharmacy patients with a cardiovascular disorder. J Pharm Health Serv Res. 2015;6(4):171–6.

Article   Google Scholar  

Jodar-Sanchez F, Malet-Larrea A, Martin JJ, Garcia-Mochon L, Lopez Del Amo MP, Martinez-Martinez F, et al. Cost-utility analysis of a medication review with follow-up service for older adults with polypharmacy in community pharmacies in Spain: the conSIGUE program. Pharmacoeconomics. 2015;33(6):599–610.

Cd C. Tercer Consenso de Granada sobre problemas relacionados con medicamentos (PRM) y resultados negativos asociados a la medicación (RNM). Ars Pharmaceutica. 2007;48(1):5–7.

Basger BJ, Moles RJ, Chen TF. Application of drug-related problem (DRP) classification systems: a review of the literature. Eur J Clin Pharmacol. 2014;70(7):799–815.

Kwint HF, Faber A, Gussekloo J, Bouvy ML. The contribution of patient interviews to the identification of drug-related problems in home medication review. J Clin Pharm Ther. 2012;37(6):674–80.

Gooch K, Culleton BF, Manns BJ, Zhang J, Alfonso H, Tonelli M, et al. NSAID use and progression of chronic kidney disease. Am J Med. 2007;120(3):1–7.

Page RL 2nd, O'Bryant CL, Cheng D, Dow TJ, Ky B, Stein CM, et al. Drugs that may cause or exacerbate heart failure: a scientific statement from the American Heart Association. Circulation. 2016;134(6):32–69.

Hungarian National Committee of pharmaceutical care. Metabolic syndrome pharmaceutical care Programme. 2009.

National Health Development Institute's primary care directorate.,alapellatasi-igazgatosag/alapellatasi-igazgatosag-feladata/ . Accessed: 22.11. 2018.

Regulation no 44/2004 MoHSFA on the ordering and issuing of medicinal products for human use. Sect. 22/A-C (2004).

Ministry of Human Capacities. A profession directive on medication review in the framework of basic pharmaceutical care. Egészségügyi Közlöny. 2013. . Accessed: 22.11. 2018.

Act XLVII of 1997 on the management and protection of health data and related personal data, XLVII. Sect. 14/A (1997).

Regulation No 41/2007 Ministry for Health on the operation, Service and registration of public, branch, manual and institute pharmacies. Sect. 36/A (2007).

Act CLIV of 1997 on Health, CLIV Sect 15 (1997).

Jiron M, Herrada L, Rojas A, Lueiza A, Vega E, Buckel E, et al. Prevalence of potentially inappropriate medication prescribing among older adults in emergency Department in Chile, vol. 24; 2015. p. 102.

Download references


Thanks for all the pharmacist, pharmacy, general practitioner and patient involved in the project.

No funding was obtained for this study.

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to our privacy policy but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

University Pharmacy Department of Pharmacy Administration, Semmelweis University, Hungary; 7-9 Hőgyes Endre street, Budapest, H-1092, Hungary

András Szilvay, Orsolya Somogyi, Attiláné Meskó, Romána Zelkó & Balázs Hankó

You can also search for this author in PubMed   Google Scholar


ASz co-ordinated the collection of data, planned the processing of the data and performed their processing. ASz wrote the first manuscript. OS and BH created the research plan, OS managed the conduct of the research and co-ordinated pharmacists involved. AM did the statistical analyzes. RZ co-ordinated the publication process OS, RZ, and BH contributed to the discussion and reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Balázs Hankó .

Ethics declarations

Consent for publication.

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Szilvay, A., Somogyi, O., Meskó, A. et al. Qualitative and quantitative research of medication review and drug-related problems in Hungarian community pharmacies: a pilot study. BMC Health Serv Res 19 , 282 (2019).

Download citation

Received : 23 November 2018

Accepted : 22 April 2019

Published : 03 May 2019


Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Pharmaceutical care
  • Community pharmacy
  • Vitamin K antagonist
  • ACE inhibitor

BMC Health Services Research

ISSN: 1472-6963

quantitative research about medicine and its contribution

  • Search Menu
  • Browse content in Arts and Humanities
  • Browse content in Archaeology
  • Anglo-Saxon and Medieval Archaeology
  • Archaeological Methodology and Techniques
  • Archaeology by Region
  • Archaeology of Religion
  • Archaeology of Trade and Exchange
  • Biblical Archaeology
  • Contemporary and Public Archaeology
  • Environmental Archaeology
  • Historical Archaeology
  • History and Theory of Archaeology
  • Industrial Archaeology
  • Landscape Archaeology
  • Mortuary Archaeology
  • Prehistoric Archaeology
  • Underwater Archaeology
  • Urban Archaeology
  • Zooarchaeology
  • Browse content in Architecture
  • Architectural Structure and Design
  • History of Architecture
  • Residential and Domestic Buildings
  • Theory of Architecture
  • Browse content in Art
  • Art Subjects and Themes
  • History of Art
  • Industrial and Commercial Art
  • Theory of Art
  • Biographical Studies
  • Byzantine Studies
  • Browse content in Classical Studies
  • Classical History
  • Classical Philosophy
  • Classical Mythology
  • Classical Literature
  • Classical Reception
  • Classical Art and Architecture
  • Classical Oratory and Rhetoric
  • Greek and Roman Epigraphy
  • Greek and Roman Law
  • Greek and Roman Archaeology
  • Greek and Roman Papyrology
  • Late Antiquity
  • Religion in the Ancient World
  • Digital Humanities
  • Browse content in History
  • Colonialism and Imperialism
  • Diplomatic History
  • Environmental History
  • Genealogy, Heraldry, Names, and Honours
  • Genocide and Ethnic Cleansing
  • Historical Geography
  • History by Period
  • History of Agriculture
  • History of Education
  • History of Emotions
  • History of Gender and Sexuality
  • Industrial History
  • Intellectual History
  • International History
  • Labour History
  • Legal and Constitutional History
  • Local and Family History
  • Maritime History
  • Military History
  • National Liberation and Post-Colonialism
  • Oral History
  • Political History
  • Public History
  • Regional and National History
  • Revolutions and Rebellions
  • Slavery and Abolition of Slavery
  • Social and Cultural History
  • Theory, Methods, and Historiography
  • Urban History
  • World History
  • Browse content in Language Teaching and Learning
  • Language Learning (Specific Skills)
  • Language Teaching Theory and Methods
  • Browse content in Linguistics
  • Applied Linguistics
  • Cognitive Linguistics
  • Computational Linguistics
  • Forensic Linguistics
  • Grammar, Syntax and Morphology
  • Historical and Diachronic Linguistics
  • History of English
  • Language Acquisition
  • Language Variation
  • Language Families
  • Language Evolution
  • Language Reference
  • Lexicography
  • Linguistic Theories
  • Linguistic Typology
  • Linguistic Anthropology
  • Phonetics and Phonology
  • Psycholinguistics
  • Sociolinguistics
  • Translation and Interpretation
  • Writing Systems
  • Browse content in Literature
  • Bibliography
  • Children's Literature Studies
  • Literary Studies (Asian)
  • Literary Studies (European)
  • Literary Studies (Eco-criticism)
  • Literary Studies (Modernism)
  • Literary Studies (Romanticism)
  • Literary Studies (American)
  • Literary Studies - World
  • Literary Studies (1500 to 1800)
  • Literary Studies (19th Century)
  • Literary Studies (20th Century onwards)
  • Literary Studies (African American Literature)
  • Literary Studies (British and Irish)
  • Literary Studies (Early and Medieval)
  • Literary Studies (Fiction, Novelists, and Prose Writers)
  • Literary Studies (Gender Studies)
  • Literary Studies (Graphic Novels)
  • Literary Studies (History of the Book)
  • Literary Studies (Plays and Playwrights)
  • Literary Studies (Poetry and Poets)
  • Literary Studies (Postcolonial Literature)
  • Literary Studies (Queer Studies)
  • Literary Studies (Science Fiction)
  • Literary Studies (Travel Literature)
  • Literary Studies (War Literature)
  • Literary Studies (Women's Writing)
  • Literary Theory and Cultural Studies
  • Mythology and Folklore
  • Shakespeare Studies and Criticism
  • Browse content in Media Studies
  • Browse content in Music
  • Applied Music
  • Dance and Music
  • Ethics in Music
  • Ethnomusicology
  • Gender and Sexuality in Music
  • Medicine and Music
  • Music Cultures
  • Music and Religion
  • Music and Culture
  • Music and Media
  • Music Education and Pedagogy
  • Music Theory and Analysis
  • Musical Scores, Lyrics, and Libretti
  • Musical Structures, Styles, and Techniques
  • Musicology and Music History
  • Performance Practice and Studies
  • Race and Ethnicity in Music
  • Sound Studies
  • Browse content in Performing Arts
  • Browse content in Philosophy
  • Aesthetics and Philosophy of Art
  • Epistemology
  • Feminist Philosophy
  • History of Western Philosophy
  • Metaphysics
  • Moral Philosophy
  • Non-Western Philosophy
  • Philosophy of Science
  • Philosophy of Action
  • Philosophy of Law
  • Philosophy of Religion
  • Philosophy of Language
  • Philosophy of Mind
  • Philosophy of Perception
  • Philosophy of Mathematics and Logic
  • Practical Ethics
  • Social and Political Philosophy
  • Browse content in Religion
  • Biblical Studies
  • Christianity
  • East Asian Religions
  • History of Religion
  • Judaism and Jewish Studies
  • Qumran Studies
  • Religion and Education
  • Religion and Health
  • Religion and Politics
  • Religion and Science
  • Religion and Law
  • Religion and Art, Literature, and Music
  • Religious Studies
  • Browse content in Society and Culture
  • Cookery, Food, and Drink
  • Cultural Studies
  • Customs and Traditions
  • Ethical Issues and Debates
  • Hobbies, Games, Arts and Crafts
  • Lifestyle, Home, and Garden
  • Natural world, Country Life, and Pets
  • Popular Beliefs and Controversial Knowledge
  • Sports and Outdoor Recreation
  • Technology and Society
  • Travel and Holiday
  • Visual Culture
  • Browse content in Law
  • Arbitration
  • Browse content in Company and Commercial Law
  • Commercial Law
  • Company Law
  • Browse content in Comparative Law
  • Systems of Law
  • Competition Law
  • Browse content in Constitutional and Administrative Law
  • Government Powers
  • Judicial Review
  • Local Government Law
  • Military and Defence Law
  • Parliamentary and Legislative Practice
  • Construction Law
  • Contract Law
  • Browse content in Criminal Law
  • Criminal Procedure
  • Criminal Evidence Law
  • Sentencing and Punishment
  • Employment and Labour Law
  • Environment and Energy Law
  • Browse content in Financial Law
  • Banking Law
  • Insolvency Law
  • History of Law
  • Human Rights and Immigration
  • Intellectual Property Law
  • Browse content in International Law
  • Private International Law and Conflict of Laws
  • Public International Law
  • IT and Communications Law
  • Jurisprudence and Philosophy of Law
  • Law and Politics
  • Law and Society
  • Browse content in Legal System and Practice
  • Courts and Procedure
  • Legal Skills and Practice
  • Primary Sources of Law
  • Regulation of Legal Profession
  • Medical and Healthcare Law
  • Browse content in Policing
  • Criminal Investigation and Detection
  • Police and Security Services
  • Police Procedure and Law
  • Police Regional Planning
  • Browse content in Property Law
  • Personal Property Law
  • Study and Revision
  • Terrorism and National Security Law
  • Browse content in Trusts Law
  • Wills and Probate or Succession
  • Browse content in Medicine and Health
  • Browse content in Allied Health Professions
  • Arts Therapies
  • Clinical Science
  • Dietetics and Nutrition
  • Occupational Therapy
  • Operating Department Practice
  • Physiotherapy
  • Radiography
  • Speech and Language Therapy
  • Browse content in Anaesthetics
  • General Anaesthesia
  • Neuroanaesthesia
  • Browse content in Clinical Medicine
  • Acute Medicine
  • Cardiovascular Medicine
  • Clinical Genetics
  • Clinical Pharmacology and Therapeutics
  • Dermatology
  • Endocrinology and Diabetes
  • Gastroenterology
  • Genito-urinary Medicine
  • Geriatric Medicine
  • Infectious Diseases
  • Medical Oncology
  • Medical Toxicology
  • Pain Medicine
  • Palliative Medicine
  • Rehabilitation Medicine
  • Respiratory Medicine and Pulmonology
  • Rheumatology
  • Sleep Medicine
  • Sports and Exercise Medicine
  • Clinical Neuroscience
  • Community Medical Services
  • Critical Care
  • Emergency Medicine
  • Forensic Medicine
  • Haematology
  • History of Medicine
  • Browse content in Medical Dentistry
  • Oral and Maxillofacial Surgery
  • Paediatric Dentistry
  • Restorative Dentistry and Orthodontics
  • Surgical Dentistry
  • Medical Ethics
  • Browse content in Medical Skills
  • Clinical Skills
  • Communication Skills
  • Nursing Skills
  • Surgical Skills
  • Medical Statistics and Methodology
  • Browse content in Neurology
  • Clinical Neurophysiology
  • Neuropathology
  • Nursing Studies
  • Browse content in Obstetrics and Gynaecology
  • Gynaecology
  • Occupational Medicine
  • Ophthalmology
  • Otolaryngology (ENT)
  • Browse content in Paediatrics
  • Neonatology
  • Browse content in Pathology
  • Chemical Pathology
  • Clinical Cytogenetics and Molecular Genetics
  • Histopathology
  • Medical Microbiology and Virology
  • Patient Education and Information
  • Browse content in Pharmacology
  • Psychopharmacology
  • Browse content in Popular Health
  • Caring for Others
  • Complementary and Alternative Medicine
  • Self-help and Personal Development
  • Browse content in Preclinical Medicine
  • Cell Biology
  • Molecular Biology and Genetics
  • Reproduction, Growth and Development
  • Primary Care
  • Professional Development in Medicine
  • Browse content in Psychiatry
  • Addiction Medicine
  • Child and Adolescent Psychiatry
  • Forensic Psychiatry
  • Learning Disabilities
  • Old Age Psychiatry
  • Psychotherapy
  • Browse content in Public Health and Epidemiology
  • Epidemiology
  • Public Health
  • Browse content in Radiology
  • Clinical Radiology
  • Interventional Radiology
  • Nuclear Medicine
  • Radiation Oncology
  • Reproductive Medicine
  • Browse content in Surgery
  • Cardiothoracic Surgery
  • Gastro-intestinal and Colorectal Surgery
  • General Surgery
  • Neurosurgery
  • Paediatric Surgery
  • Peri-operative Care
  • Plastic and Reconstructive Surgery
  • Surgical Oncology
  • Transplant Surgery
  • Trauma and Orthopaedic Surgery
  • Vascular Surgery
  • Browse content in Science and Mathematics
  • Browse content in Biological Sciences
  • Aquatic Biology
  • Biochemistry
  • Bioinformatics and Computational Biology
  • Developmental Biology
  • Ecology and Conservation
  • Evolutionary Biology
  • Genetics and Genomics
  • Microbiology
  • Molecular and Cell Biology
  • Natural History
  • Plant Sciences and Forestry
  • Research Methods in Life Sciences
  • Structural Biology
  • Systems Biology
  • Zoology and Animal Sciences
  • Browse content in Chemistry
  • Analytical Chemistry
  • Computational Chemistry
  • Crystallography
  • Environmental Chemistry
  • Industrial Chemistry
  • Inorganic Chemistry
  • Materials Chemistry
  • Medicinal Chemistry
  • Mineralogy and Gems
  • Organic Chemistry
  • Physical Chemistry
  • Polymer Chemistry
  • Study and Communication Skills in Chemistry
  • Theoretical Chemistry
  • Browse content in Computer Science
  • Artificial Intelligence
  • Computer Architecture and Logic Design
  • Game Studies
  • Human-Computer Interaction
  • Mathematical Theory of Computation
  • Programming Languages
  • Software Engineering
  • Systems Analysis and Design
  • Virtual Reality
  • Browse content in Computing
  • Business Applications
  • Computer Security
  • Computer Games
  • Computer Networking and Communications
  • Digital Lifestyle
  • Graphical and Digital Media Applications
  • Operating Systems
  • Browse content in Earth Sciences and Geography
  • Atmospheric Sciences
  • Environmental Geography
  • Geology and the Lithosphere
  • Maps and Map-making
  • Meteorology and Climatology
  • Oceanography and Hydrology
  • Palaeontology
  • Physical Geography and Topography
  • Regional Geography
  • Soil Science
  • Urban Geography
  • Browse content in Engineering and Technology
  • Agriculture and Farming
  • Biological Engineering
  • Civil Engineering, Surveying, and Building
  • Electronics and Communications Engineering
  • Energy Technology
  • Engineering (General)
  • Environmental Science, Engineering, and Technology
  • History of Engineering and Technology
  • Mechanical Engineering and Materials
  • Technology of Industrial Chemistry
  • Transport Technology and Trades
  • Browse content in Environmental Science
  • Applied Ecology (Environmental Science)
  • Conservation of the Environment (Environmental Science)
  • Environmental Sustainability
  • Environmentalist Thought and Ideology (Environmental Science)
  • Management of Land and Natural Resources (Environmental Science)
  • Natural Disasters (Environmental Science)
  • Nuclear Issues (Environmental Science)
  • Pollution and Threats to the Environment (Environmental Science)
  • Social Impact of Environmental Issues (Environmental Science)
  • History of Science and Technology
  • Browse content in Materials Science
  • Ceramics and Glasses
  • Composite Materials
  • Metals, Alloying, and Corrosion
  • Nanotechnology
  • Browse content in Mathematics
  • Applied Mathematics
  • Biomathematics and Statistics
  • History of Mathematics
  • Mathematical Education
  • Mathematical Finance
  • Mathematical Analysis
  • Numerical and Computational Mathematics
  • Probability and Statistics
  • Pure Mathematics
  • Browse content in Neuroscience
  • Cognition and Behavioural Neuroscience
  • Development of the Nervous System
  • Disorders of the Nervous System
  • History of Neuroscience
  • Invertebrate Neurobiology
  • Molecular and Cellular Systems
  • Neuroendocrinology and Autonomic Nervous System
  • Neuroscientific Techniques
  • Sensory and Motor Systems
  • Browse content in Physics
  • Astronomy and Astrophysics
  • Atomic, Molecular, and Optical Physics
  • Biological and Medical Physics
  • Classical Mechanics
  • Computational Physics
  • Condensed Matter Physics
  • Electromagnetism, Optics, and Acoustics
  • History of Physics
  • Mathematical and Statistical Physics
  • Measurement Science
  • Nuclear Physics
  • Particles and Fields
  • Plasma Physics
  • Quantum Physics
  • Relativity and Gravitation
  • Semiconductor and Mesoscopic Physics
  • Browse content in Psychology
  • Affective Sciences
  • Clinical Psychology
  • Cognitive Neuroscience
  • Cognitive Psychology
  • Criminal and Forensic Psychology
  • Developmental Psychology
  • Educational Psychology
  • Evolutionary Psychology
  • Health Psychology
  • History and Systems in Psychology
  • Music Psychology
  • Neuropsychology
  • Organizational Psychology
  • Psychological Assessment and Testing
  • Psychology of Human-Technology Interaction
  • Psychology Professional Development and Training
  • Research Methods in Psychology
  • Social Psychology
  • Browse content in Social Sciences
  • Browse content in Anthropology
  • Anthropology of Religion
  • Human Evolution
  • Medical Anthropology
  • Physical Anthropology
  • Regional Anthropology
  • Social and Cultural Anthropology
  • Theory and Practice of Anthropology
  • Browse content in Business and Management
  • Business Strategy
  • Business History
  • Business Ethics
  • Business and Government
  • Business and Technology
  • Business and the Environment
  • Comparative Management
  • Corporate Governance
  • Corporate Social Responsibility
  • Entrepreneurship
  • Health Management
  • Human Resource Management
  • Industrial and Employment Relations
  • Industry Studies
  • Information and Communication Technologies
  • International Business
  • Knowledge Management
  • Management and Management Techniques
  • Operations Management
  • Organizational Theory and Behaviour
  • Pensions and Pension Management
  • Public and Nonprofit Management
  • Strategic Management
  • Supply Chain Management
  • Browse content in Criminology and Criminal Justice
  • Criminal Justice
  • Criminology
  • Forms of Crime
  • International and Comparative Criminology
  • Youth Violence and Juvenile Justice
  • Development Studies
  • Browse content in Economics
  • Agricultural, Environmental, and Natural Resource Economics
  • Asian Economics
  • Behavioural Finance
  • Behavioural Economics and Neuroeconomics
  • Econometrics and Mathematical Economics
  • Economic Systems
  • Economic Methodology
  • Economic History
  • Economic Development and Growth
  • Financial Markets
  • Financial Institutions and Services
  • General Economics and Teaching
  • Health, Education, and Welfare
  • History of Economic Thought
  • International Economics
  • Labour and Demographic Economics
  • Law and Economics
  • Macroeconomics and Monetary Economics
  • Microeconomics
  • Public Economics
  • Urban, Rural, and Regional Economics
  • Welfare Economics
  • Browse content in Education
  • Adult Education and Continuous Learning
  • Care and Counselling of Students
  • Early Childhood and Elementary Education
  • Educational Equipment and Technology
  • Educational Strategies and Policy
  • Higher and Further Education
  • Organization and Management of Education
  • Philosophy and Theory of Education
  • Schools Studies
  • Secondary Education
  • Teaching of a Specific Subject
  • Teaching of Specific Groups and Special Educational Needs
  • Teaching Skills and Techniques
  • Browse content in Environment
  • Applied Ecology (Social Science)
  • Climate Change
  • Conservation of the Environment (Social Science)
  • Environmentalist Thought and Ideology (Social Science)
  • Natural Disasters (Environment)
  • Social Impact of Environmental Issues (Social Science)
  • Browse content in Human Geography
  • Cultural Geography
  • Economic Geography
  • Political Geography
  • Browse content in Interdisciplinary Studies
  • Communication Studies
  • Museums, Libraries, and Information Sciences
  • Browse content in Politics
  • African Politics
  • Asian Politics
  • Chinese Politics
  • Comparative Politics
  • Conflict Politics
  • Elections and Electoral Studies
  • Environmental Politics
  • European Union
  • Foreign Policy
  • Gender and Politics
  • Human Rights and Politics
  • Indian Politics
  • International Relations
  • International Organization (Politics)
  • International Political Economy
  • Irish Politics
  • Latin American Politics
  • Middle Eastern Politics
  • Political Methodology
  • Political Communication
  • Political Philosophy
  • Political Sociology
  • Political Theory
  • Political Behaviour
  • Political Economy
  • Political Institutions
  • Politics and Law
  • Public Administration
  • Public Policy
  • Quantitative Political Methodology
  • Regional Political Studies
  • Russian Politics
  • Security Studies
  • State and Local Government
  • UK Politics
  • US Politics
  • Browse content in Regional and Area Studies
  • African Studies
  • Asian Studies
  • East Asian Studies
  • Japanese Studies
  • Latin American Studies
  • Middle Eastern Studies
  • Native American Studies
  • Scottish Studies
  • Browse content in Research and Information
  • Research Methods
  • Browse content in Social Work
  • Addictions and Substance Misuse
  • Adoption and Fostering
  • Care of the Elderly
  • Child and Adolescent Social Work
  • Couple and Family Social Work
  • Developmental and Physical Disabilities Social Work
  • Direct Practice and Clinical Social Work
  • Emergency Services
  • Human Behaviour and the Social Environment
  • International and Global Issues in Social Work
  • Mental and Behavioural Health
  • Social Justice and Human Rights
  • Social Policy and Advocacy
  • Social Work and Crime and Justice
  • Social Work Macro Practice
  • Social Work Practice Settings
  • Social Work Research and Evidence-based Practice
  • Welfare and Benefit Systems
  • Browse content in Sociology
  • Childhood Studies
  • Community Development
  • Comparative and Historical Sociology
  • Economic Sociology
  • Gender and Sexuality
  • Gerontology and Ageing
  • Health, Illness, and Medicine
  • Marriage and the Family
  • Migration Studies
  • Occupations, Professions, and Work
  • Organizations
  • Population and Demography
  • Race and Ethnicity
  • Social Theory
  • Social Movements and Social Change
  • Social Research and Statistics
  • Social Stratification, Inequality, and Mobility
  • Sociology of Religion
  • Sociology of Education
  • Sport and Leisure
  • Urban and Rural Studies
  • Browse content in Warfare and Defence
  • Defence Strategy, Planning, and Research
  • Land Forces and Warfare
  • Military Administration
  • Military Life and Institutions
  • Naval Forces and Warfare
  • Other Warfare and Defence Issues
  • Peace Studies and Conflict Resolution
  • Weapons and Equipment

Oxford Textbook of Medical Education

  • < Previous chapter
  • Next chapter >

Oxford Textbook of Medical Education

53 Quantitative research methods in medical education

  • Published: October 2013
  • Cite Icon Cite
  • Permissions Icon Permissions

Quantitative research in medical education tends to be predominantly observational research based on survey or correlational studies. As researchers strive towards making inferences about the impact of education interventions, a shift towards experimental research designs may enhance the quality and conclusions made in medical education. The establishment of experiment research designs, where interventions (i.e. curriculum, teaching or assessment interventions) are tested with an experimental group and either a comparison or controlled group of learners, may allow researchers to overcome validity concerns and infer potential cause–effect generalizations. There are a number of internal and external validity concerns that researchers need to be conscious of when designing their own or looking at others’ experimental research studies. The selection of a research design for any study should fit within the parameters of the stated research question or hypothesis. In quantitative research, the findings will reflect the reliability and validity (psychometric characteristics) of the measured outcomes or dependent variables (such as changes in knowledge, skills, or attitudes) used to assess the effectiveness of the medical education intervention (the independent variable of interest). It is important to remember that not all quantitative research involves experimental studies—important results can also be drawn from quantitative observational studies. This chapter outlines commonly used quantitative methods in medical education research. It explains their theoretical underpinnings, the evidence base for their use, and gives practical guidance on their application. It concludes with a section on the role of meta-analyses of quantitative research in medical education.

Signed in as

Institutional accounts.

  • GoogleCrawler [DO NOT DELETE]
  • Google Scholar Indexing

Personal account

  • Sign in with email/username & password
  • Get email alerts
  • Save searches
  • Purchase content
  • Activate your purchase/trial code

Institutional access

  • Sign in with a library card Sign in with username/password Recommend to your librarian
  • Institutional account management
  • Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Sign in through your institution

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

  • Click Sign in through your institution.
  • Select your institution from the list provided, which will take you to your institution's website to sign in.
  • When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
  • Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Sign in with a library card

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

  • Click Sign in through society site.
  • When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

  • View your signed in personal account and access account management features.
  • View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Rights and permissions
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.


Quantitative Analysis of the Effectiveness of Public Health Measures on COVID-19 Transmission

  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: [email protected]
  • Info/History
  • Preview PDF

Although COVID-19 has spread almost all over the world, social isolation is still a controversial public health policy and governments of many countries still doubt its level of effectiveness. This situation can create deadlocks in places where there is a discrepancy among municipal, state and federal policies. The exponential increase of the number of infectious people and deaths in the last days shows that the COVID-19 epidemics is still at its early stage in Brazil and such political disarray can lead to very serious results. In this work, we study the COVID-19 epidemics in Brazilian cities using early-time approximations of the SIR model in networks. Different from other works, the underlying network is constructed by feeding real-world data on local COVID-19 cases reported by Brazilian cities to a regularized vector autoregressive model, which estimates directional COVID-19 transmission channels (links) of every pair of cities (vertices) using spectral network analysis. Our results reveal that social isolation and, especially, the use of masks can effectively reduce the transmission rate of COVID-19 in Brazil. We also build counterfactual scenarios to measure the human impact of these public health measures in terms of reducing the number of COVID-19 cases at the epidemics peak. We find that the efficiency of social isolation and of using of masks differs significantly across cities. For instance, we find that they would potentially decrease the COVID-19 epidemics peak in São Paulo (SP) and Brasília (DF) by 15% and 25%, respectively. We hope our study can support the design of further public health measures.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This work is supported in part by the Sao Paulo State Research Foundation (FAPESP) under grant numbers 2015/50122-0 and 2013/07375-0, the Brazilian Coordination of Superior Level Staff Improvement (CAPES), the Pro-Rectory of Research (PRP) of University of Sao Paulo under grant number 2018.1.1702.59.8, and the Brazilian National Council for Scientific and Technological Development (CNPq) under grant numbers 303199/2019-9, 308171/2019-5, and 408546/2018-2.

Author Declarations

All relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Data Availability

All data used in this work is public available.

View the discussion thread.

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Reddit logo

Citation Manager Formats

  • EndNote (tagged)
  • EndNote 8 (xml)
  • RefWorks Tagged
  • Ref Manager
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Infectious Diseases (except HIV/AIDS)
  • Addiction Medicine (317)
  • Allergy and Immunology (621)
  • Anesthesia (162)
  • Cardiovascular Medicine (2302)
  • Dentistry and Oral Medicine (280)
  • Dermatology (205)
  • Emergency Medicine (372)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (821)
  • Epidemiology (11640)
  • Forensic Medicine (10)
  • Gastroenterology (686)
  • Genetic and Genomic Medicine (3640)
  • Geriatric Medicine (342)
  • Health Economics (625)
  • Health Informatics (2337)
  • Health Policy (921)
  • Health Systems and Quality Improvement (872)
  • Hematology (337)
  • HIV/AIDS (761)
  • Infectious Diseases (except HIV/AIDS) (13214)
  • Intensive Care and Critical Care Medicine (761)
  • Medical Education (361)
  • Medical Ethics (101)
  • Nephrology (394)
  • Neurology (3395)
  • Nursing (193)
  • Nutrition (512)
  • Obstetrics and Gynecology (655)
  • Occupational and Environmental Health (655)
  • Oncology (1784)
  • Ophthalmology (528)
  • Orthopedics (212)
  • Otolaryngology (284)
  • Pain Medicine (226)
  • Palliative Medicine (66)
  • Pathology (441)
  • Pediatrics (1014)
  • Pharmacology and Therapeutics (423)
  • Primary Care Research (411)
  • Psychiatry and Clinical Psychology (3114)
  • Public and Global Health (6037)
  • Radiology and Imaging (1238)
  • Rehabilitation Medicine and Physical Therapy (721)
  • Respiratory Medicine (815)
  • Rheumatology (370)
  • Sexual and Reproductive Health (362)
  • Sports Medicine (319)
  • Surgery (390)
  • Toxicology (50)
  • Transplantation (171)
  • Urology (144)
  • Open access
  • Published: 24 September 2018

Showcasing the contribution of social sciences to health policy and systems research

  • Stephanie M. Topp   ORCID: 1 , 2 ,
  • Kerry Scott 3 , 4 ,
  • Ana Lorena Ruano 5 , 6 &
  • Karen Daniels 7 , 8  

International Journal for Equity in Health volume  17 , Article number:  145 ( 2018 ) Cite this article

23k Accesses

4 Citations

18 Altmetric

Metrics details

This Special Issue represents a critical response to the frequent silencing of qualitative social science research approaches in mainstream public health journals, particularly in those that inform the field of health policy and systems research (HPSR), and the study of equity in health.

This collection of articles is presented by SHAPES, the thematic working group of Health Systems Global focused on social science approaches to research and engagement in health policy and systems. The issue aims to showcase how qualitative and theory-driven approaches can contribute to better promoting equity in health within the field of HPSR.

This issue builds on growing recognition of the complex social nature of health systems. The articles in this collection underscore the importance of employing methods that can uncover and help explain health system complexities by exploring the dynamic relationships and decision-making processes of the human actors within. Articles seek to highlight the contribution that qualitative, interpretivist, critical, emancipatory, and other relational methods have made to understanding health systems, health policies and health interventions from the perspective of those involved. By foregrounding actor perspectives, these methods allow us to explore the impact of vital but difficult-to-measure concepts such as power, culture and norms.

This special issue aims to highlight the critical contribution of social science approaches. Through the application of qualitative methods and, in some cases, development of theory, the articles presented here build broader and deeper understanding of the way health systems function, and simultaneously inform a more people-centred approach to collective efforts to build and strengthen those systems.

This Special Issue represents a critical response to the frequent silencing of qualitative social science research approaches in mainstream public health journals, particularly in those that inform the field of health policy and systems research (HPSR), and the study of equity in health [ 1 , 2 , 3 ]. The issue is presented by SHAPES, a thematic working group of Health Systems Global (a membership-based society which aims to convene researchers, policymakers and implementers to develop the field of HSPR) focused on social science approaches.

By bringing together this collection of articles, the special issue highlights the critical contribution of qualitative social sciences including  interpretivist, critical, emancipatory, and other relational methods to our understanding of health systems, policies and interventions. Today, political, professional and disciplinary structures continue to privilege positivist research and quantitative methods, attributing greater evidential value to the knowledge produced by these approaches. This issue builds on growing recognition of the complex social nature of health systems [ 4 ] and on the understanding that utilizing only positivist research approaches in the study of health and health systems contributes to stripping away human experience and context. Articles in this issue demonstrate the importance of employing qualitative social science methods to explore the perspectives, experiences, relationships and decision-making processes of human actors within health systems, and in so doing, help uncover and explain the impact of vital but difficult-to-measure issues such as power, culture and norms. Through their application of qualitative methods and, in some cases, development of theory, they help build a broader and deeper understanding of the way various health systems function, and simultaneously inform a more people-centred approach to collective efforts to build and strengthen those systems.

Linked by two important themes, this initial collection of six research papers and two commentaries cut across a range of social science approaches and include policy analysis, rapid ethnography, and theory driven sociological enquiry. Future papers will be added to an online thematic collection on a rolling basis.

Global policies, local realities

The ways in which global health policies are absorbed into national and subnational health systems, and their impact as they interact with local realities is a strong theme running through the issue.

Contractor, et al. [ 5 ] use rapid ethnography to explore the dissonance between tribal women’s perception of pregnancy and childbirth, and the Indian health system’s approach to maternity care in the context of a national policy that strongly incentivises facility-based birth. Drawing on five months of data collection in Odisha state, this exploratory study used qualitative methods to document how different actors perceived and experienced the policy. Unstructured group discussions explored community perceptions around pregnancy and childbirth; in-depth interviews explored women’s actual experiences and practices of pregnancy and childbirth; key informant interviews with service provides yielded contextual information about the field area and views from within the health system; and observations enabled triangulation and produced first-hand information about the location and conditions of health services and tribal areas. The authors highlight the tensions between priorities embedded within national-level policies and tribal women’s own preferences and needs when it comes to childbirth. Their narratives demonstrate how multiple financial, geographic, social and cultural factors mitigate against uptake of facility-based maternity services, and result in pressure, sometimes coercion, by local health system actors, to comply. The article demonstrates the importance of qualitative methods and grounded analysis for surfacing the unintended consequences of blanket state policies through documentation of its impacts on so-called beneficiaries.

Also focussing on India, Sriram, et al. [ 6 ] present a nuanced, contextually rich analysis, reflecting on the way that actors from high-income countries and members of the extended Indian diaspora contribute to socialisation and legitimation of a new medical speciality (emergency medicine). The research draws on a full year of qualitative data collection conducted by the first author including interviews with 76 participants across 11 towns/cities within India, review of 248 documents and observation of 6 meetings. The authors use framework analysis, applying concepts from the literature to insights emerging from the reading of the data, and brought both emic (subject; the first author is a member of the diaspora) and etic (observer) perspectives in making sense of the data. They point to the way power within these networks resulted in the rapid growth of the speciality of emergency medicine, but also influenced its evolution as a highly medicalised, tertiary-level form of care, inaccessible to the majority of Indians for structural reasons including affordability and availability. The authors note that the socialisation of domestic Indian stakeholders in this field ‘flows from a long history of LMIC (low- and middle-income country) stakeholders adopting ideas from high-income countries, driven by undercurrents of globalization and innovations in communication and technology’. Through the personal accounts of stakeholders with a range of perceptions and experiences in relation to the growth of emergency medicine, the authors interrogate and debunk the positive narrative of knowledge flow from high-income countries to LMICs. The qualitative analysis presented instead paints a complex picture, in which power influences knowledge transfer, the outcome of which is not always experienced as beneficial or positive.

Lodenstein, et al. [ 7 ] describe the contradictory role played by traditional leaders in Malawi in the pursuit of improved reproductive health outcomes. They bring attention to the power of traditional leaders, who are regarded as key to facilitating community adoption of positive public health norms including earlier and more frequent attendance at clinic-based antenatal visits. In recent times, the adoption of public health norms in Malawi has been driven by by-laws, set by traditional leaders and with often punitive consequences for those who do not comply; for example imposing fines on women who do not attend antenatal care or who are not accompanied by their husbands on those visits. While some have heralded the success of such by-laws, the authors use qualitative methods and a gendered perspective to explore these as a social process of norm formulation from the perspective of stakeholders involved in by-law creation, as well as the perspective of those affected by them. Recognising that norms are expressed in multiple ways (rules, behaviours, narratives and mechanisms of enforcement), the authors collected data from various sources (documents, observations, and interviews), so as to explore this range of expression. They show that although by-laws were meant to strengthen service uptake and improve health outcomes for pregnant women, they also resulted in the most vulnerable women bearing the moral and material responsibility for any perceived failure to meet reproductive health policy and targets. This study, which is grounded in rich contextual experience, provides important information to national and global health systems decision makers who may be considering using traditional lines of authority to enhance uptake of public health interventions.

Resources and mechanisms of redress

While the above papers describe, and to differing extents deconstruct, the ways in which health systems interact with and exacerbate broader social and structural inequities, a second harmonizing theme in this collection is the way different resources and mechanisms can be mobilised as a form of redress to such inequities.

Spanning both themes showcased in this issue, Turcotte-Tremblay, et al. [ 8 ] describe the local effects of a globally touted performance-based financing (PBF) policy in Burkina Faso. The authors examine the equity measures (such as user fee exemptions available to those holding an indigent card) within PBF, which were introduced to address inequitable access. The study is framed using Rogers’ diffusion of innovations theory. In a comparative case study design across four primary health services, the authors utilise empirical methods, including 93 interviews, discussions, observation and document analysis. Using primary data the authors are able explore the way multiple local actors, including members of local indigent selection committees, re-invented elements of the PBF equity measures over which they had control, to either increase their relative advantage or to adapt to implementation challenges and context. For example, distributing free or very low-cost medications led to financial difficulties and drug shortages at some clinics and compensatory actions intended to resolve these problems by the ‘street level bureaucrats’ running front-line services led to adverse knock-on impacts for clients. Ultimately, the authors demonstrate how local knowledge of what it means to be indigent, and the power dynamics inherent within the health services, interacted with PBF implementation to result in both ‘uncertain and unequal’ coverage of the policy.

Topp, et al. [ 9 ] report on an empirical study of a policy-driven effort to improve the social accountability of prison health services in Zambia through the establishment of prison health committees. Locating their work in the discipline of public policy, the authors use a combination of interviews, focus groups and ethnographic observation, and begin by exploring Joshi’s three domains of impact for social accountability interventions: state responsiveness (represented by facility-based prison officials), societal impact (represented in this study by inmates), and state-society relations. (represented by relations between inmates and prison officials). Their analysis reflects on the ways in which power relations became less hierarchical, and how health outcomes improved in one particular prison after the introduction of a staff-inmate committee. A second phase of analysis draws on a more theoretical and (hence) more widely generalisable model comprising three intersecting ‘axes’ of accountability: power, ability and justice [ 10 ], using these axes to examine the depth and breadth of committee impact. The authors conclude that in relation to prison health care, local context as well as national level politics and legislative reforms, “will be crucial to support democratic decision-making, authentic engagement and appropriate action” in prison health services in low-income settings.

Kapilashrami and Marsden [ 11 ] report a study of access to health-enabling resources by multiply-disadvantaged groups in a deprived part of Scotland. Drawing on human geography and political science, their research uses the theoretical concept of intersectionality – that is, “the multiple interacting influences of social location, identity and historical oppression” – and a combination of standard qualitative tools (interviews, focus groups) and more contemporary and participatory methods (notably collaborative health resource mapping). The authors find that health-enabling resources were variously material, environmental, cultural or affective, with the combined influence of these resources playing out differently for different individuals. Amartya Sen observed in his health capabilities framework, the need to consider both individual choices and societal chances [ 12 ]. Through their use of multiple qualitative methods and the application of intersectionality, the authors demonstrate how individual responsibility, and blame, for health-related behaviour choices is an impoverished explanatory framework because it overlooks the institutional, structural and environmental influences on such behaviours.

While methodologically heterogenous, the articles in this issue showcase just some of the ways in which qualitative social science methods generate important new knowledge that is sensitive to context and which can act as a means to ‘un-silencing’ voices on the margins. Greenhalgh [ 13 ] in her commentary highlights these points, discussing the important role of critical social science as an underutilised method of social critique and emancipation of oppressed groups. She notes that methods such as these ask, “whose definitions count?”; “who makes the rules?”; and “whose voice is not being heard?”

In their commentary too, Lewin and Glenton [ 14 ] note that perhaps the key role of qualitative social sciences is to represent “the views and experiences of stakeholders, including vulnerable and marginalised groups who are often not represented directly.” And indeed, this collection represents a clear body of evidence that health policies and system levers require much work. Contractor, et al. [ 5 ], Kapilshrami, et al. [ 11 ] and Lodenstein, et al.’s [ 7 ] articles, in particular, demonstrate how qualitative social science research can surface issues experienced by, and present the voices of, people on the ground, helping to hold to account global and national health systems leaders responsible for health policy and planning. In order to realise the full value of this type of evidence, however, Lewin and Glenton [ 14 ] also argue the need for more investment in our collective capacity to synthesise the knowledge generated, and to work more closely with policy users and other stakeholders to build their capacity for evidence use.

Articles in this issue demonstrate how qualitative social science methods may be used to engage and participate with actors to co-produce knowledge, evidence and even solutions for change [ 1 ]. At inception, the idea for this special issue also encompassed ambitious plans to model a participatory and empowering approach through mentorship of early career authors, as well as for those based in LMICs. These ideas align with the values of people-centredness and equity that underpin the broader mission of Health Systems Global. Many of the lead authors in this issue are early career researchers, although most are either based in, or receive substantial support from institutions in high-income settings. We therefore believe more personal and institutional investment in learning opportunities through webinars, online teaching, and one-to-one mentoring needs to be made available. This has been recently modelled through various initiatives undertaken by HSG members, affiliates, and thematic working groups. We also acknowledge the ongoing challenges experienced specifically by health system actors who work in, or alongside, services to find the time or receive the guidance necessary to write about what they do. Questions that arose in the process of collating this issue, and which require more, and deeper examination include: how should rich (practitioner) experiences be documented? Should such documentation be acknowledged as a form of research? And if so, where does it belong in a saturated, but often siloed, publication world?

Critically engaging with issues of inclusion, voice and power is vital to building equitable and people-centred health systems and must be at the heart of the research processes that support these systems. As showcased in this special issue, robust qualitative social science research is ideally suited to understanding the social systems that generate or limit opportunities for equity in health, and that must be engaged with and transformed to build truly people-centred health systems.


health policy and systems research

Health Systems Global

Low- and Middle Income Countries

performance based financing

social science approaches for research and engagement in health policy & systems

Daniels K, Hanefeld J, Marchal B. Social sciences: vital to improving our understanding of health equity, policy and systems. Int J Equity Health. 2017;16(1):57.

Article   Google Scholar  

Greenhalgh T, Annandale E, Ashcroft R, Barlow J, Black N, Bleakley A, Boaden R, Braithwaite J, Britten N, Carnevale F, et al. An open letter to the BMJ editors on qualitative research. BMJ. 2016;352:i563.

SHAPES Health Systems Global, Regional network for equity in health in east and southern Africa, emerging voices for Global Health, Daniels K, Loewenson R, George A, Howard N, Koleva G, Lewin S, Marchal B et al : Fair publication of qualitative research in health systems: a call by health policy and systems researchers. Int J Equity Health 2016, 15:98.

Freedman LP. Reflections on emerging frameworks of health and human rights. Health and Human Rights. 1995;1(4).

Contractor SQ, Das A, Dasgupta J, Van Belle S: Beyond the template: the needs of tribal women and their experiences with maternity services in Odisha, India. Int J Equity Health 2018, 17(134):13.

Sriram V, George A, Baru R, Bennett S. Socialization, legitimization and the transfer of biomedical knowledge to low- and middle-income countries: analyzing the case of emergency medicine in India. Int J Equity Health. 2018;17:142.

Lodenstein E, Pedersen K, Botha K, Broerse JEW, Dieleman M. Gendered norms of responsibility: reflections on accountability politics in maternal healthcare in Malawi. Int J Equity Health. 2018;17:131.

Turcotte-Tremblay A-M, De Allegri M, Gali Gali IA, Ridde V. The unintended consequences of combining equity measures with performance-based financing in Burkina Faso. Int J Equity Health. 2018;17:109.

Topp SM, Sharma A, Chileshe C, Magwende G, Henostroza G, Moonga CM. The health system accountability impact of prison health committees in Zambia. Int J Equity Health. 2018;17:74.

George AS, Erchick DJ, Zubairu MM, Barau IY, Wonodi C. Sparking, supporting and steering change: grounding an accountability framework with viewpoints from Nigerian routine immunization and primary health care government officials. Health Policy Plan. 2016;31(9):1326–32.

Kapilashrami A, Marsden S. Examining intersectional inequalities in access to health (enabling) resources in disadvantaged communities in Scotland: advancing the participatory paradigm. Int J Equity Health. 2018;17:83.

Sen A: Human rights and capabilities. J Hum Dev 2005, 6(2).

Greenhalgh T. What have the social sciences ever done for equity in health policy and health systems? Int J Equity Health. 2018;17(124):3.

Google Scholar  

Lewin S, Glenton C. Are we entering a new era for qualitative research? Using qualitative evidence to support guidance and guideline development by the World Health Organization. Int J Equity Health. 2018;17(126):5.

Download references


We thank Health Systems Global for its support of this and several other articles in this special issue .

Health Systems Global supported this work through payment of the Article Processing Charge.

Author information

Authors and affiliations.

College of Public Health, Medical and Veterinary Sciences, James Cook University, Townsville, 4810, QLD, Australia

Stephanie M. Topp

Nossal Institute for Global Health, University of Melbourne, Melbourne, 3002, VIC, Australia

Research Consultant, Bengaluru, Karnataka, India

Kerry Scott

Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Baltimore, 21205, USA

Center for the Study of Equity and Governance in Health Systems, Guatemala, Guatemala

Ana Lorena Ruano

Center for International Health University of Bergen, Bergen, Norway

Health Systems Research Unit, South African Medical Research Council, Cape Town, South Africa

Karen Daniels

Health Policy and Systems Division, School of Public Health and Family Medicine, University of Cape Town, Cape Town, South Africa

You can also search for this author in PubMed   Google Scholar


All authors conceived and helped draft the article. All authors reviewed, provided critical feedback on and approved the final version of the manuscript.

Corresponding author

Correspondence to Stephanie M. Topp .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Topp, S.M., Scott, K., Ruano, A.L. et al. Showcasing the contribution of social sciences to health policy and systems research. Int J Equity Health 17 , 145 (2018).

Download citation

Received : 06 September 2018

Accepted : 11 September 2018

Published : 24 September 2018


Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Health systems
  • Health policy
  • Social science
  • Qualitative

International Journal for Equity in Health

ISSN: 1475-9276

quantitative research about medicine and its contribution

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

  • Advanced Search
  • Journal List
  • v.55; Jan-Dec 2018

A Quantitative Observational Study of Physician Influence on Hospital Costs

Herbert wong.

1 U.S. Department of Health and Human Services, Agency for Healthcare Research and Quality, Rockville, MD, USA

Zeynal Karaca

Teresa b. gibson.

2 IBM Watson Health, Ann Arbor, MI, USA

Physicians serve as the nexus of treatment decision-making in hospitalized patients; however, little empirical evidence describes the influence of individual physicians on hospital costs. In this study, we examine the extent to which hospital costs vary across physicians and physician characteristics. We used all-payer data from 2 states representing 15 237 physicians and 2.5 million hospital visits. Regression analysis and propensity score matching were used to understand the role of observable provider characteristics on hospital costs controlling for patient demographics, socioeconomic characteristics, clinical risk, and hospital characteristics. We used hierarchical models to estimate the amount of variation attributable to physicians. We found that the average cost of hospital inpatient stays registered to female physicians was consistently lower across all empirical specifications when compared with male physicians. We also found a negative association between physicians’ years of experience and the average costs. The average cost of hospital inpatient stays registered to foreign-trained physicians was lower than US-trained physicians. We observed sizable variation in average costs of hospital inpatient stays across medical specialties. In addition, we used hierarchical methods and estimated the amount of remaining variation attributable to physicians and found that it was nonnegligible (intraclass correlation coefficient [ICC]: 0.33 in the full sample). Historically, most physicians have been reimbursed separately from hospitals, and our study shows that physicians play a role in influencing hospital costs. Future policies and practices should acknowledge these important dependencies. This study lends further support for alignment of physician and hospital incentives to control costs and improve outcomes.

  • What do we already know about this topic?
  • Specific physician characteristics influence a physician’s practice style as well as health care cost, delivery of care and outcomes.
  • How does your research contribute to the field?
  • Our research expands the current literature by performing an all-payer (vs single payer) analysis, using hierarchical models to estimate the amount of variation attributable to individual physicians, and partitioning the variation in hospital costs to understand the extent of influence attributable to physicians.
  • What are your research’s implications toward theory, practice, or policy?
  • We found substantial variation in hospital costs with observable physician characteristics, lending further support for payment and organizational models that align physician and hospital incentives that seek to control costs and improve outcomes.


It has been well established that health care spending varies with geography. 1 - 3 The source of this variation has been often questioned—whether it is arising from area practice patterns, patient health status, patient characteristics, price, and/or individual provider decision making. 3 , 4

An Institute of Medicine (IoM) Committee examining geographic variations in Medicare spending convened earlier this decade concluded that individual providers of care had a great deal of influence on spending. 5 The Committee found post-acute care and inpatient care had the largest amounts of variation in spending, and discovered large variations in provider behavior. 4 Recommendations from the IoM Committee stated that evidence pointed away from geographic or small area spending signatures and toward health care decision-makers. Similarly, Gottleib and colleagues 6 performed a study of spending variation controlling for patient demographic characteristics, health status, and prices between regions, and found that price contributed only a small fraction of variation in spending although patients with similar characteristics received different levels of care from providers.

Previous studies have long demonstrated that specific physician characteristics influence a physician’s practice style. 7 - 10 Several studies have assessed how well physician characteristics explain the variation in hospital resource use. 1 , 11 - 13 Other researchers profiled physicians by analyzing and comparing the effects of their characteristics on health care cost, delivery, and outcomes. 14 - 19 A recent study by Tsugawa and colleagues 20 demonstrated the existence of physician influence on Part B Medicare spending and the extent of spending attributable to physicians.

Other recent studies have examined the relationship between observable characteristics of physicians and health care spending and outcomes. For example, patients treated by graduates of foreign medical schools had lower mortality but higher Medicare Part B payments than those graduating from US medical schools. 21 Elderly patients with a female physician had lower mortality and readmission rates than male physicians. 22 In a separate study, no clear pattern was found between patient mortality and physician age for elderly patients, but patients with an older physician had higher Medicare Part B payments. 23 Also, Southern and colleagues 24 found that tenure in practice was positively associated with higher risk of mortality and longer lengths of stay in a local hospital system.

In this study, we use all-payer inpatient data from 2 states, Arizona and Florida, to analyze and quantify the extent of physician influence on inpatient hospital costs other than professional services. Hospital care accounts for 32% of national health care expenditures and is the largest expense category in 2015. 25 In addition, physicians are responsible for selecting the course of care provision and treatment, thereby influencing hospital costs of care.

Our research has 2 aims. First, we describe the relationship between hospital costs and observable characteristics of physicians including physician gender and foreign medical school graduation while controlling for patient demographics, socioeconomic characteristics, clinical risk, and hospital characteristics, although cost in this analysis cannot be distinguished between patients and payers. Second, we measure the fraction of variation in costs of hospital inpatient visits due to individual physicians, controlling for observable physician characteristics, patient demographics, socioeconomic characteristics, clinical risk, and hospital characteristics.

This article complements and expands upon the existing empirical literature in several important ways. First, our data are all-payer and do not limit the analysis to a specific payer group or patient group. This extends the previous literature as most recent studies have focused on the physician role in Part B spending variation in large Medicare samples 20 - 23 or within small, local samples. 24 Most physicians have a mix of patients covered by Medicare, Medicaid, private payers, and the uninsured, and we seek to understand their role in influencing hospital costs across all-payer groups. In addition, we use hierarchical models to estimate the amount of variation attributable to individual physicians, controlling for patient demographics, socioeconomic characteristics, clinical risk, and hospital characteristics, allowing us to partition the variation in hospital costs and to understand the extent of influence attributable to physicians. Finally, we use regression analysis and propensity score matching to further understand the role of providers on hospital costs.

We used the Healthcare Cost and Utilization Project (HCUP) 2008 State Inpatient Databases (SID) for Arizona and Florida. These HCUP SID files include all inpatient hospitalizations for nearly all acute care nonfederal hospitals in the subject states. The SID provide detailed diagnoses and procedures, total charges, and patient demographics including gender, age, race, and expected payment source (ie, Medicare, Medicaid, private insurance, other insurance, and self-pay). Physician characteristic information (eg, specialty, year of graduation from medical school, and the name of the medical school) was obtained from the Arizona Board of Medical Examiners and the Florida Department of Health. With permissions from all data partners, information was linked to the Arizona SID using both physician license number and physician name, and to the Florida SID using physician license numbers as the Florida SID do not provide physician name. i The physician represented the surgeon (operating physician), if a surgery was performed, otherwise, the attending physician who is responsible for overall care from admission to discharge. In addition, supplemental hospital characteristic and area characteristic information were obtained, respectively, from the American Hospital Association (AHA) and Area Resource Files. The total number of hospital inpatient visits during 2008 in Arizona and Florida was about 3.31 million, and about 5% were missing physician identifiers. We successfully linked 2.53 million of these visits to physician licensure databases and AHA hospital survey data. All investigators signed a Data Use Agreement. Because HCUP does not involve human subjects, institutional review board approval was not required for this study.

Our key covariates of interest were the physician’s gender, years of experience, board certified specialties, and whether they graduated from a medical school outside of the United States. We calculated years of experience as the difference between 2008 and the year the physician graduated from medical school. We created a series of dummy variables to represent the physician’s specialties of surgery, internal medicine, obstetrics and gynecology, neurology, psychiatry, pediatrics, cardiology, family medicine and general practitioners, and urology. The effect of being foreign-trained was examined by including a separate dummy variable for physicians if they graduated from a medical school outside of the United States. While physicians’ names and the name of their medical school are included in physicians’ licensure databases, physicians’ gender and the location of their medical schools are not readily available. We obtained these from various data sources and online search engines including , , , and . For physician gender, we followed a systematic assignment process requiring matching information from at least 2 independent data sources. Complete information regarding major physician characteristics was obtained for 2.53 million discharges studied in this study.

Hospital costs represent the underlying expenses to produce the hospital services. Since hospitals differ in their markup from costs to charges, we first reduced the charge for each case based on the hospital’s all-payer, inpatient cost-to-charge ratio. ii We applied hospital-specific all-payer cost-to-charge ratios, and replaced all-payer cost-to-charge ratios with group-average all-payer inpatient cost-to-charge ratios when hospital-specific all-payer inpatient cost-to-charge ratios were missing. Next, we adjusted these costs with the area wage index iii computed by the Centers for Medicare and Medicaid Services (CMS) to control for price factors beyond the hospital’s control. We also obtained information about hospital characteristics (eg, teaching status and bed size) using the AHA Annual Survey Database.

Empirical Models

Our study’s empirical models employ a hierarchical framework 26 to assess the effects of physician characteristics on the costs of hospital inpatient visits; we developed a model that controls for physician characteristics, patient demographics, socioeconomic characteristics, clinical risk, and hospital characteristics.

We reassessed the impacts of physician characteristics on costs of hospital inpatient visits using multilevel regression analysis where hospital inpatient visits were clustered by physician.

Our empirical model takes the following general form:

where variation in the intercept is predicted at level 2 by

Substituting (2) into Equation (1) yields the following single multilevel equation:

where i indexes the hospital inpatient visits and j indexes the physicians who treated the i th visit, and LogCost ij is the natural log value of the total hospital inpatient cost associated with the i th visit in the j th physician unit. Physician j is a vector of physicians’ characteristics that includes physicians’ years of experience measured as the difference between 2008 and their year of graduation from medical school, a set of dummy variables for physicians’ board certified specialties (surgery, internal medicine, obstetrics and gynecology, neurology, psychiatry, pediatrics, cardiology, family medicine and general practitioners, urology), for physicians’ gender, and for physicians who graduated from a medical school outside of the United States. Demographic i is a vector of observable patient demographic characteristics, which include age (in age/10 scale), and dummy variables for race and gender. Socioecon i includes a set of county-level dummy variables for income (low, low-medium, medium-high, and high) and for patients’ primary insurance providers (ie, Medicare, Medicaid, private, and other). Risk i includes dummy variables for the Elixhauser comorbidity index. 27 Severity i is the high-severity-measure dummy variable (with value 1 for the patient when All Patient Refined Diagnosis Related Groups (APR-DRG) severity index takes a value of 3 or 4). 28 Hospital i includes a set of dummy variables related to hospital characteristics—including teaching status, ownership type, bed size, and state (Arizona or Florida)—that may also represent unmeasured severity of illness for a patient referred to a highly capable institution. Finally, γ j represents departures of the j th physician from the overall mean that serves to shift the overall regression line representing the population average up or down according to each physician, and ξ ij is the level 1 random error. The random components of this model provide information about intraclass correlation coefficients (ICCs), which enables us to understand variation in costs of hospital inpatient visits associated with physicians’ characteristics. Our level 1 predictor variables are dummy variables except for age, which we standardized by dividing by 10. In our case, centering around the grand mean or using raw metric values did not change the direction of estimates. Therefore, we used raw metric values in our regression analysis. iv We present findings overall, and for teaching and nonteaching hospitals, as physician mix and patient complexity may vary between these types of facilities.

Sensitivity Analysis

We also developed various scenarios to test the robustness of our results. Specifically, we enhanced our model by incorporating level 2 variation not only in intercept but also in slope. Under this model, we assumed that patients had certain preferences in their choice of physician. We ran 3 models with level 2 variations within physicians by these patient characteristics: gender, severity of illness, and gender and severity of illness. Our empirical findings in these 3 models, where both intercepts and slopes varied in level 2, were parallel to our model with level 2 intercept-only variations. For the purpose of clarity, we provide the results for our base model where level 2 variations are only observed through intercepts that represent departures for each physician from the overall mean.

Some researchers claim that there is an implicit relationship between patient gender and physician gender 7 , 29 - 32 or physicians’ practice style and their graduating medical school, 34 which could introduce some degree of endogeneity into our empirical model as presented above. Although our multilevel model substantially reduces the unobservable endogeneity by clustering patients across physicians, we employed propensity score matching techniques 33 to address the potential endogeneity issues when estimating the impact of physicians’ practice style on hospital inpatient costs. We employed the propensity score nearest neighbor (NN) matching without replacement method to create subsamples of physicians based on their observable characteristics. We created our first subsample of physicians by matching female physicians with male physicians based on their observable characteristic of medical specialties, experience, foreign- versus US-trained, state (Arizona or Florida), and whether physicians practiced at both teaching and nonteaching hospitals. Then, we reestimated our multilevel model using hospital inpatient visits registered to these matched cohorts of physicians. The new estimates provide more robust findings regarding the impact of practice styles of female physicians on hospital inpatient costs when compared with their matching male cohorts. Next, we created our second subsample of physicians by matching foreign-trained physicians with US-trained physicians based on their observable characteristics of medical specialties, experience, gender, state, and whether physicians practiced at both teaching and nonteaching hospitals and reestimated our multilevel model.

The average cost of hospital inpatient visits was $9172 for all visits, $9492 for visits to teaching hospitals, and $8679 for visits to nonteaching hospitals (see Appendix Table A1 for visit characteristics). There were 7993 physicians who worked only at teaching hospitals, 4249 physicians who worked only at nonteaching hospitals, and 2995 physicians who worked in both settings for a total of 15 237 physicians ( Table 1 ). The physicians had an average of 24 years of experience. The proportion of female physicians was 26.5%, and the relative distribution working only at teaching hospitals or nonteaching hospitals, or at both, were comparable. About a third of the physicians graduated from medical schools outside of the United States, and we observed a higher prevalence at nonteaching hospitals when compared with teaching hospitals. We also observed that 16.4% of physicians in our sample were board certified surgeons and 31.7% of physicians had board certification in internal medicine. The percentage of physicians with other board certified specialties was lower: obstetrics and gynecology (8.0), neurology (2.3), psychiatry (1.2), pediatrics (12.7), cardiology (7.0), family medicine and general practitioners (7.3), urology (2.5).

Profile of Physicians at Hospital Inpatient Settings.

Note. Data include all hospital inpatient stays incurred during 2008 in Arizona and Florida. We excluded all records associated with physicians with 12 or fewer observations during 2008, which is about 1% of entire sample.

Table 1 also presents the average cost per hospital inpatient visit by physician characteristics. The average cost of hospital inpatient visits for patients visiting female physicians was $2264 lower when compared with costs for patients visiting male physicians. This difference was larger in teaching hospitals when compared with nonteaching hospitals. Similarly, we observed the average cost per hospital visit treated by foreign-trained physicians was $1191 less when compared with physicians who graduated from a medical college in the United States. Although we observed a larger difference in average hospital inpatient costs between foreign-trained and US-trained physicians who work only at teaching hospitals, there was only about $64 difference for physicians working only at nonteaching hospitals. We found sizable variation in the average cost of a hospital inpatient visit across physicians’ specialties. Patients treated by physicians with specialties in surgery, neurology, and cardiology had relatively higher average costs per hospital visit, which were $17 431, $16 496, and $14 714, respectively.

We also documented the distribution of patients’ severity of illness by physician characteristics. The results presented in Table 1 show that the percentage with high severity of illness was higher for male patients than for female patients regardless of the hospital setting. We also observed that foreign-trained physicians had a relatively higher share of high-severity patients at teaching hospitals when compared with nonteaching hospitals. Finally, we found that the relative share of high-severity patients was greater for physicians with specialties in internal medicine or family medicine and general practitioners working at nonteaching hospitals when compared with physicians with the same specialties working at teaching hospitals. However, for most of the remaining physicians working only at teaching hospitals, we observed a higher share of patients with high severity of illness when compared with physicians working only at nonteaching hospitals.

Regression Results

Linear regression results presented in column 1 of Table 2 show that the average cost of hospital inpatient visits for patients visiting female physicians was 0.1% lower than male physicians and was 0.5% lower for patients visiting foreign-trained physicians versus US-trained physicians. Each additional year of experience was associated with 4.3% lower costs. We also observed sizable variation in average costs of hospital inpatient visits across medical specialties where surgeons and cardiologists were associated with the highest average cost and pediatricians and psychiatrists were associated with the lowest average cost per hospital inpatient visit. The regression results based on hospital inpatient visits to teaching hospitals were parallel to our main results for all key covariates ( Table 2 , column 2). We also found similar results for nonteaching hospitals ( Table 2 , column 3).

Estimated Effects of Physician Characteristics on Log Inpatient Cost Per Visit.

Note. Data include all hospital inpatient stays incurred during 2008 in Arizona and Florida. We excluded all records associated with physicians with 12 or fewer observations during 2008, which is about 1% of the entire sample. All regression models include patient’s primary payers, median household income for residences in patient’s ZIP Code, and the Elixhauser comorbidity index. Level 1 is visit level and level 2 is physician level. Percent impact is calculated as (exp(coefficient) – 1) × 100. Standard errors are in parentheses.

Columns 4 to 6 of Table 2 present the results of multilevel regressions estimated separately for hospital inpatient visits to all hospitals, to teaching hospitals, and to nonteaching hospitals to assess the robustness of our earlier results derived from single-level linear regression. The average cost of hospital inpatient visits for patients visiting female physicians was 11% lower than male physicians and was 3.6% lower for patients visiting foreign-trained physicians versus US-trained physicians. Each additional year of experience was associated with 0.10% lower costs. Similar to our earlier results, we found substantial variation in costs of hospital inpatient visits across medical specialties. The multilevel regression results based on inpatient visits to teaching hospitals and nonteaching hospitals retained the same sign and statistical significance, which enhanced the validity and robustness of our results, specifically how physician characteristics impact the cost per hospital inpatient visits. v

Table 2 presents the estimates separately for 2 cohorts of physicians where the first cohort includes equal numbers of male and female physicians with a similar distribution of other characteristics, and the second cohort includes equal numbers of foreign-trained and US-trained physicians with a similar distribution of other characteristics (see matching results in Appendix Table A2 ). The estimated coefficients on key physician characteristics are highly statistically significant and have the same direction as our earlier results. In our female-male matched cohort, the regression results show that the hospital inpatient costs registered to female physicians are 10.8% lower when compared with hospital inpatient visits registered to male physicians. Similarly, the estimated effect of foreign-trained physicians on hospital inpatient costs is 3.8% lower in our second cohort where each foreign-trained physician is matched with a US-trained physician. The coefficients on physicians’ experience and medical specialties remain statistically significant and parallel to our earlier findings in the hierarchical models.

The multilevel regression results presented in Tables 2 and ​ and3 3 also enable us to empirically measure the average correlation of patients registered to the same physicians. ICC, which is calculated by dividing the level 1 variance by the sum of the level 1 and level 2 variations, describes how strongly hospital inpatient visits registered to the same physicians are correlated with each other. In general, if ICC approaches to value zero, then one might chose to ignore multilevel estimation models and analyze the data in standard ways. On the contrary, if the ICC approaches the value one, there is no variation among patients registered to same physicians, so one might aggregate the data at the physician level and run a single-level linear regression model on aggregated data. For our case, the ICC values ranged from 0.329 (0.241 / [0.241 + 0.419]) (nonteaching hospitals) to 0.364 (teaching hospitals) ( Table 2 ) before matching and 0.605 (female models) to 0.643 (0.481 / [0.481 + 0.267]) (foreign-trained physician models) ( Table 3 ) after matching which indicates modest to sizable variation among visits registered to the same physician. The ICC range of our multilevel model also empirically validates our discussion around the necessity of using a multilevel model rather than single-level linear regression model.

Estimated Effects of Physician Characteristics on Log Inpatient Spending Per Visit.

Note. Level 1 is visit level and level 2 is physician level. Percent impact is calculated as (exp(coefficient) – 1) × 100. NN = nearest neighbor. Absolute values of t -ratios are in parentheses.

In this examination of all-payer data from two states, we found substantial variation in the costs of producing these hospital services with observable physician characteristics such as physician age, gender, foreign training, and physician specialty. We found that the average cost of hospital inpatient stays registered to female physicians was consistently lower across all empirical specifications when compared with the average cost of hospital inpatient stays registered to male physicians. We also found a negative association between physicians’ years of experience and the average costs of hospital inpatient stays. Similarly, the average cost of hospital inpatient stays registered to foreign-trained physicians was significantly lower when compared with the average cost of hospital inpatient stays registered to US-trained physicians. Finally, we observed sizable variation in average costs of hospital inpatient stays across medical specialties where surgeons and cardiologists were generally associated with higher average costs and pediatricians and psychiatrists were generally associated with lower average costs. Further research should investigate the sources of the differences associated with physician characteristics.

Using hierarchical methods and random effects, we estimated the percentage of remaining variation attributable to individual physicians. Using the entire sample, the ICC was approximately 0.35, or one third of the variation was attributable to physicians. Our approach partitions the variation in hospital costs and allows physicians to practice at multiple hospitals. Other studies have employed hospital fixed effects and partitioned the remaining variation in physician costs effectively comparing physicians within the same hospital. 21 This is an important distinction in approaches and could result in slightly different conclusions based on the variation that is being partitioned (total or net of hospital fixed effects).

Our data are all-payer and focus on the underlying costs of providing care. These differ from reimbursement amounts which may be relatively standardized across hospitals through Diagnosis Related Groups (DRG) payments within payers. Our results confirm that physician behavior is associated with variation in hospital costs other than professional services and this could occur through variations in physician practice styles and treatment decision-making.

Our study is limited to data from 2 large US states, and analysis of physician behavior in other states or countries may differ. This study relies on accurate attribution of individual physicians to hospital discharges. Our study is not experimental, and is observational, revealing a retrospective view of the association between physicians and hospital costs. Physicians were not randomized to patients, so potential endogeneity exists in patient selection of physicians. We attempted to minimize the impact of potential endogeneity in physician gender and foreign-trained physicians by creating matched samples of physicians and found that the ICC increased substantially, exceeding 0.60. This result is likely due to the retention of more similar samples of physicians, where residual variation is lower, and the percentage of variation attributable to physicians is higher.

When compared with recent studies, our findings are consistent with Tsugawa and colleagues 22 who found that female physicians treating Medicare patients had lower Part B payments. However, while we found that foreign medical graduates had slightly lower hospital costs, Tsugawa and colleagues 21 also found that foreign medical graduates had slightly higher Part B spending ($47 per discharge). The difference may be in the data used; ours is all-payer and focuses on hospital costs, and Tsugawa and colleagues 21 analyze Medicare enrollees and Medicare Part B payments as well as a methodological difference. Tsugawa and colleagues employ hospital fixed effects which compares physicians practicing at the same hospital.

Historically, physician and hospitals have been reimbursed via separate mechanisms, and our results quantify the physician role in the provision of care in hospital facilities. Our study lends support to the interconnected relationship between physicians and facilities in providing care to patients. Future policies, practices, and training processes for hospital administrators and physicians should acknowledge and address these important dependencies.

This study predates large systemic changes to align incentives of physicians and hospitals including some types of Alternative Payment Models and Accountable Care Organizations, and allows a window into physician influence on hospital costs prior to the expansion of these initiatives. At the time of this study, physicians generally had fewer incentives to control hospital costs. As aligned incentives expand, repeating this analysis will be important to understand trends in cost variation and physician influence. Our study also lends further support for payment and organizational models that align physician and hospital incentives that seek to control costs and improve outcomes.


Arizona Department of Health Services and Florida Department of Health granted special permission to access physician identifiers used by the research team. The authors would like to acknowledge the following Healthcare Cost and Utilization Project (HCUP) Partner organizations for contributing data to the HCUP State Inpatient Databases (SID) used in this study: Arizona Department of Health Services and Florida Agency for Health Care Administration. A full list of HCUP Data Partners can be found at .

Profile of Hospital Inpatient Visits.

Note. Data include all hospital inpatient stays incurred during 2008 in Arizona and Florida. We excluded all records associated with physicians with 12 or fewer observations during 2008, which was about 1% of the entire sample.

Analytic Framework for Hierarchical Models

Following existing studies on multilevel models ( Bryk and Raudenbush 1992 , Rice and Jones 1997 , Carey 2000 , Diez-Roux 2000 ), a basic formal 2-level model is presented, with a single level 1 predictor and a single level 2 predictor with the intercept modeled to vary randomly at level 2. The level 1 model takes the form

Response variable Y represents hospital inpatient cost per visit, X is a predictor that varies with hospital inpatient visits, and subscripts i and j reference hospital inpatient visits and physicians, respectively. Residual ξ i j is the random error for the i th hospital inpatient visit in the j th physician unit. At level 2, variation in the intercept is predicted by

The terms β 02 represent fixed elements and β 12 , the coefficients on P j , varies for each physician. The terms γ j are the random error components and along with ξ ij are assumed to be normally distributed with zero mean. Furthermore,

Substituting (2) into Equation (1) yields the single 2-level multilevel equation:

The first 3 terms on the right-hand side make up the deterministic part of the model. The 2 terms in parentheses comprise the stochastic or residual portion, which, in this example, contains 2 random variables. Components γ j represent departures of the j th physician from the overall mean; ξ i j is the hospital-inpatient-visit-level random error. Equation (3) requires the estimation of 3 fixed coefficients, 2 variances, and one covariance component. The presence of more than one residual term distinguishes this model from standard regression models. It is straightforward to enhance this model with more predictors and higher levels.

We present the descriptive characteristics for the 2 separate subsamples of physicians obtained from the propensity score nearest neighbor (NN) matching without replacement method as explained earlier. The descriptive results presented in Table A2 shows that the distribution of physician characteristics in the matching cohorts are very similar. For example, 3978 female physicians were individually matched with 3978 male physicians based on their observable characteristics. In this sample, the relative distribution of medical specialties and mean value for experience among female physicians were very similar to those of their male physician counterparts. Similarly, Table A2 shows that 5298 foreign-trained physicians were matched with 5298 US-trained physicians. The relative distributions of medical specialties, gender, and mean value for experience for foreign-trained physicians were very close to those statistics among the US-trained physicians.

Profile of Physicians Matched Through Propensity Score NN Matching Without Replacement Method.

Note. NN = nearest neighbor.

  • Bryk A, Raudenbush S. Hierarchical Linear Models . Newbury Park, CA: Sage; 1992. [ Google Scholar ]
  • Carey K. A multilevel modelling approach to analysis of patient costs under managed care . Health Econ . 2000; 9 :435-446. [ PubMed ] [ Google Scholar ]
  • Diez-Roux A. Multilevel analysis in public health research . Annu Rev Public Health . 2000; 21 :171-192. [ PubMed ] [ Google Scholar ]
  • Rice N, Jones A. Multilevel models and health economics . Health Econ . 1997; 6 :561-575. [ PubMed ] [ Google Scholar ]

i. Arizona Department of Health and Florida Agency for Health Care Administration granted special permission to access physician identifiers used by the research team

ii. The methodology uses the hospital’s accounting report covering all patients submitted to Centers for Medicare and Medicaid Services (CMS) and is described in user guides at (accessed October 10, 2012).

iii. Costs throughout this article are inflation-adjusted. The methodology is described in user guides at (accessed October 10, 2012).

iv. We also used grand mean centering for all level 1 variables (except age, which was scaled as age/10), and we found the direction and significance of results remained same.

v. Some researchers may suggest further clustering instead of estimating 2-level multilevel regression model separately using patients’ discharge data from teaching hospitals and nonteaching hospitals. We added further clustering by hospital’s teaching status and regression results for 3-level multilevel regression model were parallel to our earlier 2-level multilevel regression results.

Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the Agency for Healthcare Research and Quality (AHRQ) under contract HHSA-290-2013-00002-C and through AHRQ intramural research funds. The views expressed herein are those of the authors and do not necessarily reflect those of the AHRQ Quality or the US Department of Department of Health and Human Services. No official endorsement by any agency of the federal or state governments or IBM Watson Health is intended or should be inferred.

IRB Statement: The Healthcare Cost and Utilization Project (HCUP) databases are consistent with the definition of limited data sets under the Health Insurance Portability and Accountability Act Privacy Rule. The Agency for Healthcare Research and Quality (AHRQ) Institutional Review Board considers research using HCUP data to have exempt status.

An external file that holds a picture, illustration, etc.
Object name is 10.1177_0046958018800906-img1.jpg


  1. Quantitative research in medicine

    quantitative research about medicine and its contribution

  2. Lesson 2- The importance of Quantitative Research across fields-2.pdf

    quantitative research about medicine and its contribution

  3. Quantitative Methods for Precision Medicine

    quantitative research about medicine and its contribution

  4. Quantitative Medicine, Quantitative Exercise

    quantitative research about medicine and its contribution

  5. (PDF) A Quantitative Analysis of Qualitative Studies in Clinical

    quantitative research about medicine and its contribution

  6. Read Quantitative Research in Human Biology and Medicine Online by

    quantitative research about medicine and its contribution


  1. Quantitative Research

  2. Quantitative Research Vs Qualitative Research

  3. Quantitative CMR Stress Perfusion Imaging: Applications in Ischemic Heart Disease and Heart Failure

  4. Exploring Qualitative and Quantitative Research Methods and why you should use them

  5. Qualitative Research VS Quantitative Research / Introduction to Biostatistics/Lecture 1

  6. Machine Learning Techniques


  1. Quantitative Study of the Characteristics of Effective Internal Medicine Noon Conference Presentations

    Results A total of 649 evaluations of 29 conferences were submitted by 153 of 185 (83%) residents in the program.Median effectiveness score was 6 (on a scale of 1 to 7). Clinicopathological conferences had .55-point higher effectiveness scores than traditional conferences (P = .011).In multivariable analyses focusing on traditional conferences, summary statement inclusion was significantly ...

  2. Quantitative Research Methods in Medical Education

    For more information on quantitative research in medical education: Bordage G: Conceptual frameworks to illuminate and magnify. Medical education. 2009; 43(4):312-9. Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: Theory and application. The American journal of medicine. 2006; 119(2):166. e7 ...

  3. Public and patient involvement in quantitative health research: A

    1. BACKGROUND. Public and patient involvement (PPI) in health research has been defined as research being carried out "with" or "by" members of the public rather than "to," "about" or "for" them. 1 PPI covers a diverse range of approaches from "one off" information gathering to sustained partnerships. Tritter's conceptual framework for PPI distinguished between indirect ...

  4. The importance of quantitative systemic thinking in medicine

    The study and practice of medicine could benefit from an enhanced engagement with the new perspectives provided by the emerging areas of complexity science and systems biology. A more integrated, systemic approach is needed to fully understand the processes of health, disease, and dysfunction, and the many challenges in medical research and education. Integral to this approach is the search ...

  5. Effects of the COVID-19 pandemic on medical students: a multicenter

    The COVID-19 pandemic disrupted the United States (US) medical education system with the necessary, yet unprecedented Association of American Medical Colleges (AAMC) national recommendation to pause all student clinical rotations with in-person patient care. This study is a quantitative analysis investigating the educational and psychological effects of the pandemic on US medical students and ...

  6. Recent quantitative research on determinants of health in high ...

    Background Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature. Methods We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019). We searched for studies that ...

  7. Quantitative medicine: Tracing the transition from holistic to ...

    This paper explores the underlying principles and key contributions of quantitative medicine, as well as the context for its rise, including the development of new technologies and the influence of reductionist philosophies. The challenges and criticisms of this approach, and the need to balance reductionist and holistic approaches in order to ...

  8. Quantitative Methods in Global Health Research

    Abstract. Quantitative research is the foundation for evidence-based global health practice and interventions. Preparing health research starts with a clear research question to initiate the study, careful planning using sound methodology as well as the development and management of the capacity and resources to complete the whole research cycle.

  9. Assessing changes in the quality of quantitative health educations

    As a community of practice (CoP), medical education depends on its research literature to communicate new knowledge, examine alternative perspectives, and share methodological innovations. As a key route of communication, the medical education CoP must be concerned about the rigor and validity of its research literature, but prior studies have suggested the need to improve medical education ...

  10. Qualitative and quantitative research of medication review and drug

    Background Pharmaceutical care is the pharmacist's contribution to the care of individuals to optimize medicines use and improve health outcomes. The primary tool of pharmaceutical care is medication review. Defining and classifying Drug-Related Problems (DRPs) is an essential pillar of the medication review. Our objectives were to perform a pilot of medication review in Hungarian community ...

  11. 53 Quantitative research methods in medical education

    Abstract. Quantitative research in medical education tends to be predominantly observational research based on survey or correlational studies. As researchers strive towards making inferences about the impact of education interventions, a shift towards experimental research designs may enhance the quality and conclusions made in medical education.

  12. Quantitative Evaluation of Translational Medicine Based on

    Research articles of translational medicine (n=1662) Non-research articles (n=1499) Research articles within non-medical fields (n=103) Figure 2. The flow chart of the quantitative evaluation of translational medicine. Research articles on translational medicine from 2011 to 2013 Descriptive finding (the number of articles and citations ...

  13. Quantitative medicine: Tracing the transition from holistic to

    Perspectives and Debates Public Health Research Introduction Medicine has a rich and complex history, shaped by a variety of factors including cultural, social, and scientific ... This paper explores the underlying principles and key contributions of quantitative medicine, as well as the context for its rise, including the development of new ...

  14. Quantitative Analysis of the Effectiveness of Public Health ...

    Although COVID-19 has spread almost all over the world, social isolation is still a controversial public health policy and governments of many countries still doubt its level of effectiveness. This situation can create deadlocks in places where there is a discrepancy among municipal, state and federal policies. The exponential increase of the number of infectious people and deaths in the last ...

  15. Acquiring data in medical research: A research primer for low- and

    Medical research historically has focused on quantitative methods. Generally, quantitative research is cheaper, easier to gather and easier to analyse. For purposes of this chapter, we will focus on quantitative research. Qualitative research is about words, sentences, sounds, feeling, emotions, colours and other elements that are non-quantifiable.

  16. Quantitative Research in Human Biology and Medicine

    Description. Quantitative Research in Human Biology and Medicine reflects the author's past activities and experiences in the field of medical statistics. The book presents statistical material from a variety of medical fields. The text contains chapters that deal with different aspects of vital statistics. It provides statistical surveys of ...

  17. A Practical Guide to Writing Quantitative and Qualitative Research

    INTRODUCTION. Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses.1,2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results.3,4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the ...

  18. Showcasing the contribution of social sciences to health policy and

    This Special Issue represents a critical response to the frequent silencing of qualitative social science research approaches in mainstream public health journals, particularly in those that inform the field of health policy and systems research (HPSR), and the study of equity in health. This collection of articles is presented by SHAPES, the thematic working group of Health Systems Global ...

  19. A Quantitative Observational Study of Physician Influence on Hospital

    Introduction. It has been well established that health care spending varies with geography. 1-3 The source of this variation has been often questioned—whether it is arising from area practice patterns, patient health status, patient characteristics, price, and/or individual provider decision making. 3,4 An Institute of Medicine (IoM) Committee examining geographic variations in Medicare ...