Quantitative and Qualitative Impact Measures
Research impact A way to describe and measure the ways in which research causes some sort of positive impact on a community. Research impact can be measured through quantitative or qualitative data.
, in librarianship and other academic disciplines, is used in a wide variety of evaluative scenarios throughout academia, from hiring decisions to tenure and promotion decisions, and for research grant evaluations. The goal of evaluating research is to understand the ways in which it not only has influenced other researchers but also potentially non-research audiences.
There is no one definition or way to evaluate research impact. Generally speaking, research impact is a way to describe and measure how research causes some sort of positive impact. The UK-based Research Excellence Framework offers a common definition of research impact: an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Describing impact varies by discipline, subdiscipline, interdisciplinarity, and type of research output, and often does not take into account the biases in how research is viewed, used, and cited. For example, women are often less credited and cited then men, which can lead to future compounding issues of females having lower citation counts then their male colleagues. Researchers of color or researchers from non-Western locations have also been found to be cited less frequently.
The most common basis for research evaluation impact is a basic quantitative (number-based) metric most commonly tracked for journal articles: the citation count, or how many times a journal article or other form of research output is cited by others. Which sources are indexed heavily influence this citation count, and several different databases handle the topic of selectivity differently, which results The section of a research article where researchers share the results from the research. This section takes the results and directly connects them to the research questions or hypotheses posed at the start of the article. Also can be called “Findings.” in fluctuating citation counts. These counts also form the basis of common metrics not only to describe the impact of individual journal articles but also the impact of journals and researchers. Often, the larger concept of research impact is translated as the suite of citation-based metrics, also known as bibliometrics Statistical analyses of academic publications used to understand how this information is created, organized, and connected. These statistics can help illuminate output and impact. Examples of bibliometrics include journal impact factor and journal usage factor., while disregarding the larger concept the term represents.
Before we dive into those metrics, it is important to know two things. First, the entire field of research impact measurement and demonstration are a very imperfect substitute for something less tangible: quality. Thus, the metrics and measures described in this lesson are not a substitute for quality. High quality scholarship is not necessarily the scholarship with the greatest impact, and what characterizes high quality scholarship is even more difficult to define and dependent upon a number of factors. So, please keep in mind — your research impact is, by and large, not a determination of your work’s quality.
The second thing to consider is that, while administrators and other evaluators often lean heavily on numbers, which they can easily compare, sort, etc., some of the best and most compelling research impact indicators are qualitative or descriptive in nature. Quantitative metrics are much more standardized and well-defined, but if you think of research impact demonstration as a way to tell a story about how your work has been influential, a fulsome story requires many different components to present a full and accurate representation. Simply put, quantitative and qualitative measures are complementary, and both are useful in demonstrating research impact. However, the institutional expectations for impact demonstration in librarianship can vary widely and do not always allow for a qualitative approach.
A useful way to understand research impact is to think about librarianship research that has impacted you — maybe it has helped inform an aspect of your librarianship, inspired you to pursue a research line of inquiry, or enhanced a presentation in which you referred to it. In those cases, how would the people behind that research be made aware of how they impacted you? Those bits of evidence help inform their research impact story, and your process will likewise help shed light on how your research has made an impact on others.
Topic 1 References
Belcher, Brian, and Janet Halliwell. “Conceptualizing the Elements of Research Impact: Towards Semantic Standards.” Humanities and Social Sciences Communications 8 (1): 183. doi.org/10.1057/s41599-021-00854-2.
Chakravartty, Paula, Kuo, Rachel, Grubbs, Victoria, and Charlton McIlwain. “#CommunicationSoWhite.” Journal of Communication 68, 2 (2018): 254–66. doi.org/10.1093/joc/jqy003.
Dion, Michelle L., Lawrence Sumner, Jane, and Sara McLaughlin Mitchell. “Gendered Citation Patterns across Political Science and Social Science Methodology The theoretical framework that informs how a researcher approaches their work and what methods are used to collect data. Fields.” Political Analysis 26, 3 (2018): 312–27. doi.org/10.1017/pan.2018.12.
Heidt, Amanda. “Racial Inequalities in Journals Highlighted in Giant Study.” Nature Career News, April 28, 2023. https://www.nature.com/articles/d41586-023-01457-4.
Ross, Matthew B., Glennon, Britta M., Raviv Murciano-Goroff, et al. “Women Are Credited Less in Science than Men.” Nature 608 (2022): 135–145. ,doi.org/10.1038/s41586-022-04966-w
UK Research and Innovation. “How Research England supports research excellence.” 2022. https://www.ukri.org/who-we-are/research-england/research-excellence/ref-impact/.
Quantitative and Qualitative Impact Measures
Research impactResearch impact A way to describe and measure the ways in which research causes some sort of positive impact on a community. Research impact can be measured through quantitative or qualitative data. , in librarianship and other academic disciplines, is used in a wide variety of evaluative scenarios throughout academia, from hiring decisions to tenure and promotion decisions, and for research grant evaluations. The goal of evaluating research is to understand the ways in which it not only has influenced other researchers but also potentially non-research audiences.
There is no one definition or way to evaluate research impact. Generally speaking, research impact is a way to describe and measure how research causes some sort of positive impact. The UK-based Research Excellence Framework offers a common definition of research impact: an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Describing impact varies by discipline, subdiscipline, interdisciplinarity, and type of research output, and often does not take into account the biases in how research is viewed, used, and cited. For example, women are often less credited and cited then men, which can lead to future compounding issues of females having lower citation counts then their male colleagues. Researchers of color or researchers from non-Western locations have also been found to be cited less frequently.
The most common basis for research evaluation impact is a basic quantitative (number-based) metric most commonly tracked for journal articles: the citation count, or how many times a journal article or other form of research output is cited by others. Which sources are indexed heavily influence this citation count, and several different databases handle the topic of selectivity differently, which resultsResults The section of a research article where researchers share the results from the research. This section takes the results and directly connects them to the research questions or hypotheses posed at the start of the article. Also can be called “Findings.” in fluctuating citation counts. These counts also form the basis of common metrics not only to describe the impact of individual journal articles but also the impact of journals and researchers. Often, the larger concept of research impact is translated as the suite of citation-based metrics, also known as bibliometricsBibliometrics Statistical analyses of academic publications used to understand how this information is created, organized, and connected. These statistics can help illuminate output and impact. Examples of bibliometrics include journal impact factor and journal usage factor., while disregarding the larger concept the term represents.
Before we dive into those metrics, it is important to know two things. First, the entire field of research impact measurement and demonstration are a very imperfect substitute for something less tangible: quality. Thus, the metrics and measures described in this lesson are not a substitute for quality. High quality scholarship is not necessarily the scholarship with the greatest impact, and what characterizes high quality scholarship is even more difficult to define and dependent upon a number of factors. So, please keep in mind — your research impact is, by and large, not a determination of your work’s quality.
The second thing to consider is that, while administrators and other evaluators often lean heavily on numbers, which they can easily compare, sort, etc., some of the best and most compelling research impact indicators are qualitative or descriptive in nature. Quantitative metrics are much more standardized and well-defined, but if you think of research impact demonstration as a way to tell a story about how your work has been influential, a fulsome story requires many different components to present a full and accurate representation. Simply put, quantitative and qualitative measures are complementary, and both are useful in demonstrating research impact. However, the institutional expectations for impact demonstration in librarianship can vary widely and do not always allow for a qualitative approach.
A useful way to understand research impact is to think about librarianship research that has impacted you — maybe it has helped inform an aspect of your librarianship, inspired you to pursue a research line of inquiry, or enhanced a presentation in which you referred to it. In those cases, how would the people behind that research be made aware of how they impacted you? Those bits of evidence help inform their research impact story, and your process will likewise help shed light on how your research has made an impact on others.
Topic 1 References
Belcher, Brian, and Janet Halliwell. “Conceptualizing the Elements of Research Impact: Towards Semantic Standards.” Humanities and Social Sciences Communications 8 (1): 183. doi.org/10.1057/s41599-021-00854-2.
Chakravartty, Paula, Kuo, Rachel, Grubbs, Victoria, and Charlton McIlwain. “#CommunicationSoWhite.” Journal of Communication 68, 2 (2018): 254–66. doi.org/10.1093/joc/jqy003.
Dion, Michelle L., Lawrence Sumner, Jane, and Sara McLaughlin Mitchell. “Gendered Citation Patterns across Political Science and Social Science MethodologyMethodology The theoretical framework that informs how a researcher approaches their work and what methods are used to collect data. Fields.” Political Analysis 26, 3 (2018): 312–27. doi.org/10.1017/pan.2018.12.
Heidt, Amanda. “Racial Inequalities in Journals Highlighted in Giant Study.” Nature Career News, April 28, 2023. https://www.nature.com/articles/d41586-023-01457-4.
Ross, Matthew B., Glennon, Britta M., Raviv Murciano-Goroff, et al. “Women Are Credited Less in Science than Men.” Nature 608 (2022): 135–145. ,doi.org/10.1038/s41586-022-04966-w
UK Research and Innovation. “How Research England supports research excellence.” 2022. https://www.ukri.org/who-we-are/research-england/research-excellence/ref-impact/.