Rachel Miles is a Digital Scholarship Librarian at the Center for the Advancement of Digital Scholarship (CADS) at Kansas State University and is K-State Libraries’ Partner on the Open/Alternative Textbook Initiative. She holds a BA in Psychology from Wichita State University and an MLS from Emporia State University. In her free time, she loves to hang out and play music with her husband. She also loves to write fiction, sew, and attend dance
classes.
IntroductionIntroduction The start of a research article providing background information and an overview of the research presented in the article.
When I was in graduate school, I had the opportunity to become a Graduate Research Assistant (GRA) for Dr. Sarah Sutton at the School of Library and Information Management (SLIM) at Emporia State University (ESU) for the last semester of my Master’s program. In May of 2015, Dr. Sutton became involved in online survey research regarding the awareness and usage of research impactResearch impact A way to determine the impact a research project/idea has had. Can be done through quantitative data (such as citation counts, h-index, and journal impact factor.) Research impact can also be described qualitatively about the benefit of this work on a community or within society. metrics, such as bibliometricsBibliometrics Statistical analyses of academic publications used to understand how this information is created, organized, and connected. These statistics can help illuminate output and impact. Examples of bibliometrics include journal impact factor and journal usage factor. and altmetrics, among academic librarians at Carnegie-classified R1 institutions. I felt honored and excited to join Dr. Sutton in her research endeavors, though and at the time, I did not know what it would mean for my career and professional advancement. My experience and exposure to this research helped launch my career, propel me into the complex field of scholarly communication, and entice me to continue research in what I consider one of the most interesting topics in academic librarianship: research impact and evaluation.
Research Impact & Evaluation – An Overview
First, I’d like to offer a brief explanation on research impact metrics and why this research began in the first place. Research impact metrics, or indicators (a more accurate term), encompasses a range of indicators that tell us something about the research we attempt to evaluate. In our research publications and presentations, we categorized them into the following:
| Research Impact Indicator | Definition |
| Journal Impact FactorJournal Impact Factor A measure used to show the average amount of times an article within a certain journal has been cited within a period of time. This factor can be used by a discipline to determine the importance or status of a journal. (JIF) | Average number of citations an article receives in a given journal within a given time frame, usually two or five years |
| Journal Usage FactorJournal Usage Factor A calculation done by taking the number of times articles have been downloaded divided by the total number of articles published within a journal for a certain period of time. This number can be used to complement a journal impact factor. (JUF) | Average number of usage statistics (e.g., downloads, page views) that an article receives within a given journal |
| Article/book citation counts | Total number of citations a scholarly article/book receives in another scholarly article/book |
| Usage statistics | Total number of article/book downloads and page views |
| Altmetrics | Online attention to academic research
(e.g., Mendeley bookmarks, number of Twitter mentions, peer reviews on Publons, public policy document citations, news media outlet citations, etc.) |
| Author h-index | Reflects an author’s total number of publications that have received at least as many citations each. E.g. a researcher with an h-index of 25 has authored at least 25 journal articles that have received 25 citations. |
| Expert peer reviews | Post-publication peer reviews (e.g., F1000, Publons, resource reviews published in library journals). A type of altmetrics. |
| Qualitative measures of impact | “Who’s saying what about research?” (e.g., context in a citation from a scholarly article or a public policy document; ) |
Stacy Konkiel from Altmetric.com launched the research project in May of 2015. She and others had previously simply made assumptions about how academic librarians thought about and used research impact indicators. When she eventually searched the scholarly literature for any type of study like this, with the exception of a couple of small institutionally-based studies, she came up quite empty-handed. As a result, she decided to launch a study of her own.
GRAing it up at ESU
I began working for Dr. Sutton in late August of 2015, after the research project had already received IRBInstitutional Review Board A group that is charged with overseeing and approving research projects. The group ensures that research projects are ethical, meet regulations and standards, and protect any human subjects involved in the research. approval.
After completing CITI training, Dr. Sutton directed my attention to analyzing a dataset from the survey responses. First, we reviewed the survey questions, and I learned some of the basics behind the purposes for launching the survey project as well as the principles of research impact indicators. At the time, I had a minimal level of understanding of research impact and evaluation, and yet I had to analyze data regarding academic librarians’ own self-assessed understanding of research impact indicators. I was intimidated, and, without doubt, I can say I had Imposter Syndrome. Fortunately, Dr. Sutton encouraged me and guided me on my journey through library research land.
Next, Dr. Sutton was interested in parsing out the data concerning respondents’ liaison areas. About a third of our respondents indicated that they were liaison or subject specialists, in which they liaised to specific departments and/or units at their institutions. Most of these respondents also stated that they liaised to multiple areas, so I had to dig into these text-based responses to determine how the liaison areas related to one another. Dr. Sutton said I could use creativity and imagination to visualize these liaison areas as a hierarchy or network.
Eventually, Dr. Sutton and I came up with a physical visualization to connect liaison duties to one another on a bulletin board with yarn, push pins, and yarn. It took much longer to complete than anticipated, but it had enormous value in my development as a researcher.
Would I take this sort of due diligence as a researcher now? No, probably not. My first attempt at creating a meaningful data visualization with the push pins and yarn was an exercise for my mind. I believe that this exercise is analogous to toddlers playing with blocks before they play with a toolbox or solve a math problem. The hands-on nature of this exercise, as I reflect on it, helped me to take a baby-step into the world of library research.
As a GRA and graduate student, I also read relevant scholarly journal articles and books. I immersed myself in the literature concerning research impact, altmetrics, scholarly communication, and digital scholarship. I also relearned key statistical methods and data visualization concepts that helped me while I worked on this project.
For instance, I began copying datasets into Excel for individual survey questions, such as, “How familiar are you with the concept of journal impact factors and the following measures of article-level impact?” Participants could answer on a Likert-scale between “1 – I know nothing” and “5 – I’m an expert.” In Excel, I converted the raw numbers into percentages and created graphs, such as this one:
Finding Meaning in Data
After creating graphs, my data started to have more meaning, especially since I am a visual person. However, a good researcher never makes assumptions based purely on data visualizations. I ran statistical analyses on subsets of data that appeared to have statistically significant relationships. For example, there was a statistically significant relationship between the levels of familiarity with the JIF and altmetrics (x 2 (4, n = 1085) = 89.201, p < .01). The graph suggests this, but by conducting the chi-square test of independence on these subsets of data, our research team was able to determine with certainty that there were statistically significant relationships between certain variables.
What I’ve just described was a repeated process for me, but before it became ingrained in my mind, I had to take the time and effort to learn how to do it. Sometimes, I spent an unnecessarily long time with datasets, especially in the beginning of this research project. In addition, certain datasets and data visualizations were never used in presentations or publications, either because they did not provide our team with enough statistically significant resultsResults The section of a research article where researchers share the results from the research. This section takes the results and directly connects them to the research questions or hypotheses posed at the start of the article. Also can be called “Findings.” , or the results were unrelated to a project’s focus.
Communication & Organization
“There are things we could have done better in our research, mainly with survey design and data managementData management The ways a researcher collects, organizes, stores, and accesses data they collect for research. Creating a data management plan allows a researcher to know what data they will be collecting and how they will store and organize it during the research project.. However, this will help me with managing the next big research project.”
I think that what every researcher needs to know is this: they will learn as they progress through each and every project. There are things we could have done better in our research, mainly with survey design and data management. However, this will help me with managing the next big research project. There are still clear takeaways from this past research project for both myself as a researcher and a professional librarian, and for the library profession as a whole. I would even go so far as to argue that our research may have a small impact on all of academia one day.
Measuring impact is tricky, and reflecting on your experience is much less measurable, but at the end of the day, I feel I’ve gained insight and wisdom into my research and into my own field of scholarly communication. However, I have more questions than when I started; isn’t that always the conundrum?
Featured image Networks via Pixabay
This work is licensed under a Creative Commons Attribution 4.0 International License
The expressions of writer do not reflect anyone’s views but their own
Pingback: Jumping Into Academic Writing and Presenting – The Librarian Parlor