targets::tar_read(titles_kept) |>
map(\(df) count(df, name = "Articles kept")) |>
list_rbind(names_to = "Reviewers") |>
# TODO: Include percent of total articles.
knitr::kable()| Reviewers | Articles kept |
|---|---|
| Daniel | 75 |
| Mario | 161 |
| Luke | 53 |
Review stages
Title review stage
During the first stage of the reviewing, we only looked at titles. Below is the code to extract the titles we each individually reviewed, so we can show some details about how many were included, excluded, and which titles we agreed on or disagreed on.
In the title screening stage, the number of titles we kept are listed in Table 1.
We want to know how many of these titles we all agreed on between the three reviewers:
common_titles <- targets::tar_read(titles_agreed_on)We seemed to agree on 24 titles, with the full title list shown in Table 2.
common_titles |>
select("Article title" = title) |>
knitr::kable()| Article title |
|---|
| UKRN ORCC Primer on Working in Open Research |
| Accelerating addiction research via Open Science and Team Science |
| The State of Open Science Practices in Psychometric Studies of Suicide: A Systematic Review |
| Eleven Strategies for Making Reproducible Research and Open Science Training the Norm at Research Institutions |
| Biomedical supervisors’ role modeling of open science practices |
| Exploring Open Science Practices in Behavioural Public Policy Research |
| UKRN Open Research Training Resources and Priorities Working Paper |
| (Why) Are Open Research Practices the Future for the Study of Language Learning? |
| Inclusive, open, and reproducible developmental science |
| Modernizing authorship criteria and transparency practices to facilitate honest, open, and equitable team science |
| The potential of inclusive and collaborative Open Research processes at the science-policy interface |
| Knowledge and practices of open science among scholars and researchers in Tanzania |
| Reducing Barriers to Open Science by Standardizing Practices and Realigning Incentives |
| Open and reproducible neuroimaging: From study inception to publication |
| Reducing barriers to open science by standardizing practices and realigning incentives |
| Lessons Learned: A Neuroimaging Research Center’s Transition to Open and Reproducible Science |
| Ten strategies to foster open science in psychology and beyond |
| Open and reproducible practices in developmental psychology research: The workflow of the WomCogDev lab as an example |
| Open and reproducible science practices in psychoneuroendocrinology: opportunities to foster scientific progress |
| Performing Open Science Culture |
| No Better Time to Practice Open Science Than Now |
| Ten simple rules for helping newcomers become contributors to open projects |
| Privacy, Anonymity, and Perceived Risk in Open Collaboration |
| Open Source, Collaboration, and Access: A Critical Analysis of “Openness” in the Design Field |
One person will review the titles and flag any paper that might be relevant but are not in agreement to keep.
targets::tar_read(titles_disagreed_on) |>
select("Article titles" = title) |>
knitr::kable()| Article titles |
|---|
| Using open and collaborative data to improve the Brazilian topographic mapping in protected areas |
| Special Issue: Open, Digital, Collaborative Project Preservation in the Humanities |
| How Openness Could Strengthen Academia’s Partnerships with the Intelligence Community |
| Open Source Software in the Public Sector: 25 Years and Still in Its Infancy |
| Open sharing of government data and enterprise decision making |
| Open-Source Software –Benefits and Drawbacks |
| What knowledge does ChatGPT have regarding open science, and can it provide good advice about it? |
| Wide open? Creative industries and open strategizing challenges |
| Open Institute of the African BioGenome Project: Bridging the gap in African biodiversity genomics and bioinformatics |
| Regional Open Science Dialogue in Southern Africa |
| Open science in health psychology and behavioral medicine: A statement from the Behavioral Medicine Research Council. |
| Towards Reproducible and Respectful Autism Research: Combining Open and Participatory Autism Research Practices |
| Open Data and transparency in artificial intelligence and machine learning: A new era of research |
| Accelerating Social Science Knowledge Production with the Coordinated Open-Source Model |
| Response to responsible research assessment I and II from the perspective of the DGPs working group on open science in clinical psychology |
| A Review on the Use of Open Source Technologies and Soft-Ware Applied to Precision Agriculture Practices |
| ‘Conditional Acceptance’ (additional experiments required): A scoping review of recent evidence on key aspects of Open Peer Review |
| Supporting Open Science with frictionless publication workflows: The Tree of Life project at Wellcome Open Research |
| Toppling the Ivory Tower: Increasing Public Participation in Research Through Open and Citizen Science |
| Accelerating the Open Research agenda to solve global challenges |
| How to move open science from the periphery to the centre |
| Three Myths about Open Science That Just Won’t Die |
| Opening up Understanding of Neurodiversity: A Call for Applying Participatory and Open Scholarship Practices |
| Open research case study - Biological Sciences |
| Open research case study - Mathematics and Statistics |
| An Open Digital Platform to Support Interdisciplinary Energy Research and Practice—Conceptualization |
| Open digital infrastructures for bridging professional cultures: the case of extreme citizen science between journalism and research |
| Promoting Open Science Through Research Data Management |
| Open Science in Developmental Science |
| Open and reproducible neuroimaging: from study inception to publication |
| Adapting Open Science and Pre-registration to Longitudinal Research |
| Opening doors to discovery: Partnerships are key to advancing open science |
| The value of open-source clinical science in pandemic response: lessons from ISARIC |
| Context dependent trade-offs around platform-to-platform openness: The case of the Internet of Things |
| Accelerating your research career with open science |
| As an open scientist, you’re good-to-get what you need |
| Digital platform openness: Drivers, dimensions and outcomes |
| Open Scholarship as a mechanism for the United Nations Sustainable Development Goals |
| Science through Wikipedia: A novel representation of open knowledge through co-citation networks |
| Open science from the standpoint of the new wave of researchers: Views from the scholarly frontline1 |
| Conducting a Meta-Analysis in the Age of Open Science: Tools, Tips, and Practical Recommendations |
| Open Science, Open Data, and Open Scholarship: European Policies to Make Science Fit for the Twenty-First Century |
| Open creative workspaces impacts for new product development team creativity and effectiveness |
| The limits to openness: Co-working, design and social innovation in the neoliberal city |
| Abandoning innovation activities and performance: The moderating role of openness |
| The faster the better? Innovation speed and user interest in open source software |
| Open science challenges, benefits and tips in early career and beyond |
| Measuring and analyzing code authorship in 1 + 118 open source projects |
| Open and Reproducible Research on Open Science Framework |
| Community radio stations sustainability model: An open-source solution |
| Prototyping open digital tools for urban commoning |
| Reinventing Scientific Journals through Open Science Practices and Self-Publication |
| ACTIVE METHODOLOGIES AS INCENTIVE TO INNOVATIVE AND OPEN BEHAVIOR |
| Promoting Open-Science and Accessible Student Training: The Open-Science fabrication laboratory model |
| OVERVIEW OF THE METHODOLOGY OF UNIVERSITIES` INTERNATIONAL RANKINGS FOR MEETING THE CHALLENGES OF OPEN SCIENCE |
| Current Status of Open Science and Statistical Analysis in The Japanese Journal of Educational Psychology: Consideration From the Severe Testing Perspective |
| Statistical Misconceptions, Awareness, and Attitudes towards Open Science Practices in Slovak Psychology Researchers |
| Open Science Perspectives on Machine Learning for the Identification of Careless Responding: A New Hope or Phantom Menace? |
| A tradition of open, academy-owned, and non-profit research infrastructure in Latin America |
| Coopetition as a driver of success for community initiatives in open research |
| Open Science for Ukraine: the Case for International Cooperation |
| Hacking Limnology Workshop and |
| Rethinking Transparency and Rigor from a Qualitative Open Science Perspective |
| N-of-1 Trials, Their Reporting Guidelines, and the Advancement of Open Science Principles |
| Promoting trust in research and researchers: How open science and research integrity are intertwined |
| Open Science for Computer Simulation |
| Development of a Concept Inventory on Open and Transparent Research Practices |
| The LOTUS initiative for open knowledge management in natural products research |
| Open knowledge management in university education in the context of digital transformation |
| Open Science, Closed Doors |
| Open Science Considerations for Descriptive Research in Developmental Science |
| Open Science Research Priorities in Health Psychology |
| Indicators of research quality, quantity, openness and responsibility in institutional promotion, review and tenure policies across seven countries |
| Open Science and Multicultural Research: Some Data, Considerations, and Recommendations |
| Open Science in Human Electrophysiology |
| Open Science at the Science-policy Interface: Bringing in the Evidence? |
| Using the Reproducible Open Coding Kit & |
| Legacy of Jon Tennant, “Open science is just good science” |
| Open and autonomous. The basis for trust in science |
| Embedding open and reproducible science into teaching: A bank of lesson plans and resources. |
| Developing and disseminating data analysis tools for open science |
| Open science practices and risks arising from misuse of biological research |
| Is Open Science Neoliberal? |
| Open Science Practices in Gambling Research Publications (2016 – 2019): A Scoping Review |
| Help Shape the Transition to Open |
| The Open Transition Takes Center Stage |
| Open Developmental Science: An Overview and Annotated Reading List |
| Open Communitition |
| Open Science, Data Analysis, and Data Sharing |
| The importance of adherence to international standards for depositing open data in public repositories |
| Open Science in Applied Linguistics: A Preliminary Survey |
| Replication and Open Science in Applied Linguistics Research |
| Trusting on the Shoulders of Open Giants? Open Science Increases Trust in Science for the Public and Academics |
| Open and Clarified Process of Compatibility Standards for Promoting Data Exchange |
| When open data closes the door: A critical examination of the past, present and the potential future for open data guidelines in journals |
| Supervised Authorship Segmentation of Open Source Code Projects |
| Dynamics of Cumulative Advantage and Threats to Equity in Open Science - A Scoping Review |
| Bridging Feminist Psychology and Open Science: Feminist Tools and Shared Values Inform Best Practices for Science Reform |
| Wikipedia at the University: engaging students and teachers in open knowledge and collaborative work |
| Systematizing Confidence in Open Research and Evidence (SCORE) |
| Organization of Free and Open Source Software Projects: In-between the Community and Traditional Governance |
| Crowdsourced meta-analysis in psychological science: An open science approach to compiling and synthesizing scientific knowledge |
| WP5 Scoping Report: Building an Open Dissemination System |
| FRET-based dynamic structural biology: Challenges, perspectives and an appeal for open-science practices |
| Evaluation of open-source learning management systems (LMS) using design science research methodology |
| Collaboration and Open Science Initiatives in Primate Research |
| Centering inclusivity in the design of online conferences - An OHBM - Open Science perspective |
| Gambling Researchers’ Use and Views of Open Science Principles and Practices: A Brief Report |
| Open Science and information literacy |
| Open Science in Kenya: Where are we? |
| We’re all in this together: recommendations from the Society of Behavioral Medicine’s Open Science Working Group |
| Questionable Research Practices and Open Science in Quantitative Criminology |
| Framework for Prioritization of Open Data Publication: An Application to Smart Cities |
| The importance of open research practices to empirical research in the evolutionary social sciences |
| Open Science in Psychophysiology: An Overview of Challenges and Emerging Solutions |
| Practice what you preach: Credibility-enhancing displays and the growth of open science |
| Navigating Open Science as Early Career Feminist Researchers |
| Governing Open Health Data in Latin America |
| When Values Collide: Why Scientists Argue About Open Science and How to Move Forward |
| Open Science at Liberal Arts Colleges |
| Research Data Recycling through Open Sharing and Reuse: A Case Study of Sustainable Digital Good Consumption in the Sharing Economy |
| Easing Into Open Science: A Guide for Graduate Students and Their Advisors |
| Doing Data Science on the Shoulders of Giants: The Value of Open Source Software for the Data Science Community |
| The future of product creation is open and community-based |
| Quality of Open Research Data: Values, Convergences and Governance |
| Open Science principles for accelerating trait-based science across the Tree of Life |
| The future of arXiv and knowledge discovery in open science |
| Towards an Open Science Definition as a Political and Legal Framework: Sharing and Dissemination af Research Outputs |
| Enhancing scientific credibility: An open science strategy for the South African Journal of Industrial Psychology |
| Co-building Open Science: Portrait of the University Library as a Publisher |
| Challenges, Open Research Issues and Tools in Bigdata Analytics |
| Collaborative open science as a way to reproducibility and new insights in primate cognition research |
| Both Questionable and Open Research Practices are Prevalent in Education Research |
| Behaviour Change for Open Science |
| From Open Journalism to Closed Data: Data Journalism in Italy |
| Open Data and Transparency: Opportunities and Challenges in the Swiss Context |
| Bots Coordinating Work in Open Source Software Projects |
| Assessing the Openness and Conviviality of Open Source Technology: The Case of the WikiHouse |
| Empowering open science with reflexive and spatialised indicators |
| The path to open data |
| How to Teach Open Science Principles in the Undergraduate Curriculum—The Hagen Cumulative Science Project |
| The advantages of |
| ``Because it is the Right Thing to Do’‘: Taking Stock of the Peer Reviewers’ Openness Initiative |
| Participation Research and Open Strategy |
| The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) |
| A Sensemaking Perspective on Open Strategy |
| The Politics of Openness |
| Orientations of Open Strategy: From Resistance to Transformation |
| Beyond Open Data Hackathons: Exploring Digital Innovation Success |
| Making Open Science Work for Science and Society |
| An open science rare diseases research initiative: the University of North Carolina Catalyst |
| Open Science and the Science-Society Relationship |
| A Perspective on Extreme Open Science: Companies Sharing Compounds without Restriction |
| On reproducibility and replicability: Arguing for open science practices and methodological improvements at the South African Journal of Industrial Psychology |
| Open Collaborative Data - using OSS Principles to Share Data in SW Engineering |
| Towards open and reproducible multi-instrument analysis in gamma-ray astronomy |
| Improving Open Science Using Linked Open Data: CONICET Digital Use Case |
| Embracing Open Science and Transparency in Health Psychology |
| Irrelevant Events and Voting Behavior: Replications Using Principles from Open Science |
| Agile Mechanisms for Open Data Process Innovation in Public Sector Organizations |
| Gifted Child Quarterly’s Commitment to Transparency, Openness, and Research Improvement |
| A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017 |
| Open Science - the new paradigm for research and education? |
| OpenCon 2018: Building a community of openness, advocacy, and activism |
| Open Principles in New Business Models for Information Systems |
| Open Science in Data-Intensive Psychology and Cognitive Science |
| Adding Standards and Procedures to the What Works Clearinghouse Handbook on the Transparency, Openness, and Reproducibility of Effectiveness Research on Education Practices, Products, Programs, and Policies |
| Whose open science are we talking about? From open science in psychology to open science in applied linguistics |
| Towards a framework for Open Data literacy in education: A systematic mapping review of Open Data skills and learning approaches |
| Open and collaborative tools for disaster management and risk reduction |
| Open scholarship in applied linguistics: What, why, and how |
| It’s All a Game: Strategies and Scenarios in Open Sharing Spectrum |
| Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education |
| Development of the Brief Open Research Survey (BORS) to measure awareness and uptake of Open Research practices |
| Improving the Computational Reproducibility of Clinical Science: Tools for Open Data and Code |
| Open Ocean: Inventing the Future of Ocean Exploration |
| Promoting FAIR Data Through Community-driven Agile Design: the Open Data Commons for Spinal Cord Injury (odc-sci.org) |
| The Pandemic as a Portal: Reimagining Psychological Science as Truly Open and Inclusive |
| Open collaborative writing with Manubot |
| Going Farther Together: The Impact of Social Capital on Sustained Participation in Open Source |
We saved a new csv file with all the disagreements and then made a copy of that list in the file resolved.csv. This file contains some titles that we disagreed on but that were briefly reviewed by one reviewer, discussed, and included in the end.
The titles we agreed on were:
targets::tar_read(titles_resolved_path) |>
read_csv() |>
select("Article title" = title) |>
knitr::kable()| Article title |
|---|
| Special Issue: Open, Digital, Collaborative Project Preservation in the Humanities |
| Coopetition as a driver of success for community initiatives in open research |
| Open and collaborative tools for disaster management and risk reduction |
| Promoting FAIR Data Through Community-driven Agile Design: the Open Data Commons for Spinal Cord Injury (odc-sci.org) |
| Collaboration and Open Science Initiatives in Primate Research |
| The Pandemic as a Portal: Reimagining Psychological Science as Truly Open and Inclusive |
| Collaborative open science as a way to reproducibility and new insights in primate cognition research |
| Agile Mechanisms for Open Data Process Innovation in Public Sector Organizations |
| Open Communitition |
The final list of titles we selected, between the agreed on ones as well as the resolved ones, are:
targets::tar_read(titles_selected) |>
select("Article title" = title) |>
knitr::kable()| Article title |
|---|
| UKRN ORCC Primer on Working in Open Research |
| Accelerating addiction research via Open Science and Team Science |
| The State of Open Science Practices in Psychometric Studies of Suicide: A Systematic Review |
| Eleven Strategies for Making Reproducible Research and Open Science Training the Norm at Research Institutions |
| Biomedical supervisors’ role modeling of open science practices |
| Exploring Open Science Practices in Behavioural Public Policy Research |
| UKRN Open Research Training Resources and Priorities Working Paper |
| (Why) Are Open Research Practices the Future for the Study of Language Learning? |
| Inclusive, open, and reproducible developmental science |
| Modernizing authorship criteria and transparency practices to facilitate honest, open, and equitable team science |
| The potential of inclusive and collaborative Open Research processes at the science-policy interface |
| Knowledge and practices of open science among scholars and researchers in Tanzania |
| Reducing Barriers to Open Science by Standardizing Practices and Realigning Incentives |
| Open and reproducible neuroimaging: From study inception to publication |
| Reducing barriers to open science by standardizing practices and realigning incentives |
| Lessons Learned: A Neuroimaging Research Center’s Transition to Open and Reproducible Science |
| Ten strategies to foster open science in psychology and beyond |
| Open and reproducible practices in developmental psychology research: The workflow of the WomCogDev lab as an example |
| Open and reproducible science practices in psychoneuroendocrinology: opportunities to foster scientific progress |
| Performing Open Science Culture |
| No Better Time to Practice Open Science Than Now |
| Ten simple rules for helping newcomers become contributors to open projects |
| Privacy, Anonymity, and Perceived Risk in Open Collaboration |
| Open Source, Collaboration, and Access: A Critical Analysis of “Openness” in the Design Field |
| Special Issue: Open, Digital, Collaborative Project Preservation in the Humanities |
| Coopetition as a driver of success for community initiatives in open research |
| Open and collaborative tools for disaster management and risk reduction |
| Promoting FAIR Data Through Community-driven Agile Design: the Open Data Commons for Spinal Cord Injury (odc-sci.org) |
| Collaboration and Open Science Initiatives in Primate Research |
| The Pandemic as a Portal: Reimagining Psychological Science as Truly Open and Inclusive |
| Collaborative open science as a way to reproducibility and new insights in primate cognition research |
| Agile Mechanisms for Open Data Process Innovation in Public Sector Organizations |
| Open Communitition |
Abstract review stage
- all will review the abstracts of the included papers
- this will create our final list of included papers
The second phase of the review is looking over the abstracts. In Table 4 below are the number of articles each reviewer kept after this second screening.
targets::tar_read(abstracts_kept) |>
map(\(df) count(df, name = "Articles kept")) |>
list_rbind(names_to = "Reviewers") |>
# TODO: Include percent of total articles.
knitr::kable()| Reviewers | Articles kept |
|---|---|
| Daniel | 20 |
| Mario | 19 |
| Luke | 14 |
Based on the abstract review, we want to know how many of these titles we all agreed on between the three reviewers:
common_abstracts <- targets::tar_read(abstracts_agreed_on)We seemed to agree on 10 abstracts, with the full article list shown in Table 5.
targets::tar_read(abstracts_agreed_on) |>
select("Article titles" = title) |>
knitr::kable()| Article titles |
|---|
| Open and reproducible neuroimaging: From study inception to publication |
| Ten simple rules for helping newcomers become contributors to open projects |
| Collaborative open science as a way to reproducibility and new insights in primate cognition research |
| UKRN Open Research Training Resources and Priorities Working Paper |
| Eleven Strategies for Making Reproducible Research and Open Science Training the Norm at Research Institutions |
| Open and reproducible practices in developmental psychology research: The workflow of the WomCogDev lab as an example |
| UKRN ORCC Primer on Working in Open Research |
| Open and collaborative tools for disaster management and risk reduction |
| Ten strategies to foster open science in psychology and beyond |
| Lessons Learned: A Neuroimaging Research Center’s Transition to Open and Reproducible Science |
One person will review the titles and flag any paper that might be relevant but are not in agreement to keep.
targets::tar_read(abstracts_disagreed_on) |>
select("Article titles" = title) |>
knitr::kable()| Article titles |
|---|
| Privacy, Anonymity, and Perceived Risk in Open Collaboration |
| The potential of inclusive and collaborative Open Research processes at the science-policy interface |
| Exploring Open Science Practices in Behavioural Public Policy Research |
| Open Source, Collaboration, and Access: A Critical Analysis of “Openness” in the Design Field |
| Collaboration and Open Science Initiatives in Primate Research |
| Reducing barriers to open science by standardizing practices and realigning incentives |
| Knowledge and practices of open science among scholars and researchers in Tanzania |
| Coopetition as a driver of success for community initiatives in open research |
| Reducing Barriers to Open Science by Standardizing Practices and Realigning Incentives |
| Open Communitition |
| Collaboration and Open Science Initiatives in Primate Research |
| (Why) Are Open Research Practices the Future for the Study of Language Learning? |
| Accelerating addiction research via Open Science and Team Science |
| Special Issue: Open, Digital, Collaborative Project Preservation in the Humanities |
| Open and reproducible science practices in psychoneuroendocrinology: opportunities to foster scientific progress |
| Performing Open Science Culture |
| Promoting FAIR Data Through Community-driven Agile Design: the Open Data Commons for Spinal Cord Injury (odc-sci.org) |
| The State of Open Science Practices in Psychometric Studies of Suicide: A Systematic Review |
Like with the title review, we saved a new csv file with all the disagreements and then made a copy of that list in the file resolved.csv. This file contains some articles that we disagreed on but that were briefly reviewed by one reviewer, discussed, and included in the end.
The abstracts we agreed on were:
targets::tar_read(abstracts_resolved_path) |>
read_abstract_yaml() |>
select("Article title" = title) |>
knitr::kable()| Article title |
|---|
| Collaboration and Open Science Initiatives in Primate Research |
| Accelerating addiction research via Open Science and Team Science |
| Promoting FAIR Data Through Community-driven Agile Design: the Open Data Commons for Spinal Cord Injury (odc-sci.org) |
The final list of titles we selected, between the agreed on ones as well as the resolved ones, are:
targets::tar_read(abstracts_selected) |>
select("Article title" = title) |>
knitr::kable()| Article title |
|---|
| Open and reproducible neuroimaging: From study inception to publication |
| Ten simple rules for helping newcomers become contributors to open projects |
| Collaborative open science as a way to reproducibility and new insights in primate cognition research |
| UKRN Open Research Training Resources and Priorities Working Paper |
| Eleven Strategies for Making Reproducible Research and Open Science Training the Norm at Research Institutions |
| Open and reproducible practices in developmental psychology research: The workflow of the WomCogDev lab as an example |
| UKRN ORCC Primer on Working in Open Research |
| Open and collaborative tools for disaster management and risk reduction |
| Ten strategies to foster open science in psychology and beyond |
| Lessons Learned: A Neuroimaging Research Center’s Transition to Open and Reproducible Science |
| Collaboration and Open Science Initiatives in Primate Research |
| Accelerating addiction research via Open Science and Team Science |
| Promoting FAIR Data Through Community-driven Agile Design: the Open Data Commons for Spinal Cord Injury (odc-sci.org) |
Reading stage
- first skimming the papers and group by themes, writing a basic free-hand summary
- then read a few papers from each theme to decide on what information to extract
- then discuss together the themes to group by and information to extract
Data extraction
- we make a data extraction template
- extract data and read the papers
Writing stage
- how to visualize?