Access Denied — The Impact of Paywalls on Research

August 9, 2023

Project Name: Assessing Climate Change Risk of Rural Coastal Plain Communities
Authors: Nathan Yang, Othmane Echchabi, Josh Malcom Manto, Yan Lu
Project Manager: Kiera O'Donnel
Project Lead: Emily Bernhardt



Our Adventure Begins

A team of students, led by researchers within the Saltwater Intrusion and Sea Level Rise (SWISLR) Research Coordination Network, embarked on an exciting mission. Our goal? To create a geospatial database that summarized the current extent of SWISLR and the knowledge surrounding it within the North American Coastal Plain. Sounds straightforward, right? Well, buckle up!

About SWISLR

You're probably wondering: what is SWISLR? SWISLR stands for Saltwater Intrusion and Sea Level Rise. Nearly all of the North American Coastal Plain (NACP) is subject to rising sea levels, land subsidence, more severe hurricanes, and more intense droughts.

The primary focus of the SWISLR Research Coordinating Network is to conduct convergence research by building a connective intellectual network and an integrated conceptual scaffolding to rapidly expand our capacity to forecast and prepare for SWISLR impacts throughout the rural communities of the Atlantic and Gulf coasts of North America.

For more information, visit the SWISLR website.

Data Hunting: The Fun and the Challenges

Back to our research project. The plan was simple: review articles, gather data, and map out the locations. However, the task quickly became monotonous. Who wants to manually sift through countless papers for latitudes and longitudes? Not us! So, we decided to automate the process. Brilliant, right? Well, not quite...

Enter the paywalls. Our initial enthusiasm hit a wall—a paywall, to be exact. In our quest to scrape data, we encountered access restrictions. But we were determined and resourceful, so we explored alternative methods to access the necessary information.

Data Processing Image

We had to ensure the papers were about SWISLR, so we manually screened research papers and applied our methods on the ones left. First, we tried using the Scopus API to scrape abstracts and titles and apply basic NLP methods to see whether location words were mentioned. We couldn't get abstracts for all papers, so we saved those to try to find them through other means. Eventually, we ended up using the ChatGPT API to extract location words or coordinates with more accuracy from both the abstracts and full text.

A Glimmer of Hope

Despite these setbacks, we eventually managed to gather a significant number of useful papers. This allowed us to access many more resources than we initially thought possible.

The Outcome

Thanks to our perseverance and a bit of luck, we successfully mapped scholarly articles on SWISLR. By integrating this data, we developed dynamic maps that visualized regional discussions and physical changes related to SWISLR. This interactivity allowed users to explore specific areas of interest, view detailed information layers, and gain valuable insights into how these environmental challenges are affecting coastal communities. Here is an example of what the map looked like:

Map Image

In August 2023, we had the opportunity to present our interactive ArcGIS map during a poster presentation. The interactive nature of the map allowed attendees to explore specific areas of interest and view detailed information layers, providing valuable insights into the regional effects of SWISLR. The presentation was well-received and showcased how our project could serve as a powerful tool for researchers and policymakers to better understand and address these pressing environmental challenges.

Poster Presentation Image

Special thanks to our project lead, Emily Bernhardt, and our project manager, Kiera O'Donnell, for their guidance and support throughout this incredible journey.