How We Conducted On-the-Ground Research During a Global Pandemic
By Aimee Ouellet, Research Analyst
New York’s sweeping criminal legal reforms went into effect just three months before the COVID-19 pandemic, which brought about its own unique set of challenges. As part of a qualitative process evaluation of how changes were implemented on the ground, we had to quickly adapt our methods to safely and meaningfully interview practitioners as well as interpret data that was now more nuanced. Here’s how we did it.
In April 2019, fueled by the increasingly abhorrent conditions at the Riker’s Island Jail Complex in New York City, New York State passed the Criminal Justice Reform Act (Act). Hailed as one of the most ambitious bail reform packages in the country, the Act focused on creating a decision-making foundation for pretrial release that was not dependent on financial resources; to do so, it made significant changes to policy and practice in key areas of pretrial decision-making.
With support from Arnold Ventures, CUNY ISLG set out to study how the initial and ongoing legislative changes were carried out. We designed a multi-year process evaluation centered around gathering the perspectives of criminal legal system stakeholders who were impacted by the legislative changes in counties across the state, with the goal of unpacking their planning and implementation efforts. Unfortunately, weeks after the legislation went into effect in January 2020, the system was struck with another major disruption: COVID-19 begun spreading in New York (and the world). The onset of the pandemic added insurmountable burdens on some criminal legal system staff and essentially paused the entire system for months.
Weeks after the legislation went into effect in January 2020, the system was struck with another major disruption: COVID-19 begun spreading in New York (and the world). The onset of the pandemic added insurmountable burdens on some criminal legal system staff and essentially paused the entire system for months.
In the face of these challenges, CUNY ISLG was left to sort out what this meant for our own research efforts. It was not only important to continue telling the story of implementation but also to understand implementation within the context of the pandemic, given its widespread impact on the criminal legal system. The pandemic required flexibility, so we adapted in several ways, including rethinking where we focused our efforts, who participated, and what we might be able to say was a result of the reform (and why), knowing that COVID-19 impacted the regular operation of the criminal legal system as well as the trends in some of the outcomes the legislation was designed to address.
See the final process evaluation here.
Rethinking Outreach Efforts as Counties Dealt with the Impact of COVID-19
Initially, to gain participation from a diverse range of counties across the state, we sought varied perspectives from counties in different regions, across major metropolitan hubs as well as in suburban and rural areas. To maximize representation and reduce overlap with other organizations conducting research on related topics, we identified the following counties for initial outreach: Albany, Monroe, Onondaga, Erie, Suffolk, Nassau, Dutchess, Washington, and all five boroughs of New York City (Bronx, Kings, New York, Queens, and Richmond).
However, very early on in our project, we recognized that COVID-19 would require us to expand recruitment efforts beyond these initial geographic areas. In large part, this was driven by limited capacity and burnout in agencies that struggled to make sense of how their regular operations would change not only in implementing the legislative changes but also in the face of the pandemic. For this reason, we first took a moment to re-strategize, waiting several months to give our points-of-contact time to regroup. Second, we decided to expand our geographic reach, switching up our original requirements of focusing deeply on a select group of counties to instead expand outreach to assess the possibility of including other counties to fill in the gaps. While simultaneously trying to secure participation in the original counties of interest, we conducted outreach to other counties in similar geographic areas (i.e., Chenango, Columbia, Cortland, Delaware, Greene, Saratoga, Schenectady, and Westchester). Outreach within these additional counties targeted a wide range of criminal legal system agencies to gauge interest in participation.
We decided to expand our geographic reach, switching up our original requirements of focusing deeply on a select group of counties to instead expand outreach to assess the possibility of including other counties to fill in the gaps.
By drawing on existing internal contacts developed from previous partnerships, internet searches of contact information, and snowball sampling techniques (i.e., asking study participants for interviewee recommendations), we successfully recruited and interviewed a total of 228 individuals from 30 agencies across 13 counties with broad representation of agency and practitioner perspectives. Most of these participants were from many of our initial counties of interest (though not all), and additional participation was secured for Columbia and Westchester counties.
Adjusting to Challenges with Individual Participation
Data collection efforts were composed of interviews and focus groups with local agency practitioners, including both high-level decision-makers and line staff, who could provide a broad view of how agencies planned for and implemented the reforms in their respective areas. Originally, interview and focus groups were planned to be in-person, conducted in two waves during targeted timelines at various intervals following implementation of the legislation.
However, the pandemic and social distancing measures largely prevented us from traveling to the targeted counties for in-person meetings for most of the project period. Additionally, many agencies were often preoccupied with other time-sensitive efforts while they were attempting to adjust their operations, not only in response to the legislative mandates but also to the implications of the pandemic on their work. It goes without saying that these efforts, of course, were prioritized over participation in a study of this nature, even for individuals and agencies who initially agreed to participate and valued the research effort.
In order to best address these challenges, we quickly pivoted to a web-based meeting platform to conduct interviews and focus groups. This allowed us to be flexible with scheduling and timelines for data collection – the virtual setting created better opportunities to set times that worked best for participants and to reschedule quickly when needed. Other remote methods were also used, such as phone calls with participants who may not have web access – particularly with respect to individuals with lived experience – and for quicker interviews as availability allowed. We also, in some instances, reduced the number of staff that we requested to interview to make it more feasible for agencies with other obligations; we were also more flexible with respect to timelines across our data collection waves. After restrictions were lifted, a few focus groups with individuals with lived system experience were conducted in person, but the virtual platform was still utilized as our default method for follow-up interviews, given its ability to reach individuals who were a greater distance from our office in NYC.
Interpreting Data Through the Context of COVID-19
Part of ISLG’s initial research efforts included compiling administrative legal system data that spanned pre-implementation baseline periods to contextualize what we heard in interviews and focus groups. We were primarily interested in using the data compiled by the New York State Office of Court Administration (OCA) and Division of Criminal Justice Services (DCJS) as a requirement of the legislation to create more transparency regarding the reforms. This dataset, which includes case-level information, provided an opportunity for anyone to explore and assess certain outcomes to better understand changes to case processing post-implementation – with some limitations.
As COVID-19 significantly altered many criminal legal processes and operations just three months after the legislation went into effect, it also introduced a significant challenge interpreting data around the legislative goals and objectives. Any examination of trends in the data pre- and post-implementation would have to consider the fact that the legislation and the pandemic were intertwined for much of 2020 and early 2021, making it difficult to tease out the reasons for any observed changes. For example, the changes to the issuance of appearance tickets (ATs, also known as desk appearance tickets, or DATs, in NYC) were projected to substantially increase in volume. Though the proportion of arrests issued a DAT in NYC did increase, nearly doubling from 27 percent in 2019 to 42 percent in 2021,[1] we also know that overall arrests and the number of DATs issued in 2020 and 2021 declined. By 2022, the proportion of arrests issued a DAT went back to pre-pandemic levels, suggesting a COVID-19-related impact as opposed to a legislative one. More data, however, over a longer period of time is needed to fully understand the impacts of the legislation.
Any examination of trends in the data pre- and post-implementation would have to consider the fact that the legislation and the pandemic were intertwined for much of 2020 and early 2021, making it difficult to tease out the reasons for any observed changes.
When possible, we worked to triangulate as many available data sources as possible including turning to our own analysis of publicly available data, as well as drawing from published reports by other research partners, webinars discussing reform-specific data, and requests of specific data points from local agencies fill in any gaps. There were numerous factors we had to consider when contextualizing this information, but the “on-the-ground” accounts offered by study participants brought a clearer picture of progress in the areas the legislation aimed to address; while interpreting data within the context of the pandemic, we worked to incorporate perspectives from participant interviews and focus groups to better understand how these results were tied to the implementation of the legislative changes.
A complete description of methods used for this study can be found in Appendix A of the full report.
[1] “Desk Appearance Ticket Arrest Analysis Data,” New York Police Department, accessed September 5, 2023, https://www.nyc.gov/site/ nypd/stats/reports-analysis/dat.page
Image by Studio Romantic on Adobe Stock.