BTOP Projects at Work: Assessing the Effectiveness of Programs

Grant funding can provide essential supports to libraries and non-profits in the face of frozen or shrinking budgets. But without proper metrics, it can be difficult to determine the usefulness and efficacy of funded resources. A non-profit agency in Washington and a library in Maine are two examples of organizations that have developed ways to assess the materials and programming provided by NTIA’s Broadband Technology Opportunities Program (BTOP) grant funding.

Delving Deeper with Metrics

The Portland Public Library (PoPL) is one of the largest libraries in Maine. The state has approximately 100 libraries, all of which received grant funding in the summer of 2010 to establish or enhance a public computing center. Through the grant, the PoPL also received six new desktop computers and ten laptops, and was one of the smaller subset of the state’s libraries that received funds to establish regional videoconferencing hubs.

The PoPL began tracking computer usage long before the site received the BTOP grant. For example, according to staff, there were approximately 125,000 annual computer use sessions prior to the new computer equipment and increased bandwidth. But staff are thinking ahead about how to determine the usefulness of the new videoconferencing equipment they will implement as part of the grant. To measure the impact of their various services and resources, library staff established a group called The Institutional Research Team.

The Institutional Research Team will use standard metrics such as computer session usage counts, but will also delve deeper into the data. According to Sarah Campbell, of the library’s Lending Services/Technical Services department,

We absolutely assume that [the videoconferencing equipment] is going to be useful. Everybody recognizes that it’s not as good as being in person, but it is going to be better. And so [we ask ourselves] how can we make those kinds of experiences really valuable, and not feel like they are secondary to being there in person?”

To capture this qualitative data, staff plan to use more comprehensive metrics, such as patron evaluations. According to Stephen Podgajny, Executive Director, staff “think the quantification is important, but [they] are also interested in understanding very clearly what needs there were, and whether they were met. And if they weren’t, what does that mean for how [they] design services to [patrons]?”

Modifying Metrics

Staff of the EdLab Group’s Community Connect Network (CCN) also plan to use more comprehensive metrics to assess the impacts of their work. The CCN project was awarded a $4.1 million dollar BTOP grant from the NTIA in the fall of 2010; itsgoal is to improve the state’s broadband adoption rates, workforce preparation, digital literacy, and justice resources, and increase access to education and training. According to its website, “the project links rural and urban resources together to serve unemployed, low-income, disabled, immigrants, and youth through over 39 libraries, non-profit organizations, public housing, community centers, and justice centers in Washington State.” 

Two of the CCN’s major tasks will relate to education and training. One component will be capacity building training on such topics as digital skill building, education, assistive technology, online legal services, and workforce preparation, with the intent to help public computing centers meet the needs of vulnerable populations. A second major piece of the grant will be an online portal that provides resources such as best practices, curriculum, evaluation tools, and tips on computer lab management, as well as content information on topics such as education, workforce preparation, online safety, legal issues, and financial literacy.

CCN staff will also work to tailor training modules to the needs of people in the community. As Executive Director Karen Manuel explained,

For the average person, [a webinar] is not something they are familiar or comfortable with. So thinking about what kinds of archived training content would be useful for different populations. In some cases, it might be videos of trainings, or screen casts or demos.”

Just as staff work to tailor training to the needs of its community, they strive to modify their metrics and assessment tools so that they can monitor the success of their past work and inform future efforts. Ms. Manuel shared that she will look to an evaluation framework developed by the University of Washington, which has done a considerable amount of work on metrics that go beyond such basics as the number of users and hours of training. For example, the University of Washington measured different benefits of training, such as subsequent employment, academic and literacy skills, and social inclusion and personal growth.

Making Use of Metrics

Anecdotal evidence of success and basic metrics such as patron attendance at trainings can help organizations gauge the success of their work. But as the MSL and the CCN have learned, with additional grants and funders often comes a need for greater accountability, and agencies must use more sophisticated metrics to provide a more comprehensive assessment of efficacy.

Connect with Others

If you received BTOP funding or are working on a broadband project and you'd like to learn from others about assessment, please join the Broadband Now listserv, where libraries and nonprofits are discussing their particular projects. Connect with others and learn from their experiences here: http://groups.google.com/group/broadbandnow