Boston Youth Arts Evaluation Project
Executive Summary & Key Findings
The Boston Youth Arts Evaluation Project (BYAEP), a three-year initiative funded by the Barr Foundation, was launched in 2008. Embarking upon this groundbreaking venture, Raw Art Works and collaborators The Theater Offensive, Hyde Square Task Force, Medicine Wheel Productions and ZUMIX sought to create a comprehensive set of evaluation tools designed specifically for youth arts organizations working with youth ages 13-23. With one hundred years of collective experience in the youth arts field, the BYAEP members had witnessed the transformative power of the arts (music, dance, visual arts and film, and theater) and were motivated to better understand and communicate the outcomes achieved by our programs, with the ultimate goal of better serving the youth in our programs. BYAEP had three goals: to develop a youth arts evaluation language and methodology based on existing research and the experiences of Boston youth arts programs; to use the new methodology to design, pilot, and implement evaluation systems for the five collaborating organizations; and, finally, to document what was learned and publish results to help other organizations implement evaluation systems for youth arts programs.
On the topic of evaluation, John Bare’s article “Evaluation and the Sacred Bundle” challenged nonprofits to “Measure what you value and others will value what you measure” (2005, p. 7). Our first step in developing a youth arts evaluation framework and a set of tools was figuring out what we value—our own “sacred bundle.” To do so, we needed an evaluation framework that encompassed our whole identity. For many of us in the youth arts field, evaluation strategies have felt particularly foreign. As artists we have struggled with how to prove “success” while honoring the integrity of the work we do. We required a framework to provide a common language and usable evaluation tools for assessing program quality as well as youth’s self-perceptions and satisfaction with the programs. We wanted to track longer-term outcomes through alumni evaluations and employ creative tools (like our drawing evaluations) for youth to express their views. We desired tools, data, and results that were easily accessible for directors of youth development arts programs and available on the web to enable a wider dialogue.
The project began with a thorough review of each BYAEP collaborator’s “logic model” (a road map from identified needs to program outcomes). It became clear that we helped youth build success in three main outcome areas: their skills of expression and art (I Create), their ability to look at themselves (I Am), and their ability to form connections with others (We Connect). The BYAEP evaluation tools align with these three outcomes.
Between 2008 and 2011, we dedicated thousands of hours to researching, developing, piloting, and modifying the framework and tools and experimenting with methods of data collection and analysis. While the work was focused locally in Boston, BYAEP engaged national consultants, presented at national conferences, and created a website and a video widely accessed by organizations in fourteen countries. Forums were held to present our work to a larger audience and to receive important feedback on our framework and tools.
In this Handbook, we have documented the foundational research that informed our framework and tool design, the process of developing our tools, and the challenges we encountered relative to implementation, data collection and analysis. In the Appendix we include detailed information about the collaborators and examples of how we have used our results, with images, quotations, and statistics. The BYAEP Workbook includes the BYAEP Evaluation Tools themselves, which can be customized to meet any organization’s needs. These also can be found at www.byaep.com.
While the data we have collected about the impact of our youth arts programs fill hundreds of pages, some highlights include:
- 87% of alumni from Medicine Wheel Productions were proud to report that in the past five years they had worked to improve their choices in life after being at Medicine Wheel.
- At The Theater Offensive, 88% of youth agreed or strongly agreed, “This program has helped me build my confidence.”
- At Hyde Square Task Force, the percentage of teens who agreed or strongly agreed, “This year, I have done something valuable for or in my community” increased from 57% in their Beginning Self-Evaluations to 100% in their Final Evaluations.
- At ZUMIX, 76% of youth agreed or strongly agreed in their Beginning Self-Evaluation, “I know where my life can improve and how to improve it,” and this rose to 86% in their Final Evaluation.
- At RAW, 96% of youth agreed or strongly agreed with the statement, “I am able to express who I am through the arts,” showing an increase of twenty-four percentage points from the beginning of the year.
The BYAEP evaluation data provided examples in which the five collaborators collectively excelled based on the average scores among all five organizations:
- 96% of youth in all of our programs either agreed or strongly agreed, “I feel the staff does a good job with this program.”
- 94% of youth could imagine successful options for their future. For some organizations, this showed an increase of over twenty percentage points from their Beginning Evaluations!
The data also showed us collectively where we needed improvement and how we improved over time:
- It was a bit surprising that we all scored lower on certain indicators such as “I am connected to my community,” where collectively only 58% of youth felt that this was true.
- All of the BYAEP collaborators worked hard in 2010–2011 to intentionally increase youth’s connection to their community. Our final scores, when averaged, increased nine percentage points to 67% who agreed, “I am connected to my community.”
The value of evaluation extends beyond the measure of program quality by also serving as a powerful developmental tool for youth. The actual request to fill out an evaluation form expresses to youth that their thoughts count. It also suggests that self-reflection and goal setting are worthy endeavors, intentional activities through which they can begin to change the course of their lives. One of the RAW alumni put it this way:
“RAW definitely taught me that it is important to evaluate one’s life. It taught me how important it is to explore how past and present experiences affect me. Every project we did required reflection on often intimate aspects of our lives and our identities. Since it was a group setting, I saw the individual journeys of my peers, who each struggled to figure out who they were. That camaraderie created a comfortable space for all to reflect and evaluate. I am proud of maturing over my college years. I am proud of graduating from college. But mostly, I am proud that I was able to evaluate my core beliefs. I am proud that I am actively making steps that allow me to continue to evaluate and strengthen what it is that makes me, me.” – Jen, age 28 (graduated from Harvard University in 2006 and is currently enrolled in a Ph.D. program in Clinical Psychology.)/
Our Key Findings
When BYAEP first began, the main concerns that our Forum 1 (October 2008) participants had about evaluating their own programs were that: 1) the project would require a significant expenditure of time and money; 2) their existing tools did not authentically match with the work; and 3) youth and staff were resistant to evaluation. Evaluation for many was a necessary but time-consuming task consisting of rigid processes and burdensome paperwork that might have appeased funders but did not add much value, insight or useable feedback to inform the work. Authentic systems and tools were needed to better describe the work and to evaluate it in a timely, cost-effective way.
BYAEP wrestled with the above concerns for over three years. We feel that we have succeeded in creating a framework and piloting tools that authentically represent the youth arts development field. We worked to gain a greater understanding of short-term, intermediate, and long-term outcomes while creating a language and culture that integrated the aesthetic (I Create), the personal (I Am), and the community (We Connect).
Challenges that we encountered included the time-intensity of evaluation data collection, analysis, and reporting, insufficient resources to compensate BYAEP collaborators’ generous contribution of time, and difficulties in designing an optimal database–one that effectively integrates evaluation data with other organizational systems, while being affordable and user-friendly. We have much more work ahead, but this Handbook presents the opportunity for a richer dialogue, offering our best attempts at honoring the courage of youth who give us feedback and adding our voice to the important work of our field.
“I have learned more about myself than I thought I could know. I now have a voice in what I want and what I am going to do.”
– Shawn, age 16
