Boston Youth Arts Evaluation Project
Overall Reflections on Evaluation & BYAEP
We have provided a lens, a framework, and tools that have helped us and others gain understanding of the complexity and effectiveness of youth arts organizations. The BYAEP Framework and tools have been instrumental in bringing all of our organizations to the next level of evaluation and program measurement. What we have collectively realized is that while a common language is incredibly useful, each organization must recognize its own dialect. There is great potential to positively influence the work that happens in your organization by understanding more fully what is happening in your programs. Having the framework is a great jumping-off point, and we highly encourage you to modify the actual surveys to meet the needs of your individual organizations. While many of the surveys and questions have produced powerful results, there may be some that are not applicable to the specific mission of your organization, and there will be other questions that need to be added. Make these tools work for you, do not make yourself work for them, and don’t be afraid to shift things around a bit so that everything fits together.
We learned—and urge you to embrace—the POWERFUL story that can be expressed through a combination of NUMBERS, STORIES, and IMAGES. This is the three-legged stool on which we can stand and see/be seen in the most complete and impactful way (see Appendix).
Believe in the power of the work you do. Seek to understand it. Boldly show its beauty and nurture the weaker areas so that they may become strong. There is also magic that happens in every strong arts organization that cannot, and perhaps even should not, be measured. We must always allow the art itself to be a bold voice in this process. As Eliot Eisner said, “Neither words nor numbers define the limits of our cognition; we know more than we can tell!we need art forms to say what literal language cannot say” (Eisner, 2004).
Questions to Further Explore
- How can we most accurately see the “whole picture” in our analysis? Can we work to invent new tools supported by research and funders that can better take into account the complexity and relational nature of our field, of our youth, community and the age in which we now live? What are other ways beyond traditional control groups, logic models, and “pre- and post-tests” that can more effectively measure success through an individual’s personal growth, essential skills, and community engagement? How might this system better address the intensity, breadth, and duration of the youth that are involved in our programs?
- Are there ways that we can collaborate through shared data systems, measurement tools, and outcomes so that we could better address and assess our impact while saving time and money? How might we find shared solutions by fostering further cross-sector collaborations? Who might fund these dynamic collaborations that will bring forth effective systems of measuring impact? Where can we find funding to support a mentorship program of professional evaluators to work with arts organizations?
- How do we most effectively weave the creative arts into the process of evaluation so that it becomes more accessible and feels less like a “test” for youth and teachers and more like artistic reflection and critique?
- How can we more dynamically report on our discoveries? How can we display our findings in ways that more directly speak to the artistic nature of our organizations in strong and beautiful ways?
Next Steps
The complex and relational nature of the work we do is very difficult to assess, and often the tools we are given to measure our progress do not suit our circumstances. Bare (2010) beautifully describes the traditional logic model as a string of dominoes, where he states that when we knock one over, we will get a chain reaction that knocks over every other perfectly lined-up domino in our “If! Then!” theory of change. However, we all know that we do not live in a linear and simple world. Bare challenges us to think of our world rather like the game of pick-up sticks, where our interconnections are messy and where it is virtually impossible to touch one item without all other aspects being affected. It is, in fact, more like a game where we are all moving the sticks at the same time and it “is nearly impossible to predict exactly how the pile will move” (Bare, 2010, p. 89). We work in complex worlds, and what we attempt to see and evaluate is often the “in-betweenness” of things. As we engage with youth in our arts programs we especially value all the feedback we can get directly from them. We know the power in these stories far outweighs the quantifiable statistics of control groups. As a field we are now challenged to build better tools that can give us better feedback to match the complexity of our systems and the issues at hand.
The good news is that we are all beginning to talk about our complex systems in very similar ways—ways that seem to do justice to this complexity. Since the publication of the BYAEP Framework, we have found many other frameworks that speak in similar terms. A report released by the Community MusicWorks (Wolf & Holochwost, 2009) and one from the Mosaic Youth Theatre in Detroit (Gutti & Spencer, 2008) look at outcomes directly corresponding to those of BYAEP. As the youth development field launches further research and has further discussions, we collectively can enlarge our “capacity to be strong” and create tools that more powerfully align with these very similar outcome areas.
What we need is a movement that challenges the orthodoxy of “domino” views of the world and more realistically sees the “pick-up sticks” world in which we live. We need tools that can better help us to view our worlds and we need database systems that are up to the challenge of recording and reporting on the things that matter. We need support for these efforts. We need to creatively design tools and systems that can dynamically inform our efforts, ones that are committed to the outcomes but also have the flexibility to help us “adapt and adjust in terms of what it will take to produce the desired impact” (Bare, p. 103).
We can no longer do this alone. As Kania and Kramer noted in their article Collective Impact, and as we have experienced over the past three and a half years, “No single organization, however innovative or powerful, could accomplish this alone! large-scale social change comes from better cross-sector coordination rather than from the isolated intervention of individual organizations! examples suggest that substantially greater progress could be made in alleviating many of our most serious and complex social problems if nonprofits, governments, businesses, and the public were brought together around a common agenda to create collective impact” (2011, pp. 36-38).
We have a ways to go, but we have the courage and the skills to navigate through complexity. It is what we do well.
“It’s the artists of the world, the feelers and thinkers, who will ultimately save us, who can articulate, educate, defy, insist, sing and shout the big dreams.”
– Leonard Bernstein (Trilling, 2010)
