By Allison Russo
This piece is part of a larger series examining more equitable practices in arts research and consulting. Our goal is to reflect openly and self-critically on the underlying processes, assumptions, and structures driving inequity and what would make the arts and culture sector more equitable.
Despite the incredible diversity of arts organizations, each strives daily towards the same goal: improvement of their services and impact. Yet, it is a privilege to access quality evaluation services that would measure an organization’s impact, illuminate gaps and weaknesses in programming, and improve accountability to supporters.
Large organizations with annual operating budgets of hundreds of millions of dollars are able to access outside services for deep research or special projects, in addition to their own team of consistent internal evaluators. Smaller organizations with budgets under $1 million make up a majority of arts organizations in many disciplines and locations; for example, orchestras of that size make up 85% of the membership of the League of American Orchestras. For these organizations, evaluation work typically falls on the shoulders of program directors, who are often skilled at measuring artistic growth, but in the dark about other elements of impact.
Even with awareness of the importance and value of outside evaluations services, these smaller organizations face barriers to access. First is the prohibitive cost for evaluation services. Although the results of evaluation work can provide data that would be valuable to attain robust and reliable funding, ironically the organizations cannot pay for the evaluation services without that funding first. Another “catch-22” is the desire to build capacity with the guidance of an evaluator, but also needing the capacity to fully utilize the data and analysis they provide.
There are also emotional and cultural barriers to accessing evaluation services. Evaluation is an emotionally charged and highly personal process. Quality evaluation work challenges a staff to face their reality, and some leaders and organizations are not prepared for the emotional component. There is also an issue of trust; not all evaluators are trained to be asset-based, trauma-informed, and culturally-responsive. Improperly obtained or handled data can do harm to a community. This is particularly limiting for organizations that serve and/or are run by BIPOC individuals or other marginalized communities.
What works?
There are many organizations and associations that offer free or low-cost professional development sessions on evaluation. However, ongoing training and individualization to a unique mission is key to evaluation work.
One model that is working for dozens of arts organizations here in Newark, New Jersey, is the collective impact evaluation work of Arts Ed Newark (AEN). An alliance of about 100 arts education organizations, AEN has provided nearly one million dollars in professional development, individual coaching, and collective data analysis over the past six years. The hyper-local, long-term, and collective nature of this evaluation work has provided a culturally-relevant learning community for arts organizations of all sizes.
The keys to the success of this effort has been a unique combination of collective efforts– including a survey tool that has allowed evaluation leader Dr. Deborah Ward to tell the impact of arts education across the massively complex city of Newark– and also individualized training and coaching to build capacity for each unique organization.
In this way, local arts councils, alliances, and other umbrella organizations can leverage their larger budgets and networks to obtain the funding needed to provide external evaluation services on a large scale. For AEN, this work was primarily supported through NEA grants, which are out of reach for smaller organizations. AEN organizations are only responsible for an annual partnership fee of just $150-$500, depending on the size of the organization. This model of providing low-cost, high-quality, long-term, and culturally-responsive evaluation services is a potential combatant to the inequities of access to evaluation services for other urban areas.
This is piece is part of our series on equity in the arts. You can read other posts here:
Researcher-Practitioner Partnerships in the Arts: Dr. Steven Holochwost, a WolfBrown Principal, writes about his ten year journey evaluating the New Jersey Symphony’s Character, Achievement, and Music Project (CHAMPS). He reflects on the nature of their research-practitioner partnership and insights that to a new program design.
Arts Outcomes Worthy of Pursuit – Joanna Borowski and Samuel McDonald of the New Jersey Symphony’s Education and Community Engagement Department share outcomes they considered worthy of research for their Training Ensemble.