California Arts Council: Portfolio Evaluation

California Arts Council a State Agency logo

Team: John Carnwath, Alan Brown, Salvador Acevedo, Anh Thang Dao-Shah, Shalini Agrawal

California Arts Council (CAC) selected the team of WolfBrown and Scansion, Inc. to conduct a portfolio evaluation of its grantmaking. The purpose of this work was to gain critical perspective on CAC’s grantmaking and lay a foundation in logic for making future investment decisions.

The consultant team was selected, contracted, and ready to begin the evaluation work in early 2020, just as the onset of the Covid-19 pandemic thrust the cultural sector into an unprecedented crisis. At the time, CAC offered approximately 18 grant programs, including several that are legislatively mandated. In response to the needs of the sector, CAC suspended most of its grant programs in 2020-21 and 2021-22 and quickly pivoted towards distributing relief funding to artists and organizations.

Rather than postpone the evaluation work, CAC sharpened its focus and carried forward. An initial track of evaluation planning work yielded a work plan with three primary components:

Field Scan of Equity in Arts Funding in California

The Field Scan offers a deep analysis of California’s arts infrastructure and access to funding through the lenses of equity and access, and thus provides critical context for the other components of the evaluation. Methods included a scan of the existing literature, extensive quantitative analyses of CAC data resources and third party data on nonprofit organizations in California, as well as qualitative data from arts stakeholders in three communities across the state. Our collaborator in preparing the quantitative analysis was the National Assembly of State Arts Agencies (NASAA). The Field Scan allows us to understand the scope of CAC’s funding efforts in the larger context of California’s arts ecology, and to assess how equitable CAC’s grant distribution is relative to other sources of funding. It also offers a sweeping view of California’s nonprofit infrastructre of arts organizations, which might be helpful to other sector stakeholders.

Field Scan resources include:

  • Equity Challenges in California’s Arts Ecosystem, Report to California Arts Council, by John Carnwath (an integrative summary of all Field Scan research, 36 pages)
  • An Analysis of Equity in Nonprofit Arts Funding in California, Executive Summary (17 pages) and Technical Report (74 pages), Prepared by National Assembly of State Arts Agencies
  • Portrait of an Arts Ecosystem: Imperial County, by John Carnwath and Sarina Guerra
  • Portrait of an Arts Ecosystem: South Los Angeles, by John Carnwath and Anh Thang Dao-Shah
  • Portrait of an Arts Ecosystem: Fresno, by Salvador Acevedo and Nikiko Masumoto

Grantmaking Business Process Evaluation

The Grantmaking Business Process Evaluation explored the specific inputs and work steps in the grantmaking processes at CAC, how the grant programs intersect with each other in the larger portfolio, how communications and decision-making about the grantmaking process flow within the organization, and where the system is overloaded or stressed. Efficiency, effectiveness, and equity in funding were the three guiding principles we assessed in CAC’s business systems, with special emphasis on racial equity. The Business Process Evaluation contributed much to the Agency-Level Theory of change in pointing out stress points in the grantmaking process and revealing possiiblities for structural changes that could reduce or eliminate stressors and improve the model.

Business Process Model Evaluation deliverables (currently not publicly available) include:

  • Grantmaking Business Process Evaluation, Final Report, July 2022, by Salvador Acevedo
  • Grantmaking Process Diagram

Portfolio Review and Agency-Level Theory of Change

Given the complexities and shifting sands of CAC’s portfolio of funding programs, our basic approach to the portfolio evaluation was to first develop a framework in logic for CAC’s intended outcomes from the totality of its work (i.e., an Agency-Level Theory of Change), and then take stock of the “fit” of existing grant programs, and consider what types of different or additional investments would contribute to CAC’s success, outcome by outcome. The Evaluation Task Force was our thought partner throughout this process. The starting point for this work was the 2019 Strategic Framework, which established goals and aspirations and provided a racial equity decision tool, but stopped short of offering a specific framework for “balancing the portfolio” of grant programs.

In many public and private funding agencies, grant programs tend to be added, expanded or dropped over the years without a strong rationale, especially during times of budgetary growth or decline. Decisions are made on rolling, case by case basis, often without the benefit of a logic model or theory of change that might argue for or against a particular investment proposal. In fact, we have learned from this process just how difficult it is for an agency with such a broad purview as CAC to interrogate its investment decisions through the lens of logic because of the extraordinary amount of thought work necessary to create the Theory of Change in the first place. It is this very capacity we hope to have built through our work with CAC.

Portfolio Review and Agency-Level Theory of Change deliverables include:

  • Agency-Level Theory of Change, version 1.0 (not publicly available)
  • Summary Report on 2021-22 Grantmaking Evaluation, which includes a Portfolio Review and discussion of the “fit” of current and past grant programs with the Theory of Change

Find out more on the California Arts Council’s website.

Read John Carnwath’s post about this project for our On Our Minds newsletter here.