Appendix 1: Methodology | Diversity in the Orchestra Field 2023

No dataset is fully representative, and all statistics are subjective: conventions and biases inform the way we collect, analyze, and interpret quantitative data. Here, we outline the methods and definitions that have shaped the study and explain the steps we have taken to minimize harm.

Data sources

This report uses annual data collected mostly from the League’s Orchestra Statistical Report (OSR), with some data drawn from its Salary and Benefits Survey.

151 adult U.S. orchestras participated in OSR20, representing 29.9% of the 523 League of American Orchestras’ adult orchestra members.

180 U.S. orchestras completed a separate Demographics Survey in January 2023, which captured staff, musician, board, and conductor gender and race/ethnicity demographics as of December 31, 2021 and December 31, 2022. Twenty-four of these orchestras are youth orchestras, which were excluded from the analysis on musicians, staff, and board to remain consistent with OSR data, leaving a data cohort of 156 orchestras.

The League’s Orchestra Statistical Report and Demographics Survey both collect data from orchestra administrators. Orchestra administrators are encouraged to collect demographics data direct from staff, board members, musicians, music directors and conductors in order to optimize inclusion and data accuracy. In the 2022-2023 season, between 36% and 50% of the demographics data received by the League was self-reported by individuals in this way. We refer in the report to individuals “identified as” rather than “identifying as” because in many cases individuals were assigned racial/ethnic and gender categories to them. The League is working to encourage all orchestras to collect self-reported demographics data.

Because the majority of data is not self-reported, we are not able to make any analysis of the intersectionality between racial/ethnic and gender identities, or to report on the intersections between different racial and ethnic identities.

Definitions

Please see the definitions outlined earlier in this report.

Data representativeness

We are confident that the data included in this report is representative of the League’s membership overall, when disaggregated by two budget categories (larger budget and smaller-to-medium budget orchestras).¹ With a response rate of 29.9% from the League’s 523 adult orchestra members in FY20, the estimates meet the recommended sample size for a 95% confidence level, which is advisable in ensuring representativeness of the sample.²

The representation of smaller-to-medium budget orchestras within the League datasets improved significantly between 2014 and 2023. Survey improvements since 2014 increased the sample representation of smaller-to-medium budget orchestras and by the 2019-2020 survey the representation of smaller-to-medium budget orchestras had increased by 13 percentage points since 2014. The representation of smaller-to-medium budget orchestras increased still further when, for the years 2022-2023, the League used a new survey that separated orchestras demographics from the financial and operational data collected in the OSR.

Budget Size	OSR14 Survey	OSR20 Survey	Demographics Survey 2023	League Population FY20
Larger Budget Orchestras (Groups 1 to 3) (16% of League Member Population, 2020)	53%	40% (-13 pct. points)	35%	16%
Smaller-to-Medium Budget Orchestras (Groups 4 to 8) (84% of League Member Population, 2020)	47%	60% (+13 pct. points)	65%	84%
Total Respondents	106 Adult Orchestras	151 Adult Orchestras	156 Adult Orchestras	523 Adult Orchestras

For certain populations within our dataset, our sample size is relatively small. These populations include people identified as nonbinary, as American Indian and Alaska Native, and as Native Hawaiian and Other Pacific Islander. The relatively small sample sizes of each of these populations means that they often go unreported, and that they consequently suffer a loss of power. For this report, we decided to include statistics for each group, for each orchestra role, while avoiding further disaggregation by budget category. We also stated and accepted that lower data confidence in the reporting of each of these demographic groups is a reasonable cost of ensuring representation.

Our sample size is also relatively small for two of the orchestra role populations in our dataset: music directors and top executives. For these two orchestra roles, we have protected data confidence by largely excluding budget group analyses.

The total League membership of 523 adult orchestras in FY20 does not represent the entire U.S. orchestra field.³ However, the activity of League member orchestras (programming, labor force, revenue, expense) can be considered representative of all U.S. orchestra activity, since the vast majority of non-member orchestras report annual expenses to the IRS that indicate a very small scale of operations.

Analytical approach

Data can be subverted through its analysis and interpretation, inadvertently perpetuating harmful stereotypes. The steps we have taken to minimize harm in the writing of this report include the following:

• Acknowledging that diversity of representation is only one aspect of inclusion. 
• Centering the experiences of all historically marginalized racial/ethnic and gender groups in our analyses and reporting.
• Being clear where our small sample size inhibits our ability to disaggregate data or report on trend lines.
• Introducing U.S. population comparators.
• Consulting widely with orchestra community members who identify with a wide range of racial/ethnic and gender identities.

Within this framework, having aggregated the data using Stata software, we conducted a range of analyses including the following:

• Descriptive analysis. These results described and summarized data, in order to show patterns in the racial/ethnic and gender diversity of the orchestra field, including strengths and weaknesses in representation, and differences between larger and smaller-to-medium budget orchestras.
• Constant sample analysis. A constant sample analysis of 55 orchestras taking part in the OSR in 2010, 2015, and 2020 helped to validate the results of the descriptive research, as the cohort of orchestras supplying data had shifted over time.
• Comparative analysis.
  • Throughout the report, we compare the representation of particular racial/ethnic and gender groups in certain orchestra roles with the U.S. population. These comparisons are not necessarily intended as strategic benchmarks, but rather as a constant unit of comparison that aids future interpretation. To supplement the U.S. population data comparators, we identify peer art form and nonprofit sector comparisons wherever possible. In many cases, differences in data categories prevent comparison.
  • We also compare the representation of particular racial/ethnic and gender groups in certain orchestra roles, distinguished by orchestra budget category. We recognize that smaller-to-medium budget orchestras employ fewer staff than larger budget orchestras, and the musicians they employ may work for multiple orchestras; only larger budget orchestras are typically able to provide full-time musician employment. Larger budget orchestras typically have the financial leverage needed to pay higher salaries and to hire consultants. Smaller-to-medium budget orchestras typically operate within local hiring markets, whereas larger budget orchestras typically operate within national hiring markets.
• Longitudinal trend analysis. Trend analyses for all orchestra roles, broken down by orchestra budget group wherever sample size permitted, allowed us to assess representation growth and decline over time. Trends were considered sufficiently meaningful to be eligible for reporting in the text when they met the following criteria:
  • For populations with a proportion of 0% to 2.9% relative to the total sample, a 0.5-percentage point change triggered a mention in the text.
    • Example: From 2014 to 2023, the proportion of nonbinary musicians increased by 0.6 points, from 0.0% to 0.6%.
  • For populations with a proportion of 3% to 9.9% relative to the total sample, a 1.5-percentage point change triggered a mention in the text.
    • Example: From 2014 to 2023, large increases in Hispanic/Latinx (2.5% to 4.8%) and Multiracial (0.8% to 2.4%) musicians.
  • For populations with a proportion of 10% to 29% relative to the total sample, a 3-percentage point change triggered a mention in the text.
    • Example: From 2014 to 2023, the proportion of women conductors increased meaningfully by 10.9 points, nearly doubling from 12.9% to 23.8%.
  • For populations with a proportion of 30% or higher relative to the total sample, a 5-percentage point change triggered a mention in the text.
    • Example: From 2014 to 2023, the proportion of women top executives slightly decreased by 1.9 percentage points, from 51.9% to 50.0%. However, women top executives grew meaningfully by 6.8 percentage points in larger budget orchestras, from 24.1% to 30.9%. 

---

Footnotes:

1. The margin of error (MOE) is a statistical term that describes how many percentage points the results will differ from the League population value. Given the sample size, the statistical MOE for FY20 is ±6.012 percentage points. A margin of error of ±6 percentage points at the 95% confidence level means that if we administered the OSR survey 100 times, we would expect the result to be within 6 percentage points of the true population value 95 of those times. The MOE is affected by participation rates in each year’s survey, yet throughout the reporting period it remains reasonable in ensuring representativeness of the sample, at under +10 percentage points at the 95% confidence level.
2. FY20 was used as a basis for calculating the overall confidence level because it was the final year of data collection within the Orchestra Statistical Report. 
3. The total U.S. population of adult, youth, and community orchestras was estimated at 1,200 orchestras in 2014 (Voss, Voss, and Yair 2016).

---

Jump to another page:

• Introduction
• Overview
• Key Findings
• Recommendations
• Definitions
• Technical Notes
• Detailed Analysis 1: By Orchestra Role (see full PDF)
• Detailed Analysis 2: By Demographic Group (see full PDF)
• Detailed Analysis 3: By Role and Budget Size (see full PDF)
• Detailed Analysis 4: Focus on the Pandemic Years (see full PDF)
• Appendix 1: Methodology
• Appendix 2: Stories of Progress in Orchestras
• Appendix 3: References
• Acknowledgements

---

This project was supported in part by an award from the Research Grants in the Arts program at the National Endowment for the Arts, Grant #1863433-38-20. Additional funding was provided by a generous grant from the Mellon Foundation, and by public funds from the New York City Department of Cultural Affairs in partnership with the City Council.