By Benjamin Anderson ’15 (Master’s in Economics of Public Policy).
Ben is a Data Strategist for Made in Durham, a non-profit organization in North Carolina (United States) that works to improve education and career outcomes for local youths.
This article originally appeared on Made in Durham’s website.
In the past few weeks, there have been a barrage of media reports about educational achievement and, more generally, life outcomes for the youth of Durham.
The positive news is that these issues are receiving attention, but the downside is that the reports may be more harmful than helpful. At its best, data optimizes decision-making, but at its worst data can be deceptive and divisive.
Specialized knowledge is required to leverage data for decision-making, whereas selectively reporting figures requires some effort but no expertise. In the latter scenario, the ambiguity of statistical assumptions predisposes the audience to personal, as well as, framing bias. Those who go through the effort to produce data often have an agenda, and therefore, have incentives to make claims which imply causes and solutions. Data is dangerous when misused. It can create tension, undermine trust and unity, and result in costly adverse decision-making.
One key characteristic of amateur statistics, aside from lacking an experimental design, is that they do not account for the fact that outcomes are a function of many different variables. For example, schools clearly play a crucial role in influencing academic attainment, but a report drawing relative comparisons between attainment outcomes within or across cities usually implicates unidentified failures of one school district versus another while all but ignoring the effects of transportation, affordable housing, food, healthcare, and social support accessibility, as well as people’s different lived experiences, including traumatic exposure of various kinds.
Reactivity to outcomes is strongly linked to bias and emotion. Making decisions about problems and solutions based exclusively on outcomes is the logical equivalent to going with your gut.
Descriptive statistics alone have a tendency to reinforce what we already think we know rather than helping us to gain an objective understanding of the issues because we often overestimate our understanding of the context. Shards of truth may be buried in our presumptions or between the different storylines, but other times the truth isn’t within sight.
If one wanted to know what public schools are doing right and what positive changes could be made, the reported outcomes would not meaningfully increase understanding. This would be like a college basketball coach using the Ratings Percentage Index (RPI) to make game plans. The RPI is simply a function of outcome variables that are influenced by other, more influential variables over a team’s success, such as shot selection, rebounding, ball control and many others.
Similarly, objective inference about the determinants of academic achievement are impossible when we simply have some measure of the output, like grade level proficiency, graduation rates or achievement gaps. Summarized outcomes do not even begin to untangle the multifaceted causal factors of student achievement, or even point to which factors are within the schools’ control and which are shaped by other institutions that govern infrastructure, real estate development, credit markets and criminal justice.
Good intentions often lead to unintended consequences. Calculating outcomes or deriving slightly new definitions of them does not enhance the cultural or intellectual competence of our community, its citizens or the institutions within it.
This is troubling because the extent of harm done with every report that subjectively frames and selectively reports data will never be known. A symptomatic obsession can enable data to have a negative social impact, leading to the proliferation of economic and racial segregation, adverse selection of people and funds from public schools, victim blaming and the marginalization of objectivity. The focus needs to shift from symptoms to solutions.
Data should be collected and analyzed in a way that enables us to separately identify effects on outcomes, including those determinants within the school’s control and those outside, so that all can be addressed in order of impact and feasibility. Robust evaluations should yield insight, pointing out specific causal factors that affect outcomes that the schools, nonprofits policy and citizens can address.
Applying a scientific lens to social issues transforms data from punitive to instructive. Careful investigation using valid quantitative methods can help us gain an understanding of the inferences that the data will and will not permit. Through empirical analysis, we have the opportunity to disentangle the effects that different factors have on certain outcomes. This is powerful because it enables us to create informed strategies.
Subsequently, when we know how our potential actions will affect an outcome, a cost-benefit analysis can help decide which evidence should be brought to action. Operating in the public and nonprofit sectors, the cost-benefit analysis goes beyond fiscal considerations to examine social returns. Combining these empirical tools puts us in a position to optimize social welfare. Data or analysis vacant of these characteristics will result in suboptimal decision-making.
An empirical basis for decision-making that respects the complexity of determinants on outcomes and the tradeoffs between various actions or lack of action should be utilized at all levels – from the systemic to the programmatic. A symptomatic focus and a preoccupation with a single area will not result in systemic improvement. As institutions, organizations and programs, our goal should be to improve, which can only be achieved through learning.
Durham has great potential to grow while enhancing the well-being of all, including the most marginalized. Continuous improvement requires the commitment of people in the public, private, and social sectors to work together.
Part of analytical integrity is the acknowledgement that sometimes our data tells us nothing at all. If we truly care about addressing systemic issues, lack of information is a strong argument for why we should build more robust datasets that incorporate variables across institutions and the socio-economic environment. This requires a willingness to coordinate and to learn. Importantly, these actions imply the willingness to change.
The Made in Durham partnership exists to address issues of the highest importance. It is the job of data is to increase the role of evidence in the partnership’s decision-making, and because of the gravity of these decisions, I also feel an ethical accountability to this work.
If we aren’t asking the right questions, data can lead to costly decisions that undermine improvement. As members of the community, we should all be able to ask the right questions to hold decision-makers accountable to analytical standards that drive improvement.
Regardless of what the outcomes show now, or anytime in the future, what we should be asking is: what are the causes of these outcomes, what are their magnitudes, and thus, what can we do to improve.
