LH Martin Institute

What Do the Data Say?

22 October 2012, by Prof. Victor (Vic) Borden

Victor (Vic) Borden, Ph.D. is Professor of Educational Leadership and Policy Studies at Indiana University Bloomington. He is currently a senior advisor to the Executive Vice President for University Regional Affairs, Planning, and Policy. Vic has served in senior institutional research positions at Indiana University, Purdue University and George Mason University, after starting his institutional research career as a research analyst at the University of Massachusetts in 1984. Vic is an active contributor to several professional associations, most notably, the Association for Institutional Research, of which he is a Past President.

Higher education institutions internationally face a vexing array of issues that are challenging historically autonomous and collegial modes of governance and operation. Academic concerns with the production and dissemination of knowledge as well as the preservation of culture have been augmented by higher education’s increasing role in training a qualified labor force for regional and national economic development. The growing global, market-driven competition among “world class” institutions, fueled by world university rankings, promotes the channeling of scarce resources toward the production of highly specialized knowledge and elite credentials. At the same time, political demands persist, often without commensurate funding, for access and opportunity to higher education among all members of society and especially those who have been historically under-represented in the academy and consequently in economic and political leadership roles in society.

There is a silver lining to these complicated and increasingly contentious issues confronting higher education. They reveal the increasing importance of postsecondary education to society. However, they have also stimulated accountability demands from more diverse stakeholders for a broader array of responsibilities, which further promotes the shift in governance and management from collegial to more professionalised, hierarchical and bureaucratic modes. The recent world-wide recession has not helped. The decline in public resources has spurred an increased reliance on tuition fee revenues, shifting more of the cost burden to the student, which further evokes demands for accountability from the student/consumer market and reinforces the professional/administrative capacities required to effectively respond to these demands.

Another dimension to this “new normal” is a growing emphasis on data-driven decision making. Although many people rightfully question the limitations of a “quantitative/reductionist” approach to managing people, programs and resources, especially in the academic domain, most recognise that quantitative measures and other forms of evidence can contribute constructively to the myriad decisions made regarding what courses and curriculum to offer, and how to distribute scarce resources to shape and achieve program- and institutional-level goals.

There is an emerging array of methods, tools and techniques for using data and other forms of evidence to inform the academic enterprise. To name a few: student persistence and completion models, faculty workload studies, induced course-load matrices, net tuition optimisation models, activity-based costing, peer benchmarking, student learning outcomes assessment, surveys of student engagement and, newer entrants into the arena, learning analytics and performance dashboards. As tools, each of these can be used to build things of beauty. But they can also be used to demolish them.

Using constructively the tools of any trade requires putting them into the hands of competent crafts people who are engaged in building or modifying things according to well-designed plans that meet both community and professional standards. But there are many colorful sayings and quotations about the inappropriate use of statistics that convey how a data- or evidence-informed approach can lead to undesirable outcomes. Interestingly, several of these sayings are misattributed to Mark Twain (aka Samuel Clemens), such as “Figures don’t lie but liars figure,” and “There are three kinds of lies: lies, damns lies and statistics.” (Best evidence suggests these quotations can be attributed to 19th century U.S. government statistician, Carol Wright and 19th century British statesman, Benjamin Disraeli, respectively.) Another such aphorism that I enjoy using in statistics classes I teach is, “He uses statistics as a drunken man uses a lamp-post…for support rather than illumination” (attributed to the Scottish poet, novelist and literary critic Andrew Lang). I particularly like the “drunk and lamp-post” metaphor as it also appears in a joke about a passerby who, upon seeing a drunken man searching for lost car keys at night under a lamp-post asks if that’s where he lost them, to which the drunkard replies, “no, but the light is so much better here.”

How then do we promote constructive use of quantitative methods and tools to guide higher education institutions toward pursuing their objectives individually and collectively to the satisfaction of our publics? Nobel laureate Herbert Simon suggested that the power of information is not related to its content but rather to how it captures and directs people’s attention. Data can be used to promote a specific agenda. But they can also be used to solicit diverse, insightful interpretations among the intellectually gifted individuals who are typically involved in the higher education enterprise. In a chapter of the recently published “Handbook of Institutional Research,” Adrianna Kezar and I describe a “collaborative organisational learning” framework for shifting emphasis away from a data-driven focus, per se, and toward evidence-informed discussions that solicit diverse viewpoints to illuminate the complex issues we now confront [1].

So what do the data say? Data do not speak for themselves. For better and for worse, we speak for them. Given the power of information and well-crafted analyses to direct people’s attention toward constructive solutions, the usefulness of the same analytic prowess for debunking inappropriate use of information, and the great potential for misuse when access to data and the capacity to analyse it effectively are unevenly distributed, we are best served by promoting the growth of such capacities equitably across the spectrum of the higher education enterprise. To paraphrase an aphorism that is growing in popularity, if we do not come to the table with credible evidence, we may find ourselves on the menu.

Vic Borden will run a workshop on Leading Effective Institutional Research in Melbourne next month. 

Reference

1. Borden, V. M. H. & Kezar, A. (2012). Institutional research and collaborative organizational learning. In R. D. Howard, G. W. McLaughlin, & W. E. Knight (Eds.), The handbook of institutional research (86-106). San Francisco: Jossey-Bass.