Today, we begin the second phase of our empirical study of the Illinois Supreme Court’s civil decision making – a data analytic look at the Court’s oral arguments. For this addition to the Supreme Court database, we reviewed the videotapes on the Court’s website of all 233 oral arguments in civil cases from January 1, 2008 through December 31, 2014. Over on our sister blog the Appellate Strategist this morning, you’ll find my list of do’s and don’ts for appellate oral arguments, drawn from those 233 arguments.
The role played by oral argument in the appellate decision making process has been a major subject of discussion in the appellate bar for many years. In some jurisdictions, observers have suggested that oral argument has little impact on the court’s decision. Others claim that oral argument is largely an opportunity for the Justices to debate with each other, with the attorneys merely facilitating the discussion. On the other hand, many appellate justices have reported that although oral arguments seldom change the result of a case, arguments frequently influence the scope and language of the court’s decision.
Although law and political science professors have been applying empirical methods to studying appellate decision making for more than sixty years, oral arguments have begun to attract researchers’ attention only recently.
The earliest study appears to be Sarah Levien Shullman’s 2004 article for the Journal of Appellate Practice and Process. Shullman analyzed oral arguments in ten cases at the United States Supreme Court, noting each question asked by the Justices and assigning a score from one to five to each depending on how helpful or hostile she considered the question to be. Once seven of the ten cases had been decided, she divided her observations according to whether the Justice ultimately voted for or against the party. Based upon her data, she made predictions as to the ultimate result in the three remaining cases. Shullman concluded that it was possible to predict the result in most cases by a simple measure – the party being asked the most questions generally lost.
John Roberts addressed the issue of oral argument the year after Shullman’s study appeared. Then-Judge Roberts (at the time, two years into his tenure on the D.C. Circuit) noted the number of questions asked in the first and last cases of each of the seven argument sessions in the Supreme Court’s 1980 Term and the first and last cases in each of the seven argument sessions in the 2003 Term. Like Shullman, Roberts found that the losing side was almost always asked more questions. So apparently “the secret to successful advocacy is simply to get the Court to ask your opponent more questions,” Judge Roberts wrote.
Professor Lawrence S. Wrightsman, a leading scholar in the field of psychology and the law, took an empirical look at U. S. Supreme Court oral arguments in a 2008 book. Professor Wrightsman chose twenty-four cases from the Supreme Court’s 2004 term, dividing the group according to whether they involved what he called ideological or non-ideological issues. He then analyzed the number and tone of the Justices’ questions to each side, classifying questions as either sympathetic or hostile. Professor Wrightsman concluded that simple question counts were not a highly accurate predictor of ultimate case results unless the analyst also took into account the tone and content of the question.
Timothy Johnson and three other professors published their analysis in 2009. Johnson and his colleagues examined transcripts from every Supreme Court case decided between 1979 and 1995 – more than 2,000 hours of argument in all, and nearly 340,000 questions from the Justices. The researchers isolated data on the number of questions asked by each Justice in each argument, along with the average number of words used in each question. The study concluded, after controlling for a number of other factors that might explain case outcomes, all other factors being equal, the party asked more questions generally wound up losing the case.
Professors Lee Epstein and William M. Landes and Judge Richard A. Posner published their study in 2010. Epstein, Landes and Posner used Professor Johnson’s database, tracking the number of questions and average words used by each Justice. Like Professor Johnson and his colleagues, they concluded that the more questions a Justice asks, all else being equal, the more likely the Justice will vote against the party, and the greater the difference between total questions asked to each side, the more likely a lopsided result is.
So far as we’re aware, our study is the first attempt to apply these statistical techniques to any appellate court below the U.S. Supreme Court. We track the number of questions asked by the Court to each side and by each Justice in every argument, noting the first questioner to each side’s counsel. Later in this phase of our analysis, we will study the relationship between the Justices’ questions, the ultimate vote of the Court and whether the Justice agreed or disagreed with the majority. Join us back here tomorrow as we begin our analysis of the data.
 Sarah Levien Shullman, “The Illusion of Devil’s Advocacy: How the Justices of the Supreme Court Foreshadow Their Decisions During Oral Argument,” The Journal of Appellate Practice and Process, Vol. 6, No. 2 (Fall 2004), pp. 271-293.
 John G. Roberts, Jr., “Oral Advocacy and Re-emergence of a Supreme Court Bar,” 30 Journal of Supreme Court History 68 (2005).
 Lawrence S. Wrightsman, Oral Arguments Before the Supreme Court: An Empirical Approach (2008).
 Lee Epstein, William M. Landes and Richard A. Posner, “Inferring the Winning Party in the Supreme Court from the Pattern of Questioning at Oral Argument,” 2010 Journal of Legal Studies 39 (2010).