4401992531_9f9a6b63b8_zYesterday, we addressed the principal question discussed in most previous scholarship looking at question patterns in appellate oral arguments: does the losing side average more questions?

Today, we turn to a related question: do a party’s chances of winning fall as the questioning becomes more lopsided? What about the vote – is counsel who gets far more questions than his or her opponent likely looking at a 7-0 loss?

Let’s begin with the simplest way of addressing the question. For the entire seven year period between 2008 and 2014, losing parties averaged 4.43 questions more than winning parties. However, the standard deviation of that variable was 11.56 – meaning that the difference between losing parties’ total questions and winning parties’ total questions varied widely, including a not insignificant number of cases in which the winning party received more questions.

Nor was the wide variation in questioning restricted to split decisions. Losers of unanimous decisions averaged only 4.24 questions more than their opponents. That variable had a standard deviation of 11.998 – again suggesting a wide variety of results.

Nevertheless, it is true that a party’s chances of winning fall as the difference in the number of questions asked the two sides increases. Between 2008 and 2014, parties getting at least five questions more than their opponent were 48-106 – a winning percentage of only 31.69%. Parties getting ten questions or more than their opponent were even less likely to win, going 21-71 over the seven years – a winning percentage of 22.83%. But as arguments became even more lopsided, parties’ chances of winning didn’t fall much more. Parties who received at least twenty questions more than their opponents were 5-21 over the seven years, a winning percentage of 19.23%.

Let’s try plotting the data on a scatter chart. The horizontal axis below is the difference between total questions asked the losing party and total questions to the winner. Where the winner received more questions, we report a negative number. The vertical axis is the votes for the losing party minus the votes for the winner – in other words, the magnitude of the loss, ranging from a one vote difference up to seven.

If more lopsided questioning suggested that the case would ultimately not be close, we would expect this data scatter to move from the lower left, suggesting small question differences when the difference in votes is small, to the upper right – large question differences in 7-0 cases. But if anything, the data appears to have something of an hourglass shape:

Table 48 AWe confirm the implications of the scatter plot by calculating the correlation between the margin of questions and the difference in votes. Correlation is the statistical measure which determines to what degree two variables move together. If two variables are completely linked – an increase in one leads to an equal increase in the other – then the correlation is one. The correlation between the questions margin and the votes margin, however, is only 0.02086. Thus, although the odds against a party winning slowly lengthen as questioning gets more lopsided, getting far more questions than your opponent is not a reliable indicator that counsel is in danger of a 7-0 loss.

Next week, we’ll continue to analyze what we can predict from oral arguments in which one side receives significantly more questions than the other. Does heavy questioning suggest a longer opinion?

Image courtesy of Flickr by Hector Alejandro (no changes).

2200500024_e93db99b61_zLast week, we looked at whether the level of disagreement on the Illinois Supreme Court affects the average number of questions at oral argument. Today, we begin our analysis of the central question addressed in most of the previous scholarship: does the losing side tend to get the most questions?

For the entire period from 2008 through the end of 2014, the answer was clear: yes. For the entire seven years, winners (regardless of which side they were on) averaged 13.64 questions, while losers averaged 17.1 questions. Breaking the data up year-by-year reveals that winners were questioned more heavily during only one year of the period – 2009.

Table 45 ABut one can easily imagine that this measure might vary depending on whether the ultimate winner is the appellant or the appellee. For example, if one supposes that the Court is reluctant to reverse without good reason, it would follow that a losing appellee might be questioned significantly more heavily than an appellant about to lose his or her case would be. So does the Court average more questions when it’s reversing?

In fact, there is little evidence of reversal aversion in the Court’s question patterns. In only three of the seven years did the Court average more questions when it reversed than it did in affirmances.

Table 46In the chart below, we divide out the data by winning and losing appellants, and winning and losing appellees. Several interesting points are worth noting about this chart. First, note that in five of the seven years of our study, losing appellants got more questions than any other kind of party. For both appellants and appellees, losers averaged more questions from the Court than winners did. In all seven years, winning appellees get the fewest questions. It appears to make no difference to the Court’s patterns whether the decision is to affirm or reverse – for both appellants and appellees, losers average more questions than winners in every year from 2008 through 2014.

Table 47Tomorrow, we’ll address a related question – can the size of the difference between the number of questions asked one side and the other predict the result? In other words, is an appellee who gets triple the questions the appellant does measurably more likely to lose than one who gets double?  What – if anything – can the margin tell us about the vote?

Image courtesy of Flickr by Colin Kinner (no changes).

327122302_bbc4a3935b_zYesterday, we addressed the data on average questions asked to each side by the Illinois Supreme Court in unanimous and non-unanimous cases. We concluded that there was no evidence that disagreement on the Court had a consistent effect in either direction on the total number of questions.

But of course, “non-unanimous” is an aggregate statistic, including 6-1 decisions. Let’s take another look at the data, this time dividing the cases according to whether there was one or no dissenters, versus more sharply divided 5-2 and 4-3 votes.

For the entire seven year period, both appellants and appellees averaged at least slightly more questions in cases where the Court was closely divided:

Table 43A

Once again, we turn to the year-by-year data. Although the yearly data on 2-3 dissenter cases should be read with some caution since the Court typically decides only a few closely divided civil cases each year, the numbers are once again equivocal. There is some indication that levels of questioning are drifting downwards. Appellants were asked more questions on average in unanimous and near-unanimous cases in 2008, 2011, and 2014. Appellees were asked more questions in lopsided cases than in closely divided cases in 2009, 2010, 2011, 2012 and 2013.

Table 44A

Join us back here next week and we’ll turn to the issue addressed by most of the previous scholarship on oral arguments: does the Supreme Court ask more questions of the losing side?

Image courtesy of Flickr by Alexander Henning Drachmann (no changes).

4555108439_c3aba7565b_zLast week, we began our analysis of questioning patterns on the Court by addressing the most basic question of all: which side gets the most questions? Today, we address a related question: does it matter if there is disagreement on the Court?

Certain theories about the role of oral argument in appellate decision making would suggest that the answer is yes. If the Court has already reached a tentative decision before argument, members of the Court might well take the opportunity to try to reach their colleagues on the opposite side of the case. In courts where oral argument is the court’s first real opportunity to turn its attention as a group to the particular appeal, one might expect, as Justices perceive disagreement on the Court, for questions to multiply. On the other hand, if Justices are primarily interested in testing their own preliminary conclusions, it might not matter whether or not the Court is divided.

Looking at the entire seven-year period, we see that the unanimity of the Court appears to have had no impact at all on questioning patterns. Appellees averaged exactly the same number of questions before unanimous and non-unanimous Courts over the entire period. Appellants were actually asked a few more questions when the Court was unanimous.

Table 41

Turning to the year-by-year data, we see little evidence that the presence of disagreement among the Justices influences how “hot” the bench is at oral argument in any consistent way. Appellants averaged more questions when the Court is unanimous in 2008, 2010, 2011 and 2014, but fewer in 2009, 2012 and 2013. Appellees averaged more questions before a unanimous Court in 2008, 2009, 2010 and 2013, but fewer in 2011, 2012 and 2014. Average questions to both sides before unanimous courts may be drifting downwards slightly, but average questions from non-unanimous courts show no such trend.

Table 42

Given that non-unanimity appears to have at best an equivocal impact on the Court’s questioning patterns, we can conclude that this data offers no support to the view that the Justices’ questions are primarily aimed at persuading colleagues.

Tomorrow, we will continue this line of investigating by disaggregating this data further, looking at the overall question patterns before the most closely divided (two- and three-dissenters) Courts.

Image courtesy of Flickr by Roland O’Daniel (no changes).

 

3392883329_00b509024e_zToday, we begin our analysis of the questioning patterns in civil cases at the Illinois Supreme Court.

But first, a few ground rules to govern our inquiry. Data is always assigned to the year in which the case was decided, not the date of its argument. Therefore, the data for 2008 will always be incomplete, since eighteen civil cases decided during 2008 were argued in 2007. What constitutes a single question is defined by the Justice, not by counsel. Therefore, if counsel interrupts the Justice in mid-sentence to agree with a preliminary assertion, what follows is counted as the same question. On the other hand, several members of the Court often ask questions in multiple steps, stopping to ask counsel whether he or she agrees with a preliminary assertion before leading counsel to an ultimate point. In such instances, a new question begins every time the Justice invites counsel to comment. Since the data is derived from reviewing the videotapes rather than printed transcripts, this enables us to avoid what might be a slight tendency to overcount, depending on the habits of a particular Justice or counsel.

We begin with the broadest question of all – which side tends to get the most questions?

Table 39

Several things are noteworthy about this table. First, there is some suggestion that activity levels during oral argument may be declining. With the exception of an interesting one-year spike in the data in 2012, the Court’s total number of questions asked during oral argument has declined each year since 2009. Although we have no data for the 2007 arguments, if the 2008 data is extrapolated to cover all forty-two 2008 decisions, that year would have seen the heaviest questioning of all.

Also, with the exception of the years 2010-2011 for appellants and 2011-2012 for appellees, the total number of questions asked the first two speakers in oral argument has decreased each year. Curiously, questions during rebuttal have remained fairly steady until 2013.

Finally, with the exception of the incomplete data in 2008 and cases decided during 2012, when one adds the appellant’s opening remarks to rebuttals, appellants have received more questions than appellees in every year. This suggests an interesting twist on the research we discussed yesterday. Previous researchers have concluded that the U.S. Supreme Court tends to ask more questions of the side which will eventually lose. Does this preliminary result suggest that the Illinois Supreme Court asks more questions of the side which lost below? One can certainly imagine a judicial approach which would lead to such a result, where the Court tends to examine the position of the party which lost below more searchingly than the Appellate Court winner in order to determine whether reversal is justified.

But these are aggregate statistics, dependent on the varying numbers of cases each year. Let’s look at the average questions per case in each segment of argument:

Table 40

Average questions to the appellant have outpaced average questions to the appellee every year aside from 2008 and 2012, although the difference separating the two sides may be drifting downwards. For example, in 2011, appellants averages 6.41 questions more than appellees in opening remarks and rebuttal, while the margin was only 3.88 questions in 2014. Total questions asked to the appellant has decreased every year of our study, aside from a slight uptick in 2014. On the other hand, average questions asked of both sides hasn’t changed much during our period, dropping slightly from 2009 to 2011 before drifting upwards again in 2014 at 30.12. As with total questions, the average number of questions asked during rebuttal has remained relatively constant.

Next week we’ll turn our attention to a related question: is the number of questions affected by whether the Court is divided on the merits of the case?

Image courtesy of Flickr by MrHicks46 (no changes).

4810954845_13f12b6948_zToday, we begin the second phase of our empirical study of the Illinois Supreme Court’s civil decision making – a data analytic look at the Court’s oral arguments. For this addition to the Supreme Court database, we reviewed the videotapes on the Court’s website of all 233 oral arguments in civil cases from January 1, 2008 through December 31, 2014. Over on our sister blog the Appellate Strategist this morning, you’ll find my list of do’s and don’ts for appellate oral arguments, drawn from those 233 arguments.

The role played by oral argument in the appellate decision making process has been a major subject of discussion in the appellate bar for many years. In some jurisdictions, observers have suggested that oral argument has little impact on the court’s decision. Others claim that oral argument is largely an opportunity for the Justices to debate with each other, with the attorneys merely facilitating the discussion. On the other hand, many appellate justices have reported that although oral arguments seldom change the result of a case, arguments frequently influence the scope and language of the court’s decision.

Although law and political science professors have been applying empirical methods to studying appellate decision making for more than sixty years, oral arguments have begun to attract researchers’ attention only recently.

The earliest study appears to be Sarah Levien Shullman’s 2004 article for the Journal of Appellate Practice and Process.[1]  Shullman analyzed oral arguments in ten cases at the United States Supreme Court, noting each question asked by the Justices and assigning a score from one to five to each depending on how helpful or hostile she considered the question to be. Once seven of the ten cases had been decided, she divided her observations according to whether the Justice ultimately voted for or against the party. Based upon her data, she made predictions as to the ultimate result in the three remaining cases. Shullman concluded that it was possible to predict the result in most cases by a simple measure – the party being asked the most questions generally lost.

John Roberts addressed the issue of oral argument the year after Shullman’s study appeared.[2] Then-Judge Roberts (at the time, two years into his tenure on the D.C. Circuit) noted the number of questions asked in the first and last cases of each of the seven argument sessions in the Supreme Court’s 1980 Term and the first and last cases in each of the seven argument sessions in the 2003 Term. Like Shullman, Roberts found that the losing side was almost always asked more questions. So apparently “the secret to successful advocacy is simply to get the Court to ask your opponent more questions,” Judge Roberts wrote.

Professor Lawrence S. Wrightsman, a leading scholar in the field of psychology and the law, took an empirical look at U. S. Supreme Court oral arguments in a 2008 book.[3] Professor Wrightsman chose twenty-four cases from the Supreme Court’s 2004 term, dividing the group according to whether they involved what he called ideological or non-ideological issues. He then analyzed the number and tone of the Justices’ questions to each side, classifying questions as either sympathetic or hostile. Professor Wrightsman concluded that simple question counts were not a highly accurate predictor of ultimate case results unless the analyst also took into account the tone and content of the question.

Timothy Johnson and three other professors published their analysis in 2009. Johnson and his colleagues examined transcripts from every Supreme Court case decided between 1979 and 1995 – more than 2,000 hours of argument in all, and nearly 340,000 questions from the Justices. The researchers isolated data on the number of questions asked by each Justice in each argument, along with the average number of words used in each question. The study concluded, after controlling for a number of other factors that might explain case outcomes, all other factors being equal, the party asked more questions generally wound up losing the case.

Professors Lee Epstein and William M. Landes and Judge Richard A. Posner published their study in 2010.[4] Epstein, Landes and Posner used Professor Johnson’s database, tracking the number of questions and average words used by each Justice. Like Professor Johnson and his colleagues, they concluded that the more questions a Justice asks, all else being equal, the more likely the Justice will vote against the party, and the greater the difference between total questions asked to each side, the more likely a lopsided result is.

So far as we’re aware, our study is the first attempt to apply these statistical techniques to any appellate court below the U.S. Supreme Court. We track the number of questions asked by the Court to each side and by each Justice in every argument, noting the first questioner to each side’s counsel. Later in this phase of our analysis, we will study the relationship between the Justices’ questions, the ultimate vote of the Court and whether the Justice agreed or disagreed with the majority.  Join us back here tomorrow as we begin our analysis of the data.

Image courtesy of Flickr by Andrew Steele (no changes).

—————————————————————-

[1]               Sarah Levien Shullman, “The Illusion of Devil’s Advocacy: How the Justices of the Supreme Court Foreshadow Their Decisions During Oral Argument,” The Journal of Appellate Practice and Process, Vol. 6, No. 2 (Fall 2004), pp. 271-293.

[2]               John G. Roberts, Jr., “Oral Advocacy and Re-emergence of a Supreme Court Bar,” 30 Journal of Supreme Court History 68 (2005).

[3]               Lawrence S. Wrightsman, Oral Arguments Before the Supreme Court: An Empirical Approach (2008).

[4]               Lee Epstein, William M. Landes and Richard A. Posner, “Inferring the Winning Party in the Supreme Court from the Pattern of Questioning at Oral Argument,” 2010 Journal of Legal Studies 39 (2010).

13567059383_dc71a5f599_zToday we conclude our analysis of voting patterns over the past fifteen years’ worth of civil cases on the Illinois Supreme Court by considering the entire fifteen year period – which Justices voted with the majority most often when the Court was split? The data below encompasses the entire tenure on the Court for every currently serving Justice other than Justice Freeman. For Justice Freeman and all retired Justices, the data encompasses all civil cases from January 1, 2000 through December 31, 2014. During that period, former Chief Justice Fitzgerald was most often in the majority in non-unanimous civil cases, followed by Justices Thomas, Miller, Theis, Heiple, McMorrow and Chief Justice Garman:

Table 38.1

Join us here next Tuesday morning as we begin the next phase of our data analytic analysis of the decision making of the Illinois Supreme Court.

Image courtesy of Flickr by Tim Evanson (no changes).

1898178998_1a77d72802_zLast week, we began our study of the voting patterns on the Illinois Supreme Court between 2010 and 2014 by considering the agreement rates between the Justices in non-unanimous civil cases. Now we turn to a different aspect of the issue: the question of swing votes on the Court.

For most of the period, the data reflect our numbers on agreement rates, with Chief Justice Garman and Justices Thomas and Karmeier voting with the majority in the vast majority of non-unanimous civil cases. Justices Theis and Burke vote with the majority only slightly less often, although Justice Freeman is not far behind:

Table 37 ATurning to the subjects most frequently heard by the Court, we find little disagreement among the Justices on tort cases, with six of the seven Justices nearly always in the majority:

Table 37 BNext, we turn to non-unanimous decisions in the field of civil procedure. Here, we find more disagreement, with the Chief Justice and Justices Thomas and Karmeier somewhat less often in the majority: [1]

Table 37 CTomorrow, we’ll conclude our look at the voting dynamics on the Illinois Supreme Court by looking at the individual Justices’ voting data for the entire period – which Justice was most often in the majority?

Image courtesy of Flickr by Tony Webster (no changes).

 

 

————————————————–

[1]              The Court handed down no non-unanimous decisions in the field of civil procedure in either 2010 or 2011.

2969665423_cb5861053f_zYesterday, we addressed swing voter patterns at the Illinois Supreme Court between 2005 and 2009. Today, we turn to agreement rates between the Justices during the final five years of our study period, 2010 to 2014.

The Court’s unanimity rate between 2010 and today presents us with a bit of a conundrum. In 2010 and 2011, the Court decided more than seventy percent of its civil cases unanimously, but in 2012 and 2013, the Court’s unanimity rate abruptly dropped nearly twenty points. During the first two terms of 2014, the Court’s unanimity rate has risen back to its trend level of 77.8%.

The reason for the two-year dip in the Court’s unanimity rate is not evident from examining the Court’s voting patterns during the final five years of our study period. Chief Justice Garman had a high agreement rate with Justices Thomas and Karmeier, Justice Burke and increasingly, Justice Theis. Justices Thomas and Karmeier both had quite high agreement rates with their fellow Republican Justices. Justice Theis voted even more often with Justice Garman in our three-year floating averages for 2013 and 2014 – nearly eighty percent of the time.

Table 35 A

Chief Justice Kilbride and Justice Freeman were the two leading dissenters on the Court, with relatively low agreement rates with the other Justices:

Table 35 B

Since three-year floating averages do not reflect the reason for the sharp dip in 2012-2013 in the Court’s unanimity rate, we turn to the single-year statistics. This data suggests that rather that the two voting blocs – Justices Garman, Thomas, Karmeier, Burke and Theis and Chief Justice Kilbride and Justice Freeman – remained relatively steady during these two years. Instead, the decrease in unanimity appears to have been the result of the two voting blocs moving further apart, agreeing with each other less often.

Table 36

Next week, we’ll conclude our study of the 2010-2014 period by considering the data on swing voting patterns. Then, we’ll begin the next phase of our study of the Court’s civil decision making.

Image courtesy of Flickr by Kelley Minars (no changes).

5139846952_1dd049fcf8_zLast week, we began our intensive look at the voting dynamics at the Illinois Supreme Court in civil cases between 2005 and 2009. Today, we assess the results of our study of agreement rates by turning to the issue of swing voters. How often was each Justice in the majority when the Court was divided?

Table 34 A

Again, the data largely confirms our tentative conclusions based on agreement rates. Justice Fitzgerald, Chief Justice Thomas and Justice Freeman were in the majority in eighty percent or more of all divided decisions. Justices Burke and Garman were only slightly behind.

Let’s disaggregate the data and take a look at the Court’s most common subject matter, tort law.

Table 34 B

This table suggests a highly united Court in the area of tort law, with Chief Justice Thomas and Justices Freeman, Karmeier and Fitzgerald voting with the majority between eighty and ninety percent of the time in most years. Justice McMorrow and Justice Garman ranked only slightly behind, agreeing with the majority in between seventy and eighty percent of the Court’s tort cases.

Tomorrow, we’ll turn our attention to agreement rates during the final five years of our study period, 2010 through 2014.

Image courtesy of Flickr by Lisa Parker (no changes).