Questionable arguments about serious statistics

Commentary by Robert Steinbuch
 
I write to continue the interesting discussion with Aaron Taylor regarding the proper role of the LSAT in law school admissions.  (See the first article here, my rebuttal here and his response here). 
 
Unfortunately, Taylor confuses some simple statistical truths, both in general and regarding LSAT scores and bar passage, specifically: First, correlations can be explained through any of the following: causation, reverse causation, or "third" variables. Second, nobody actually thinks that LSAT scores cause bar outcomes or vice versa. After all, how can a score on one exam cause a score on another? And, third, LSAT scores demonstrably help predict bar success due to at least one underlying common causal factor of both, i.e., likely some measure of skill. Empirical analysis is complex, and clichés and truisms do nothing to simplify it.  
 
In Taylor's original National Jurist article, he argued against the current use of the LSAT in admissions decisions by asserting a dearth of studies on the predictive value of LSAT scores on bar performance. Taylor was seemingly unaware of my coauthored 2016 empirical study of admissions data from my school (Univerity of Arkansas, Little Rock) for a period that covered Taylor's own time at the school as admissions dean. So, I specifically directed him to my law review article, which used regression analysis to demonstrate that LSAT scores and undergraduate GPAs do predict bar passage.  
 
Taylor asserts that correlation studies are insufficient alone to prove the predictive power of the LSAT. I obviously agree. That's why I provide the very regression analysis that Taylor seeks — as well as a discussion of Richard Sander's.
 
Paradoxically, after repeatedly questioning the relationship between LSAT scores and bar performance, he tacitly admits the predictive value of the LSAT when he (1) relies on the Texas Tech study demonstrating it and (2) argues that the predictive power of LSAT scores on bar passage could be caused by the third variable of wealth. One wonders why the discussion, therefore, didn't begin with this concession, but hopefully we can now bury this now resolved confusion. 
 
Irrespective of whether the cause of the association between LSAT scores and bar performance is underlying wealth, skill, or entirely something else, LSAT scores nonetheless remain useful in predicting success on the bar exam. In other words, even if arguendo we couldn't now explain the source of the predictive relationship between LSAT scores and bar passage, we could nonetheless act on it in the same way we could — using Taylor's unintentionally applicable example about the increase in both ice cream sales and murder during hot weather — put more police on the streets when ice cream sales go up. (Of course, in reality, we'd just deploy more cops when it gets hot, because we can readily discern what underlies both phenomena.)
 
And, critically, whether a law school graduate fails the bar exam based on wealth effects, skills, or other reasons, doesn't change the painful reality that he has, in fact, not passed the only exam that would have allowed him to practice law — likely with significant debt, no less. If a graduate cannot pass a bar exam, he most likely would've preferred not to have incurred the actual and opportunity costs of attending law school in the first place. (No doubt a few graduates choose not to practice law, but they are uncommon, particularly at practice-oriented schools.) And admissions committees not only do well in considering this rational preference, they need to ensure that they aren't falsely suggesting to admittees unrealistic chances of success. 
 
Upon finally conceding that LSAT scores, like those I presented, not only correlate with bar success but do, in fact, predict it, Taylor seeks to downplay how strong the predictive value is. He references a study putting that measure at 13%. This is where, finally, Taylor and I have a real disagreement.
 
As I previously discussed, Richard Sander's seminal Stanford Law Review article on the mismatch phenomenon described the predictive value on bar performance of LSAT scores and undergraduate G.P.A.s as 35%. Sander aptly called this relationship "impressive," because we're using independent variables to explain perhaps the most complex phenomenon in existence: human behavior. Taylor would presumably disagree with Sander's characterization of the usefulness of the association; he's free to do so. But Sander insightfully explains that no other factor predicts more than 5% of bar performance. So, even if 13% is accurate, it's indeed remarkable. 
 
I'm happy to debate the usefulness for admissions decisions of having a pre-consideration test that soundly predicts 13% or more of the outcome on the bar exam (before having attended one day of law school), but Taylor doesn't actually do that. He merely asserts that the exam has an "outsized role in the admissions process," but never discloses how much weight is, in fact, given to LSAT scores by law schools individually or collectively. He seems to use the LSAT as a Boogeyman to bemoan what he sees as an underrepresentation of minorities in law schools. But the notion that LSAT scores are used to shift law school seats from minorities to non-minorities is demonstrably wrong.
 
Ultimately, Taylor hasn't offered any meaningful viable alternative to considering LSAT scores and undergraduate G.P.A.s of applicants to law school. Both aid in predicting whether applicants will pass the bar exam. Taylor can spend as much time as he sees fit positing whatever theories he may have as to why; but, in the meantime, that association remains very real and useful.  And Taylor's expressed ignorance about the confounding factor(s) shouldn't be used as a cudgel to avoid consideration of LSAT scores in the admissions process.  
Categories: