Employers, like most people, tend to trust their intuitions. But when employers decide whom to hire, they trust those intuitions far more than they should....The headline way overstates the point in the text, which compares "unstructured interviews" with "specific aptitude tests." But Sunstein enthusiastically presents research that is skeptical of human intuition and sanguine about the objectivity of tests. And he doesn't give any sign of noticing any complexity in the idea of what it means to "succeed."
A lot of evidence suggests that... employers will stubbornly trust their intuitions -- and are badly mistaken to do so. Specific aptitude tests turn out to be highly predictive of performance in sales, and general intelligence tests are almost as good. Interviews are far less useful at telling you who will succeed.
What’s true for sales positions is also true more generally. Unstructured interviews have been found to have surprisingly little value in a variety of areas. For medical school interviews, for example, they appear to have no predictive power at all: in terms of academic or clinical performance, those accepted on the basis of interviews do no better than those who are rejected. In law schools, my own experience is that faculties emphasize how aspiring law professors do in one-on-one interviews -- which usually provide no information at all about how they will do as teachers or researchers....
In fact, some evidence suggests that interviews are far worse than wasteful: By drawing employers' attention to irrelevant information, they can produce inferior decisions. For example, people make better predictions about student performance if they are given access to objective background information, such as grades and test scores -- and prevented from conducting interviews entirely....
Where do we stand these days on the subject of "objective" tests? I've seen them disparaged over the years. Is liberal opinion turning in favor of these these tests? I remember, circa 1990, hearing a very famous law professor denounce the LSAT as evidence of absolutely nothing. I suggested that the LSAT was at least useful in giving those who'd squandered or botched their college education* a chance to show what they're capable of doing now, and she fiercely stood her ground. The LSAT has no value other than its negative value as a vector of discrimination.
But perhaps objective-test meritocracy is on the upswing. I wonder why. What's in the air these days? And can the air be objectively tested?
__________________________
* I was thinking of myself. I arrived at college in 1969 with a headful of ideas from Bob Dylan, the hippie movement, "The Way of Zen," and the Sermon on the Mount. I exited college with a BFA and a major in painting. But after 5 more years of youthful foolery, I had nailed a 99th percentile on the LSAT. Didn't that mean something? Or was I an inappropriate interloper? A third of a century later, I believe I was.
Posts
48 comments:
Prof. A: you would have done well at anything you put your mind and heart into. I enjoy your lawyerly stuff here but more generally your "take" on things and especially your ear for words, for what people are "really" trying to say (or conceal). Would any of that come through in an unstructured interview? Well, what is a blog?
Prof. A: you would have done well at anything you put your mind and heart into. I enjoy your lawyerly stuff here but more generally your "take" on things and especially your ear for words, for what people are "really" trying to say (or conceal). Would any of that come through in an unstructured interview? Well, what is a blog?
I took the LSAT a year and a half after I left college and was running an endloader or a jackhammer - and enjoying it. I, too, scored very highly on the LSAT, with no prep and a hangover.
At times I've felt like an interloper, too.
"in terms of academic or clinical performance, those accepted on the basis of interviews do no better than those who are rejected." I don't get how you would measure these against one another, since the rejected group didn't get in/get the job.
That's amazingly stupid, Sunstein.
Don't judge, lest ye be judged, I guess.
Ignore your brain, your wisdom, and your experience, because something better is over here on the exam table.
If you don't interview them, how do you figure out which ones are hot?
Of the standardized tests I took I thought the LSAT came closest to testing pure IQ (the SAT, at least back then, seemed to measure in good part what culture you came from and how many books you had read). One of the surprising correlative findings in the Bell Curve (which Sunstein seems to echo) was that for almost every job (including incredibly menial and boring ones) the employees with the higher IQ will perform better.
"Prof. A: you would have done well at anything you put your mind and heart into. I enjoy your lawyerly stuff here but more generally your "take" on things and especially your ear for words, for what people are "really" trying to say (or conceal). Would any of that come through in an unstructured interview? Well, what is a blog?"
The "unstructured" part of a law school admissions application is the personal statement (and all the soft variables, like what you've majored in and done with your time). On those things, combined with my LSAT, should a law school have admitted me or not? A good law school accepted me as others quite sensibly rejected me. I believe I got into NYU because of the LSAT score, and not any diversity or interestingness I might have seemed to offer. So I got a chance. Did I use it better than the next person in line? Probably not.
If I had it to do over again, I wouldn't go to art school or law school, but who is that person, the more real me than the me that's here now to ask that question?
There's no entrance test for the law professor job. There's a base line of "objective" things that you've got to hit, but after that, it's very weird, very much based on the narcissistic ideation of people who either think they lucked out getting to the inside or who need to believe they truly belong here. And I'm saying that as someone who once testified under oath in federal court about how we hired lawprofs. (I had the resumes of 4 or 5 lawprofs in front of me and had to explain why each of them was hired and why the plaintiff was not.)
"Prof. A: you would have done well at anything you put your mind and heart into."
That's a trick statement. I was going to disagree, but now I'll agree, because the condition is so severe.
I've put my mind and heart into this blog and I believe I've succeeded. I don't think I've put my mind and heart into anything else. Not any other commercial enterprise anyway. I'd like to think I've put my mind and heart into some of my family relationships at least some of the time.
Virgil Hilts, the SAT and standard IQ tests are pretty much the same thing. The LSAT and GMAT are close cousins.
I've helped interview potential hires for my company over the last 6 years. I don't know what an unstructured interview is, we always came prepared with a set of questions to ask.
Quality of answers played a small part, attitude played a much bigger part. ( If I ask you a technical question, don't point out your wonderful credentials, just answer the question. And don't tell me you don't need to know that because it is not relevant to the types of problems you want to work on. You're interviewing at a small company, you will work on whatever we need you to work on. )
Of course, HR already ensured that we only saw people with the correct credentials, and while our process weeded out some obvious duds, we hired a few duds as well.
Steve Hsu has several columns on test validity, e.g.
Holistic evaluation of applicants = noise? (Or worse?) or "The measure of success"
Having spent many years earlier in my career interviewing and hiring software developers and related personnel, I came to the the conclusion that there was a single best predictor as to actual performance: the recommendation of someone who had worked with this person and whose opinion I trusted. All other predictors were fallible, and sometimes disastrously so.
I'm very surprised that only one of links to the scholarly literature references Daniel Kahneman. I guess Kahneman's work is now boilerplate for current researchers.
The headline way overstates the point in the text, which compares "unstructured interviews" with "specific aptitude tests."
One of the more interesting bodies of aptitude tests out in the wild are the programming interview questions given for software engineering jobs. The goal of the interview question is to see if an applicant can break down the question into logical components, map the component to an algorithm or model, and then, perhaps, apply real-language constructs such as Java. The candidate is judged by their grasp of the necessary logic, the efficiency of their solutions, their alertness to potential problems with their solution, and their understanding of how to apply it.
When I began interviewing for developer jobs after working for years as a designer, I realized I was in an entirely different world. No one cared about my portfolio. Instead I was being asked to talk through user interface problems.
So I think an academic study specifically on this paradigm could be quite interesting.
AA wrote, "I've put my mind and heart into this blog and I believe I've succeeded."
Agree. And it shows.
Althouse said ...
Didn't that mean something? Or was I an inappropriate interloper? A third of a century later, I believe I was.
Sorry to hear this. Although most of the focus in getting into professional school is on the choice made by the school in the longer term it is the choice of career made by the student that is most important and that choice is usually made on very shaky grounds. Of the people applying for medical school very few have any real vocation for medicine, they are pushed by their parents, want to make a lot of money without any aptitude for entrepreneurship or simply perform well on standardized tests and can't think of anything else to do.
I tried several things at the start of my working life and was lucky enough fail quickly at some and to get some apprentice-like opportunities in others so that I wasn't going in blind. For many people a period of thrashing around before committing to a career is a good thing, although parents usually have some difficulty seeing this.
I used to interview medical school applicants. The admissions office told me I was the only faculty member who got the interview reports in on time and who seemed to spend much time on them. I suspect that many faculty members were far more interested in their own research than teaching or interviewing applicants. I used to spend considerable time on their life stories and was looking for real world experience instead of fake "volunteer" time or peace marches.
We had a doctor convicted of second degree murder in Orange County about 30 years ago. He was a plastic surgeon and allowed a women to die post-op in his office, rather than call 911. He was a graduate of a very prestigious medical school in the east and a very highly ranked residency. The court got letters from the school describing what an outstanding student he was and he had been admitted to medical school at age 19. He was a sociopath and I often wondered who interviewed him and if they asked him any questions about his life. For doctors, test scores are not enough.
My class had a 19 year old freshman. Fortunately, he went into Psychiatry where he could do no harm.
"want to make a lot of money "
I had one classmate who admitted this one time in premed. He flunked out the first year.
Does anyone ever find the perfect fit? Aren't we all interlopers, intruders in the dust. I remember basic training. We all had the same haircut and wore the same clothes. There was sleep deprivation and close order drill. I remember feeling part of the unit when we marched. That's the only time I remember fitting it, and I didn't really want to be there.
Yeah, and then fire departments try to hire according to test results and it becomes racist.
Tests of cognitive ability are the best single predictor of first year success in graduate programs. That is not an opinion, but empirical fact. Google Frank Schmidt and John Hunter.
That said, other factors certainly play into success, such as focus, persistence, emotional hardness, written and spoken verbal skills and the like. That's why Psychology grad schools (used to) look at a combination of GRE scores, grades, letters of recommendation and research experience. Now, things are quite a bit more political and focused on factors that increase skin color diversity.
Henry said...
One of the more interesting bodies of aptitude tests out in the wild are the programming interview questions given for software engineering jobs. The goal of the interview question is to see if an applicant can break down the question into logical components, map the component to an algorithm or model, and then, perhaps, apply real-language constructs such as Java. The candidate is judged by their grasp of the necessary logic, the efficiency of their solutions, their alertness to potential problems with their solution, and their understanding of how to apply it.
When I began interviewing for developer jobs after working for years as a designer, I realized I was in an entirely different world. No one cared about my portfolio. Instead I was being asked to talk through user interface problems.
So I think an academic study specifically on this paradigm could be quite interesting.
I agree with this. I love programming because it so much more constrained and definite than my real job and outcomes can be assessed immediately. When things are proving difficult in my real job I retreat to programming just as others I know retreat to mathematics, for this reassuring certainty.
For most careers there are so many variables and different aspects to the job that blanket statements about aptitude are unlikely to have much validity. For a programming job the degrees of freedom are greatly reduced, the code either works well and efficiently or it doesn't. There are other factors, of course, but they are more secondary than they are in most other jobs.
IMO,
People who do poorly on high stakes cognitive testing (whether due to lack of ability or very poor test taking skills) tend to downplay LSAT, MCAT and GRE scores. SJWs downplay the scores if the do not help achieve "the proper" diversity.
Interlopers make the world go round. And besides, Juries do not rate you by your LSAT.They rate you by your persona and your dramatic speaking skill set.
I've put my mind and heart into this blog and I believe I've succeeded. I don't think I've put my mind and heart into anything else.
Holy Althouse Agonistes!
Ann, I like to think of you as a person I would be friends with, had life put us in each other's path outside of this blog. This comment, plus the "interloper" remark sure sound as though as the final semester winds down, you are pondering the paths not taken. Suck it up, girl! You did what you did and you are who you are. Your sons are productive citizens, you have led a productive life, you have Meade, the whole city of Madison to roam around in with your camera, and a whole universe to contemplate and comment thoughtfully on. So what if you weren't a cool painter in NYC? Who you are is fine. Just fine. Your Reader, Oso
The LSAT was spot-on when it came to my law school performance.
Sadly.
The Brits claim tomfoolery as a descendant of Tom O'Bedlam, the South finds a relation to Uncle Tom, but patriarchy is everywhere.
I think it's rather common for high achievers to feel like they never really belonged in the field. Does "Fake it 'til you make it" ring a bell? I suspect that really smart people tend to regret being channeled into one thing more than someone with lesser abilities who are just content to be getting by. Always wondering about how you might have succeeded better in another area.
Test taking is both an art and a science. Took both the LSAT and the test for business school back to back, took a prep class for the LSAT and scored in the top 1%, but scored 100 points lower on the one for B School, so ended up with an MBA most of a decade before a JD. little things are important - like how and when to guess. LSAT was calculated from number of right answers minus a proportion of wrong answers, which meant, if I remember correctly, that you shoul definitely guess if you can get it down from 5 to 3 answers. B School was right minus wrong, so guessing was bad. That sort of thing. I think analytically, and very quickly, so was able to double check all my MBE answers, finish better than a half hour early each session, and still ace it (top 2% that time). Next brother is a mathematical genius, but scored 100 pets lower on the math SATs than the rest of us. Slow and thorough. Probably the same on the LSATs, but he is probably the better lawyer of the two of us. As my father said of the two attys who trained him - one jumped to his conclusions and was usually correct, the other slowly ground it out and was always correct.
The other problem with going just with the LSAT is that grades as an undergrad do matter in LS. They show a willingness to work hard. You need the IQ for understanding the material, but you aren't going to excel in LS if you don't work hard and have good study habits. I know this from personal experience - top LSATs, but only decent grades meant top maybe third of my LS class, beat out by people almost as smart who worked much harder. A lot of them. Yes, you have to be decently bright to be a good atty, but it is a lot of boring work, which is where the good study habits come in. Luckily, I found a specialty (patent law) that was both intellectually and technically challenging, more than just a lot of hard work.
Test taking and the LSAT are interesting. My grades were decentish, but nothing special. It was the LSAT that got me into law school.
I went to BYU law school, which surprisingly enough has very high entrance requirements; about the same as an Ivy. They have small class sizes, they are a good school, and they get tons of applications. So everyone admitted is pretty bright.
Oddly, the person with the highest LSAT score there was someone with a dance degree. And we had several very good music degrees that did quite well. I was surprised by that at first; since dance and music aren't thought of much as a "logical" area. We didn't have very many math majors either.
I did ok in law school, but nowhere near the top ten. But then, I didn't waste away either like those poor souls.
--Vance
When I went to law school it was all about the LSAT. If you were in-state, you just needed to be a graduate which took a C GPA. I was told that if your LSAT was marginal, the admissions people pushed hard to discourage enrollment. Two out of three were flunked out. It worked out fine.
My second year a new dean came. He raised the entrance requirements, devalued the LSAT, and started recruiting the pointy heads. Our class gift was a bench with a plaque, "From the last of the carpenters."
Once the pointy heads got there, class discussions were mostly echoes of the prof's point of view except for the few third years. Grades you know. We've seen how that turned out for the legal profession.
Oso: what you said about the Prof. Keep the blog!
All: great comments about testing and predictive power. I think interviews of mature candidates are best focused not on resume but real problems. "Suppose you had problem X. Tell me how you would attack that?" Also? Word of mouth from the trust network.
I'd love for Professor Althouse to comment on this story at my old law school. A professor wore "black face" as part of her costume in her own home for a Halloween party. She now is looking at losing her job and other professors are behind the effort to fire her.
http://www.oregonlive.com/education/index.ssf/2016/11/uo_law_professor_apologizes_fo.html
I think interviews of mature candidates are best focused not on resume but real problems. "Suppose you had problem X. Tell me how you would attack that?"
I take it one step further: "Give me an example of a time when you faced problem X and tell me how you solved it."
Are people embracing objective tests for employment or school admissions? The can't. It is illegal.
Companies ueds to use tests for employment and sometimes those were just IQ tests. The smart liberals decided that those tests were racist (maybe sexist too, I dunno). Employers needed some other screening mechanism so they switched to caring a lot about college. Suddenly everyone needed to have gone to college, whether the job itself really needed it or not. To stand out you need to graduate from a BETTER school, and better almost certainly means more expensive. Everyone needs a college degree now and there is intense competition for better schools...so college is expensive and kids have tremendous debts.
But now you think objective tests for employment might be OK? Awesome; thanks smart liberal elites.
I think, as a general matter, Sunstein is correct. The problem isn't that interviewing prospects is ineffective in itself, but rather that most interviewers are simply not up to the task of doing it properly, and I would argue that many of the ones who are up to it get weeded out by the rules laid down in human resources departments.
"Sorry to hear this."
Oh, don't feel sorry for me. I didn't say I suffered. I feel very fortunate, and I don't know what pitfalls would have lain ahead on other paths. If I can look back and say I should have done something else, I, looking back, am the person that got here. The me that wants different choices is the me that wants not to exist. How can I want that? How can I imagine the other me that exists now after a lifetime of making other choices? Who is that strange, nonexistent person, and does she wish she'd gone to law school?
"I'd love for Professor Althouse to comment on this story at my old law school. A professor wore "black face" as part of her costume in her own home for a Halloween party. She now is looking at losing her job and other professors are behind the effort to fire her."
I'm just seeing this now, after I've already done such a post. Hope you like it!
Presidential campaigns are equally useless predictors. Let's see those grades and standardized test scores. Obama passed the bar but Hillary flunked it (the DC one; she passed the Arkansas bar exam).
One of the more interesting bodies of aptitude tests out in the wild are the programming interview questions given for software engineering jobs.
I have mixed thoughts about these. They're reasonable tests, but they're also very easy to game. Read an algorithms textbook for prep and you'll nail every one of them. It's a reasonable way to weed people out, but it's not difficult enough to discern any real talent.
If you select on objective merit you won't get enough NAMs, hence the soft stuff.
"Specific aptitude tests turn out to be highly predictive of performance in sales, and general intelligence tests are almost as good." Careful, Cass. Wouldn't want to be called racist, would you? But yes, IQ scores predict lots of things, correlate well with a bunch of other scores, and in good prog conscience couldn't possibly be used to select job candidates. Because, you know.
"AA wrote, "I've put my mind and heart into this blog and I believe I've succeeded."
Agree. And it shows."
Agee with AA and Begley.
"If I can look back and say I should have done something else, I, looking back, am the person that got here. The me that wants different choices is the me that wants not to exist. How can I want that?" Fair enough. But the now-person looking back can reasonably recognize something in the then-person and wish it had been amplified. Regret is not entirely irrational. In this case, it may be the regret of not realizing that your literary talents were greater than your talent in drawing and painting (albeit of a very particular sort), that they were good enough to make a career in literature of a certain sort (as you have now done in fact), and that you coulda been somebody (which you have become belatedly).
Many business firms would like to use general intelligence tests if they could, but then they would be sued by Cass Sunstein's erstwhile colleagues for discrimination since blacks and hispanics generally fare worse on general intelligence tests than whites and asians.
As for Prof. Althouse's question about law school admissions. One, personal statements are useless, even if they haven't been ghostwritten, as many are. Two, if there were a better way to systematically predict success in law school than the LSAT and GPA, I'd say go for it, but I haven't seen any. LSATs give some measure of whether the student can learn to analyze problems the way lawyers do, and GPAs are some measure of how hard a student works. Not perfect but better than the alternatives.
How can I imagine the other me that exists now after a lifetime of making other choices?
If the many worlds hypothesis is correct, you both exist.
Apologies for no cites, but I have only first hand experience to go on.
There is a very strong correlation between Air Force Officer Qualifying Test scores and success in pilot training.
So strong, that below about the 85th percentile, there's no way to get in, because the failure rate starts to skyrocket, and failure is very expensive.
I guess that matters more when failure could mean dead people.
But after 5 more years of youthful foolery, .....we need to hear more about these 'lost years'.
Post a Comment