The Limits of IQ Tests
Until now, the only cognitive skills measured were those initially identified to predict schoolchildren's academic performance—and traditionally such skills have been measured using IQ tests. Although IQ tests were not originally intended for use in business, studies have shown that these instruments predict work performance at least as well as competency interviews do (the most common assessment tool used today for hiring and promotion) and about ten times better than personality tests do. That's because some of the thinking skills that support academic success are also crucial to executive performance.

Yet IQ testing is not widely used as a way to identify top talent (though it plays an indirect role, as companies may choose to hire people with degrees from elite schools). The skills that IQ tests assess represent a fraction of a person's existing cognitive abilities. Some of the skills measured—such as vocabulary, arithmetic, and spatial reasoning—have almost no relevance to managerial work. Moreover, the topics tested would seem academic and elementary—indeed, almost insulting—to people with extensive professional experience.

The format is also ill suited to business. Executives rarely if ever confront problems that have just one right answer; nor do they have the option of picking one answer from several choices listed. IQ test questions don't assess the practical, on-your-feet thinking skills needed in business. What's more, these tests have been repeatedly accused of racial and gender bias.

Yet, despite these very real shortcomings, IQ tests are still a better predictor of managerial success than any other assessment tool. The business world's reluctance to use intelligence testing of any kind (other than assessments of emotional intelligence, which is really about personality and style) has robbed companies of a powerful tool for evaluating candidates for employment or promotion. It is, however, possible to create a comparable measure of intelligence for executives, one that tests for the skills managers need—such as evaluating the quality of data or accurately identifying the core issues in a conflict—and in a format that more accurately emulates the real business environment.

Interviewing for Intelligence
The most common interviewing methodology is the "past behavioral interview" (PBI). A PBI includes questions about a person's experiences performing certain activities—such as managing deadlines or resolving conflicts—but does not include personal questions. This form of interview has become the accepted best practice over the past thirty years, and, in fact, the PBI is a good predictor of performance. It can explain about 25 percent of the variances in performance among employees.

Still, PBIs miss a lot of what determines executives' success. That's because they don't measure what they claim to. Take two sample PBI questions. "What is the strategic direction of your company or division, and how did you go about developing it?" is designed to assess someone's competence at devising strategy. And "Describe a situation in which you had to interact with a difficult colleague and resolve a conflict" is supposed to test a person's capacity to handle conflicts. Surprisingly, you can just as accurately predict an executive's ability to devise strategy based on her answer to the second question as you can based on her answer to the first. This is not just a single example unique to these two competency questions; the same circumstance holds true for any competency question.

Research by professors Jesús F. Salgado and Silvia Moscoso of the University of Santiago de Compostela in Spain explains why. A person's performance on any behavioral interview question is dominated by the same three qualities: experience, job knowledge, and social skills. A candidate with a long work history has lots of compelling examples to draw from when asked to recount events that might illustrate a particular competency. A candidate's job knowledge—specifically, his awareness of industrial and managerial best practices—can make it easier for him to punctuate his answers with stories that will earn him high marks from interviewers. And a candidate who can relate his stories in a positive, likable manner has a distinct advantage over someone with inferior social skills. Because each question in the behavioral interview essentially assesses the same qualities, there's no need for the grueling three-to-four-hour sessions favored by hiring managers today. They need only ask enough questions to get a reliable appraisal of the candidate's work experience, job knowledge, and social skills.

Despite their advantages, behavioral interviews really only establish a candidate's minimum qualifications; they don't identify star talent. A candidate's experience, for example, is obviously an important hiring factor, but we all know seasoned executives who aren't stars. Similarly, being likable doesn't mean you have the intellectual horsepower to be a stellar leader. In short, behavioral interviews measure knowledge, not intelligence. Knowledge is information acquired through experience or formal training. Intelligence is the skill with which someone uses knowledge to solve a problem. Knowledge questions require people to recite what they have learned or experienced, while intelligence questions call for individuals to demonstrate their abilities.

So how do you measure executive intelligence? The best way is to use questions that require candidates to demonstrate their skills in an interview format. For such a measure to assess intelligence, it must raise questions and situations that the candidate has never confronted. The more novel the situation, the less rote knowledge can be applied and the more cognitive ability is required to render an answer.

The interview format is a departure for intelligence tests, which have traditionally been presented as written, multiple-choice exams because their developers believed that human judges could not make objective assessments. But that's not true. The Educational Testing Service recently changed the format of the Scholastic Assessment Test (SAT) to include essay writing. The move was controversial, because it introduced a human element into the judging, presumably making scores less objective. But, in fact, research had shown that other essay-based standardized tests, such as some Advanced Placement exams, were in many cases better predictors of academic success than the SAT. That's because most university students are graded on essay exams; almost none of their grades are derived from multiple-choice testing. It turns out that the best way to predict how well people will write essays in the future is to test them using an essay format today. In other words, to most accurately predict someone's performance, you must closely mimic the context in which the individual will have to perform.

The same holds true in the office. Executives exchange information through conversations, questions are posed, and decisions are made on the fly. The most accurate predictor of business performance would have to imitate these dynamics, and human evaluators are far and away the best judges of such interactions.

Rather than concentrating on academic subjects, executive intelligence tests should focus on the particular cognitive subjects associated with executive work: accomplishing tasks, working with and through others, and judging oneself. The questions shouldn't require specific industry expertise or experience. Any knowledge they call for must be rudimentary and common to all executives. Only then can a hiring manager be assured that the disparities among job candidates are because of differences in their processing power, not in their knowledge. And the questions should not be designed to ask whether the candidate has a particular skill; they should be configured so that the candidate will have to demonstrate the skill in the course of answering the question.

Imagine you want to determine whether someone can critically examine underlying assumptions and can anticipate likely unintended consequences. Rather than ask the candidate to recount an occasion in which she did either of these things, you must present a fact-based situation in which she would need to apply such skills. An executive intelligence evaluation designed to test these abilities might look like this:

You are the CEO of a large software company. Your prices are being severely undercut by both domestic and foreign competitors. Your executive team recognizes a desperate need to cut costs. Your COO concludes that the answer is to outsource most of the company's programming to foreign subcontractors, thereby reducing labor costs. In fact, your COO has already received a number of bids from service firms in both India and South Korea. What questions do you have about his proposal?

A candidate displaying a high level of executive intelligence while answering this question would explain that the core assumption underlying the COO's conclusion needs to be confirmed—that is, outsourcing automatically equals cheaper production. She might point out that there may be indirect costs (up-front investment, ongoing customer service, and software development issues) involved with such a move that must be considered. Further, she would cite the probable unintended consequences of the COO's proposal, such as how using a distant workforce might affect productivity or labor relations.

Excerpted with permission from "Hiring for Smarts," Harvard Business Review, Vol. 83, No. 11, November 2005. and published online in HBS Working Knowledge

Justin Menkes (jmenkes@executiveintelligence.com) is a managing director of the Executive Intelligence Group, a New York-based consulting firm partnered with Spencer Stuart and focused on the assessment of executive talent. Menkes earned a Ph.D. in organizational behavior from Claremont Graduate University, where he studied with the late Peter Drucker.