Quantifying the intangible: Determining the performance of knowledge workers

Measuring the performance of workers on an assembly line is simple: Count the objects produced and find out how long the process took. That should reveal the productivity of the factory workers. But how do you determine the performance of knowledge workers? Observing and measuring what they do is an enormous challenge. They often work unscripted and alone. The workers themselves may be unaware of the individual steps they take when carrying out an assignment. Professor Robert St. Louis and Assistant Professor Michael Lee of the W. P. Carey School's Department of Information Systems and Robyn Raschke of the University of Nevada, Las Vegas, are tackling this issue in an unusual study that could hit close to home for the researchers.

Measuring the performance of workers on an assembly line is simple: Count the objects produced and find out how long the process took. That should reveal the productivity of the factory workers. But how do you determine the performance of knowledge workers?

Observing and measuring what they do is an enormous challenge. They often work unscripted and alone. The workers themselves may be unaware of the individual steps they take when carrying out an assignment. Assessing the output of knowledge workers also can be daunting. Knowledge comes in many forms, some of which are not very tangible. Knowledge can be impossible to quantify.

Professor Robert St. Louis and Assistant Professor Michael Lee of the W. P. Carey School's Department of Information Systems and Robyn Raschke of the University of Nevada, Las Vegas, are tackling this issue in an unusual study that could hit close to home for the researchers.

They are in the midst of surveying more than 3000 faculty members at business schools across the United States in hopes of determining how institutions assess the performance of these academics and what the consequences are for job satisfaction. "We want to understand how the design of measurement systems leads to employee satisfaction or dissatisfaction, which, in turn, can lead to a good or not so good institution to work at," Lee says.

Business schools looking inward

The researchers began with a pilot survey of faculty at their own institutions — the W. P. Carey School and University of Nevada, Las Vegas — and now are expanding the study to include a cross-section of US business school faculty. The survey will be administered anonymously. St. Louis expects faculty members and administrators to welcome this critical look at management practices in academia.

"I've been on many personnel committees, and I think people really are genuinely dissatisfied with the measures that we're using to evaluate academics," St. Louis says. The survey will try to ascertain whether business school faculty are assessed using standard academic outcome measures — such as number of articles published and scores on student evaluations — or whether some other standard is used.

The researchers want to find out if business schools with different orientations use different management approaches. The investigators plan to separate the survey results into two groups, one for faculty who are at research-led institutions and the other for faculty at teaching-led schools. Measures that are appropriate for one group may not be appropriate for the other, the researchers suspect.

Agency theory

Most large organizations, including academic institutions and businesses, must deal with what is known in economics and political science as the "principal-agent problem." One party, the principal, delegates work to another party, the agent. Typically, the interests of principal and agent conflict, and the principal must find ways to discover what the agent is doing.

Management experts have identified three ways for principals to direct and measure the performance of agents: outcome controls, behavior controls, and social or clan controls. "Outcome controls come from the old factory model, in which there are easily measurable things that you can use to evaluate assembly workers," Lee says. "But when you are a knowledge worker in a research institution, it is not so easy to measure outputs."

Behavior controls involve monitoring the processes that take place when work is accomplished, then assessing performance in each of the steps in the process. Again, this approach runs into difficulties when applied to knowledge workers.

"You can measure every single thing a person does, but it's intrusive, time-consuming and costly," St. Louis says. "It would be much better if you could figure out some other way to get people to carry out the mission of the organization." St. Louis believes social or clan controls may be the best way to manage knowledge workers.

He says this means developing an organizational culture that knowledge workers can buy into and also hiring workers who are a good fit. "You have to be really careful about the hiring process," St. Louis says. "Some companies do that. At Google, it's a six-month period before somebody actually gets hired. The same is true for many of the really high-powered consulting companies. They know it's going to be really tough to monitor people based on either outcome or behavioral measures."

Managing a school of management

While St. Louis and Lee will await the data from their survey before reaching any conclusions, they suspect that academic institutions generally follow the outcome-based approaches that arose from the factory model. "Ultimately it's the outcome that counts in a lot of institutions," Lee says.

"How many research articles have you published? What are your scores on student evaluations? Where do your students go after they graduate? What's proportion of students get jobs? These are all outcome measures." Outcome measures can be deceptive, according to Lee.

"You don?t have to be the best researcher to publish high quality articles if you work with good researchers. You don't have to be a quality teacher to get a good evaluation — you can incentivize students in many ways to give you good evaluations You can game the outcome measure because it is only the outcome that counts, not how the outcome is achieved."

For teaching-led institutions, behavioral measures may be a more effective way to assess faculty and at least should be included in the mix of evaluation methods, according to Lee. "You could measure time spent with students outside of class or time getting your courses developed and up to date," Lee says. "You could measure time spent thinking about the different types of students in your class and how you could use different teaching methods to get your message across. These are all behavioral measures."

How to measure research

In research-led schools, outcome measures are the main way faculty performance is assessed, according to Lee. "Outcome measures are easily observable in research," he says. "Faculty are asked how many tier one journal publications did you produce this year? How many tier two journal publications? How many research presentations did you do?"

Lee believes a mix of behavioral and outcome measures would be more appropriate in a research-led institution. "You could look at attending research methods classes or on learning from expert researchers. These are behavioral measures, which are harder to observe but are an important ingredient in research quality," Lee says.

One of the main goals of the study is to find out whether academics are happy with the job performance measures used in their institutions. Job satisfaction among employees is a key ingredient for success in an organization, according to the researchers.

"We could be wrong in our assumptions," St. Louis says. "Maybe people are actually thrilled with outcome measures. That would mean we would need to re-examine the theory. The theory says this is not the way you ought to be managing knowledge workers."

Lessons beyond academia

Lee and St. Louis say they hope their study will prompt business schools to do some introspection. If academic institutions are going to advise others about best management practices, then these schools should make sure they are being run in an effective manner themselves, according to the researchers.

"The essence of this is to get us to really think about what we're preaching to the world and how we can apply it ourselves," Lee says. "A university should really be a model organization for everyone." While this study is on faculty in business schools, it should have relevance for other knowledge workers, according to St. Louis and Lee.

"I think what we will find by looking at two distinct types of academic institutions — research-led and teaching-led — is that we need to design systems that take into account all of the different cultures and strategies we find in organizations," Lee says.

"We are getting more and more knowledge workers and fewer and fewer assembly line workers," St. Louis says. "We need different types of systems to evaluate knowledge workers, and this is true not just in academia but in industry. We've got to get this right if in fact we are going to have effectively run organizations."