systematicHR

The intersection between HR strategy and HR technology

, ,

Survey Design 101 – Part 2: Which questions should I keep or drop?

sjeong Avatar

Guest Author:  Stephen B. Jeong, Ph.D.

In Part 1 of Survey Design 101, we discussed two broad topics related to survey design – choosing the right topics and creating quality questions.  Survey design (or questionnaire development), however, is not complete until you can show that all or most of the redundant questions have been filtered out from the final set.  Moreover, this “redundancy” is often only visible through statistical analysis (i.e., factor analysis, discussed below) conducted after the data have been collected.  In other words, the initial draft of survey questions needs to be treated as just that – an initial draft. It’s only after the first data collection and subsequent revision that survey design can be said to be complete.  Beyond this, additional data should be used – collected on an annual or biennial basis – for continued refinement of the survey questions.

One common method used to refine survey questions is through factor analysis – a data reduction technique.

Factor analysis has been around for nearly a century (see Charles Spearman and intelligence testing); and although the mathematics involved – linear algebra – may seem intimidating, the concept is simple – it’s a technique used to reduce a large number of variables into a smaller set by examining the interrelationships among the variables. Fortunately for most of us, understanding how it can be used to improve the quality of our survey is all that’s necessary.

A key premise behind factor analysis is the idea that many can be reduced to few.  Imagine yourself in Munich for their annual Oktoberfest. You would undoubtedly see thousands people from all walks of life. Now, if I were to “group” these people based on some meaningful category – e.g., nationality, height, weight, or even the type of beer they are drinking – the resulting number of groups would be fewer than the thousands of individuals on which those groups are based. Factor analysis is very similar to this. Rather than people, however, we’re now talking about survey questions.

When you conduct a factor analysis on survey data collected from your employees, you’re asking the program “group” the survey questions in some conceptually meaningful way.  If you’re thinking to yourself that survey questions are already organized into meaningful groups or categories – e.g., training, benefits, supervision, and so on – you’re right.  In fact, if the survey was designed properly and the factor analysis done correctly, you may find that factor analysis results show a perfect match between your survey items and your survey categories.  Unfortunately, this will be rare. More often than not, you will find that a portion of the survey questions can be omitted, re-categorized or refined.

Bottom line here is that when it comes to employee surveys, factor analysis is an important tool that can be used to help answer the question – Which questions should I keep or drop?  It is an important step that will help to clarify the conclusions drawn from results of other advanced analyses typically conducted on survey data.

Stephen B. Jeong, is currently the Managing Director of Waypoint People Solutions – www.waypointps.com, a human capital consulting firm that focuses on high precision employee diagnostic surveys using cutting-edge measurement technology and methodologies. He holds Ph.D. in Industrial-Organizational psychology from the Ohio State University and has been advising private, public, and government organizations since 2000.  He can be reached at stephen.jeong@waypointps.com.

sjeong Avatar

3 responses to “Survey Design 101 – Part 2: Which questions should I keep or drop?”

  1. Mark Cruijff Avatar
    Mark Cruijff

    The problem with factor analysis is that you first need to perform the survey (in a pilot group?) as you need the results of the survey to do the factor analysis. If the survey is one that is repeated on a regular basis this will work. However, if it’s a survey you will run once it’s probably better to rely on your survey design experience/knowledge/common sense to prevent unnecessary questions and create adequate groupings of questions.

  2. Stephen B. Jeong Avatar

    Excellent point Mark. First year data collection is a prerequisite for taking advantage of factor analysis. My experience has been that a one-time survey reveals very little useful information about an organization and the information that does stand out tend to somewhat unreliable. When I think of engagement surveys, the thing that comes to mind is a brand new pair of leather boots – the real value comes after you wear it a few times. Similarly, the value of annual engagement surveys increases each year that you’re able to compare back to the previous. Thanks for pointing that out Mark.

  3. […] Survey Design 101 – Part 2: Which questions should I keep or drop? Systematic HR Thursday, January 28, 2010 Moreover, this “redundancy” is often only visible through statistical analysis (i.e., Jeong, is currently the Managing Director of Waypoint People Solutions – http://www.waypointps.com , a human capital consulting firm that focuses on high precision employee diagnostic surveys using cutting-edge measurement technology and methodologies. READ MORE […]