A Look Inside Norming Assessments
Anyone who works in research in the behavioral sciences learns to appreciate the value of data. Psychological research, as it is practiced in North American universities and those who follow the same model, has been dubbed the science of first year university students. Most of the undergraduate and graduate studies, and probably many published articles relied on that easy to gather convenience sample that is those enrolled in Psychology 101. Most studies tend to require filling up some battery of questionnaire and relinquish part of your personal information (after signing a vaguely worded consent form).
That is not to say that psychological research is not valid. But behind any flashy headline such as: “The more (A) you do/have/eat/consume tend to produce/increase/decrease/eliminate (B) . One should always ask the question, what was really the research question and most importantly where did the data come from?
Working for a people analytics company has meant having access to quality data. To construct proper instruments to measure psychological constructs and assess human characteristics, targeted data must be collected. The Internet has made it faster to collect such data, particularly because the final product is already a digital tool. However, there are many caveats to using online data, and reputable sources must be vetted.
One of the traditional processes that has proven effective in the modern construction of psychological assessments and in collection of data for people analytics is the Pilot Study. The purpose of a pilot study is to test the feasibility of the instrument[1] and the quality of data gathering. As tempting as it is to rely on conclusions from a dataset of hundreds of cases, a pilot study is conceived and executed to check that the instrument works. Item development procedures are enhanced by testing them in a smaller sample size than the one that finally will be used to determine norms. Sometimes, software and delivery issues can be detected that could ruin the entire project. User experience evaluation also can be extracted from pilot studies.
A pilot study could also help to mirror what would be the final Norming Process. Norming refers to creating the transformations necessary to convert raw data into usable and interpretable information. A Norm is the parameter that will determine how the construct of interest is distributed in the target population. When a test administrator or a coach highlights a report showing that the individual being evaluated has a high level of Hardiness or is below the mean in Stress Management, those statements are based on parameters determined using a Norm Sample that describes the target population. In most of the products we develop at MHS Assessment, we thrive to get Census Matching to the U.S. population[2]. We collect data that will be representative in specific demographic characteristics (Ethnicity, Gender, Educational Level, Employment Status) to our target population.
We ensure that not only the respondents are valid, but that they also conform to the overall demographics of the people who would ultimately utilize our tools. That is not to say that the instrument cannot be used in other latitudes. Beyond the norm sample, and depending on market requests and financial feasibility, different comparative studies can be made, and have been made, to ensure that the interpretation of a report is the most accurate and appropriate. Translations are the first step, but ideally and with time, regional studies can corroborate the Norms or create adjustments as needed.
[2] https://www.census.gov/en.html
