This blog post is adapted from an article published in the Summer 2019 edition of the British Psychological Society’s Assessment & Development Matters journal. It can be accessed here.
Back in 2016, we conducted research with 90 participants competing in the latter stages of an Air Traffic Controller selection process. We used the Risk Type Compass to explore deeply-rooted risk dispositions and identify any recurring personality characteristics in the sample.
Our analysis indicated a proportion of Deliberate Risk Types that differed dramatically from that of the wider population and, equally unexpectedly, a complete absence of Excitable Risk Types. The original piece was published in the Autumn 2016 edition of Assessment & Development Matters and on our blog.
Since our initial analysis, we’ve been able to collect additional findings on a further 131 individuals completing the same recruitment process.
Given the nature of the job and the difficulty of the recruitment process, we began our analysis with a couple of questions:
❓ Will our second Air Traffic Controller sample reflect a similar Risk Type distribution to the first?
❓ How do these second findings compare with our General Population sample of over 13,500 participants?
As the above bar graph indicates, the answer to these questions was a resounding yes.
As before, Deliberate Risk Types were several times more likely to occur in the Air Traffic Controller sample. Additionally, Excitable Risk Types were completely absent from the combined sample of 221 Air Traffic Controllers.
These findings are clear and emphatic evidence of Reproducibility.
What is Reproducibility?
Reproducibility is an essential cornerstone of the scientific method and has become a hot topic in psychology in recent years, as efforts to reproduce the findings of published academic research have yielded very mixed results – see Open Science Collaboration (2015) for details. In reproducibility studies, researchers aim to replicate previous studies using comparable methodological frameworks and measures. Failing to do so casts doubt on the conclusions and utility of past research.
So, what drives such impressive reproducibility in our Risk Type Compass research?
In a word – RELIABILITY.
What is Reliability?
In the realm of psychometrics, reliability acts as an umbrella term for several important metrics. Arnold et al (2016) define it as follows:
An indicator of the consistency which a test or procedure provides. It is possible to quantify reliability to indicate the extent to which a measure is free from error.
The reliability coefficients of the Risk Type Compass are included in the technical manual and our research outputs (e.g. our latest test-retest findings).
The reliability of the Risk Type Compass was also independently reviewed by the British Psychological Society’s (BPS) Psychological Testing Centre. A summary of the review is available here. Several facets of reliability were assessed by the BPS Test Review:
– Internal Consistency: a measure of scale reliability determined by inter-item correlations.
– Test Retest/Temporal Stability: correlations between two scale scores obtained from the same individual at two different points in time.
– Equivalence Reliability: comparison between the short form and standard form of the Risk Type Compass.
– Split Half Analysis: comparison between the first half and second half of the Risk Type Compass.
The BPS Test Review uses a four-star rating system for each facet described above. The independent reviewers awarded the Risk Type Compass a total of 15 stars from a possible 16 for the reported coefficient strengths across the four criteria. This is an exceedingly high rating.
Conclusion
The Risk Type Compass has provided considerable evidence for its ability to provide consistent results across multiple dimensions of reliability. It also differentiates very effectively, with the findings above representing one of many examples.
In conclusion, our Risk Type Compass research provides strong evidence for the proficient measurement characteristics of the Risk Type Compass, for its effectiveness and reliability in differentiating individuals and for its utility and in relation to the world of work and across a wide range of applications.
The Author
Dr Simon Toms is a Principal Research Psychologist with Psychological Consultancy Ltd, and a Chartered Psychologist and Associate Fellow of the British Psychological Society.
References
Arnold, J., Randall, R., Patterson, F., Silvester, J., Robertson, I., Cooper, C. L., Burnes, B., Harris, D., & Axtell, C. (2016). Work Psychology: Understanding Human Behaviour in the Workplace, 6th ed. Harlow: Pearson.
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349(6251), retrieved November 8, 2018, from here.