Scientific Validation of VoiceSignals: Part 1
.svg)
Introduction
Psychological assessments have been used for over a century to predict key behavioral outcomes across various situations. For example, personality assessments have been shown to predict which candidate will perform best in a job[1], which applicant will default on a loan[2], and which shopper will purchase a product or service[3]. Forecasting these behaviors provides organizations with more reliable data, improving strategic decision-making and, ultimately, financial performance.
However, such impressive outcomes can only be realized if the psychological assessment that an organization uses is sufficiently accurate. In technical terms, an accurate assessment is known as a valid assessment, and standards exist within the field of psychological science for evaluating the validity of an assessment. This 3-part series contains the validity evidence for the psychological assessments embedded in VoiceSignals' People Intelligence Platform© (PIP).
1. Measure Development
The first step in the development of the PIP© was to create accurate measures of the target psychological characteristics. A review of the relevant literature in the psychological sciences was conducted to identify the psychological characteristics of interest.
The Psychological Characteristics
The design objective for the PIP© was to develop a tool that helps organizations achieve business success by forecasting and influencing key revenue-driving behaviors among their employees and customers. Therefore, an investigation into the behavioral sciences was conducted to identify the characteristics of individuals that best predict their future behavior across a wide variety of situations (e.g., purchasing habits, loan payments, job performance). It was identified that personality traits and emotional states are two psychological characteristics that reliably predict a wide variety of behaviors in both the long term (i.e., personality traits) and the short term (i.e., emotional states).
The following sections describe the development of the personality trait and emotional state measures.
Personality Traits
There is a large consensus in the psychological literature around a model of personality that proposes five broad factors that explain the majority of differences between people's behavioral styles. This model is known as the Big Five model or Five-Factor Model (FFM). One of the most frequently used measures of the Big Five is the NEO-PI-R which identifies six sub-factors, or "facets," under each of the Big Five dimensions, resulting in 30 facets (see Table 2). Due to the scientific evidence supporting five broad factors and thirty sub facets of personality, we decided to use this structure as the basis for our personality framework. This framework was used to create definitions and rating scales with descriptive anchors for each of the thirty facets (see Table 3 for an example). These definitions and rating scales were worded so that they could be used to rate a speaker on the thirty personality facets by hearing their voice alone.
Emotional States
Unlike the body of academic literature on personality, research on emotions has yet to reach a consensus on the general structure of human emotions. There is a consensus that between six and ten distinct basic-level emotions exist. After reviewing the existing emotional frameworks, Plutchik's Wheel of Emotions was selected due to its wide use in clinical settings and its proposition of the existence of eight basic emotions (see Figure 1), which fits within the consensus mentioned above.
The Wheel of Emotions identifies three levels of intensity for each of the basic emotions and eight more complex emotions that are combinations of the basic emotions (for example, love is a combination of joy and trust). A measure of emotional states was developed from this framework that observers could use to rate the intensity of emotions present in a speaker. This measure contains sixteen emotion variables, each with three intensity levels, following the framework proposed by Plutchik. An example of the description and rating scale for one of the emotions in this measure is shown in Table 4.
Mindset
In addition to developing a measure of the eight basic and eight complex emotions mentioned above, measures of a speaker's general attitude and state of well-being were also developed. These two measures are grouped under the category of "Mindset" as they are both reflections of an individual's current state of mind.
Measure Review and Revision
After the measures were developed, they were reviewed by a group of academically and professionally trained psychologists. Minor revisions to wording resulted from the reviewer's feedback. As discussed further in Section 9, this provided preliminary evidence of content validity, given that the measures were developed and reviewed by trained psychologists and were based on previously validated frameworks.
2. Data Collection
The first step in the development of the PIP© was to create accurate measures of the target psychological characteristics. A review of the relevant literature in the psychological sciences was conducted to identify the psychological characteristics of interest.
The Psychological Characteristics
This step involved the collection of audio files containing speech data and served two purposes. The first purpose was to investigate the reliability of the new measures described in Section 1. The second purpose was to develop high-quality labeled and rated speech data for the AI algorithms to learn from.
Summary
This blog series details the development and validation of VoiceSignals' PIP©. The PIP© has been designed and developed by experts in AI engineering and psychological assessment and validated according to best practices in psychological science. The validation evidence presented in this series shows that the algorithms embedded in the PIP© are valid predictors of important psychological characteristics and can be applied in real-world settings. The PIP© has been developed to continue to improve its predictions as it learns from more data, and therefore the validation evidence will become increasingly more vital with time. Accordingly, the validity evidence contained in this series will be updated periodically.
[1] Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: a meta‐analysis. Personnel psychology, 44(1), 1-26.
[2] Karlan, D., Mullainathan, S., & Robles, O. (2012). Measuring personality traits and predicting loan default with experiments and surveys. Banking the world: Empirical foundations of financial inclusion, 393-410.
[3] Kassarjian, H. H. (1971). Personality and consumer behavior: A review. Journal of marketing Research, 8(4), 409-418.
[4] American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC.
[5] Costa Jr, P. T., & McCrae, R. R. (1995). Domains and facets: Hierarchical personality assessment using the Revised NEO Personality Inventory. Journal of personality assessment, 64(1), 21-50.
[6] Weidman, A. C., Steckler, C. M., & Tracy, J. L. (2017). The jingle and jangle of emotion assessment: Imprecise measurement, casual scale usage, and conceptual fuzziness in emotion research. Emotion, 17(2), 267.
[7] Plutchik, R. (2001). The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American scientist, 89(4), 344-350.
[8] Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2), 155-163.
[9] Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019). Speech emotion recognition using deep learning techniques: A review. IEEE Access, 7, 117327-117345.
[10] Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd Edn. New York, NY: Academic Press
James is an Industrial/Organizational Psychologist and a demonstrated industry leader in the applied use of Artificial Intelligence (AI) and Machine Learning to enhance psychological assessment. He has a track record of developing innovative AI-based psychological assessments and his work has frequently been presented at industry conferences. James is a true scientist-practitioner and leverages his broad experience as both a research scientist and consultant to provide world-class solutions for customer needs.
%201.svg)
