Scientific Validation of VoiceSignals: Part 2

Part 2 of 3: Our Rating Process, Development, and Agreement

01. Rater Selection

A crucial step in developing the PIP© was creating a group of experts in psychological assessment who could provide accurate ratings of the psychological characteristics identified in (part 1 of our series). Job descriptions for this role were developed, and applicants were assessed based on their academic credentials and experience with psychological assessment. This group of raters has a combined 190 years of experience in psychological assessment and represents a diversity of races, age groups, and gender. This diversity in rater demographics further helps to mitigate bias in the AI algorithms.

02. Rating Application Development

A propriety application was developed to make the rating process highly effective and efficient. This rating application, named the Rating Application for Traits and Emotions (RATE)©, contains the measures and presents the speech files to the raters. The cohort of raters then rated each speaker in a speech file. This process is highly automated and enables a high quantity of speech files to be rated quickly. The combination of transparent and standardized measures and expert raters also resulted in high-quality labels and ratings of the psychological characteristics of each speaker.

03. Rating Process

Based on an extensive review of the literature, each of the raters received initial training on the RATE application and best practices for perceived (versus self-report) psychological assessment. Rater productivity and performance were monitored using a standardized report. Indicators of performance were a series of reliability, agreement, and rating error metrics. Assessors received ongoing feedback, training, and oversight.

04. Rater Agreement Evidence

When using multiple raters to rate an individual on a psychological characteristic, such as a person's degree of extraversion, it is essential to assess the level of agreement among those raters. In developing the PIP©, the examination of rater agreement served two purposes. First, the results from rater agreement analyses revealed information regarding the quality of the new measures (described in Part 1).

Second, speech files with high rater agreement can be considered high-quality and used to train the AI algorithms. Two different metrics of rater agreement were used depending on the psychological characteristic in question.

Emotional States

For emotions, which have a binary "present or not present" rating, an agreement threshold of a substantial majority of expert raters endorsing that a specific emotion was present in any 5-second segment of speech was used.

Personality Traits

For personality traits, which have a continuous instead of binary distribution, the Intraclass Correlation Coefficient (ICC) was used to assess rater agreement. The ICC shows the degree to which raters agree on a scale from zero to one, with higher scores representing higher agreement. There are different versions of ICC, and researchers are guided to select the metric best suited to their specific use case. For these analyses, ICC3k was used, which provides the level of agreement among a fixed set of raters when the mean of the rater's ratings is the value of interest.

Best practice guidelines recommend the following interpretation of ICC values[8]:

• ICC values greater than .90 indicate excellent rater agreement
• ICC values between .75 and .90 indicate good rater agreement
• ICC values between .50 and .75 indicate moderate rater agreement
• ICC values less than .50 indicate poor rater agreement

The average ICC3k value for the personality traits showed a high degree of agreement.

Mindset

The ICC measure of rater agreement was also used for the Mindset psychological characteristics as they also have a continuous rather than binary distribution of ratings. The average ICC3k value for the Mindset characteristics indicates excellent rater agreement.

Read Part 3 Here

Download the full validation study here.

Summary

This paper details the development and validation of VoiceSignals' PIP©. The PIP© has been designed and developed by experts in AI engineering and psychological assessment and validated according to best practices in psychological science. The validation evidence presented in this paper shows that the algorithms embedded in the PIP© are valid predictors of important psychological characteristics and can be applied in real-world settings. The PIP© has been developed to continue to improve its predictions as it learns from more data, and therefore the validation evidence will become increasingly more vital with time. Accordingly, the validity evidence contained in this paper will be updated periodically.