Research & Development of our Assessment

The Assessment: Brief Research Overview

The Assessment is a carefully constructed assessment instrument designed to help individuals understand their personality and reflect on important aspects of their work and personal lives. It is administered online and takes ~40 minutes to complete. On a seven-point scale, individuals rate the extent to which a set of descriptive items (e.g. “I am very disorganized”) applies to them. The output comprises a person’s score on each of 12 major traits, 36 sub-traits (called facets), and 5 independent scales. A person’s score is also reported on 28 Archetypes derived from an algorithm based on facet scores. Finally, an extensive narrative report provides details on the meaning of the person’s scores on traits, facets, archetypes and additional outputs currently under development. These narratives are unique to our Assessment.

The Assessment has several distinctive features that differentiate it from other personality instruments. First, it is based on contemporary developments in personality and organizational research, including that carried out by Professors Brian R. Little and Adam Grant, who are a core part of our team. Second, it utilizes advanced psychometric techniques, including novel algorithms for creating profiles and Archetypes. Third, the Assessment is unique in measuring traits that Ray Dalio has identified over his decades of running a successful business, including his concept of Shaper personality.

Well-designed measures of personality traits can predict highly consequential outcomes. By combining personality data with real-life performance data from within Ray Dalio’s company, Bridgewater Associates, the traits measured by the Assessment have been shown to have high internal consistency, retest reliability, and validity. They compare favorably to other personality assessments in predicting job performance and well-being.

Research Base of the Assessment: Creation of the Assessment Item Pool

The Assessment is based in part on the Big Five personality assessment, an extensively researched and respected perspective within personality and organizational research. This framework was augmented by insights from our own personality research and Dalio’s business experience.​

The first step in creating the Assessment was to create an item pool of several hundred short questions such as “I enjoy parties,” and “I seldom procrastinate.” These questions/items were modeled on (and sometimes drawn directly from) existing public domain resources, primarily the International Personality Item Pool (IPIP). IPIP contains an open-source data bank of over 3,000 items that are organized into scales to measure dimensions of personality (e.g. “I enjoy parties” is one item that would, together with several other similar items, comprise the Big Five trait of Extraversion; “I seldom procrastinate” would be one item on a multi-item scale to measure the Big Five Trait of Conscientiousness.)

We also included several dozen items of our own invention, tapping traits beyond the Big Five that we judged to be relevant to organizational goals—including some uniquely identified with our own research, such as whether a person is a “Giver or Taker” or is “Person-Oriented” and items drawn from the Personal Projects Q-Deck procedure. We also created new items that drew from Dalio’s ​frameworks—designed to tap into traits such as Humility and Toughness. Many of these are unique to the Assessment. The Assessment Item Pool currently comprises ~600 items and is continually being revised and expanded. 246 are currently used within the Assessment.

Scale Development Strategy: An Iterative Process

During Scale Development we created Dimensional Scales (e.g. Extraversion) from groups of items that are hypothesized to tap into that dimension (e.g. enjoy parties). The procedure we used to construct the dimensions was a combination of rational and psychometric analysis.

Rational Analysis:​ We initially created Dimensional Scales that represented each of the Big Five dimensions of personality: Openness, Conscientiousness, Extraversion, Agreeableness, and Stability, as well as a variety of other trait and facet concepts that were of interest to the parties involved (e.g. humility, leadership orientation, etc.). When possible, these items were taken from the IPIP item pool and adopted based on the extensive pre-existing research behind these items. They have been found to be both reliable and valid indicators of basic personality traits and are established predictors of consequential outcomes.

In an extensive iterative process extending over 16 months, consensus was reached between team members on the rational alignment of items with Big Five traits and several others. Typically, 7-10 items were selected from the Assessment item pool to characterize each of the traits.

Psychometric Analysis​: We performed detailed statistical analyses on the rational scales to ensure the highest psychometric standards. The online test was administered to two large Mechanical Turk samples (our primary study from which our reliabilities and norm group are drawn had an N > 3,000 individuals of mixed age, gender, and race), and care was taken to screen out those who may have not been paying attention to their answers. The test was also administered to an internal Bridgewater sample. 

We used Factor Analytic, Scaling and Network Analysis programs in R, SPSS, JAMOVI, and JASP to examine the internal consistency, construct validity, and test-retest reliability and factor structure of the traits and facets. The results were encouraging. Measures of internal consistency (alpha and omega) for traits were generally excellent. Trait alphas averaged .85 and trait omegas averaged .88. Due to the scales being shorter, alphas and omegas for facets are predictably lower than for traits, but we still see strong results.  Facet alphas averaged .74 and omegas averaged .82. These values are on par with those from the NEO-PIR, the gold standard for personality assessment; these Big Five trait scales, as reported in their Manual, had average alphas for traits of .88 and facets of .71.

Our strategy emphasized the creation of traits and facets with high levels of internal consistency. While this is a useful measure, we augmented it with data on test-retest reliability. An MTurk sample of participants were retested after a two-week period. The retest reliabilities were excellent. The mean retest reliability for traits was .92 and for Facets it was .86. These compare very well with retest reliabilities for other research-based personality tests. 

Construct validity was assessed by examining correlations between the trait dimensions and a set of outcome measures related to well-being and rated job effectiveness. There were strong linkages with different measures of well-being (R = .53) and moderate linkages with job performance evaluations (R =.30). The results are consistent with contemporary research in personality. Well-being is best predicted from scales tapping into Stability (Composed) and Extraversion. Conscientiousness (Detailed and Reliable) was a strong predictor of job effectiveness. 

We also examined correlations between trait dimensions and performance data within Bridgewater Associates, which showed moderate to strong linkages with internal ratings on related competencies (e.g. internal ratings on Creativity, Holding People Accountable, Organization, Shaping Change, etc…). R ranged from .27-.61, with an average of .4. These results compared favorably with other research-based assessments in predicting performance in the job.

We performed factor analyses with both items and facets and they partially confirmed the rational grouping of items into 12 trait dimensions, although there was moderate overlap between some of the facets. Based on these analyses, 12 Trait dimensions each with 3 facets were chosen. An additional set of 5 dimensions (without facets) were also created for a Total of 12 Traits, 36 Facets, and 5 Independent Dimensions. We also include a Validity/Social Desirability scale that we score for everyone, but don’t report automatically in our narrative profiles for all retail users. The creation of the facet structure is unique to our instrument with many of the facets representing dimensions not assessed by other personality tests.

Throughout scale development there was an extensive iterative exchange among the development team to ensure that we were in sync on creating an instrument that was both psychometrically rigorous and relevant to the distinctive needs of understanding personality in an organizational context.

Archetype Analysis: A Novel Approach to Clustering Individuals

Simultaneously with the development and refinement of the Assessment Traits and Facets, we created an algorithm for analyzing each participant’s status on a set of 28 Archetypes including Technician, Strategist, Shaper, Quiet Leader, Protector, Promoter, Implementer, Problem Solver, Campaigner, Planner, Thinker, Orchestrator, Investigator, Inventor, Inspirer, Impresario, Helper, Growth Seeker, Explorer, Entertainer, Enforcer, Individualist, Peacekeeper, Critic, Commander, Coach, Artisan, and Adventurer. We adopted a novel Q-methodology procedure that weighted each facet in terms of the positive and negative features of each Archetype (e.g. for the Helper Archetype high positive weights were given to the Empathy facet and low weights to the Tough facet). This creates a vector score for each individual on each of the 28 Archetypes. Again, both the algorithm and thematic descriptions of the Archetypes are distinctive to our assessment.

We look forward to continuing to develop and improve the test over time. For personality researchers with more detailed questions about our methodology, please address questions to support@principles.com, and we will do our best to provide additional information.