Home
Doc LobeTeaching, Learning & Research Firm
Research · Assessment development

Assessment Research: Validity and Reliability

How Doc Lobe develops, tests and norms its four research instruments: the Roots Personality Type Assessment, Love in Color, the Paths in Color Career Assessment and the Cultural Belonging and Racial Identity Survey (CBRIS). This page has the instrument matrices, permission to use, item blueprints, an expert review form, and reliability and norms as responses come in.

Content validity (I-CVI, S-CVI)Cronbach's alphaItem analysisExpert review formNormsPermission to use
Instruments and permission

Using a Doc Lobe instrument

All four instruments are original works by Micheal McGee. Educators, counselors and researchers may use them only with the author's permission. Each instrument matrix has the theoretical blueprint, every item, a short form, scoring and the validation plan.

Conditions of use
  1. Contact the author for permission before using any instrument with students, clients or study participants. Use the request form below or email doclobe23@gmail.com.
  2. Participants must take the instrument on doclobe.com and choose "For me and for Doc Lobe research." This adds their anonymous answers to the norming study that every Doc Lobe instrument follows. Do not copy the items into another survey platform.
  3. Use the study code Doc Lobe gives you. Share the link with your code (for example, doclobe.com/cbris.html?code=YOURCODE) so your group's anonymous summary results can be sent to you.
  4. Cite the author in any presentation, report, syllabus, thesis or publication, using the citation on each card.
  5. No reliable internet? Use the Offline Collection Station on a shared device and upload the exported file later. On doclobe.com, answers from a dropped connection are saved and sent automatically when the device reconnects.
  6. Obtain IRB or ethics approval for research. The instruments are research versions for education and research, not clinical or diagnostic tools.

Request permission to use an instrument

Which instrument(s)?

For researchers

Your data, ready for SPSS, SAS, R and Stata

Researchers with a study code receive a data package for their group. It works in any statistics program, and the scores it computes match the scores participants see on doclobe.com.

Doc Lobe calculates

Built-in results

  • Scale means and standard deviations for your group
  • Cronbach's alpha and item analysis (item–rest r, alpha if deleted)
  • Content validity indexes from the expert panel
  • Norms and percentile comparisons once 200 people share
In every package

Data, codebook and code

  • Anonymous item-level data (CSV) with background answers and scale scores
  • A codebook with every item, label and scoring rule
  • SPSS syntax (.sps), SAS code (.sas), an R script (.R) and a Stata do-file (.do)
  • A README with steps and the citation
The code runs

Analyses included

  • Reverse scoring and scale scores
  • Alpha with item-total statistics; omega in R
  • Descriptives and scale correlations
  • Exploratory factor analysis; confirmatory factor analysis and invariance in R (lavaan)
  • Group comparisons by any background variable
Data cleaning

Careless responses screened

  • Duplicate submissions
  • Speeding: under 2 seconds per item
  • Straight-lining and long strings of the same answer
  • Inconsistent answers (opposite poles, even-odd consistency)
  • Test codes such as TEST and DEMO

Flags travel with the data; the code analyzes clean responses by default.

Other programs

JASP, jamovi, Excel, Mplus

Open the CSV directly. JASP and jamovi run reliability and factor analysis from menus; flag the reverse-scored items listed in the codebook.

Try a sample package

These packages use 200 rows of simulated practice data, so you can test the code in your program before your study begins. They are not real results.

Offline collection

Upload answers collected without internet

Sites with little or no internet collect answers on the Offline Collection Station and export one file. Upload it here from any connection to add the anonymous answers to the Doc Lobe norming study, or email it to doclobe23@gmail.com.

Files are read on this device. Answers that were already sent from this browser are skipped, so a file can be uploaded again safely.

Qualitative research

Written answers: patterns, codes and Dedoose export

Each assessment ends with 2–3 optional open-ended questions. Load responses above (Netlify CSV or offline file), or paste your own interview or focus group text. Doc Lobe shows first-pass patterns and helps you tag answers with a codebook; people then build the themes. Everything runs on this device.

Load responses in the Reliability section or the offline upload above to begin.

Frequent words

Share of answers that use the word. Tap a word to read it in context.

Key phrases

Two- and three-word phrases used in at least two answers.

Codebook

One code per line: Code name: keyword, keyword*, phrase. A star matches word endings (mentor* matches mentors and mentoring). Starter codes come from the theory behind each assessment; rename, merge and add codes as you read.

Answers and tags

Tap a code under an answer to add or remove it. Your changes are included in the export.

Research plan

From draft items to a validated assessment

Step 1 · now

Content validity

A panel of 6–10 experts, such as psychologists, counselors, educators and community members from the communities served, rates every item for relevance and cultural responsiveness. Items need an I-CVI of at least .78, and each scale needs an S-CVI/Ave of at least .90 (Polit & Beck, 2006).

Step 2 · now

Pilot and reliability

As people take the assessments and choose to share anonymous answers, Doc Lobe computes Cronbach's alpha and item statistics for each scale. A target of α ≥ .70 is common for new research scales. About 200 responses or more give stable estimates.

Step 3 · next

Structure and fairness

With 300 or more responses, factor analysis tests whether items group as intended, omega supplements alpha, and measurement invariance testing checks that the scales work the same way across cultural groups (Putnick & Bornstein, 2016).

Ethics

Consent and IRB

Sharing is opt-in, limited to adults 18 and older, and anonymous. Before results are used in a thesis, dissertation or publication, the study should be reviewed by an Institutional Review Board.

Content validity

Item blueprints and preliminary checks

Content validity starts with a blueprint showing that every construct is covered by enough well-written items, then expert judgment. The checks below are computed from the items themselves.

Blueprint

Expert panel results

For Doc Lobe: load expert review files (CSV) to compute I-CVI, S-CVI and Lawshe's CVR for the current assessment. Files stay on this device.
Expert review form

Rate the items (for invited panelists)

For each item, rate relevance (how well it represents its construct) and cultural responsiveness (how well it reflects the experiences of people of color without stereotyping): 1 = not, 2 = somewhat, 3 = quite, 4 = highly. Add a comment or suggested rewording if you like.

Reliability

Cronbach's alpha and item analysis

For Doc Lobe: load the CSV export of the roots-research, love-research, career-research or cbris-research form from Netlify (Forms → the form → Download as CSV) to compute alpha, norms and study-code reports. The file stays on this device. Then use Download published results to update the site's norms.
Sources

References

  1. Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006
  2. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
  3. Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393
  4. Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8
  5. Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39(1), 103–129. https://doi.org/10.1016/j.jrp.2004.09.009
  6. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
  7. Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
  8. McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
  9. Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.
  10. Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467.
  11. Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
  12. Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the critical values for Lawshe's content validity ratio. Measurement and Evaluation in Counseling and Development, 45(3), 197–210.