Home
Doc LobeTeaching, Learning & Research Firm
Research Methods Lab · Measurement

Psychometrics Lab

Measure what matters, and measure it well. Learn how psychological tests are built and checked, try a reliability calculator with real numbers, and browse a library of free and open instruments for research in psychology, counseling, mental health and education.

Reliability and validityCronbach's alpha calculator60+ open instrumentsCulture and fairnessLesson plan
Key ideas

What makes a measure trustworthy

Psychometrics is the science of measuring psychological qualities such as anxiety, belonging or conscientiousness. Because these cannot be seen directly, researchers build instruments and then gather evidence that the scores are consistent and mean what they claim to mean.

Step 1

Constructs and operational definitions

A construct is the idea being measured, such as "academic stress." An operational definition states exactly how it will be measured, such as total scores on the Perceived Stress Scale. A clear construct definition guides every later choice.

Consistency

Reliability

  • Internal consistency: do the items hang together? (Cronbach's alpha, McDonald's omega)
  • Test-retest: are scores stable over time when the trait should be stable?
  • Inter-rater: do two trained observers agree? (Cohen's kappa, ICC)

Reliability is a property of scores in a particular sample, so report it for your own data.

Meaning

Validity

  • Content: do the items cover the whole construct?
  • Construct: do scores relate to similar measures (convergent) and not to unrelated ones (discriminant)? Does the factor structure fit?
  • Criterion: do scores predict a real outcome now (concurrent) or later (predictive)?

Validity is an argument built from many kinds of evidence (Messick, 1995).

Scores

Scoring, norms and cutoffs

Follow the developer's scoring rules exactly, including reverse-scored items (on a 1–5 scale, a reversed 2 becomes 4). Norms compare a score with a reference group; cutoffs flag scores worth a closer look. A cutoff from one population may not fit another.

Fairness

Measurement across cultures

A test can work differently across languages and cultures. Good practice includes careful translation and back-translation with community review (International Test Commission, 2017), checking measurement invariance before comparing group means, and looking for items with differential item functioning. Also ask whether the construct itself means the same thing in each culture.

Ethics

Using tests responsibly

  • A screening score is not a diagnosis; it signals who may need a full evaluation.
  • Get informed consent, protect confidentiality and store data securely.
  • Use instruments within your training; some require a licensed professional.
  • Have a referral and safety plan before asking about suicide, trauma or abuse.

See the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014).

Try it

Cronbach's alpha calculator

Paste survey responses: one row per person and one column per item, separated by spaces, commas or tabs. The sample below shows 12 people answering a 6-item scale (1–5). Item 4 is worded negatively, so try reverse-scoring it.

Your data stays in your browser. Nothing is uploaded.

Reading alpha. Many researchers treat about .70 as acceptable for research and .80–.90 as good, but there is no magic number. Alpha rises with more items, assumes every item measures the construct equally well, and a very high value (above about .95) can mean items are redundant. McDonald's omega relaxes some of these assumptions. A negative or low item-total correlation often means an item needs reverse-scoring or does not fit.
Open instrument library

Free and open instruments for research

Measures you can use in psychology, counseling, mental health and education research without buying a license. Each entry describes what the instrument measures, its length and audience, how it can be used, and the original citation.

Before you use any instrument: get it, with its scoring instructions, from the developer or official source, and read the current terms of use (free to use is not always free to change, sell or put online). Doc Lobe does not reproduce item wording. Instruments marked Screening help decide who needs a full evaluation; they do not diagnose. Report reliability for your own sample, and check that the measure has been validated with people like your participants.

Choosing a measure

Eight questions to ask before you choose

1. Does it match my construct?

Read the items and the developer's definition. "Stress," "distress" and "burnout" are different constructs.

2. Was it validated with people like mine?

Check age, language, culture, and clinical or community samples in the validation studies.

3. How reliable are scores in similar samples?

Look for internal consistency and, where relevant, test-retest evidence in published studies.

4. Is there validity evidence for my purpose?

A measure built for research may not be suited to individual decisions or diagnosis.

5. Are there versions in my participants' languages?

Use published, validated translations rather than translating on your own when possible.

6. What are the terms of use?

Confirm it is free for your purpose and whether you may change items or put it online.

7. How long does it take?

Short forms reduce fatigue and missing data, especially in combined surveys.

8. Could any items cause harm?

Plan support and referrals for questions about suicide, trauma, abuse or discrimination, and get ethics board approval.

Watch and read

Videos and recent research

Short talks and peer-reviewed articles published in the last ten years (2016 or later) that support this lesson. Videos and article links need an internet connection.

Videos

Peer-reviewed articles

  • Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465.Shows how hidden or unjustified choices about measures weaken research, with questions every researcher should answer.https://doi.org/10.1177/2515245920952393
  • Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378.Found that many published studies report little validity evidence for their scales; offers recommendations.https://doi.org/10.1177/1948550617693063
  • McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433.Explains alpha’s assumptions and alternatives such as omega, in plain language.https://doi.org/10.1037/met0000144
  • Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484–501.A step-by-step guide to choosing and computing omega.https://doi.org/10.1177/2515245920951747
  • Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90.The standard guide to testing whether a measure works the same way across groups before comparing them.https://doi.org/10.1016/j.dr.2016.06.004
  • Levis, B., Benedetti, A., & Thombs, B. D. (2019). Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: Individual participant data meta-analysis. BMJ, 365, l1476.Pooled data from thousands of participants to show how well PHQ-9 cutoffs screen for depression.https://doi.org/10.1136/bmj.l1476
  • Plummer, F., Manea, L., Trepel, D., & McMillan, D. (2016). Screening for anxiety disorders with the GAD-7 and GAD-2: A systematic review and diagnostic metaanalysis. General Hospital Psychiatry, 39, 24–31.Summarizes the sensitivity and specificity of the GAD-7 across studies.https://doi.org/10.1016/j.genhosppsych.2015.11.005
  • Soto, C. J., & John, O. P. (2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117–143.A model of how a modern personality measure is built and validated.https://doi.org/10.1037/pspp0000096
For instructors

Lesson plan preview: Psychometrics in practice

A ready-to-teach lesson that takes students from constructs to reliability analysis and fair instrument selection.

Quick check

Test your understanding

Sources

References

  1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
  2. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
  3. International Test Commission. (2017). The ITC guidelines for translating and adapting tests (2nd ed.).
  4. McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
  5. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741–749.
  6. Instrument citations appear with each entry in the library above.