What is psychometrics?

Contents:

• How psychometrics and tests are related
• How psychometrics came to education and why it is here
• Where psychometricians work and do they work with assignments for the Unified State Exam

Candidate of Sciences in Education, Academic Director of the Master's program "Teaching and Assessment as a Science", and a research fellow at the Center for Psychometrics and Measurement in Education at the Institute of Education of the National Research University Higher School of Economics. Specializing in research in psychometrics and educational measurement, which enables the application of scientific methods to improve the quality of the educational process.

In this article, you will learn about key aspects of the topic, which will help you gain a deeper understanding of the subject matter. We will examine important factors, the influence of various elements, and offer helpful tips. Read on to gain new knowledge and skills in this area.

• How are educational tests and the science of psychometrics related?
• When and why did psychometrics appear?
• Why do psychometricians distrust neural networks and are skeptical of calls to study digital traces?
• What can and cannot be measured by tests in education?
• What do psychometricians dislike about the modern Unified State Exam and why they still consider it an advantage rather than a disadvantage?

How are psychometrics and tests related?

Psychometrics is a field of psychology that deals with the measurement of psychological characteristics and personality traits. It includes the development and use of various tests and methods to assess such aspects as intelligence, emotional state, personality traits, and behavior. Psychometrics plays a vital role in psychological counseling, education, and personnel selection, providing objective data for decision-making. A key aspect of psychometrics is the reliability and validity of tests, which enable accurate and reproducible results. The development of psychometrics continues actively, taking into account new research and advances in psychology and statistics.

Historically, psychometrics is a discipline that deals with the measurement of human qualities in the social sciences. Psychometrics began with the evaluation of test quality. A test is considered high-quality if all its tasks measure the same quality or skill. Otherwise, the final score loses its informative value. For example, if you combine questions on mathematics with questions on Russian language and geography, the final score will be meaningless. Thus, to ensure the reliability of test results, it is necessary to strictly adhere to the principle of homogeneity of measured parameters.

Modern psychometrics is a science that studies human behavior as a whole. This field uses complex statistical models that allow for a deeper understanding of mental and emotional processes. One of the most relevant areas is network psychometrics, which applies models from statistical physics to the analysis and description of human behavior. This approach helps identify patterns and relationships between various aspects of behavior, which in turn facilitates more accurate predictions and interpretations of social phenomena. Modern psychometrics encompasses numerous areas related to learning analytics and machine learning. One key task is determining when a student has mastered the material and is ready to move on to the next topic. Tests continue to serve as the basis for such decisions. Each question in a test is a behavioral indicator that reflects the internal processes occurring in the student's mind. Since we cannot directly measure changes in mental state and knowledge, we examine their manifestations in behavior and analyze the corresponding behavioral characteristics. This allows us to more accurately assess the level of material acquisition and adapt the educational process to the individual needs of students.

Modern psychometrics is both a separate science and and a set of practical skills that are applicable across a variety of fields. This discipline develops and evaluates methods for measuring psychological characteristics such as intelligence, personality traits, and emotional state. Psychometrics is widely used in psychology, education, personnel selection, and sociological research, highlighting its versatility and importance. Modern approaches to psychometrics incorporate statistical methods for analyzing and interpreting data, allowing for the creation of reliable and valid instruments for assessing mental processes. Thus, psychometrics is not only a theoretical foundation but also a practical tool for a deeper understanding of the human psyche. The answer to this question depends on which psychometrician you speak with. I believe psychometrics is a branch of computational behavioral science, closely related to sociology, neuroeconomics, and other areas of neuroscience. Another important aspect is the study of consumer behavior in marketing, which also falls under this category. If your goal is to find out which button color attracts more customers, you need to conduct user behavior analysis. This will allow you to more accurately determine preferences and improve website conversion.

Psychometrics is a field where statistics and behavioral data intersect. This field finds application in various disciplines, including psychology, sociology, and marketing. Major international conferences on psychometrics confirm this approach, showcasing a wide range of topics, from causal analysis to the application of machine learning in marketing research. Thus, psychometrics plays a vital role in data analysis and interpretation, allowing us to better understand human behavior and develop effective strategies based on the obtained results.

A psychometrician is more than just a test developer. Their role covers a wide range of tasks, including data analysis and interpretation, assessment of mental characteristics, and the development of measurement methods. Psychometricians apply scientific approaches to the creation of reliable and valid assessment instruments, making them important players in the fields of psychology and education. They are involved not only in testing but also in research activities aimed at improving existing methods and developing new assessment instruments.

Classical psychometrics, based on task development, remains a relevant and important field. However, with the advancement of technology, the boundaries between psychological assessment and marketing research are becoming increasingly blurred. In both cases, conclusions about human behavior are based on data analysis. Thus, modern psychometricians not only develop tasks but also actively analyze the data obtained, which allows for a deeper understanding of people's motivations and behavior in various contexts.

Modern psychometrics is developing in several key areas. First, technology is being actively integrated into the testing and assessment process. The use of online platforms and mobile applications allows for more efficient and accessible psychometric research. Second, the emphasis is shifting to individualized assessment approaches, taking into account the unique characteristics of each respondent. This includes adaptive tests that adjust to the user's knowledge and skills. Third, there is growing interest in multidimensional assessment, which uses complex methods to analyze various aspects of personality and intelligence. There is also increasing attention to ethical issues in psychometrics, including data protection and testing fairness. All of these trends contribute to a more accurate and objective understanding of human behavior and abilities.

In the field of computational psychometrics, psychometrics relies on well-labeled and structured data. It utilizes a variety of behavioral indicators and tasks whose reliability we are confident in. This approach allows us to obtain accurate and objective results, which makes computational psychometrics an important tool in psychology and related fields.

Currently, there is a trend towards the use of Behavioral indicators based on natural user behavior. This data is unstructured and requires machine learning for processing. The use of such indicators helps increase the authenticity and naturalness of the results assessed. However, it is worth noting that this can also lead to potential threats to the validity of the conclusions. It is important to carefully analyze such data to minimize risks and ensure the reliability of the information obtained. Neural networks are already actively used in psychometrics. These technologies allow for the analysis and interpretation of psychological data with high accuracy. Using machine learning algorithms, neural networks can process large volumes of information, which contributes to a deeper understanding of human behavior and cognitive processes. The use of neural networks in psychometrics opens up new possibilities for diagnostics, forecasting, and individualization of approaches in psychology, making this field more effective and scientifically sound. Neural networks are rarely used in psychometrics, the main reason for this is their lack of interpretability. When a developer encounters a dissatisfied respondent who disagrees with the test results, in classical psychometrics they can explain why a particular score was obtained based on the respondent's response profile and actions. When using neural networks, the situation becomes significantly more complex: it is difficult to understand what factors led to the assignment of a specific score. This lack of transparency creates problems for both test developers and respondents, limiting the widespread use of neural network technologies in psychometrics.

How Psychometrics Came to Education and Why It's Here

Psychometrics as a science began to develop in the late 19th century, when scientists began using quantitative methods to study human psychological characteristics and intelligence. Francis Galton is considered the founder of psychometrics, who used statistical methods to analyze individual differences in his research. In the following decades, psychometrics gained popularity, especially with the development of testing and personality assessment. The 20th century saw the emergence of standardized tests and methods that are still used today to measure intelligence, academic ability, and other psychological characteristics. Thus, psychometrics continues to evolve, integrating new approaches and technologies, making it an important field in psychology and education. Psychometrics, as a scientific field, was founded by psychologists who began by measuring intelligence. The first test, considered fundamental in this field, is the Binet-Simon IQ test, introduced in 1905. Some researchers argue that the origins of psychometrics can be traced even earlier, to the laboratory of Wilhelm Wundt, who, beginning in the 1870s, studied the intensity of sensations and how we perceive various stimuli. This study was an important step towards understanding the human psyche and developing methods for its quantitative assessment.

The trend towards psychometrics in the educational sphere arose in The United States, where a culture demands that tests be validated for college admissions and other purposes, be used without proof of reliability. If a test cannot be proven reliable, its use is rendered ineffective. As a result, the United States developed a team of experts dedicated to standardizing tests and validating their ability to measure their stated qualities. Over time, such tests became the gold standard for assessing human qualities. This practice has been adopted by other countries seeking to develop human capital, such as the Netherlands and Belgium. Psychometric tests play an important role in educational systems, allowing for a more accurate assessment of students' abilities and potential, which contributes to improving the quality of education and preparing qualified professionals. When psychometricians discuss tests, they typically refer to multiple-choice questions. These questions effectively assess respondents' knowledge and skills, providing a structured approach to measuring their psychological and cognitive characteristics. Multiple-choice tests are often used in research and practical psychology for standardized assessment of personality traits and intelligence. Psychometrics provides the ability to work with a variety of task types and data. A behavioral indicator may extend beyond simply answering correctly. For example, tasks may simulate computer games, where the behavioral indicator is the actions performed by the respondent during the gameplay. In such scenario-based tasks, not only the respondent's choices are analyzed, but also the speed of decision-making. Important indicators include the sequence of actions, keystrokes, and clickstream. Thus, any information relevant to assessing a specific ability can be used in psychometric analysis.

Psychometrics and the study of digital footprints in education are closely related. Psychometrics, as the science of quantitatively measuring psychological characteristics, provides tools for assessing students' educational achievements and progress. At the same time, the digital footprints left by students during their learning process contain valuable information about their interactions with educational resources and platforms.

Digital footprint analysis helps identify patterns in student behavior, which in turn helps optimize educational processes and tailor learning materials to individual needs. Thus, the integration of psychometrics and digital footprint analysis contributes to a deeper understanding of educational outcomes and improves the quality of learning.

With the increasing digitalization of education, it is important to recognize how these two areas can mutually enrich each other, providing new opportunities to improve the educational experience and enhance the effectiveness of teaching.

In my experience, discussions about digital footprints often fail to yield meaningful information. Digital footprint data collection is largely driven by industry interests and political agendas, where the 'faster, higher, stronger' principle dominates. In contrast, psychometrics is an academic discipline for which carefully considered conclusions and the absence of alternative explanations are critical.

Psychometrics develops statistical models based on an understanding of how scientific knowledge functions. The first step is to formulate a theoretical framework based on previous research. This is followed by an analysis of psychological processes: their nature, manifestations, and interrelations. Only then are statistical models created, which allows for a deeper understanding of mental phenomena and their interactions.

Marketing applications of Data Science require a mathematically oriented approach, which is not always based on a deep theoretical model. This approach focuses more on predicting trends than on explaining them, which can complicate the interpretation of the obtained results. Collecting digital data for marketing purposes may prove ineffective for scientific analysis if it is not based on a theoretical foundation, but only on tactical product-related needs. This underscores the importance of integrating theory and practice in Data Science to achieve more accurate and valid conclusions in marketing.

There are aspects of education that cannot be assessed using standardized tests. These include creativity, critical thinking, and emotional intelligence. These qualities play a significant role in the development of a well-rounded personality and a successful career. Tests often focus on memorization of facts and problem solving, but fail to adequately assess innovative abilities or teamwork. Furthermore, soft skills such as communication and leadership are also left out of testing. It's important to remember that education isn't just about knowledge, but also about developing a holistic personality capable of adapting to change and interacting effectively with the world around us. Almost any human quality and ability can be measured, including mathematical skills, creativity, critical thinking, and intelligence. Modern assessment methods allow for a comprehensive analysis of various aspects of mental activity, which makes it possible to better understand the individual characteristics and potential of an individual.

The rest is related to theoretical The basis and process of operationalization.

Operationalization is the process of transforming abstract concepts and theories into measurable and observable variables. This approach allows researchers and practitioners to pinpoint how specific ideas can be implemented in practice. Operationalization plays an important role in the social sciences, psychology, and other fields where it is necessary to quantify complex phenomena. With proper operationalization, it is possible to create clear and reproducible methods for collecting data, which, in turn, contributes to a deeper understanding of the processes and phenomena under study. It is important to keep in mind that successful operationalization requires careful selection of indicators and measurement methods so that they adequately reflect the concept under study and its characteristics.

Operationalization is the process of identifying measurable elements in a phenomenon or property that can be observed. For example, to assess academic motivation, it is necessary to clearly define its nature, manifestations, and the behavior of highly motivated people. This includes an analysis of the actions and inactions characteristic of highly motivated people, which allows for the creation of objective criteria for measuring this psychological aspect. A detailed understanding of operationalization aids in research and the practical application of knowledge in psychology and education.

The theoretical model of motivation may remain universal for various groups—schoolchildren, students, and adults. However, behavioral indicators should vary depending on the context. When translating tests, it is important to adapt them to cultural characteristics, as they significantly influence the perception of tasks in different countries. In addition, gender differences must be taken into account so that respondents' results depend solely on the measured parameters, and not on gender, origin, native language, or other factors. Proper adaptation of tests and consideration of these aspects contribute to more accurate and objective results.

Where do psychometricians work and do they deal with assignments for the Unified State Exam?

Psychometricians, specialists in psychometrics and psychodiagnostics, are used in various fields, including education, psychology, and human resources management. Their work is particularly relevant in the education sector, as they assess and analyze academic achievement and identify individual student characteristics. Graduates of the Master's program at the Higher School of Economics (HSE) successfully apply their knowledge and skills in educational institutions, developing and implementing methods for assessing student performance and psychoemotional well-being. Psychometricians help create test tasks and conduct research aimed at improving educational processes. They also work in scientific research, analyze data, and develop recommendations for educators. Thus, the demand for psychometricians in education remains high, opening up a variety of career opportunities for HSE graduates. Many specialists choose research careers in the commercial sector, including UX and product research, data analytics, and assessment in HR and EdTech. Every year, a significant number of graduates continue their education in postgraduate programs, seeking to deepen their knowledge and skills for a successful academic career.

In the field of education, psychometrics is becoming increasingly relevant, although awareness of its necessity varies. Psychometricians, working not only in Russia, often face the problem of justifying their value. They typically come with the statement, "We will identify the flaws in your tests." However, not everyone is interested in hearing about the shortcomings of their assignments. Therefore, many psychometricians turn to product research, where the skills of collecting, analyzing, and interpreting data are in high demand. This allows them not only to apply their knowledge but also to make a significant contribution to the development of educational technologies.

The Federal Institute for Pedagogical Measurements (FIPI), which develops the Unified State Exam (USE), does indeed employ specialists in psychometrics. Psychometricians analyze and evaluate mental processes, which allows for the creation of objective and reliable tests. Their work includes the development of knowledge assessment methods, as well as the analysis of results, which contributes to improving the quality of exams and the objectivity of assessment. This is important to ensure the fairness and effectiveness of the education system.

Leading psychometric experts in Russia played a key role in the development of the Unified State Exam (USE). However, since its introduction, the exam has undergone significant changes.

Currently, the Federal Institute for Pedagogical Measurements (FIPI) employs qualified specialists developing tasks for the Unified State Exam (USE). However, unfortunately, I know only a few of these professionals. The scale of the USE is so vast that psychologists and psychometricians cannot fully cover all its aspects. The most experienced specialists I know are not involved in state educational measurement. This creates potential risks to the quality of the USE and may negatively impact the assessment of student knowledge. Measures are needed to attract a wider range of experts to this important area.

In Russia, besides the USE, there are no alternative tests that could provide high-quality monitoring studies with reliable psychometrics. It is important to note that the education system does not conduct sufficient research monitoring educational progress and providing formative feedback. This creates gaps in the assessment of student knowledge and in the possibility of their further development.

The shortcomings of psychometrics in the current USE are the subject of much discussion among experts. Firstly, many note that tests do not always accurately reflect students' knowledge and skills. This is due to the Unified State Exam's frequent focus on memorization of facts rather than a deep understanding of the material. Secondly, some critics point to the insufficient adaptation of tests to the specific characteristics of different schools and regions, which can lead to unfair results. It is also believed that standardized tests may fail to take into account students' individual abilities and creativity. As a result, such shortcomings can negatively impact student motivation and their attitude toward learning. Therefore, it is important to continue the discussion and seek ways to improve the assessment system to make it more objective and effective.

A modern test should effectively measure the abilities of as wide a range of respondents as possible. The largest group among them are average respondents, as confirmed by the normal distribution law. Previously, a reliable assessment of the lower and middle parts of the distribution was achieved using Section A, which offered multiple-choice answers. However, after its removal from the Unified State Exam to combat guessing, the test became more focused on assessing strong students. As a result, students in the bottom half of the distribution began to receive unreliable grades. This creates problems in objectively assessing student knowledge and skills, which is important to consider when designing tests.

Fighting the guessing game is an important task. Guessing something can be easy, but this strategy often leads to ineffective results. Instead of relying on chance, it is better to use analysis and knowledge. By taking a thoughtful approach, you can significantly increase your chances of success. It is important to develop skills that help you make informed decisions rather than guessing. By relying on facts and experience, you can avoid common mistakes and achieve the desired result.

The main argument in favor of guessing-based tasks is their fairness. If a participant manages to guess the answer, then they truly achieved it independently. Even if someone accidentally chose the correct answers to two or three questions, it is worth rewarding them with these points. In the general population of participants, such "guessers" will be a small number. To minimize the influence of guessing on results, it is advisable to increase the number of answer options.

The Unified State Exam (USE) has become dependent on expert assessment, which creates additional risks to the objectivity of the results. An expert always represents a variable that can negatively impact fair assessment. If a student's work is assessed by two strict experts, they may receive a low score. If the work is assessed by two lenient experts, the score will be inflated. Thus, the final grade becomes a kind of lottery over which the student has no control. This raises the need to revise the USE assessment system to improve its fairness and transparency.

Test creators today strive to reduce expert intervention to ensure fairness and increase the level of participation of respondents in the testing process. This makes results more objective and facilitates a more accurate assessment of participants' skills and knowledge.

Tests are often criticized, particularly for the potential for students to specialize in certain types of tasks. However, it's worth considering the real benefit of tests. They allow students to quickly assess their level of knowledge and understanding of material, as well as identify areas of weakness in their preparation. Tests can serve as an effective tool for reinforcing material learned and preparing for exams, and they can also encourage students to delve deeper into topics they find difficult. When used correctly, tests can become part of a comprehensive approach to learning, combining various forms of assessment and active learning methods.

Tests, such as the Unified State Exam, have an important advantage: they are measurable. Although pedagogy combines scientific and artistic aspects, modern education has also become an industry in which measurable indicators and scalability are key. From this perspective, the Unified State Exam represents more of an advantage than a disadvantage in the context of educational quality. However, further work on improving it is necessary.

Test cramming does occur. Each test not only assesses specific knowledge but also reflects general test-taking skills. For example, participants may understand commonly worded questions differently. This is one of the key problems associated with testing. It is important to consider that successful completion of a test may depend not only on the level of preparation but also on habits with the question format. Therefore, to objectively assess knowledge, it is necessary to minimize the influence of testing skills on results.

Psychometricians are actively addressing this issue, developing various methods to reduce the influence of testing skills on results. Teaching children not the subject, but rather preparing for tests, is not a flaw in the tests themselves. If a student has genuinely prepared for tests, this is no reason not to appreciate their efforts. They have spent a lot of time preparing and deserve high marks for their efforts and achievements.

Rework the text so that it remains on-topic, without adding unnecessary information. Optimize it for search engines and supplement the content if necessary. Avoid using emojis and unnecessary symbols. Don't add structured sections like numbers or symbols; simply provide plain text.

Read also:

• What is evidence-based education?
• Why neuroscience education: interview
• What research do methodologists need?