How Evidence-Based Education Works in EdTech: The Yandex Practicum Experience

Contents:

• What is evidence-based education and what "evidence" is used
• How an evidence-based approach helps students complete their studies
• How an evidence-based approach makes education student-centered

On November 15, Yandex held its annual online conference on education, YaC/e. During the event, representatives of Yandex Praktikum shared their experience in applying an evidence-based approach to teaching. This topic was discussed in detail in two presentations. It is important to note that an evidence-based approach allows for improving the quality of educational programs by drawing on scientific research and practical data. This promotes more effective acquisition of knowledge and skills by students, which makes learning more focused and effective.

• Maria Kovaleva, Head of the Educational Technologies Laboratory at Yandex Praktikum, presented a report entitled "Methodological Expertise and Data Science in Search of Evidence in Education";
• Dmitry Abbakumov, Chief Psychometrician of the company, presented a report entitled "The Curious Case of Benjamin Bloom and Other Secrets of Evidence-Based Education".

We present the key ideas from their presentations.

What is evidence-based education and what "evidence" is used

The speakers expanded on the concept of evidence-based education, describing it as a process based on scientifically valid data. An important aspect is regularly conducting statistical research that helps determine how students learn, which educational methods are most effective, and how to improve the quality of education. The collection and analysis of digital data plays a key role in this process. Maria Kovaleva noted that this approach involves scaling up successful educational methods, as well as using artificial intelligence and machine learning to optimize the educational process. Maria uses the metaphor of a pyramid to explain the evidence-based education system. At the base of this pyramid lies data, which serves as the foundation. The next level includes experience applying this data in real-world settings, and the top of the pyramid is statistical research, which allows for the discovery of new insights. Excluding any of these levels can lead to the collapse of the entire system. For example, if an educational product is developed without support from data and is based solely on subjective opinions about what effective learning should be—even if these opinions come from an experienced teacher or methodologist—this can lead to errors, as such ideas are often based on false stereotypes. Furthermore, relying on data without the support of statistical research is also insufficient, as the same data can be interpreted differently, leading to an incomplete picture. Thus, to create an effective educational system, it is necessary to take into account all three levels of the pyramid to ensure its reliability and effectiveness.

Dmitry Abbakumov emphasizes that the primary goal of evidence-based education is the selection of effective pedagogical technology. This technology should ensure optimal presentation and explanation of educational material, which will help each student achieve their individual educational goals.

An evidence-based approach improves communication with students and increases their trust. For example, a study conducted by one American educational platform showed that informing a student that practicing before a test helps them achieve better results 60% of the time is more effective than simply telling them, "Please practice, and you'll succeed." This approach allows students to better understand the value of training and increases their motivation to learn.

Dmitry identified three key aspects of evidence-based education:

• high-quality data, without which there can be no evidence;
• valid mathematical models, that is, those that represent reality in a mathematical formula;
• interpretability and transparency.

Maria Kovaleva emphasized the key aspects that should be considered when conducting research and analyzing data in order to improve an educational product. It is important to understand the influence of various factors, such as the learning context, user preferences, and current trends in the educational field. Effective data analysis allows you to identify the strengths and weaknesses of a product, and adapt it to the needs of the target audience. Taking these phenomena into account contributes to the creation of higher-quality and more in-demand educational content.

• The paradox of research interests. The point is that both desirable and undesirable values ​​are important for research. For example, to analyze students' willingness to recommend a course, you need to study the attitudes of not only those who recommend it but also those who don't. Although, for business as a whole, it is, of course, important to have as few undesirable values ​​as possible.
• The paradox of applicability of findings. Every course development team wants research to be conducted specifically among their students, because students in different programs differ. And when data is obtained for a specific course, you want to compare it - what about others?
• The paradox of individual actions. Educational product developers always want to know the effect of a single action on a business metric - but in most cases, a result is achieved through a combination of actions, not a single one.
• The paradox of data fascination. When working with data, you must remember that there is something beyond it. That is, there is no need to cross out the methodological expertise.

Also study:

To successfully design a course, a methodologist must conduct a series of studies. First and foremost, it's crucial to identify the target audience, their needs, and their level of expertise. Surveys, interviews, and focus groups can be used to gather information about learners' preferences and expectations.

The next step is to analyze existing educational resources and courses already available on the market. This will help identify content gaps and identify unique offerings that can be incorporated into the new course. The methodologist should also review user reviews of similar courses to understand what works and what doesn't.

It's also worth conducting research on current educational trends and new technologies that can be used in the course. This includes examining online learning methods, interactive elements, and multimedia resources. It's important to consider how these elements can increase engagement and learning effectiveness.

Competitor analysis is equally important. Studying their courses and approaches will allow you to understand the strategies used by other methodologists and how you can improve your product.

As a result of this research, a methodologist will be able to create a course that will meet the needs of the target audience and stand out from the competition, which, in turn, will increase its attractiveness and success in the educational services market.

How an evidence-based approach helps students complete their studies

Evidence-based education significantly improves the effectiveness of learning. The example given by Maria Kovaleva shows that confidence in one's own knowledge contributes to students successfully completing a course. Therefore, it is important to consider and overcome impostor syndrome, which students often experience, feeling that they lack knowledge and cannot cope with the volume of information. This feeling can be "immobilizing" and prevent students from reaching their full potential. Creating a supportive learning environment and using confidence-building techniques can significantly improve learning outcomes.

Maria emphasized that sharing real-life stories of successful graduates is essential for boosting students' confidence. These stories demonstrate that even they experienced doubts and believed they couldn't succeed, but ultimately overcame the challenges. Mentors who have already established themselves as experts also sometimes share their experiences with impostor syndrome. This experience demonstrates to students that doubts are a normal part of the path to success and helps them cope with such feelings.

It is important to improve specific areas of the curriculum that may be weakening students' confidence. This applies to the most complex topics, where students often encounter difficulties and are at risk of dropping out. To prevent such situations, complex topics should be broken down into simpler steps and assignments should be gradually made more challenging. This approach promotes more effective learning and strengthens student confidence.

To increase student confidence, it is important to emphasize the practical value of the material they are learning. Maria Kovaleva provided a clear example from a data analysis course where she observed a decline in student confidence while studying a certain topic. Analyzing the content, it became clear that one complex term was explained using abstract concepts, which hindered understanding. Students were unable to connect theoretical explanations with real-world tasks. After supplementing the explanations with practical examples, student confidence increased significantly. This confirms that the use of real-world cases in teaching facilitates better learning and increases student confidence.

It is important to demonstrate to students how various terms and ideas are applied in real life. Practical tasks and projects, as well as collaboration with real clients, help students understand the value of the knowledge they are gaining. However, there is a caveat: presenting beginners with overly complex projects early on can lead to demotivation. Students are beginning to realize not only their accomplishments but also the significant amount of information they still need to master. Therefore, it's important to find a balance between the complexity of the tasks and the students' level of preparation to maintain their motivation and interest in learning.

How to maintain a balance in learning? Maria shared her observations about the decline in confidence among students in the data analysis department. Data analysis is a truly complex field, where students encounter cumulative and complex concepts throughout their learning process, making it difficult to systematize and integrate them. In response to this problem, the designers developed not only projects for each sprint but also a consolidated project that simulates real-world conditions and incorporates the skills acquired in previous stages of learning. The result was clear: students' confidence declined before completing the consolidated project, but increased significantly after its completion. This suggests that the consolidated project helps students overcome a crisis of confidence in their knowledge and skills.

The question of how Yandex Practicum assesses students' confidence in their knowledge is relevant. According to the speaker, questionnaires are used for this purpose. This method allows us to obtain objective data on the perception of our own knowledge and skills by students, which, in turn, helps to improve the educational process and adapt programs to the needs of students.

Read also:

Imposter syndrome is a psychological condition in which in which a person doubts their achievements, considers themselves unworthy of success, and fears exposure. This phenomenon is often found in people who have achieved significant success in their careers or studies but lack confidence in their abilities. Understanding impostor syndrome is important for boosting self-confidence and achieving new goals.

Overcoming impostor syndrome begins with becoming aware of your feelings and thoughts. It's important to acknowledge that many people experience these doubts, and this is normal. It's important to analyze and challenge negative thoughts, replacing them with more positive and realistic ones. Setting goals and keeping a progress journal will help you document your achievements and build self-confidence.

Connecting with colleagues or a psychologist can provide new perspectives and support. It's important to surround yourself with people who inspire and motivate you, and to share your experiences. Using these strategies can help combat impostor syndrome and promote personal growth and development.

How an evidence-based approach makes education student-centered

Dmitry Abbakumov shared that at Yandex Praktikum, they drew inspiration from an approach known as Mastery Learning. This method was developed by American psychologist Benjamin Bloom, who is also known for his taxonomy of learning objectives. This system categorizes learning objectives from simple to complex, allowing for an effective organization of the educational process and promoting deep learning. Mastery Learning is a method in which learning a new topic is possible only after the previous one has been fully mastered. This approach emphasizes the individual pace of each student, allowing them to delve into the material without rushing. If students encounter difficulties, they are provided with the necessary support to achieve success. Mastery Learning promotes a deeper understanding of the subject and improves overall learning effectiveness, creating the conditions for successful acquisition of knowledge. Dmitry Abbakumov emphasized that Bloom concluded that even the most qualified teachers and high-quality educational materials cannot be universally applicable to all students. Each person perceives information differently, which leads to different learning outcomes.

The psychologist proposed conducting formative assessments, which serve as an important stage in the educational process. This stage allows us to determine how successfully a student has mastered the educational material offered for study and practice. The results of the assessment influence the student's further educational path. If the student has successfully mastered the material, they can move on to the next topic or receive additional assignments to deepen their knowledge. If the assessment results indicate difficulties in mastering, the student is provided with support. In his report, Dmitry Abbakumov called this process an "intervention" – an intervention that can include alternative educational materials adapted to the individual needs of the student. Thus, formative assessments are a key tool for effective learning and supporting the personal development of each student.

The Mastery Learning strategy is based on two key components. The first component includes formative assessments, which help identify differences in students' knowledge levels and needs. Based on this assessment, alternative interventions are proposed that accommodate individual differences using a variety of learning materials. These materials may include retellings, additional assignments, or examples, depending on the course content. Furthermore, various workflows can be used throughout the learning process, such as group work, pair work, or individual assignments. Thus, by varying the content and format of the learning process, we can achieve high results for each student.

Dmitry emphasized that the Mastery Learning method has proven its effectiveness over its forty-year history. Unlike the traditional approach to teaching, where most students perform average, with some showing a mixture of high and low achievement, the Mastery Learning method helps the majority of students achieve excellent results. This approach allows each student to master the material at a deep level, which significantly improves the overall level of academic performance and the quality of education.

However, this approach has limitations. It is important to consider that its effectiveness may vary depending on specific conditions and circumstances. In addition, the application of this method may encounter certain difficulties, such as lack of resources, lack of information, or implementation difficulties. Given these aspects, it is necessary to carefully analyze the situation and evaluate possible alternatives to achieve the best results.

• Alternative interventions. For many years, it was believed that Bloom's approach is limited by the impossibility of organizing a wide variety of interventions for different students, since personalizing learning to such an extent is very expensive.
• Bulkiness. To obtain a high-quality, valid, and reliable result of formative assessment, a voluminous test is required, which significantly complicates the application of the approach.
• Untimely. Mastery Learning assumes that a student must cover a significant portion of the educational material (for example, a topic) before undergoing assessment. It turns out that, faced with difficulties and disappointed with their educational experience, a person may drop out of education even before they receive support, and even before they figure out what kind of help would be useful.
• Lack of focus. Testing based on the results of one large topic does not allow us to accurately identify the part that caused difficulties for the student.

Read also:

Effective management of cognitive load is a key aspect of successful learning. Cognitive load refers to the amount of information a person can process simultaneously. To optimize learning, it's important to consider several factors.

The first thing to consider is the complexity of the material. Teachers and instructors are encouraged to break information into small, easily digestible chunks. This helps students focus on a single task without overwhelming their memory. Using techniques such as concept maps or diagrams can significantly simplify the process of perceiving information.

The second aspect is the use of active learning methods. Incorporating assignments, practical exercises, and discussions into the learning process promotes deeper assimilation of the material. Active student engagement helps reduce cognitive load, as they can apply knowledge in practice.

Furthermore, it is important to consider the individual characteristics of students. Each person has their own pace of information processing and level of preparation. A personalized approach to learning helps adapt the material to the needs of each student, which in turn reduces cognitive load.

Also, it is worth paying attention to the organization of the learning environment. Cleanliness, order, and the absence of distractions contribute to increased concentration and effective learning. Using technology and digital tools can facilitate access to information and diversify the ways it is presented. Managing cognitive load in learning requires a comprehensive approach, including adapting materials, active learning methods, individualization, and creating a comfortable learning environment. This will help improve knowledge acquisition and make the learning process more effective. Artificial intelligence and data analysis now make it possible to effectively overcome the limitations of Mastery Learning. Yandex Praktikum developed an innovative diagnostic technology called Delta. Dmitry Abbakumov provided a detailed explanation of its operating principles. Delta uses intelligent algorithms to assess knowledge levels and identify learning gaps, allowing for personalization and increased effectiveness of the learning process. This solution opens up new opportunities for students and teachers, making learning more adaptive and effective. The course includes topics, which are divided into lessons. Each lesson is divided into elements such as tasks, questions, quizzes, and learning materials. For each of these elements, starting with the smallest and moving to larger levels, a cognitive feasibility metric is calculated. This metric reflects how accessible a given element is to the average learner.

The product team and instructional designers have a clear understanding of the typical course student and develop educational content based on this understanding. As actual students take the course, cognitive feasibility is measured, allowing us to assess how closely the team's understanding of the average learner's capabilities corresponds to reality. This helps determine whether students are mastering the material. Deviations from expectations are designated as deltas, which can be either positive or negative. Negative deviations indicate that students found the material more challenging than expected, while positive deviations indicate that the material was easier than expected. If certain course elements generate a significant number of deviations, this serves as a signal for content revision. Thus, the educational material is fine-tuned to the needs of the typical student, eliminating discrepancies between the product team's expectations and actual learning outcomes.

Deviation analysis conducted for each student serves as the basis for individualized support. This analytical algorithm also helps predict successful course completion. As Dmitry Abbakumov notes, among students who drop out, a significant proportion have a negative delta—23%, compared to only 1% of those who successfully complete the course. This underscores the importance of promptly identifying problems and providing the necessary support to improve student retention.

The speaker noted that the problem of cumbersomeness in the Delta system is solved by analyzing data on how students cope with each course element. Assessment is performed during the learning process, which allows for more accurate progress tracking and identification of student difficulties. This approach promotes the individualization of the educational process and increases the effectiveness of learning.

The problem of student unfocus is solved by collecting data on many small elements. This allows for the identification of specific moments when students experience difficulties during the learning process. By analyzing the details, it is possible to better understand which aspects require additional attention and support, which contributes to more effective learning and improved academic performance.

The problem of untimely learning has not yet been fully resolved, as student difficulties often become apparent only after they arise. However, Dmitry Abbakumov announced a new version of the technology – "Delta+." This technology uses data on students' deltas for each task to predict potential difficulties in the next topic. Thus, "Delta+" allows for proactive work and the prevention of learning problems.

Yandex Praktikum claims that the "Delta" technology is universal. The speaker emphasized that the development of this technology took into account its applicability both in small groups of students and in large EdTech companies. This makes Delta a flexible tool for educational organizations of various sizes and formats.

Read also:

• "We have little evidence-based approach in the daily work of teachers"
• 6 reasons why psychometrics is needed in EdTech
• Checklist: 5 ways to increase the completion rate of an online course
• 11 metrics to evaluate the quality and effectiveness of an online course
• Improvement loop: how to improve an already launched online course