Contents:
The online education market is actively searching for effective methods of assessing the quality of courses that can become the standard for both companies and users of online schools. In early April, a significant joint project between the HSE Institute of Education and the Skillbox product and methodological team was announced at the Moscow International Education Fair (MIEF). The project's goal is to develop a new model for assessing the quality of online education. At the event, participants presented the results of the research phase, which included an analysis and synthesis of existing global models for assessing the quality of educational technologies and products. This innovation could significantly increase trust in online education and help users make informed course choices.
Work in the field of online education is becoming increasingly important, as there is currently no generally accepted definition of quality in this field. The concept of quality encompasses a wide range of aspects, and virtually every company in the EdTech market uses its own criteria to evaluate the quality of its courses. This diversity of approaches creates difficulties for users in choosing the most effective educational programs. To improve the quality of online learning, it is important to develop uniform standards and evaluation criteria that will help both educational institutions and students navigate the diversity of market offerings.
Within a single company, departments may interpret the concept of a "quality educational product" differently. Students also form their own perceptions of quality online learning and evaluate courses based on these perceptions. The question is whether these different approaches can be combined to objectively assess the quality of educational programs. It is important to find a balance between professional standards and student expectations to create universal evaluation criteria that reflect the true value of online learning.
At the "Quality EdTech: Designing the Student Experience" conference, organized by the School of Education, Otus, and Netology, Maria Rakultseva, an educational product methodologist with the Strategy and Results team, shared Netology's approach to solving current problems in education. In her presentation, she highlighted key points regarding creating a high-quality educational experience for students. Maria's key ideas emphasize the importance of adapting educational programs to student needs and implementing modern technologies to improve learning effectiveness.
How Netology Researched Perceptions of Quality
Maria shared that she joined the Netology team six months ago. At that time, the company lacked a unified process for monitoring product quality and a mechanism for making data-driven product decisions. This created difficulties in product management and impacted its development. Maria noted that the introduction of a systematic approach to data analysis and quality significantly improved the decision-making process and increased the team's effectiveness.
The online school offers a wide variety of courses, varying in format, topic, duration, level of preparation, and student goals. Teams working on different areas perceive and evaluate the quality of the educational product differently. Therefore, as Maria noted, it was necessary to create a unified framework for evaluation while taking into account the unique characteristics of each course.
Maria and her colleagues began by conducting research to align the company's perceptions of quality and compare them with the opinions of the target audience—students. As part of this work, two studies were conducted: an internal one, aimed at assessing current quality standards, and an external one, focused on understanding student expectations for online courses. These studies will help improve the educational process and enhance the quality of services provided.
The internal study covered 22 in-depth interviews with the company's managers and employees, Including producers, product managers, methodologists, expert relationship managers, and members of the support and maintenance team. These interviews provided a deeper understanding of internal processes and identified key aspects impacting team performance. The research findings provided valuable insights for further improving engagement strategies and enhancing the quality of services provided.
The external study consisted of two stages. In the first stage, Maria's team conducted 13 in-depth interviews with students. The sample included both students who had positively evaluated their studies at Netology and those who had decided to return the course, as well as students from other online schools. This diverse sample provided valuable insights into the quality of education and student satisfaction.
In the second stage, Maria and her colleagues conducted a survey of 1,019 students of the Netology online school who had been studying for at least a month or had completed their studies in the past six months. This study aims to understand students' opinions and experiences, which will help improve the quality of the educational process and tailor courses to student needs.
According to the research results, the company's employees identify several key criteria that a high-quality educational product must meet. First, it must offer relevant and practical information that promotes professional skill development. Second, the availability of materials is important, ensuring easy access to learning resources at any time. Interactivity is also essential to engage learners and maintain their motivation. Furthermore, the product should be adapted to different user levels to meet the needs of both beginners and more experienced professionals. Finally, providing feedback and opportunities for participants to share their experiences and ask questions is crucial. These criteria form the basis for creating an effective educational product that will be in demand and useful for employees.
- it leads the student to the goal;
- meets market demand;
- meets student expectations and has good feedback;
- solves educational problems;
- contains competent materials;
- the course is attended by good experts;
- the result stated in the course positioning corresponds to reality.
The results of interviews and a survey of students demonstrated that the key factors for a quality course are:
- relevance of the content;
- a good teacher;
- detailed feedback during the learning process;
- accessible presentation of the material;
- a lot of practice and work on real projects;
- optimal pace and format of training;
- good Support.
Students were also asked to what extent they felt certain factors were expressed in Netology courses. Maria noted that this information was very useful for identifying the product's strengths, as well as those aspects that require additional attention and improvement. This allows the Netologia team to better adapt their courses to student needs and improve the quality of training.
Read also:
Before starting to design a course, it is important for a methodologist to conduct a series of studies that will help determine the needs of the target audience and form the course structure. Key aspects include analyzing the target audience, studying existing educational programs, and assessing current trends in the chosen field.
The first step is defining the target audience. It is necessary to find out who will be studying, what knowledge and skills they have, and what goals they set for themselves. This can be done through surveys, questionnaires, and interviews with potential students. The data obtained will help tailor the course content to specific needs.
Next comes a study of existing educational programs. An analysis of similar courses will help identify the strengths and weaknesses of competitors, as well as identify unique offerings that may attract the attention of students. This stage includes a study of the curricula, materials, and teaching methods used in other courses.
It is also important to consider current trends in the educational field and in the specific field of knowledge. It is necessary to research the latest developments, methods, and technologies that can be integrated into the course. This will help make the course more relevant and relevant.
Conducting this research will provide the methodologist with the necessary information for successful course design, making it more targeted and effective for learners.
Maria and her colleagues compared the factors identified during internal and external research to form a holistic picture of a quality course. They associated each key factor with a measurable indicator. As the speaker noted, for some aspects, there are obvious metrics, such as the Customer Satisfaction Index (CSI), which assesses student satisfaction with various aspects of the learning experience, including interaction with instructors and the speed of homework assessment. However, selecting an appropriate metric for assessing content relevance proved more challenging, as specialized indicators for this purpose had not been developed.
Netology reassessed all previously recorded quality metrics. The goal of this process was to determine their validity and representativeness, as well as to identify those that can be truly relied upon. Maria's team compiled the selected metrics and placed them on a universal SJM—a student journey map. This map clearly demonstrates at what stage of the student's interaction with the course each metric is measured and the conclusions that can be drawn based on the collected information.
Maria and her colleagues analyzed the metrics used by the Russian and international EdTech markets. The study identified current trends in assessing the quality of educational products, allowing for a better understanding of the factors that influence the success and effectiveness of educational technology solutions.
The results of this work contributed to the development of a new quality control system based on a hierarchical metrics matrix. This system allows for more effective monitoring and evaluation of product quality, ensuring high standards and compliance. Using a hierarchical metrics matrix improves data analysis and facilitates informed decision-making in quality management.
How the new quality control system works at Netology
The new evaluation system focuses on the quality of the educational product in combination with revenue. As Maria noted, quality and financial indicators are closely interrelated, since high-quality products retain students for the long term and encourage them to make repeat purchases. For example, after completing a professional retraining course, students can purchase additional courses to develop in their profession or update their skills. Furthermore, course quality significantly impacts referral rates, that is, the percentage of new students coming to the educational institution based on recommendations from graduates. It is important to consider that a high quality rating for the educational product can be a key factor in attracting new clients and increasing revenue.
The developed metrics matrix is a tree-like structure with horizontal levels and vertical branches. The main target metric is revenue. Its formation is influenced by first-level metrics, such as referrals and repeat purchases. These metrics, in turn, depend on second-level metrics, which include:
- the percentage of returns due to students being dissatisfied with the product quality;
- NPS (Net Promoter Score) of the product - an indicator of students' willingness to recommend courses to others;
- overall CSI - an index of student satisfaction with the school's courses;
- retention rate - the user retention rate, which reflects student engagement, their activity in interacting with the courses;
- student achievement of their goals as a result of training.
Maria noted that these metrics are the most relevant to the current goals of the company and accurately reflect the results of the research.
When analyzing data on educational institutions, the indicators are detailed at various levels. It begins with an overall school analysis, followed by a focus on individual areas, course sets, specific courses, lessons, modules, and content units. Responsible staff members oversee monitoring of metrics at each level. At the first level, these are product managers; at the second level, educational product managers and team leads; and at the third and higher levels, instructional designers, expert relationship managers, and coordinators. This approach enables effective management of the educational process and improvement of learning quality. There is an interconnected metrics structure organized at different levels. For example, the Customer Satisfaction Index (CSI) is divided into more specific metrics, such as expert effectiveness, content quality, and support level. At subsequent levels, these metrics become more detailed, allowing for CSI analysis for individual modules and specific content units. This allows for a deeper understanding of how various aspects influence overall customer satisfaction and optimize processes to improve results.
Netology has identified key principles of interaction with This system and approaches to product decision-making. These principles help optimize work processes, improve product quality, and enhance user satisfaction. A primary focus is on data analysis, customer feedback, and continuous process improvement, allowing the company to remain competitive in the market.
- Work on improving quality can be done both top-down and bottom-up. For example, if an indicator at the upper levels has dropped or increased unexpectedly, you can drill down along the corresponding "branch" and determine the cause. You can also track how changes at lower levels of the system affect the indicators above.
- Each employee responsible for product quality at a given level makes decisions based on the relevant indicators, metric links, and their dynamics, and acts within their area of responsibility.
- When making decisions, the corresponding level and "branch" are taken into account—that is, again, to solve a problem, you need to see its place in the overall structure and consider what influences the indicators at a particular level.
Netology is currently testing the effectiveness of a new system aimed at accelerating decision-making and improving the quality of educational products. By 2024, the online school team plans to learn how to predict the behavior of key metrics. For example, if one metric shows a decline or increase, it is important to understand how this will affect other indicators. The study also examines the perception of course quality from the perspective of employers. As Maria noted, the goal is to look at students not only as consumers of educational services, but also as a "product" that the school brings to the labor market. This approach will allow for a better understanding of market requirements and the adaptation of educational programs for the successful training of specialists.
What's wrong with the COR metric?
The metrics system used by online schools lacks the popular COR (Completion Rate) metric—the percentage of students who complete their studies. Maria Rakultseva made this observation in her model. In another discussion at the "Designing Reskill vs. Designing Upskill" conference, it was discussed why "Netology" does not use this metric as the primary one. Arina Pantina, Educational Product Manager for Creative Industries at Netology, shared her thoughts on this matter. Discussing the importance of metrics in education highlights the need to select appropriate indicators for assessing learning effectiveness and student engagement. Arina noted that at Netology, the COR metric is used solely as an auxiliary tool for analyzing changes in other metrics. She noted several significant limitations of this metric. First, there is no universal methodology for calculating COR, which allows for various approaches to be used. This creates the potential for data manipulation. Arina also shared that she is aware of several "relatively fair" methods that can increase COR to 100%. Some course types do not require maximum COR, and this is for their own reasons. For example, long-term professional retraining courses attract students seeking to master a new profession and find employment or freelance work. These students often drop out mid-course, achieving their goal before completing it. In this case, a low course completion rate (COR) is not an indicator of the quality of the educational product, as the student is satisfied with the outcome—they get what they were looking for. This point was also expressed by Nikita Podlipsky, Managing Director of Ultimate Education, at a seminar on the prospects of EdTech held by the HSE Institute of Education in February. Educational institutions should consider this aspect to more accurately assess the effectiveness of their courses and student satisfaction.
Arina Pantina noted the importance of measuring changes that occur in students during their studies and after course completion, for example, in the context of their career development. However, such assessment is often challenging, as achieving change requires both time and active effort on the part of students. This underscores the importance of a personalized approach to learning and the need to support students in their professional development aspirations.
Learn more about education by subscribing to our Telegram channel. We regularly share relevant news, helpful tips, and interesting materials. Join us to stay informed about the latest trends and developments in education.
Learn more about our topic. We offer up-to-date information and helpful tips to help you better understand the subject. Our articles are based on verified data and researched materials, making them a reliable source for everyone. We strive to provide readers with the most relevant and useful resources so you can easily navigate the topics discussed. Don't miss the opportunity to expand your knowledge and improve your skills. Continue exploring our content and stay on top of the latest trends.
- What is an online course audit and how can a methodologist conduct it?
- How EdTech is looking for approaches to assessing the quality of online courses?
- The improvement loop: how to improve an already launched online course?
- How to evaluate the effectiveness of an online course?
- How to measure the quality of the educational experience in EdTech. And why do it?