Contents:
Training with a guarantee of employment: "Methodologist - from beginner to professional"
Find out moreThis fall, a group that combined the efforts of the Skolkovo EdTech hub, Skypro University, the Smart Ranking research agency, and the EdYes analytics company, proposed to the leaders of the online education sector the need to develop a kind of code for This code should include clear criteria for assessing the quality of continuing professional education (CPE) with the goal of improving students' educational achievements and their competitiveness in the labor market.
In early October, Skolkovo hosted a conference and open workshop on "Criteria for Assessing the Quality of Continuing Professional Online Education." During these events, participants identified twelve metrics for assessing the quality of continuing education in an online format.
Twelve proposed indicators of the effectiveness of online education for analysis:
- referral ability, the propensity to recommend a course;
- analysis of the degree of satisfaction with various components of the educational process;
- conversion coefficient/success rate ― whether the graduate managed to achieve the set educational goals;
- COR, the audience retention rate, and the completion rate.
- the degree of employer confidence in a particular educational institution or course program;
- student return on investment (the time required for the training to justify its costs);
- the significance of the course content;
- Cooperation of educational institutions with business representatives and the employment process.
- Analysis of changes in student salaries over time.
- the share of attendance in the current moment;
- the presence of a safe space in the course and an environmentally friendly atmosphere in communication;
- creation of a favorable atmosphere, involvement of specialists and alumni.
As part of the discussion, the School of Education organized a meeting with experts in the field of educational technology to explore a less obvious aspect related to the quality of the educational process. Typically, such conversations emphasize learning outcomes, such as obtaining a new specialty or improving qualifications in the context of continuing professional education. However, this time the participants focused on the additional effects of the educational process: they found out whether the student received insights that were not included in the program, how comfortable they felt throughout the course, and whether it was possible to create an active community among students.
Undoubtedly, assessing these indicators is a complex task. But is it really worth striving for, and if so, for what purpose? We invite you to hear the opinions of experts on this issue. The full video recording of the discussion on the topic "Metrics of the quality of the educational experience, not the result: should it be assessed and how to do it?" can be found on YouTube.
Key Elements that Determine the Quality of the Educational Experience
Asya Fursova, Dean of the School of Education and moderator of the meeting, proposed starting with identifying the components that make up a student's educational experience, and also finding out which of these elements influence the perception of the quality of education. The discussion participants highlighted several key aspects.
Mikhail Sverdlov, Director of Business Development at Skypro, opened his discussion by stating that most people extremely rarely set specific educational goals for themselves. He cited the results of an internal analysis conducted at the Skyeng group of companies, which showed that a significant number of adult learners are unaware of the purpose for which they begin learning, for example, English.
Only 20% of Skyeng students are able to clearly articulate their learning goals, while 60% of them lack a clear understanding and rely on general questions: "Perhaps I need this for work, study, or something else?" They answer: "Yes, of course, I also want to watch Netflix in English." In fact, we have a good idea of which of them actually use English for watching Netflix. Furthermore, 20% of students conclude their conversation with the methodologist during the introductory lesson with the phrase: "Listen, can't we just pay and start learning, and then we'll figure it out?"
According to Mikhail, there are certainly those who set clear goals for themselves, such as learning English for emigration or improving their qualifications after leaving their job. However, this is more the exception than the rule. The speaker emphasizes the importance of paying attention to how consciously a person approaches their educational program selection. In this context, the student attrition rate, or return rate, can be a useful metric, demonstrating the number of students who realized they hadn't chosen the right course and decided to leave.
Marina Litvinova, Product Director and Head of the Guild of Methodologists at EdMarket, shared her experience with student attrition from a course training technical administrators for online schools. It turned out that one of the main reasons students leave the course is incorrect expectations. The curriculum includes a significant technical load, requiring students to independently master certain tools. Many of them were not ready for such requirements.
EdMarket methodologists have made changes to the program of an open promotional webinar, the purpose of which is to familiarize with The technical administrator profession. Information is now presented more clearly, helping potential students better understand what they will be doing. The introductory lesson included a detailed discussion of the tasks students will face, including specific examples. These measures allowed us to more effectively filter out students who don't meet the course requirements, prepare students for the upcoming course load, and provide them with the necessary guidance.
Marina Litvinova highlighted an important issue that arises during group learning: students with completely different goals can be in the same group. Some genuinely want to master a new profession and gain a deeper understanding of the subject, others are looking for a quick way to earn money to pay off their loan obligations, and still others come to the courses for the social interaction. This situation raises a number of questions: how to effectively support the learning of such a diverse group, how to organize individual support from mentors, and what criteria should be used to evaluate the educational experience of people with different goals.
Therefore, EdMarket uses student feedback to assess their satisfaction with the support service. In addition, such indicators as NPS, intermediate and final COR, as well as other metrics are assessed at each stream.
Read also:
Evaluating the effectiveness of an online course is an important task that allows you to understand how successfully it achieves its educational goals. There are several approaches and criteria for analysis.
First, it is worth paying attention to the level of participant satisfaction. Surveys and reviews can provide valuable information about how well the course meets student expectations, how accessible the material is, and how comfortable the learning environment is.
Second, it is necessary to assess knowledge acquisition. For this, you can use tests and assessments that will help determine how successfully students have mastered the material. Analyzing the results of such assessments will help identify the strengths and weaknesses of the course.
In addition, it is worth considering student engagement. Active participation in discussions, completing assignments, and interacting with the instructor can serve as indicators of student interest and motivation.
An equally important aspect is the analysis of long-term results. Assessing how acquired knowledge and skills are applied in real life will also provide an idea of the course's effectiveness.
Finally, it is worth considering technical aspects, such as the stability of the platform, the availability of materials, and the quality of video lessons. All these elements combined help create a complete picture of the effectiveness of online learning.
Maxim Skryabin, Director of Education at PostNauka.Academy, emphasizes that for people starting out in a new profession, it is crucial to be surrounded by like-minded individuals. Marina Litvinova supports this statement, noting that student interaction during training—whether in a small group or project team—provides experiences unavailable in individual lessons and promotes the development of soft skills.
Mikhail Sverdlov emphasized the importance of communication and support from mentors and fellow students for students.
"I am delighted to observe the activity in our community chats on Slack and the Miroboards used by our students studying in the ISA bootcamp program. This interaction has an extremely positive impact on their motivation. The fact is that they are accustomed to a learning format where one teacher leads many students, perceiving themselves not as individuals, but as part of a class or group. However, our mentors, like those at EdMarket, provide each of them with individual support and personalized feedback. By working in teams and observing each other's successes, students receive additional incentive and motivation for further development."
Asya Fursova noted that the OECD Learning Compass 2030 analytical review highlights key global trends in education, such as well-being, which implies a sense of security and well-being, and sustainability. The importance of aligning the values of students and other participants in the educational process is becoming increasingly significant.
"How might this manifest itself in education? A simple example: if everyone is given plastic bottles, then a person with a highly developed environmental consciousness will perceive this as something unpleasant and illegal. At the same time, if the educational process is designed with an emphasis on the well-being of all its participants, this is immediately noticeable. For a student, this can become an important metric related to values."
Methods for Assessing Educational Process Indicators
When it comes to parameters that play a key role in the educational process, a logical question arises: is it possible to evaluate them in any way? Experts have identified several methods for accomplishing this task.
To assess the development of interactions among students, measurable parameters such as the speed of response to questions in chats, the number of discussion participants, and the ratio of useful messages to the total number of students can be used. Mikhail Sverdlov shares this opinion. The expert believes that such indicators can be tracked on any platform where communication between students takes place.
Marina Litvinova noted that individual groups such as "Who else is taking the psychology course?", "Are there any guys from St. Petersburg among us?" or "Any newcomers?" According to the speaker, such small communities can also serve as an indicator of the activity of the entire group.
"By observing the emergence of a particular chat, we can count how many such groups have formed from the main educational chat. Then, if we have the opportunity to enter these groups so as not to disrupt their atmosphere, we can analyze how small mastermind groups consisting of 5-7 participants are forming. It is important to find out how often they meet, how active they are in discussions, how many posts appear daily, whether they are open to new members, and whether their small community is growing."
Maxim Skryabin noted that students' activity can be measured not only by their chat conversations but also by any collaborative activities. For example, this could be demonstrated in their ability to self-organize and regularly hold online meetings via Zoom.
He also noted that analyzing community metrics is part of social learning analytics—a field that studies quantitative aspects of human interaction, such as the density of connections. Other sociological approaches can be used to assess the experience of an individual student, for example, methods related to the analysis of the ego-network, which is a set of social connections of a particular individual.
Read also:
Observational learning is the process by which an individual acquires new knowledge or skills by observing The actions of others. This is a form of learning based on perception and imitation, allowing people to master behavior without direct practical experience. This approach is particularly relevant in the context of social interactions, where interaction with others plays a key role. Learning is viewed as a social process because it occurs in the context of interactions between people. Social aspects influence how and what we learn, as our environment, cultural norms, and interactions with others shape our cognitive activity. Thus, learning is not an isolated event, but is closely linked to the society and culture in which we find ourselves.
While the metrics of social interactions are generally quite clear, the question arises of how to assess the personal experiences of students, especially in the context of their values, comfort levels, and other individual factors.
Marina Litvinova emphasizes that the results of learning and the accompanying changes in personality can be unexpected.
"This event occurs unexpectedly for the person himself: he comes to study and suddenly begins to reconsider his life priorities, realizing that he has developed other interests. He begins to value himself more, may decide to quit his job or leave a toxic relationship that negatively affects his life. Perhaps he finds the courage to go to an interview. Such changes related to personal growth can have a positive impact not only on his own life, but also on the lives of people around him, including his family.
Read also:
Learning as a means of transformation: what is transformative learning for adults.
Certainly, quantitative Assessing such changes is a complex task. However, as Marina asserts, such personality transformations can be reflected in students' responses to questions like "Would you recommend us?" and "How did you enjoy the training?" These responses form the basis for the key CSI and NPS metrics.
To track and influence these parameters, EdMarket employs feedback and reflection methods. Specifically, a few weeks after the start of the course, students are asked to complete a questionnaire on their motivation. The questions in this questionnaire are aimed at identifying students' satisfaction with the classes, factors that facilitate or hinder learning, self-support strategies for learning, and the opportunities offered by the support service.
At the end of each module, students are given the opportunity to reflect on what they would like to praise themselves for. The final reflective stage takes place during the defense of their theses. At this point, graduates are asked to share their thoughts about how they felt at the beginning of their studies, what changes have occurred since then, and what plans they have for the future.
Maxim Skryabin confirmed his point of view on the importance of reflection, relying on scientific research. He noted that data shows that students who use reflective approaches in the learning process perform better on tests compared to those who do not use such methods.
Read also:
How can a course that is already in progress be improved?
Implementation?
Why is it important for companies to track and improve educational experience metrics?
Creating a community, transforming a student's personality, and their satisfaction with the support they receive during the learning process are all certainly important. However, the question arises: do these aspects have practical value for business? Why should EdTech companies evaluate these metrics and strive to improve them?
In practice, as demonstrated by expert opinions, there is a close relationship between educational process indicators and learning outcomes. Mikhail Sverdlov gave an example from Skyeng, which illustrates that the level of student comfort has a direct impact on both their academic achievement and the company's financial results.
The company found that students at the beginning level of English learning drop out of their studies significantly faster than their more experienced peers. On average, beginners completed only 15 lessons, while more advanced students completed between 28 and 50. To understand the reasons for this, Skyeng surveyed students and found that they experienced difficulties and discomfort during the learning process due to high expectations from teachers. Thus, students' personal experiences were reflected in specific business metrics.
As a result, the company created a new program, revising its approach to teaching. The quality of education remained unchanged, and the number of students leaving the institution decreased: just a few months after the launch of the updated course, the average number of lessons for new students increased to 23.
Marina Litvinova shared a similar experience. After the first wave of the COVID-19 pandemic and the introduction of restrictive measures, many people with no technology background joined EdMarket courses. Even using online editors presented certain difficulties for them.
"Google Docs" became the first major hurdle. Many people struggled with their first homework assignment. This was especially true for adults, such as teachers in their 40s and 50s, as they find it difficult to ask for help. We invested significant effort in developing support instructions and additional materials for the introductory lesson to ease users into the experience and help them master the platform. Although this seemed like a purely technical issue, many felt intimidated by even asking questions. Creating a certain atmosphere where there was no shame in asking questions and fear of mistakes was extremely important, because everyone understood that learning occurs through mistakes.
Support Marina developed approaches for working with students who lack basic digital skills. Additionally, at the beginning of each course, students are given the opportunity to master basic digital literacy principles. This has led to a decrease in the number of students who drop out due to initial difficulties, fear of asking questions, and self-doubt. According to Marina, such problems can only be resolved through personal interaction and empathy.
The Ideal Educational Experience in the Modern World
Almost two years have passed since the world faced the COVID-19 pandemic, and restrictions have entered our lives, creating an atmosphere of uncertainty and anxiety. How do these conditions affect the educational process of students in continuing professional education, and what measures can the company take to improve it?
Marina Litvinova rightly noted that even before the pandemic, students could experience personal circumstances that interfere with their studies, such as heavy workloads, health issues, or unexpected situations. However, such cases were the exception. Currently, students are facing more frequent illnesses, post-COVID consequences, and psychological exhaustion, yet they maintain a desire to continue their education despite difficult conditions. According to the expert, this creates significant challenges for EdMarket's support team: it is necessary to adapt the curriculum, develop ways to integrate the educational process into the student's life, or discuss possible timelines for their return to classes.
Marina noted that such issues are resolved individually, often through in-person meetings, as it is sometimes difficult for a person to accurately convey the essence of the problem through correspondence. Care team mentors share examples and methods with colleagues for working in various situations, but there are no universal rules for such cases within the company—there are too many details and nuances.
The specialist emphasized that employees need support in the face of high information and psychological stress. To this end, the company organizes meetings with a psychologist. EdMarket is currently exploring the possibility of providing similar assistance to students who have faced difficult circumstances.
Maxim Skryabin noted that at present There has been a significant increase in interest in psychological support. He also mentioned that one of the most popular courses on the Coursera platform in 2020 was "The Science of Wellbeing." Maxim emphasized that this focus on mindfulness and self-regulation influences the development of educational programs.
"When discussing the formation of an educational environment, it is important to strive to create the most comfortable and safe space possible, where people can freely make mistakes while feeling supported. After all, in recent years, especially after the pandemic, some aspects can be perceived much more acutely."
Experts unanimously agree that support functions in education—such as tutors, mentors, and curators—are becoming increasingly important. The successful functioning of support and guidance services directly impacts business results. While objective indicators can reveal shortcomings in the educational process, only through personal, trusting communication can we gain a deeper understanding of the causes of these problems and develop effective solutions.
Read also:
- Digital courses on the concept of well-being can actually significantly contribute to improving psycho-emotional well-being.
- The methods for motivating adults to learn can differ significantly from the methods used for children. To successfully motivate them, it is necessary to consider their life experiences, needs, and goals. It is important to create conditions that allow them to recognize the value of knowledge and skills.
First, it is necessary to set clear goals and objectives. Adults want to understand why they are learning and how the knowledge they gain will apply to their lives or careers. Therefore, it is important to emphasize the practical application of the material being studied.
Second, individual interests should be taken into account. Learning will be more effective if it is linked to personal hobbies and professional ambitions. This way, you can choose topics that will genuinely interest students, rather than being imposed from the outside.
A third important aspect is creating a comfortable and supportive atmosphere. Adults often experience fear of failure or judgment, so it's important to provide an environment where they can freely exchange opinions and ask questions.
Furthermore, it's worth using a variety of learning formats. Blended methods, such as online courses, seminars, and group sessions, can make the process more varied and engaging. It's also important to encourage independence and initiative, allowing students to choose their own pace and approach to learning.
Finally, regular feedback and support from teachers and colleagues can play a key role in increasing motivation. Recognition of successes and achievements stimulates the desire to move forward and overcome new obstacles.
- Five methods for increasing the completion rate of online courses.
- Excerpts from the month regarding the level of education and skills of teachers.
- Will higher education give way to professional courses?