EdTech: New Methods for Assessing the Quality of Online Courses

Contents:

• Methods for evaluating the effectiveness of online educational programs
• For whom and in what circumstances can EdTech be most effective?
• Graduates: how does their employment reflect their level of education?
• Should EdTech employ rigorous student selection, like in universities?

This fall, the quality of additional online education became the subject of active discussions. It all began with a debate on quality indicators organized by the Skolkovo EdTech Hub, Skypro University, the research agency Smart Ranking, and the analytics firm EdYes. As a result of the conference and workshop, the participants identified 12 key criteria that characterize the quality of EdTech education.

12 metrics that can be discussed to assess the quality of online education:

• referral potential, desire to recommend this course;
• analysis of the degree of satisfaction with various components of the educational process;
• conversion coefficient / success rate ― was the graduate able to achieve the educational goals;
• COR, audience retention rate, and completion rate.
• the degree of employer trust in a particular educational institution or course;
• return on investment for the student (how quickly he can justify the costs of his education);
• the significance of the course content;
• cooperation of educational institutions with employers and the employment process.
• change in student income level over time.
• the current share of visits;
• the existence of the program a safe environment and sustainability in terms of interaction;
• the presence of a favorable atmosphere, the inclusion of specialists and graduates.

The School of Education approached this topic from a new perspective and engaged experts to discuss additional aspects not directly related to the curriculum. The focus was on elements such as changes in students' personalities and their sense of comfort, the creation of an active community, and other factors. All of these elements together shape the quality of the educational experience.

At the November Yandex conference, known as Yet another Conference on Education (YaC/e), topics related to the quality of online education were also discussed. What are the key aspects to consider when assessing the quality of education in EdTech, especially in the context of continuing professional education? In what directions should EdTech companies move to achieve success?

As part of the discussion, conducted under the leadership of Daria Ryzhkova, founder of the Smart Ranking agency, Alexander Laryanovsky, Managing Partner of Skyeng, and Dmitry Krutov, Founder of Skillbox, presented their expertise. A recording of the event can be found on YouTube, and we present the key points made by the experts.

Methods for Assessing the Effectiveness of Online Educational Programs

Alexander Laryanovsky notes that the online education market in Russia has reached a stage where we can talk about the real quality of EdTech products—this indicates that the initial "foam" has subsided. In this context, the expert identifies two key parameters. The first is the degree to which students' expectations match their learning outcomes. It is important that these expectations are based on conscious understanding of desired outcomes, shaped by previous educational experience. "An example would be if we went into a coffee shop, and although we might not know how to make coffee, we can still evaluate how well our expectations matched what we were served," he explains.

Alexander highlights the share of academic costs in a project's economics as the second key aspect of its quality. This means that a truly high-quality product should be characterized by the fact that, under the same parameters, the majority of expenses are directed toward content creation, as well as toward remunerating qualified teachers and methodologists, as opposed to spending on marketing and design.

Dmitry Krutov expressed doubt that the EdTech market has reached a stage where the issue of online education quality has become critical.

"I believe we are still in an era when the market is oversaturated with content, and the process of its digitalization will continue for another five years. We will be digitizing a huge number of materials, some of which will be of high quality, while others will not. In about three to five years, a serious discussion of quality issues will begin. <…> More companies will emerge, and the market will become more diverse and multifaceted."

Nevertheless, as Dmitry notes, an important characteristic of quality assessment in EdTech is already evident: it is focused on the product itself, rather than on the needs and expectations of students. Existing metrics primarily assess the work of producers or methodologists, but do not provide information on the extent to which the course met the needs of those who completed the training. The head of Skillbox believes that the EdTech industry should strive to develop customer-focused metrics. However, this raises the question: how can we determine whether a course meets students’ expectations if only a small portion of them make an informed choice of an educational product?

When we analyze our usual Looking at metrics like NPS, CSAT, and COR, you can see that they generally reflect our effectiveness. However, right now, it's crucial to focus on customer-focused metrics that will help us understand how accurately people have set their expectations. Essentially, NPS, CSAT, and COR demonstrate that our methodologists and producers have done an excellent job, creating high-quality content, selecting a successful storyboard, and using modern, expensive equipment. However, I believe we need to shift our focus to other metrics—those that reflect the voices of learners and students—and engage in active discussions around these new metrics.

For whom and in what circumstances can EdTech be most effective?

People planning to begin training may often have very vague ideas about its essence and the benefits it can bring.

Dmitry Krutov emphasizes that the opportunity to carefully examine and try out various training options plays a key role in setting realistic expectations. He cited a common example of someone signing up for a Python programming course, seeking to change their profession to a higher-paying one, and only during the course of the training does they realize that they would actually be much more interested in becoming an interior designer. In such cases, Skillbox offers the opportunity to change the curriculum and hosts open webinars discussing various professions and career guidance courses.

Alexander Laryanovsky, while acknowledging the need to help clients shape their expectations, emphasizes another key element. He points out that the EdTech industry can effectively engage only with a motivated audience, but this audience remains extremely limited. "The number of people interested in self-development and who have the time and resources to learn is <…> small."

Most adults work beyond their means, striving to earn the median salary of 30,000 rubles per month. According to Alexander, all of them completed school education, and many of them continued their education at universities, but they perceive the knowledge they acquired in educational institutions as of little use. Nevertheless, according to the expert, for these people, acquiring professional skills can be a chance to develop and improve their standard of living.

"Undoubtedly, according to official data, about one-third of the population lives below the poverty line. When we discuss the issue of quality, we must consider not only those highly motivated people who are easy to communicate with and who absorb information, but also those who find themselves in more difficult circumstances. It is important to be able to communicate with them in a language they understand and achieve positive results in this environment."

Alexander went into more detail about how EdTech companies should learn to attract a wider audience and find ways to interact with it, instead of Focus exclusively on resourceful and motivated users, in his Facebook post. The main idea of ​​his reasoning is that in the long term, success will be achieved by those EdTech representatives who can measure their achievements not only by increased income, but also by the actual number of trained people who were able to achieve significant changes in their lives thanks to the knowledge they acquired.

Graduates: How does their employment reflect the level of education?

Dmitry Krutov emphasizes that, in matters of continuing education and subsequent employment, in addition to professional skills, an important aspect is a person's ability to adapt to the atmosphere and culture of both the profession itself and a specific company.

There are two key components in the candidate evaluation process. The first is to check compliance with hard skills - that is, the basic skills and knowledge necessary for performing a specific job. The second part is more cultural. Here, it's important to assess how the applicant's personality fits into the corporate culture, as well as the state of their soft skills. For example, will they be able to adapt and become part of a team at companies like Yandex, Skyeng, or Skillbox? And how can we influence this aspect of the assessment?

Dmitry emphasized the importance of creating an educational environment that helps students immerse themselves in the context in which they will work in the future.

Many people like to divide education into online and offline formats. However, I believe that neither we nor our colleagues focus exclusively on online education. We work with hybrid learning formats, where the key task is to create a fully-fledged educational environment. This is achieved through organizing conferences, meetups, workshops, seminars, as well as in-person review of homework assignments, diploma projects, internships, and other events.

During the conference, one participant raised an interesting question live on air: "Why are employers often distrustful of candidates with online course diplomas?" This sparked a lively discussion. In response, Dmitry Krutov pointed out that in 2021 alone, Skillbox helped more than five thousand of its graduates find employment, including representatives of well-known companies. This calls into question the claim of employer distrust. Moreover, employers always have the option of requesting graduates' final projects completed during their studies to verify the quality of the knowledge acquired.

Alexander Laryanovsky, answering a similar question, emphasized two key points. First, he noted that the reputation of the formal vocational education system, including universities, leaves much to be desired, especially when it comes to institutions outside the top tier. Second, he highlighted the existing stereotype that the state is the country's primary employer. Indeed, it is very difficult to get a job in government agencies or companies without a higher education, namely a university diploma - course diplomas are not considered in this context.

Alexander also noted that People are attracted to education for more than just finding a job. He emphasized that education can trigger releases of dopamine and serotonin, which, simply put, brings the joy of achieving something through one's own efforts. While it's difficult to determine exactly how to measure these hormones, the enjoyment of the process is nonetheless noteworthy. If a person becomes even slightly happier, then the effort expended was definitely not in vain.

Should EdTech employ rigorous student selection, like universities?

During a discussion on the divide between higher education institutions and EdTech, moderator Daria Ryzhkova suggested that a truly high-quality education may not be accessible to everyone without exception. As an example, she cited the words of the rector of a private university: "We have a very strict selection process, training is difficult, and the thesis is a real test. As a result, very few students remain, but I am confident that they become highly qualified specialists with all the necessary knowledge."

Alexander Laryanovsky commented on this situation, pointing out that student selection is not a blessing, but rather indicates problems at universities that are unable to scale. He also noted that educational institutions interact with goal-oriented people, but in reality, education as a means to change lives is needed by entirely different categories.

Of course, it's easy and pleasant to support those who have achieved success. Of course, various selections can be conducted among them, similar to how the best athletes are selected in professional sports. However, let's still try to make the whole country a little healthier.

Dmitry Krutov noted that many university graduates begin retraining in EdTech after completing their studies. He believes that countering this phenomenon is inadvisable. Instead, it's worth focusing on creating partnerships between universities and EdTech companies, where each party could solve its own problems.

Read also:

• The most successful EdTech startups in Russia and the CIS in 2021.
• In the field of educational technologies, there are tools that are not fully represented in our market.
• There are five key issues regarding the vocational education system and its interaction with the labor market.

  First, one of the most pressing problems is the mismatch between educational programs and the real needs of employers. Educational institutions often fail to adapt to rapidly changing market demands, which leads to a shortage of qualified specialists in certain fields.

  Second, the low level of practical training of students should be noted. Many educational institutions focus on theoretical knowledge, which does not always enable graduates to successfully cope with real-world work tasks.

  The third problem is related to insufficient funding for vocational education. Limited resources often prevent schools and colleges from updating equipment, implementing modern technologies, and attracting qualified teachers.

  The fourth difficulty is the lack of a clear system of career counseling and support for students in their job search. Many graduates do not know how to effectively present their skills and obtain the desired position.

  Finally, the fifth problem is the low level of interaction between educational institutions and businesses. Weak ties hinder the exchange of experience and knowledge, which in turn complicates the training of specialists who meet current market demands.

• In recent years, there has been active interaction between higher education institutions and commercial organizations in the continuing professional education (CPE) market. Universities, possessing a scientific base and pedagogical resources, offer a variety of programs that allow people to improve their qualifications and master new specialties. On the other hand, companies strive to tailor education to their needs, offering practical courses and training that meet current labor market demands.

  This segment of education is divided into several areas. Universities typically focus on the theoretical aspects of knowledge and laying the foundation for further professional growth. They develop curricula that include both general subjects and specialized courses aimed at training specialists in specific fields.

  Companies, in turn, strive to use their resources and employee time as efficiently as possible. They often offer short, intensive programs that emphasize practical skills and current industry trends. Thus, organizations often partner with educational institutions, creating joint courses that address the needs of both students and employers.

  This collaboration allows universities to attract more students, and companies to provide their employees with relevant knowledge and skills. As a result of this symbiosis, both parties benefit and contribute to the development of the professional environment as a whole.