What could be common standards for assessing the quality of online courses: research results

Contents:

• What assessment models did the researchers consider?
• What general conclusions did the researchers draw?
• Why this study was conducted?
• How the study was commented on at the MMCO?
• How the study data will be used?

What Evaluation Models Have Researchers Considered?

The first methods for evaluating digital educational products began to develop in the 1990s, in particular, Douglas Merrill's model, presented in 1995. However, these approaches are outdated, as technology and its role in learning have undergone significant changes. Today, technology not only transmits information to students but also actively influences the learning process, interaction, and student engagement. Modern evaluation models take into account various criteria, such as the effectiveness of material assimilation, content adaptability, user interaction, and the possibility of feedback and support from instructors. These aspects help ensure higher quality learning and better perception of educational content.

• ease of access, use, and updating;
• design (UX and UI);
• the ability to provide students with feedback;
• professional development of the teacher;
• student and teacher satisfaction;
• ensuring student motivation and opportunities for self-education;
• time of inclusion in the educational process.

The researchers analyzed and summarized 22 standards and models for assessing the quality of educational technologies and products available on the global market. These standards were developed both at universities and in associations of educational organizations. They can operate at the level of individual universities, associations of several institutions, as well as at the national and international levels. Among these standards and models, several examples can be highlighted, which are described in detail in the research report.

• The iNACOL standards system, developed in 2011 in the United States to assess the quality of online courses and teaching in the online environment.
• The Quality Matters set of standards was also developed in the United States by a group of universities and is very popular in many states. It can be used at different levels of education and is primarily focused on ensuring the quality of online product development.
• The TELAS model, developed by the Australasian Society for the Use of Computers in Higher Education in collaboration with university professors, is for the assessment and accreditation of online learning in the higher education sector.
• The Online Learning Consortium (OLC) quality metrics system for administering online programs was also developed in the United States. It is aimed at use in both educational and product management activities.
• USDLA - standards developed by the American Distance Education Association.
• International online course certification system EOCCS. It was developed by EFMD - a non-profit international association for management development, which is a recognized worldwide accreditation body for business schools, their programs, and corporate universities.
• The Blackboard Exemplary Course Program assessment system from Anthology - an association of four global companies in the field of educational technologies: Blackboard, Campus Management, Campus Labs and iModules.
• The Sloan-C 5 Pillars of Quality Model. It was developed by Sloan-C - a consortium of universities offering online programs and specializing in the study and improvement of education quality.
• Open Course Quality Assessment Model OSCQR. Developed by SUNY Online, the State University of New York, and partnered with the Online Learning Consortium to become the standard course quality assessment tool.
• A system for assessing the learning effectiveness of online courses developed at a public university in Hong Kong.
• The "A Quality Review for Online Courses" tool, developed in Canada.

The study's authors emphasize that course quality assessment can vary depending on the tool used. It can focus on various aspects, such as course design, learning objectives, content of learning materials, technologies used, and other important factors. This diversity of approaches allows for a more accurate and comprehensive assessment of the effectiveness of the educational process.

Some models and standards assess quality at the product development stage, while others implement assessment throughout the entire life cycle. For example, the model proposed by Mona Ternus and Debbie Faulk, as well as the model developed by Weiyuan Zhang and Yu Ling Chen, serve this purpose. These approaches help ensure high quality at all stages, from concept to operation of the product, which contributes to its longevity and user satisfaction.

What general conclusions did the researchers make?

The authors of the study identified both similarities and differences between the various models and assessment systems. Similarities concern key elements, such as approaches to assessment and the use of standards. However, differences in implementation methods and specific evaluation criteria can impact the final results. These aspects are important to consider when selecting the appropriate model for specific evaluation purposes.

• The methods reviewed offer different assessment frameworks and different levels of indicators. Management criteria are the most common, followed by teaching practices, teacher and student support in third place, and the use of technology and assessment in fourth place. And the most common outcome of online learning is student and teacher satisfaction, not objective learning outcomes.

• Different models may have significantly different goals for developing, adapting, and using frameworks for assessing the quality of online courses.
• Assessment tools used within a single university make it possible to systematize best practices in the development and implementation of courses and include them in methodological recommendations for instructors. However, tools developed taking into account the needs of a specific university often cannot be used in other educational contexts.
• The most popular assessment tool at the national level is the Quality Matters set of standards.

Why this study was conducted

The authors of the study emphasize that the rapid development of information technology and its increased availability have contributed to the emergence of many EdTech projects created without the participation of the state or traditional educational institutions. However, this has given rise to a number of issues related to the quality of educational products, particularly online courses. It is important to address these issues to ensure the effectiveness and reliability of new educational formats.

• There is no single approach to what constitutes quality. It can be assessed differently in different educational contexts—and in practice, quality is assessed differently by different companies.
• Private educational projects rarely conduct research to select the optimal solution in terms of quality.
• If they do conduct research, they typically focus on data on user engagement and feedback. Half of the companies use exclusively internal data sources for analysis, which is not very objective. And few examine the effectiveness of products in the context of measuring educational outcomes.

During a discussion at the Moscow International School of Economics (MISO), Kirill Savitsky, leading expert at the Laboratory of Digital Transformation of Education at the Institute of Education at the National Research University Higher School of Economics, emphasized that 51% of companies are confident in the availability of data confirming the effectiveness of their educational products. However, actual figures show that only 7% of them are able to provide objective evidence to support their stated results. This highlights the importance of robust analytics in education and the need for greater data transparency to improve the quality of educational decisions.

Study additional materials:

Measuring the quality of educational experience in EdTech is an important task for improving the effectiveness of educational processes. This allows us not only to evaluate the effectiveness of educational programs but also to identify areas for improvement. A high-quality educational experience encompasses various aspects, such as user-friendliness of the interface, accessibility of materials, interactivity, and student engagement.

Several methods can be used to measure the quality of the educational experience. Surveys and questionnaires help collect user feedback, while data analytics allows you to track student behavior and progress. It is also important to benchmark against other educational platforms to understand where you can improve.

Why measure the quality of the educational experience? Firstly, it helps increase user satisfaction and engagement. Secondly, the data obtained helps develop more effective educational strategies and tailor content to student needs. Thirdly, a high level of educational quality helps strengthen the platform's reputation and attract new users.

Therefore, regular measurement and analysis of the quality of the educational experience in EdTech are necessary for creating a competitive product and ensuring successful learning.

How the study was commented on at the Moscow International Conference on Educational Standards

At the Moscow International Conference on Educational Standards, experts discussed important reasons for the need to develop uniform standards for evaluating digital educational products. The challenges the market faces in implementing these criteria were also discussed. The discussion covered key aspects that influence the development of evaluation criteria and identified barriers hindering the effective development of this field. Lyudmila Kevdina, Head of Quality at RATIO and Director of External Communications at Maximum Education, emphasizes that the EdTech market is currently experiencing active growth, with a steady increase in the number of online schools. These educational institutions vary in size and types of products offered, making it difficult for users to choose the right option. A lack of understanding of the selection criteria can lead to disappointment with the results, which, in turn, creates the risk of conflict in the information space. Last year, we witnessed significant events that attracted public attention. When such stories come to light, regulators begin to carefully analyze them. In the current situation, it is important to develop a criteria model that will allow clients to clearly understand how to evaluate and select educational products. Transparency in this process is essential for all participants in the educational process and the system as a whole.

Lyudmila emphasized that not only RATIO, but also other market players, are currently striving to develop methods for assessing the quality of educational products. She believes this is a positive trend. The availability of different approaches will allow them to be compared and ultimately create a single universal assessment model applicable to educational products as a whole.

Kirill Savitsky noted that the main problem of EdTech companies is the use of an excessive number of different metrics to assess the quality of educational products. This leads to confusion and complicates the objective assessment of the effectiveness of educational solutions. Instead of focusing on a few key indicators, companies often process information, which complicates the decision-making process and reduces transparency. Effective assessment of educational products requires simplified metrics and a clear understanding of their impact on learning outcomes.

When we have large amounts of data, humanities specialists who have studied analytics strive to understand and evaluate it. As a result, they develop a multitude of indicators—between 90 and 100—each of which they would like to include in the evaluation model. However, an important question arises for specialists with a technical background: if, for example, the first indicator has a weight of 40 percent, the second 30 percent, and the third 20 percent, how much is left for the remaining 87 indicators? And should they really be considered? Effective use of data requires not only collection and analysis but also an understanding of the significance of each indicator in the overall model. This underscores the importance of high-quality analytics and the correct approach to data interpretation.

Marina Kharakhordina, Head of the Methodology Department at Skillbox, noted that EdTech companies are actively seeking criteria for assessing the quality of educational courses. For example, several years ago, the Higher School of Economics created a tool for assessing the extent to which graduates achieved their goals. Many major market players have adopted this tool and discussed whether it is the only way to assess quality. As a result, it turned out that there are many approaches to evaluating educational programs, which emphasizes the importance of an integrated approach in this area.

Now Smart Ranking in collaboration with The "Pachka" messenger is conducting a new study, in which industry experts are once again sharing their opinions on important metrics for assessing quality. According to Daria, the survey results showed that the metrics system has become significantly more complex since the first study. While previously the focus was on standard indicators, now each participant has their own unique approach and methodology. This underscores the importance of individualization in quality assessment, which can significantly impact business development and performance.

All participants are genuinely interested in a unified methodology, allowing them to compare their results with others. We are confident that the development of a universal approach will become an important benchmark for both the entire online education market and for online school clients. This will help improve the quality of education and increase trust in educational platforms.

Nikita Podlipsky, Managing Director of Ultimate Education, raises the important issue of the need for a unified methodology for measuring educational quality. He agrees that specific criteria for evaluation are necessary, but notes the difficulties that arise in creating a universal model. For example, student completion rates at online schools can vary widely: at one school, it can reach 90%, while at another, it's only 25%. This doesn't necessarily indicate poor quality of education at the second school, as for some students, completing the course and earning a diploma may not be as critical. In some professions, a diploma isn't a mandatory requirement for employment. Therefore, according to Podlipsky, attempts to create a unified system for assessing the quality of education can lead to incorrect conclusions and fail to take into account the individual characteristics of various educational institutions.

Check out additional materials:

In 2024 The EdTech market will continue to evolve, driven by new technologies and the changing needs of educational institutions and students. Key trends are expected to include artificial intelligence, which will be increasingly used to personalize learning, and adaptive educational platforms, which will help accommodate individual student needs. Virtual and augmented reality will also play a more prominent role in educational processes, providing new opportunities for interactive learning.

Furthermore, we will see the growing popularity of online courses and microcourses, which offer more flexible learning opportunities. Educational institutions will strive to integrate these solutions into their programs to meet the demands of the modern labor market. Importantly, an emphasis will be placed on accessibility and inclusiveness of education, which will expand the audience and make learning more accessible to everyone.

Thus, the EdTech market in 2024 promises to be dynamic and innovative, providing new opportunities for both educational institutions and students seeking high-quality education.

How the research data will be used

The ultimate goal of this large-scale project, which explores approaches to assessing the quality of educational products, is to create a quality standard for online courses for the Russian EdTech industry. The model is expected to be developed by the end of 2024 and will include several key components for assessing the quality of educational programs. Specific evaluation criteria will be defined for each of these components. Leading experts in online education in Russia are planned to be involved in the model validation process, which will ensure a high level of reliability. After testing is complete, the model will be presented to a wide audience. The standard will consist of four main blocks: business process organization, product development, product implementation, and its market position, each of which will contain its own set of criteria. This approach will ensure a high level of quality for online courses and increase their competitiveness in the market.

This approach will allow users to choose programs that best suit their needs, while course creators will be able to develop relevant and in-demand educational programs that fully meet student needs. This approach contributes to improved educational quality and increased student satisfaction, which in turn impacts the success of courses and their market popularity.

The development of quality standards, as Marina Kharakhordina emphasized at a recent discussion, could lead to the creation of an online course navigator. This online platform will allow users to easily find courses using clear and understandable criteria. In turn, developers of educational products will be able to take these criteria into account when creating their courses, which will improve the quality and relevance of offers to user needs.