The Strengths and Weaknesses of ChatGPT in Online Course Development

Table of Contents:

• How the Study Was Conducted
• ChatGPT's Strengths for Course Design
• ChatGPT's Weaknesses
• So, Can ChatGPT Be Useful for Course Developers?

How the study was conducted

The research team carried out preparatory work and collected the necessary input data.

• the title of the future course ("Makerspace as a space for learning");
• its short and extended description;
• target audience;
• planned learning outcomes;
• course duration and format.

Based on the collected data, the first prompt for ChatGPT was developed: "Using the backward design model and the provided information, create an outline for a four-week online course for master's degree students. Include expected outcomes for each module, assessment methods, practical sessions, and independent work assignments."

The researchers also developed six additional prompts to deepen each component. For example, you could use the prompt: "Form discussion questions based on the materials in Module X" or "Create five quiz questions based on the materials in Module X, including two multiple-choice questions, two true-or-false questions, and one essay-style question." These approaches help add variety to the learning process and increase student engagement.

Each of the four research team members sequentially entered all seven prompts into ChatGPT and saved their results in documents. A SWOT analysis was then conducted, a method that identifies both external and internal factors related to the object of analysis and assesses their impact, whether negative or positive. The name of the method comes from the English categories by which factors are classified: strengths, weaknesses, opportunities, and threats. This approach is an effective tool for deep analysis and informed decision-making.

• Strengths;

• Weaknesses;

• Opportunities;

• Threats.

ChatGPT Strengths in Course Structure Development

As a result of the analysis, the experts identified several key advantages of ChatGPT. This model demonstrates a high degree of understanding and text generation, which makes it useful in various fields. Among the strengths of ChatGPT is its ability to perceive context, which allows it to support meaningful and coherent conversations. It is also effective in creating original content, which contributes to its use in marketing and educational projects. In addition, ChatGPT quickly adapts to a variety of topics, ensuring the relevance and accuracy of information. These features make ChatGPT a powerful tool for users seeking high-quality text solutions.

• It clearly articulates learning objectives and intended outcomes. When defining objectives and outcomes, course developers typically avoid using words like "know" and "understand"—they don't capture specific student actions, and such outcomes are difficult to measure. ChatGPT also appears to adhere to this approach and uses appropriate wording: "use," "create," and so on. Furthermore, according to the researchers, the neural network suggests tasks and assessment criteria that align with the stated learning objective of each module.

• It offers appropriate tools and materials. According to the researchers, ChatGPT has compiled a high-quality set of tools and materials for use in the course—for example, it has collected resources that will allow students to delve deeper into the topic of educational makerspaces and learn more about methods for conducting creative activities with schoolchildren.

• It formulates discussion questions that encourage critical thinking. The experiment's authors noted that ChatGPT offers insightful questions that encourage students to reflect on what they've learned and connect new knowledge with their personal experiences. Moreover, according to the researchers, these questions will also help instructors critically evaluate course content and highlight the most important aspects of a given module.

• It can reference materials from other modules. The neural network, according to the researchers, automatically references video lectures and texts from previous modules when generating assignment instructions, without any additional prompts. And in discussion questions, he often refers to students’ practical work.

ChatGPT provides course designers with a number of opportunities, the main one being reducing the workload of specialists. The neural network allows for the rapid generation of a course structure, including agreed-upon learning objectives, assignments, and assessment methods. Furthermore, ChatGPT effectively handles the labor-intensive task of searching and selecting information on a given topic. However, as the researchers note, errors are possible, so it is important to manually check the relevance and timeliness of the sources used. Using ChatGPT in course design can significantly improve work efficiency and accelerate the creation of high-quality educational content. By delegating some tasks to a neural network, instructional designers have more time to develop personalized learning. This allows them to implement scaffolding methods for various learning activities and more deeply develop strategies for increasing student engagement throughout the course. Using neural networks helps optimize the workflow, allowing them to focus on creating effective educational solutions and improving the learning experience.

ChatGPT Weaknesses

The experiment revealed ChatGPT's weaknesses in pedagogical design. Key shortcomings include limited ability to adapt to individual student needs, insufficient depth of analysis of complex topics, and a lack of emotional intelligence, which can hinder interaction with students. There is also a lack of critical thinking and original idea generation, which is important for creating high-quality educational content. These aspects may negatively impact the effectiveness of ChatGPT in educational processes.

• The neural network has a poor understanding of context. For example, researchers used ChatGPT to develop the structure of an online course for master's students, but the AI ​​did not take these conditions into account when suggesting specific learning activities. According to the authors, almost all activities required students to be present in person in the classroom. ChatGPT also poorly estimates the time required to complete a given task. For example, the AI ​​gives students just a week to design a creative activity, select a makerspace and coordinate the activity with its management, recruit participants, conduct the activity, and evaluate the results—which, of course, is impossible.

• The selected sources and materials are often unreliable. ChatGPT is known to frequently hallucinate—citing non-existent authors and recommending publications that cannot be found in any database. The experimenters encountered this in this case as well. In their opinion, this is one of the neural network's most serious shortcomings.

• The task instructions lack specificity. High-quality task instructions should be specific and detailed enough for students to easily understand what exactly they are supposed to do. ChatGPT's capabilities weren't very successful in this regard either. As the researchers note, the neural network tends to use vague wording and describe tasks in very general terms.

• Discussions are structured in a monotonous manner. Although ChatGPT, according to the researchers, generates discussion questions efficiently, they follow the same template across all modules. In a real course, such an approach would bore students.

• Lack of depth in test questions and tasks. The researchers noted that the tests generated by ChatGPT only tested factual knowledge—that is, essentially, memorization of information. Such tests can't assess, for example, how students understood a particular principle or whether they can apply it in different contexts. The same applies to tasks that require writing a short essay—the questions for these were also often quite standard and general.

• Frequent failures in answer generation. The authors report that system errors frequently occurred during the generation process—for example, ChatGPT would only produce a plan for two modules instead of the requested four and stop. And after a prompt like "Keep going," I just started over.

Researchers have identified quality control as the main threat in the field of content generation. They believe this problem manifests itself on several levels. Users frequently encounter glitches in the search engine generation process, receiving varied results for the same queries. Furthermore, search results may contain factual errors and links to non-existent sources. This highlights the need for improved algorithms to ensure more consistent and accurate information quality.

So, can ChatGPT be useful for course developers?

As a result of the experiment, the researchers concluded that ChatGPT can be a valuable tool in the initial stages of course design. It allows you to quickly create an initial prototype of the course structure, which can then be refined independently. The neural network also demonstrates its effectiveness in the process of developing educational materials, providing methodologists with ideas for assignments, discussion questions, and additional sources of information. This makes ChatGPT an indispensable assistant in the educational process, contributing to the improvement of the quality of curricula.

The authors of the experiment emphasize that instructional designers must be confident in their subject area to effectively use ChatGPT in the design process. Without in-depth knowledge, it is impossible to identify and correct errors that may arise in AI, as well as refine the course structure and educational materials. It's important for instructional designers to possess sufficient expertise to maximize the potential of ChatGPT and create high-quality educational content.

The researchers raise an important question: is it always advisable to reduce time spent at the initial stages of course design? They argue that when a methodologist independently searches for information sources, selects the most relevant materials, and structures them, this facilitates a deeper understanding of the course topic. This approach allows for the precise formulation of goals and objectives, as well as the connection between them and student actions. Thus, the researchers emphasize the importance of finding a balance between the use of artificial intelligence tools and independent creative activity.