Code

Why Machine Learning Models Don't Make it to Production: 3 Reasons

Why Machine Learning Models Don't Make it to Production: 3 Reasons

Free Python course ➞ A mini-course for beginners and experienced coders. Four cool projects for your portfolio and live chat with the speaker. Click and find out what you can learn in the course.

Learn more

Senior Data Scientist at Rosbank, teacher of Data Science courses at PyMagic and author of the YouTube channel miracl6. With deep knowledge in data analysis and machine learning, I strive to share my experience and train new professionals. My YouTube content covers various aspects of data science, including practical examples and tips for applying modern technologies to real-world projects.

Anastasia shares her experience in machine learning, her career as a developer, and development opportunities in data science. She covers key aspects of this rapidly growing field, including the skills needed for a successful career and the various paths specialists can take. Anastasia also shares personal stories and recommendations to help beginners and professionals develop in data science.

According to research by consulting firms Gartner and McKinsey, 50-80% of machine learning models do not reach the production stage. This means that a significant portion of projects developed by data scientists remains untapped. We discussed the three main causes of this problem and how to solve it with Anastasia Nikulina.

For reference: what is a machine learning model and production?

A machine learning model is an algorithm that processes data and transforms it into valuable information. This process involves analyzing and identifying patterns in large volumes of data, which allows for predictions and informed decisions based on the results. The use of machine learning models is finding application in various fields, including finance, medicine, marketing, and technology, helping to optimize processes and improve operational efficiency.

At a steel foundry specializing in the production of metal for spacecraft, technologists spent years experimenting with various additives. They recorded the results of their research to optimize the production cycle and achieve the required alloy quality. This process requires high qualifications and in-depth knowledge of materials. Creating a reliable and durable metal for the space industry is critically important, as the safety and effectiveness of space missions depend on it. Continuous improvement of technologies and production methods allows the plant to remain competitive in the market and meet the growing demands of the space industry. Optimization of the production cycle is made possible by the implementation of a machine learning model. This program is capable of automatically identifying combinations of more cost-effective additives and predicting their compatibility. Reducing production costs by even 1% will result in significant savings for the plant, potentially amounting to millions of rubles. Using machine learning in production not only reduces costs but also improves process efficiency, which is an important step toward the company's sustainable development. Manually mixing new additives instead of modeling can result in incorrect proportions, leading to deviations from standards. As a result, the resulting batch of metal will be used to smelt shovels instead of producing spacecraft. This will negatively impact the plant's financial performance and lead to losses. Therefore, it is important to strictly adhere to technological processes and proportions when using new additives.

A data scientist cannot simply take random data and create a working machine learning model based on it. The model development process involves many steps and requires a deep understanding of the data, its structure, and context. This complex cycle includes data collection, cleaning, and analysis, algorithm selection and tuning, and model testing and validation. We will discuss this process in more detail in future materials.

It is important to understand that until a model is trained on a large set of statistically significant data, it cannot be implemented in production. In our case, this means that such a model will not be able to effectively optimize metal production processes at the plant. Without high-quality training, the program will produce incorrect calculations and will not be able to provide real savings. Training on adequate data is key to achieving high results and implementing a system into workflows.

In this article, we will examine the main obstacles to the successful integration of machine learning models into business processes and their implementation in production. Machine learning models have enormous potential, but they often face various challenges that hinder their full application in real-world settings. Understanding these obstacles will help companies more effectively develop and implement AI-based solutions, turning them into useful business tools.

Reason 1 is that many users encounter specific issues that can negatively impact their experience. These issues can range from technical errors to a lack of information. It is important to understand that addressing these issues not only improves overall efficiency but also increases user satisfaction. Providing high-quality support and providing up-to-date information are key factors in improving customer engagement and increasing customer loyalty.

Not all employees welcome the introduction of technology

Machine learning models outperform humans at many tasks related to processing large volumes of data and automating repetitive processes. They effectively cope with calculations, forecasting, and classifying information. However, many employees are concerned about the impact of these technologies on their jobs. The development of artificial intelligence and automation can raise concerns, as some professions may be at risk. It is important to understand that technology is designed not only to replace humans but also to improve their productivity and efficiency.

Consider an HR department, where employees traditionally use Excel to prepare reports. One day, a data scientist comes into their office with an innovative algorithm designed to automatically screen resumes and filter out unsuitable candidates. This technology has the potential to reduce the HR department's time by 80%. Implementing such solutions not only optimizes processes but also allows for greater focus on more strategic tasks, such as improving the quality of recruitment and enhancing team performance. HR professionals may resist change, perceiving algorithms as a threat to their position. They may fear being fired, demoted, or facing increased workloads. In some cases, this leads to sabotage: a data scientist may receive incomplete resume databases or distorted candidate requirements. Incorrect data negatively impacts a machine learning model, rendering its results unreliable. As a result, such a model will not be able to work effectively in a production environment, which undermines its value to business.

Employees can sabotage technology that can harm their careers. Attitude: better to work ineffectively than to lose their salary. Infographics: Polina Vari for Skillbox Media

Sabotage in the implementation of new technologies should be eliminated through negotiations. Data scientists need to convey to employees the importance of the machine learning model and its benefits for their work. To increase persuasiveness, it's worth studying the experience of companies successfully implementing similar technologies. For example, banks can review the case of Rosbank, which uses machine learning to automate processes to combat money laundering and terrorist financing. This will help employees understand how modern technologies can improve their tasks and increase efficiency.

The implementation of a financial monitoring system has significantly simplified the process of searching and comparing information for specialists. Now they can quickly prepare reports and make informed decisions on questionable transactions that require mandatory bank control. This improvement increases operational efficiency and facilitates more accurate identification of potentially risky transactions.

Other Rosbank cases demonstrate how machine learning models contribute to the effective management of bank branches and retail networks. The use of modern data analysis algorithms allows for process optimization, improved customer service, and improved financial performance. Machine learning helps identify trends and predict customer needs, which, in turn, facilitates more effective resource planning and strategic development. Thus, Rosbank actively implements innovative technologies, making it a leader in the banking sector.

It is important to convey to employees that the implementation of new technologies will not lead to staff reductions. Negotiations will be ineffective as long as employees fear losing their jobs and perceive algorithms as a threat. For example, a meeting could be organized in the HR department to compare the machine learning model with Excel. It should be explained that these are merely tools that complement human work and perform different functions. This approach will help relieve tension and create a more positive perception of change.

In the next step, demonstrate the value of working with data. Explain how analytics and data processing can improve business processes, increase efficiency, and facilitate informed decision-making. Emphasize the benefits your company will receive by using data for strategic planning and operational optimization.

  • As it is now: manual processing of resumes, staff turnover, a shortage of qualified specialists.
  • As it could be: an algorithm will match candidate skills with job requirements and eliminate unsuitable resumes. HR specialists will be able to conduct more interviews, test candidates more thoroughly, and hunt valuable employees from competitors.

Once employees grasp the concept of machine learning, you can move on to finalizing negotiations and requesting the necessary data. In the future, the success of the project will depend on the qualifications and experience of the data scientist.

Employees are more likely to support the technology if they see its benefits and are not worried about their career future. Infographics: Polina Vari for Skillbox Media

Reason 2. This category includes several key factors that significantly influence the situation. These factors can be related to both internal and external circumstances. It is important to consider that each cause may have its own unique characteristics and consequences, which must be analyzed for a deeper understanding of the problem. Applying a comprehensive approach to researching causal factors will help identify key trends and develop effective strategies to address them.

There aren't enough people on the team

Data scientists collaborate like players on a football team. Each team member plays a role in the development and implementation of a machine learning model. Initially, specialists collect and process data, which forms the basis for subsequent analysis. Then, the model is tested to identify its strengths and weaknesses. It's also important to set up an infrastructure that will ensure the stable operation of the model in production. The code written for the algorithms requires thorough testing and optimization. After deploying the model, key metrics must be monitored to evaluate its effectiveness and performance. Therefore, each step in the model development process requires coordinated teamwork to achieve high results in machine learning.

Football players do not possess the same skills across different game roles—successful play in goal, defense, and attack requires specialized training. Similarly, in data science, each specialist has their own area of ​​expertise, which is necessary to complete specific tasks within a project. If a company doesn't attract a sufficient number of qualified specialists with diverse skills, the likelihood of successfully implementing a machine learning model in production is significantly reduced, which can lead to delays and a decrease in the quality of the final product.

In most large companies, one person cannot know everything and quickly complete all the tasks required to launch a model Machine learning. Infographics: Polina Vari for Skillbox Media.

Data Scientist is a multifaceted profession that combines statistics, machine learning, mathematics, analytics, data visualization, and business processes. Although in theory there are universal data scientists who have deep knowledge in all of these areas, in practice such specialists are rare. Larger projects may require a team of various experts. Let's consider the main roles that may be needed to successfully work on data analysis projects.

  • Data Architect. Responsible for quality standards - establishes rules for collecting, storing, processing, and using data.
  • Data Engineer. Engaged in storing, processing, and retrieving collected data - implements what is described in the data architect's project.
  • Data Analyst. Works with statistics: finds patterns in numbers, visualizes information, and prepares various reports.
  • Machine Learning Engineer. Conducts tests, configures automation, and performs post-production maintenance of machine learning models.
  • Business Intelligence Developer. Collects the data obtained by the team, translates it into understandable language, and presents it to the company's management.
  • Database Administrator. Monitors the database in which data scientists store information for each project: fixes errors, performs backups, configures secure access, and ensures compatibility between different software versions.

The number of specialists required depends on the specific task and deadlines. To avoid project disruption, it is important for the responsible employee, together with the data scientist team, to develop a plan for launching the model into production in advance, taking into account the company's available resources. This plan must indicate which specialists are lacking and the reasons for this shortage. Based on this information, the manager will be able to decide whether to expand the staff or postpone development.

If the tasks are evenly distributed among data scientists, there is a better chance that the development will move according to plan and the model will be released to production. Infographics: Polina Vari for Skillbox Media

Reason 3 is that many users encounter problems related to poor service quality. This may manifest itself in the form of a slow response to inquiries, lack of information, or insufficient support from specialists. Improving the quality of service is an important aspect for increasing customer satisfaction. Providing fast and effective assistance can significantly increase loyalty and trust in the brand. Understanding user needs and promptly resolving their problems also contribute to a positive company image.

The company is not ready for launch

Imagine: you see an advertisement, click on the ad, and are redirected to an empty online store. This seems illogical. Entrepreneurs will not spend an advertising budget before organizing the supply of goods and setting up sales on their website. To avoid such situations, it's important to prepare your online store in advance, ensuring product availability and a user-friendly interface for customers. Proper store setup and management will help attract customers and increase sales. An online store owner is dissatisfied with the work of his warehouse staff, as their actions are causing significant losses for the company. Frequent theft, the creation of false reports, the late disposal of expired products, delays in shipping, and the selection of unscrupulous suppliers create serious problems. All these factors negatively impact the business's profitability and reputation. Effective warehouse management and employee oversight are key to minimizing risks and improving the overall performance of the online store. The entrepreneur seeks to avoid traditional problem-solving methods, as they require significant effort: updating staff, selecting responsible employees, implementing quality standards, and creating compliance monitoring mechanisms. Instead, he plans to engage a team of data scientists to optimize logistics using algorithms. The goal is to develop a program capable of autonomously managing a warehouse network, which will significantly improve the efficiency of business processes. This approach will not only reduce time costs but also ensure a higher degree of automation and management precision.

The online store owner hopes that implementing a machine learning model will improve business process management and resolve employee-related issues. Machine learning technologies will optimize tasks, increase efficiency, and reduce human error risks.

Modern technologies make it possible to train programs to generate reports, forecast inventory levels, and perform various logistics processes. However, for such algorithms to operate successfully, data scientists require accurate and complete data on the business processes being optimized. Without high-quality data, the algorithm will not function effectively, highlighting the importance of collecting and analyzing information in logistics. Optimizing processes based on reliable data allows companies to increase efficiency and reduce costs.

If a company is a mess, machine learning models won't fix it—everything will stay the same or become even more confusing. Infographics: Polina Vari for Skillbox Media

Machine learning models are not a one-size-fits-all solution for all businesses, and not every company is ready for their implementation. To avoid unnecessary waste of time and budget, managers should determine in advance which process they plan to optimize, how the algorithm can help, and what changes will occur after implementation. A clear understanding of these aspects will allow for more effective interaction with data scientists and ensure the successful launch of machine learning projects.

The optimal situation is when the initiative to launch a machine learning model comes from company employees. Internal experts with deep knowledge of local business processes are able to most accurately identify areas for improvement. They know which aspects can be optimized and can offer effective solutions based on real business needs. Supporting such initiatives not only improves process efficiency but also fosters innovative approaches, which in turn leads to improved market competitiveness.

A warehouse manager faces the challenge of employees manually checking expired goods daily. This leads to overtime and hinders other duties. Therefore, the manager initiates the development of a system that will automatically monitor product expiration dates. Such a system will help optimize the workflow, allowing employees to effectively complete tasks during a shift. Automated control of expired goods will not only reduce the workload for employees but also improve the overall efficiency of the warehouse.

If the manager takes the warehouse manager's opinion into account, this is a signal to consider a machine learning model. It is important to consult with data scientists and analysts, assess the development costs, and determine the potential benefits for the company. Implementing machine learning can optimize warehouse processes, improve inventory management, and improve demand forecasting. This approach will not only help reduce costs but also improve the business's competitiveness.

The company's next steps depend on the forecast. It can either begin developing the project or abandon the idea if the budget doesn't allow for its implementation. Alternatively, the manager may decide to increase the warehouse employees' salaries, which will be more cost-effective than hiring a team of data scientists.

A manager with an understanding of the company's real needs and challenges can select the most effective tools to solve existing problems. Everything depends on good planning, and sometimes the use of machine learning models is the best option. Effective planning is the basis for rational work and successful implementation of algorithms in the production environment.

Machine learning models cannot magically generate profit. They are just a tool that sometimes helps in work. Infographics: Polina Vari for Skillbox Media

What to do if all else fails and the project is shelved

If you participated in the development of a machine learning model, but it wasn't implemented in production, don't be upset. This is a natural part of the work process. You gained valuable experience, continued to receive a salary, and can now focus your efforts on new projects. Development setbacks are opportunities for growth and learning, which will help you in future machine learning endeavors.

If your project is not a trade secret, consider writing a blog article to share your experience and expand your portfolio. You can also prepare a presentation for an IT conference, where you can present your case to other data science specialists. Participating in such events not only helps you systematize your acquired knowledge but also contributes to the creation of valuable professional contacts. Speaking at conferences also helps you overcome public speaking anxiety, which can have a positive impact on your career advancement.