Turning Chaos into Order: What You Need to Know About Data Science

Contents:

• Data Science
• What Data Science consists of
• From IT to finance. In which areas is Data Science in demand?
• How data science changes our lives for the better

Data Science

The first idea of ​​what Data Science is comes from the literal translation of the phrase. Data science means "science of data" or "science of working with data." To summarize, all natural sciences are based on the collection, storage, and analysis of information, followed by its systematization and drawing conclusions. These are then used to formulate hypotheses and make forecasts. A predictive model is the result of the work of a data scientist, who, to varying degrees, even included Archimedes and Newton.

But hundreds of years ago and today, the volumes of information differed by thousands and thousands of times—today, we have to analyze vast quantities of data. Big Data, a vast array of information. To hypothesize that the Earth has a gravitational field, Isaac Newton needed to record and analyze the fact that an apple fell from a branch to the ground. To predict the number of people who will want to buy a domestic car priced over two million rubles in the first ten days of next year, we will need to analyze a huge flow of information using various tools, including machine learning and its highest level—artificial intelligence. A Data Scientist, a scientist and expert in the field of analytical and statistical work with large data sets, works with such volumes of information and automation tools.

Data Science stands at the intersection of several classical and new sciences: mathematics, statistics, analysis and predictive analytics, machine learning, Big Data, etc. This interdisciplinary field allows us to achieve the desired results - to structure data and create mathematical algorithms based on it. and present predictive models for making informed and balanced decisions.

What Data Science consists of.

Data Science is divided into three components: data collection and storage, processing and analysis. Let's consider each of the components.

Data collection and storage. Building the foundation

In order to process and analyze information, it must be collected. Therefore, collection is the first stage in the work of a Data Scientist. The final result directly depends on the completeness, relevance and representativeness of the collected data.

To collect information, a data scientist uses various tools. Both well-known and cutting-edge:

• Surveys and Engagement — classic telephone surveys, paper questionnaires, online forms, internet quizzes;
• Data from educational, medical and social organizations;
• Tools for collecting internet statistics — sensors on websites, webvisors, automated web scraping technologies (ed.: obtaining data directly from website pages), "pixels" in some social networks;
• Feedback received from electronics and household appliances operating on the IoT (Internet of Things) principle, GPS devices;
• Reports and databases companies, banks, online stores.

The list is endless. The higher the qualifications of a Data Science expert, the more tools will be in his professional arsenal.

It is equally important to ensure proper storage of Big Data. For this task, the following are used:

• Data Warehouse (data warehouse) - specialized database management systems. Information comes there from various sources, passing through filtering and structuring. In simple terms, such a database can be called a set of tables with data and relationships. The most well-known DBMS are: ClickHouse, Greenplum, Exasol, Teradata, Vertica.
• Data Lake (data lake) is a huge storage for "raw", unsorted data of different types without any order or sorting. It can contain everything from Word documents and commercials to downloads from CRM systems.

Data warehouses have been the subject of extensive, sometimes confusing, literature—articles and entire books. We need to understand that this is a complex, responsible, and important process. Working with big data in Data Science typically begins with a "lake."

Data Processing. Building Walls

To maximize the usefulness of existing information, it must first be processed and cleaned—brought into a form suitable for analysis. A variety of tasks can be solved at this stage: from combining a large number of tables into a single array to total optimization of the final dataframe (tables). There are many technologies and techniques for this, including: Removing duplicates. Sometimes data in different arrays (for example, downloading private car sale ads from different sites) can completely coincide, and significantly. They must be removed. aria-level="1">Removing inconsistencies.If we take car rental as an example, the same cars may be offered at different prices in different places. Simply removing all the values ​​is not always the right solution - sometimes you have to leave one option or look for an algorithm for combining them.

Special automation tools perform such work. Although a Data Scientist is directly related to programming, he is also a mathematician, statistician, and analyst. He can independently create a Python script. This will help to further understand the essence of mathematical cleaning algorithms and gain good practice in creating valid code. It will also help to analyze the results and summarize them.

Data Analysis. We Get Results in Convenient Forms

After cleaning, the data is converted into the desired format. Then it is systematically analyzed, conclusions are drawn, and predictive models are built.
During the analysis process, which is called Data Mining, the obtained information undergoes final sorting. Various indicators are used for this. Here are just a few of them:

• Standard deviation. An indicator that reflects the amount of fluctuation or dispersion (edit: of available values. The lower this indicator, the closer the set of values ​​is to the average.
• Median. An indicator that separates the upper part of the sample from the lower one. Unlike the mean, the median is less dependent on peak values ​​at the upper and lower boundaries, which allows for a better representation of the center.
• Correlation. A mutual statistical relationship between several random variables, in which a change in the value of some variables causes changes in others.

To present the results of analytical work, their competent visualization is important: graphs, charts, pivot tables, structured diagrams, etc. Such visual information helps to improve perception.

Various tools are used for visualization: free and paid, simple and not so, multifunctional and highly specialized. Among the free ones, Google charts stand out. They are enough for quickly creating diagrams and graphs. The prices for paid tools vary enormously - from several tens of dollars (Tableau, Qlik) to several hundred and even thousands of dollars monthly (Power BI, Fusion Charts). Some tools require serious training and certain technical skills from a data science specialist. Others, like Juicebox and Tableau, are suitable for those without even minimal technical experience.

From IT to finance. In which areas is Data Science in demand?

Today, Big Data is talked about literally everywhere. And rightfully so. Therefore, Data Science finds application in various fields. Here are some examples.

• Entrepreneurship. Big data makes traditional business analysis and marketing research more qualitative. It allows for more accurate forecasting of the creation of popular products and the opening of promising areas of activity. For example, statistics on the deterioration of drinking water led many years ago to the creation of a new product – bottled drinking water. But back then, the analysis was done manually and took a long time, while now it is automated and fast.
• Meteorological services.Modern weather forecasts are based on processing a huge amount of multi-vector information.
• Financial sector.Data science specialists create algorithms that help make loan decisions.
• Healthcare.Technologies that allow for automatic diagnoses are being increasingly implemented. And this is the result of big data analysis using machine learning and artificial intelligence technologies.
• IT industry.Data Science is used to create chatbots, neural networks, search engine algorithms, etc.

The list of areas where Data Science is needed is endless. Here we can mention agriculture, where science is used to forecast crop yields. And logistics - to predict profitability and optimize routes. In the social sphere, applications for people with disabilities allow them to move around the city based on the prompts of a virtual assistant. And the application itself is loaded with all sorts of descriptions of objects obtained precisely from Big Data. In this regard, the demand for Data Science will only grow.

Data science changes our lives for the better.

Data scientist is a profession that requires knowledge of programming, specific technical skills, and mathematical and analytical abilities. Humanities students will have to work hard to recall and radically supplement the knowledge acquired in high school. You must be able to work with databases, have programming skills in Python and SQL, and be able to use big data tools such as Hadoop and Apache. In addition, good technical English is important for the job. This will help you gain knowledge from reliable primary sources, which are almost always in English.

However, this specialty also has many prospects. Even small businesses understand the importance of working with big data today. Data scientists turn chaos into order, transforming arrays of disordered data into useful information and highly accurate forecasts. Thanks to these specialists, companies receive more accurate pictures of their target audience and create truly needed goods and services. And users receive only targeted and interesting advertising, taking an invisible part in the creation of new products. And without exaggeration, data science changes our lives for the better.

And if you want to get to know the profession better, the editors of Skillbox.by recommend studying thematic literature and professional communities.

Literature:

• Bruce Andrew, Bruce Peter "Practical Statistics for Data Scientists". A book for specialists with experience, technical skills, and knowledge of the R programming language.
• J. Grass "Data Science. Data Science from Scratch". A practical guide for quickly entering the profession without experience and technical training. The book describes the basics of writing algorithms in Python, mathematical analysis, and statistics.
• Kennedy Berman, "Python Essentials for Data Science." The latest tutorial on mastering Python, the number one language in data science.

Professional Communities:

• Data Science by ODS.ai— a Telegram channel that positions itself as the first and oldest resource of its kind. It was created by members of the Open Data Science community. Here they talk about deep neural networks, computer vision, processing and understanding of live and natural language, bots, etc.
• Data science | Machinelearning— Russian-language Telegram channel about artificial intelligence, data science, and machine learning. We publish case studies, training and advisory materials, forecasts, and industry statistics.
• Data Science Notes is a Russian-language channel where you can find not only articles, but also entire books on data science.