Code

BeautifulSoup in Python: How to Scrape Websites in 3 Steps

BeautifulSoup in Python: How to Scrape Websites in 3 Steps

Course with employment: "The profession of Python developer"

Find out more

Data parsing is performed using various programming languages, such as Python, JavaScript, and Go. While the choice of tool isn't crucial, some languages ​​offer convenient parsing capabilities thanks to specialized libraries. For example, Python's Beautiful Soup library greatly simplifies the process of extracting information from HTML and XML documents.

In this article, we'll cover the basics of parsing in detail, focusing on the structure of an HTML request. We'll walk through the process of parsing weather data from Yandex, demonstrating the practical application of this knowledge. We'll also provide a recording of a workshop in which our web development expert will demonstrate how to create a web parser from scratch. This will give you a deeper understanding of the parsing process and its potential for extracting data from web resources.

What is parsing and why is it needed?

Parsing, or web scraping, is the automated process of collecting data from websites. This technology allows for the extraction of various types of information, such as articles and headlines, which is especially useful for journalists and sociologists. Programs that collect and process data from the internet are called parsers. They analyze the structure of web pages and extract the necessary data, significantly simplifying research and information analysis. Parsing is becoming increasingly relevant in a rapidly changing information environment, providing users with access to up-to-date data from various sources.

Parsing is used to solve various problems. With its help, Telegram bots can extract information, which they then present to users. Marketers use parsing to collect data from social networks, and businessmen get the opportunity to study information about their competitors. Effective data parsing allows you to optimize business processes and make more informed decisions based on up-to-date information.

There are several methods of data parsing. One is to use an API provided by the service developers. An alternative approach is to extract information directly from the HTML code of web pages. It's important to understand how interaction with servers on the internet occurs and how HTTP requests work. Let's take a closer look at these aspects.

HTTP Requests, XML, and JSON

HTTP (HyperText Transfer Protocol) is a protocol that enables data transfer between a client and a server. It got its name because it was originally intended for exchanging hypertext documents written in HTML. Today, HTTP is used not only to transfer web pages, but also to transfer various types of data, including images, videos, and other media files. This protocol is the foundation of the World Wide Web and plays a key role in ensuring communication between users and internet resources.

To understand how HTTP works, it's important to consider its client-server data transfer architecture. The client, in this case your browser, creates a request and sends it to the server. The server processes the request, generates a response, and returns it back to the client. In this example, the browser acts as a client requesting information from the server.

The request includes three main elements:

  • Request line: specifies the transfer method, the HTTP version, and the URL the server is accessing.
  • Message header: the message itself sent to the server, its parameters, and additional information.
  • Entity body: the data transmitted in the request. This is an optional part.

Let's look at an example of a simple HTTP request that we use to get a weather forecast. This request allows us to interact with an API to obtain up-to-date information about weather conditions. The HTTP request is sent to the server, which processes it and returns the data in a format convenient for analysis. Using the correct request, we can obtain information about temperature, precipitation, wind, and other meteorological parameters. This approach allows developers to integrate weather data into apps and websites, improving the user experience and providing up-to-date information.

A GET request to the URL https://yandex.com.am/weather/ is used to obtain weather information. This request is sent via the HTTP/1.1 protocol and allows users to obtain up-to-date data on meteorological conditions in Armenia. This request provides information on the current temperature, humidity, wind speed, and the forecast for the coming days. Using this API helps users stay informed about weather changes and plan their actions based on weather conditions.

This request has three key components.

  • _GET — the request method. The GET method allows you to retrieve data from a resource without changing it.
  • /https://yandex.com.am/weather/ is the URL of the site we are accessing.
  • HTTP/1.1_ indicates the HTTP version.

The response to the request includes three main components: _HTTP/1.1 200 OK_. In this format, the HTTP protocol version is indicated first, followed by a numeric response code and a text explanation. There are many different responses, and there is no need to remember them all. Instead, you can refer to the documentation, which provides detailed explanations for each code. This allows you to quickly find the information you need and better understand how HTTP responses work.

HTTP requests can be formed in various formats, among which the most common are XML and JSON. These formats have their own characteristics and advantages. XML (eXtensible Markup Language) provides a flexible structure that allows you to describe complex data using tags, making it convenient for representing hierarchical data. JSON (JavaScript Object Notation), in turn, is more lightweight and easier for humans to read, making it a popular choice for web applications and APIs. Both formats are actively used for transferring data between the client and the server, and the choice between them depends on the specific requirements of the project.

JSON (JavaScript Object Notation) is a lightweight data exchange format based on JavaScript. It represents text data that is easy for humans to read and write. JSON is widely used to transfer structured information between the client and the server, as well as in various web applications. Due to its simplicity and versatility, JSON has become the standard for working with data in modern APIs and web services. The format supports basic data types such as numbers, strings, arrays, and objects, allowing you to effectively organize and structure information.

To retrieve data in JSON format, you need to correctly construct an HTTP request. It is important to specify the correct request method, such as GET or POST, and set the necessary headers, including ‘Content-Type’ and ‘Accept’, so the server can recognize the required format. Make sure the request URL is correct and points to the correct resource. Also, be sure to check the request parameters if they are required to retrieve specific information. The correct structure of the request guarantees the successful receipt of data in JSON format.

The text structure is divided into five key logical parts.

  • var requestURL — a variable pointing to the URLs with the necessary information;
  • var request = new XMLHttpRequest () — creating a new instance of the request object from the XMLHttpRequest constructor using the new keyword;
  • request.open (‘GET’, requestURL) — opening a new request using the GET method. Be sure to specify our variable with the URL;
  • request.responseType = ‘json’ — explicitly designating the received data format as JSON;
  • request.send () — sending a request to receive information.

XML is a markup language that establishes a set of rules for encoding documents in text format. Unlike JSON, XML offers greater complexity and flexibility in data structure. For example, XML allows for a more detailed description of data hierarchies, making it useful for representing complex structures.

To retrieve data from the server in XML or HTML format, you must use the same library as for working with JSON. However, in this case, the responseType parameter should be set to Document. This will allow for the correct processing and extraction of information from the received document.

Choosing an HTTP request format is an important step in developing web applications. Although JSON is perceived as simpler and easier to read, the correct choice of format depends on the specific task and requirements of the project. In future materials, we will take a closer look at the various HTTP request formats, their advantages and disadvantages, and situations in which each can be most effective.

Today, we will cover the basics of web scraping using standard Python libraries. You will learn how to effectively extract data from web pages and work with various useful tools for automating this process. Web scraping allows you to gather information from the internet for analysis, research, or other purposes. Once you master these skills, you can create your own scripts to process the data and improve your workflows.

The Three Stages of Scraping

Scraping is an effective way to extract information from web pages. One of the easiest ways to learn the basics of scraping is to write a simple program. In this case, we'll create an application that displays the weather forecast for your city. This will help you understand how the web scraping process works and master the basic tools and libraries needed to extract data from websites.

To achieve this goal, we will go through three consecutive stages.

  • We will include libraries that will help us parse information using Python (see our article for how to install Python on Windows, macOS, and Linux).
  • We will visit the website from which we plan to parse information and study its source code. It will be important to find the elements that contain the required information.
  • We will write the code and parse the data.

Including Libraries

Including libraries is an important step in developing web applications. Proper integration of libraries can improve the functionality and performance of your project. Make sure you are using the latest versions of libraries to avoid compatibility and security issues. To include libraries, use the `<link>` tags for CSS and `<script>` for JavaScript. It is also recommended to include libraries through a CDN, which ensures faster loading and optimized website performance. Don't forget to check the documentation of each library for proper configuration and use.

Different programming languages ​​offer their own libraries for parsing data from websites. In JavaScript, the most common is the Puppeteer library, and in Python, Beautiful Soup. These tools have similar operating principles, which facilitates the process of extracting information. However, before you start parsing with Python, you need to install and configure the Python runtime on your computer.

Writing code in a text document requires using certain techniques. There are several effective ways to help you with this process.

  • Use the terminal on macOS or Linux, or use the command line on Windows. To do this, you will first need to install Python on your system. We wrote about this in detail in a separate article.
  • Use one of the online editors that lets you work with Python code without installing it: Google Colab, python.org, onlineGDB, or another.

After installing Python on your computer or using an online code editor, you can start importing the necessary libraries. Importing libraries is an important step, as they allow you to extend the functionality of your project and use ready-made solutions for various tasks. Make sure you have installed all the necessary dependencies before starting development.

BeautifulSoup is a powerful library for processing HTML and XML documents. It simplifies parsing, navigation, and modification of data structures. Including BeautifulSoup in your project is easy: just install the library using pip and import it into your code.

To work with web requests, you need to use the requests library, which allows you to send requests to the target site. You can import it with just one line of code. This library greatly simplifies the process of interacting with web pages by providing convenient methods for performing GET and POST requests, processing responses, and managing headers.

All libraries are ready to use. They allow you to extract the source code of a website and search for the necessary information within it.

The Beautiful Soup library is an indispensable tool for parsing HTML and XML documents. It is often pre-installed in development environments and Jupyter Notebook, but in some cases it may be missing. If an error occurs when trying to import the library, run the command to install it. After successful installation, you can try importing again and continue working with the library.

Examining the Website Source Code

We will use the Yandex.Weather website to get weather forecasts. Go to the site and enter the name of your city in the search bar. In this example, we are looking at Moscow. This resource will help you get up-to-date weather conditions in your region, including temperature, precipitation, and other important parameters.

Pay attention to the address bar, as it will be useful later: https://yandex.com.am/weather/?lat=55.75581741&lon=37.61764526. This URL contains weather information, including latitude and longitude coordinates. Using such addresses helps you get up-to-date weather data for specific geographic points.

The address bar often does not include the city name; instead, it displays the geographic coordinates of the location for which current weather information is provided. In this case, it is the center of Moscow.

Now let's analyze the page's source code and determine the area where the current temperature is displayed. Pay attention to the highlighted block in the website screenshot.

Screenshot: Yandex.Weather / Skillbox Media

To view HTML code, open the Code Inspector. To do this, use the following keyboard shortcuts: ⌥ + ⌘ + I in Google Chrome on macOS, and Ctrl + Shift + I or F12 on Windows. The Code Inspector is a separate browser window containing several tabs for analyzing and editing HTML, CSS, and JavaScript. This is a tool that allows developers and web designers to quickly view the structure of a web page, check styles, and debug code.

Screenshot: Yandex.Weather / Skillbox Media

There's no need to switch between tabs, as all the necessary information is presented on the first page.

Now you need to find the block in the code that contains the temperature value. To do this, you need to sequentially expand the code blocks located inside the tag. This can be done by clicking the ▶ symbol.

To determine that we're moving in the right direction, use the developer tools. When you hover over a code block in the code inspector, the corresponding area on the site is highlighted, allowing you to visually connect the code with interface elements. By stepping through the HTML structure, you can find the specific element you're interested in. This is a useful approach for optimizing work with code and simplifying the development process.

In this case, we've gone through several levels of nested elements. We started with the element with the class "b-page__container", then moved to the first element with the class "content xKNTdZXiT5r0Tp0FJZNQIGlNu xIpbRdHA". Next, we found the element with the class "xKNTdZXiT5r0vvENJ", after which we looked at the element with the class "fact card card_size_big". We continued the path to the element with the class "fact__temp-wrap xFNjfcG6O4pAfvHM", then to the element with the class "link fact__basic fact__basic_size_wide day-anchor xIpbRdHA", and finally got to the element with the class "temp fact__temp fact__temp_size_s". The "temp fact__temp fact__temp_size_s" class will be key to performing the following steps.

Screenshot: "Yandex.Weather" / Skillbox Media

Writing Code and Getting the Necessary Information

Continue working with commands in the terminal, command line, IDE, or online Python code editor. At this stage, you need to use the included libraries to obtain temperature values ​​from the element. First, it is important to check that the libraries are working correctly to ensure that further operations are performed correctly.

Save the URL of the page from which data will be parsed in a variable. This is a necessary step to start working with web content. Specifying the correct URL allows you to access the required information and efficiently extract data for further processing.

Let's create a request to the server and analyze the received response.

In this case, we receive the following response:

The "200" response indicates that the requests library is working correctly and that data has been successfully transferred from the server. This means that the server successfully processed the request and returned information from the page.

Now we can get the source code using the Beautiful Soup library. In this process, we'll extract data and immediately display the results. Beautiful Soup is a powerful HTML and XML document parsing tool that simplifies extracting information from web pages. We'll create a Beautiful Soup object, passing it the HTML code, and then use the library's methods to extract the desired elements. Displaying the result on the screen will help us verify that the code is executed correctly and the data extracted is correct.

Once the process is complete, the full page code is displayed on the screen.

Screenshot: Skillbox Media

To extract only the required block of code containing the temperature value, we'll use the Beautiful Soup library and its find() function. This will allow us to efficiently find and retrieve the required data without unnecessary information. This way, we'll focus exclusively on the element that stores the temperature, simplifying further data processing.

The find() function takes two arguments that allow it to efficiently search for elements in a data collection. The first argument specifies the search criteria, and the second argument sets the context in which the search will be performed. This function returns the first element that matches the specified criteria, or null if no such element is found. Using the find() function allows you to optimize the process of searching in an array or object, making it faster and more convenient. This function is widely used in programming for working with arrays, objects, and other data structures, which makes it an important tool for developers.

  • an indication of the type of HTML element in which the search is performed;
  • the name of this element.

In this example, the code will look like this:

To display the result on the screen, we use the print function. This simple yet powerful tool allows you to display data and messages in the console, which is especially useful for debugging and analyzing code. With print, you can easily check how the program is executed and what values ​​​​are passed to various variables. Using this feature is the foundation for user interaction and providing information about the program's operation.

We deliver results that meet your expectations and requirements. We consider every detail to ensure high quality and customer satisfaction. Our team of professionals uses proven methods and technologies, allowing us to achieve our goals. We strive to continuously improve and optimize all processes to provide the highest level of service. Ultimately, the result of our work is the successful completion of tasks and the desired outcome for our clients.

Let's remove the unnecessary HTML tag with the class to leave only the temperature values. To do this, we use the text property, which allows us to extract precisely the information we need without unnecessary elements. This optimizes our content and makes it more readable.

We successfully retrieved the current city temperature using the Yandex.Weather website and the Beautiful Soup library for Python. This library provides powerful tools for parsing HTML and extracting the necessary data. You can use it for a variety of tasks, such as integrating weather data into widgets on your website or creating a bot that informs users about weather conditions. Using Beautiful Soup allows you to automate the process of obtaining up-to-date information and improve the user experience.

Website Scraping with Python - A Masterclass for Beginners

If you're new to web scraping and want to create your own parser, for example, to automatically generate Excel reports, we recommend checking out this webinar by Mikhail Ovchinnikov, a leading software engineer at Badoo. This webinar explains the basics of Python and the key principles of web scraping using an accessible example. From the very beginning of the video lesson, you'll be able to run a simple parser and master reading data in HTML and JSON formats. This course will be a great start for your journey into the world of web scraping and programming.

Parsing Dynamic Websites with Python and Selenium

The free Selenium library provides the ability to emulate web browser actions, allowing you to "mask" script web requests as user actions in browsers such as Google Chrome and Safari. This is crucial, as many websites are able to detect bots and block IP addresses that send automated requests. Using Selenium helps bypass these restrictions, providing more reliable access to information and resources on the internet.

There are two main ways to avoid blocking: by learning the basics of HTTP and Python's principles of working with web technologies and then creating your own emulator, or by using ready-made tools. In this context, Selenium is one of the most effective and convenient solutions for automating interactions with websites. This tool emulates user actions, significantly reducing the risk of blocking and enabling tasks related to parsing and testing. Using Selenium not only simplifies the development process but also improves the stability of working with various web resources.

Mikhail Ovchinnikov shared information on how to effectively use the library. His presentation covered key aspects of working with resources available in the library and provided practical recommendations for users. He emphasized the importance of searching for information correctly and using catalogs, which significantly speeds up the process of finding the necessary materials. Mikhail also discussed the opportunities the library provides for research and educational activities, including access to digital resources and specialized databases. These guidelines will help everyone maximize the library's potential and improve their teaching and research.