Code

How to Scrape Websites with JavaScript and Node.js

How to Scrape Websites with JavaScript and Node.js

Course with employment: “Profession Frontend Developer”

Find out more

RIA Novosti journalist and author of the Dark Lord of Communications channel provides current news and analytical materials. His professional experience and deep understanding of current media trends enable him to create high-quality content. The channel publishes articles on current events, politics, and social issues, making it a valuable source of information for a wide audience.

Links are an important element in web content, facilitating navigation and improving the user experience. They allow users to quickly navigate between pages and find the information they need. It's important to structure links correctly so they are understandable to both users and search engines. Using keywords in link text can increase a page's visibility in search engines. It's also worth paying attention to internal and external links. Internal links help distribute page weight and increase user time on the site, while external links can enhance the authority and trust of your resource. Link optimization helps improve SEO and increase website rankings in search engines.

Web scraping, or web parsing, is the automated process of collecting data from the internet. This method has many applications in professions related to information processing. For example, by parsing articles from various websites using JavaScript and the Node.js framework, we can demonstrate how it can be useful for modern journalists, PR specialists, and marketers, who, at first glance, may not have programming experience. Parsing allows you to efficiently extract and analyze data, simplifying work with information and increasing productivity in various fields.

If you have a source website and you want to review all the articles on it to study a specific topic or create a news collection, the process can be time-consuming due to the large number of pages. In such a situation, it would be reasonable to first compile a list of all publications and then filter the required materials. This will significantly simplify the search and speed up the process of information analysis. Using tools for automatic data extraction from a website can be an excellent solution for this task.

How Website Scraping Works

The process of collecting data from a website can be characterized by the following steps. First, you need to determine the target site and the type of information you want to extract. Then, you should use tools to analyze the site structure and its HTML code to understand where the desired data is located. After this, the information is collected using software tools such as web scrapers. The obtained data can be structured and saved in a convenient format for further analysis. It is also important to consider the legal aspects of data collection to avoid infringing the rights of content owners. The correct approach to data collection will allow you to effectively use the obtained information for analysis and decision-making.

  • We determine the source site and the desired data.
  • We determine the pagination method (page navigation) and the structure of the site code.
  • Using any of the many possible methods, we make sequential network requests for each page. If the site has an API, we use it; if not, use other tools.
  • Convert the obtained data into a convenient format.
  • Save the final data to a file.

Success in online business is largely determined by high-quality website analysis. It is important to identify the key aspects that influence its effectiveness. It is necessary to study current traffic indicators, evaluate user behavioral factors, and conduct a competitor analysis. In addition, you should pay attention to content optimization and technical parameters of the site to improve its visibility in search engines. This will help identify strengths and weaknesses, which will help in the future develop a strategy to increase conversion and improve the user experience.

  • How the transition to the next page occurs. This is necessary for the parser to do everything automatically; otherwise, the collection will end on the first page. Usually, this happens when you click a button like "Next" or "Next page" - and the parser simulates the click.
  • The correct and exact place where the required materials are contained in the site's HTML markup. To do this, you will need to determine the location (nesting) of blocks, as well as their selectors.

Requests should be sent "politely", allowing for a certain delay to avoid negative impact on the source site. For example, it is not recommended to run a loop that makes hundreds of instant requests to all archive pages simultaneously. This can lead to server overload and blocking of your IP address. A correct approach to sending requests contributes to more efficient and secure data extraction.

Copyright infringement is strictly prohibited. Before developing a scraper, you should carefully read the user agreement, which may contain direct prohibitions on automatic data collection. Compliance with these rules will help avoid legal consequences and protect the rights of content owners.

For the scraping example, I chose two sites with different approaches to pagination. The first site uses a "Next Page" button, allowing you to navigate to the next set of data. The second site implements infinite content loading, which ensures smooth browsing without the need to navigate to a new page. These differences in pagination affect parsing methods and require an adapted approach depending on the site structure.

Installing the Necessary Software

Our parser, developed in JavaScript, will run in the Node.js runtime. To improve data processing efficiency, we will use additional modules, such as axios for HTTP requests and jsdom for working with HTML documents. This will allow us to extract and analyze information from web pages, ensuring high speed and accuracy of processing.

  • Using the JavaScript language, we will declare variables and constants, as well as run functions and loops.
  • The Node.js framework will allow you to do all this not in the browser, but through the Windows command line.
  • The fs module (short for file system) built into Node.js allows you to work with the computer's file system to create files with the result.
  • The additionally downloadable axios module will allow you to conveniently make HTTP requests by links.
  • The additionally downloadable jsdom module will allow you to parse the received result as a DOM tree, as if it were done in a browser.

Let's start by installing Node.js. First, go to the official website and download the installation file. Install Node.js by following the on-screen instructions. Once the installation is complete, you'll be able to use JavaScript code through the command line, allowing you to run JS files and execute individual commands. This provides a convenient and efficient way to work with JavaScript.

When you install Node.js, you'll get the npm package manager, which allows you to install various modules, such as axios and jsdom. To do this, open a command line and run the commands npm install axios and npm install jsdom one at a time, waiting for each package to install. You can install the modules either in the default folder or in your project folder, depending on your preference.

We've selected additional modules based on their popularity, as evidenced by weekly download statistics in the npm directory. Frequent use of these solutions is a sign of their reliability and effectiveness.

Parsing a website with the first type of pagination: individual pages, click-through

In the classic approach, each page of the site containing materials is a separate unit, and the user initiates the transition by clicking. For effective parsing, it is necessary to sequentially visit all pages, stopping at each one to extract the necessary data. This process continues until all available pages are exhausted. This method ensures systematic information collection and allows for careful processing of the site's content.

Let's study how the transition is implemented on the website of the professional magazine "Journalist" and parse it. This site was chosen for parsing for several reasons. Firstly, it provides interesting content for analysis. Secondly, the site structure allows for easy extraction of the necessary data. Finally, "Journalist" is an authoritative resource in its field, making it a suitable subject for research and information extraction.

  • First, Skillbox Media editor "Kod" and I agreed that this is a cool magazine :)
  • Second, the magazine's pagination structure makes it suitable for demonstrating the technology.
  • Third, the editors of "Journalist" kindly agreed to help us.

The site contains over 160 pages of materials spanning six years, which in total amounts to almost 3,000 articles. This will create an HTML file containing a list of article titles with corresponding links. This file will become a convenient tool for navigation and searching for information on the site, allowing users to easily find the materials they need.

We are considering a method of navigation between pages. The transition is carried out by clicking the "Read more" button below the articles. This button sends a request to the server, forming a URL in the form «https://jrnlst.ru/node?page=2». As a result, additional materials related to the next set of content are loaded on the current page. This approach improves user experience, allowing users to easily find and view related articles without leaving the current page.

We will choose the second method available on the site, using links in the format «https://jrnlst.ru/?page=[page number]». These links allow you to load specific pages with articles. The numbering starts with page zero, which is the main page, although this information is not always explicitly stated.

We determine the last page on which to complete the collection of information. At the time of writing, I have determined that the last page is page number 162. On this page, instead of a transition button, there is a simple inscription below the articles, "That's all for now."

I found it quite easily. I started following links with arbitrary page numbers, starting with «page=200» (this was just a guess) and gradually decreasing the numbers. It is important to keep in mind that the result depends on the site’s features, how long it has existed, and the expected frequency of updates. In the end, we counted 163 pages, including page zero (the main one).

To effectively extract information from HTML code, you need to tell the parser where exactly the desired content is located. Using web development tools built into the browser, you can analyze the code structure and determine the path to the desired headings. They are located as follows: element with the class «block-views-articles-latest-on-front-block» → first element with the class «view-content» → all elements with the class «flex-teaser-square» (in turn) → in each of them, the first element with the class «views-field views-field-title» → and, finally, in each of these elements, the first tag ‘a’, which is a hyperlink with the title of the article. This approach allows you to quickly and accurately find the necessary data for further analysis or processing.

Location of the block with the article Screenshot: Evgeny Kolesnikov for Skillbox Media

Now that we have all the necessary data for parsing, let's begin automating the process of collecting materials. Automating information collection will significantly simplify and speed up the work, allowing us to focus on analyzing and processing data. An effective parser will help us extract relevant materials, which will improve the quality of content and increase its relevance.

Our parser will include two files: a JS file containing the main code, and a BAT file that will allow you to launch the parser with a simple click. This approach ensures convenience and ease of use, and also allows you to easily integrate the parser into your workflows. The JS file will be responsible for the data processing logic, and the BAT file will simplify the launch of the parser for users unfamiliar with programming.

  • Let's create a file named "JJ Articles Parser.js" (JJ is a convenient abbreviation for "Journalist Magazine" - no magic there). This file will contain almost all of our executable code.
  • Let's create a start.bat file and write the following commands in it:

Everything is crystal clear here:

  • The first line is to use the cd command to change to the desired drive and folder.
  • The second line launches the Node.js interpreter and immediately passes our JS file to it for processing.
  • The pause command prevents the command line from closing after the code is executed.

Now let's move on to the parser code. In this section, we will focus on the key aspects of developing an effective parser that can extract data from various sources. We will cover the main functions and methods that will help in creating a reliable parsing tool. It is important to keep in mind that optimal code not only gets the job done, but is also easily scalable and maintainable. We will also focus on error handling and ensuring correct interaction with web pages to ensure the parser operates reliably under a variety of conditions.

You can examine the code on the Pastebin platform.

Creating program code requires precision and accuracy. During development, it is important to monitor the line count to ensure the code is readable and maintainable. In this case, it is optimal to stay within 50 lines, including comments and notifications that will help the user understand what is happening in the program.

Comments play a key role in the code, as they explain the logic and structure of the program. Console notifications are also useful for monitoring program execution and identifying possible errors. Proper use of these elements not only improves the quality of the code but also makes it easier to edit later.

Strive to ensure that every line of code is meaningful and functional. Avoid unnecessary complexity and duplication to maintain compactness. Ensure comments are clear and informative, explaining every important part of the code.

Ultimately, writing a program within the 50-line limit requires attention to detail and the ability to express thoughts concisely. This improves not only the quality of the code but also the overall perception of the project.

This parser's concept is based on data analysis and processing. It collects information from various sources, structuring it for further use. The parser's primary task is to extract the necessary information from web pages or other data formats, such as APIs. The process includes the following steps: sending requests to target resources, receiving responses, parsing HTML code or JSON format, and extracting and saving the necessary data. This approach allows for the efficient processing of large volumes of information and the automation of routine tasks, significantly accelerating the data retrieval process. Using a parser can be useful in a variety of areas, including marketing, competitor analysis, and market research.

  • Connect the necessary modules.
  • Define constants: the number of site pages, the main part of the link (except for the page number, which changes).
  • Define the starting values ​​of the main variables: the start of the traversal from page zero and zero request latency, which will constantly increase.
  • Define the main parser function called paginator(), which contains almost all the code.
  • The last line runs this function.

The paginator() function plays an important role in organizing navigation through content pages. It is responsible for breaking large amounts of data into smaller, easier-to-read parts, which significantly simplifies the process of finding information. Pagination helps improve the user experience, allowing users to easily navigate between pages and find the content they need. Furthermore, proper implementation of the paginator() function can positively impact SEO, as search engines index structured content better. Understanding and configuring this function is key to creating an efficient and user-friendly web application.

Within this function, another one is implemented—getArticles(), which generates a link to the next page using an immutable "base part" and a page number. It performs a GET request using the axios library and processes the resulting DOM tree using the jsdom module. The function then extracts all links from the current page, converts them from relative to absolute, saves the result to a file, and increments the page number for the next request.

The for loop that runs the internal getArticles() function is scheduled to run with an increasing delay. The set delay is 10 seconds, which does not significantly impact the site's performance and reduces overall execution time. This time is recommended by the site's developers in the crawl-delay directive of the robots.txt file, although not all developers adhere to this practice, as this directive is considered deprecated. Each new run of the function initiates a request to the previous page, since the variable responsible for the page number is incremented with each run. This allows for efficient data processing and minimizes server load.

The getArticles() function is executed until the value of the variable responsible for the next page number exceeds the set constant, which represents the total number of pages. When this happens, code execution stops, and a notification is printed to the console. This prevents the parser from endlessly attempting to access the site, which is impractical. Optimizing the data retrieval process allows for efficient use of resources and ensures that the parser operates within specified constraints.

Result of parsing the table of contents of the "Journalist" magazine. Screenshot: Evgeny Kolesnikov for Skillbox Media

Once the code is complete and configured correctly, simply run it by clicking on the start.bat file and monitor the process in real time. In about half an hour, an HTML file will be generated containing a list of all 2920 articles with links, as planned. This process allows you to efficiently collect and organize information, which greatly simplifies further work with the content.

Scraping a website with the second type of pagination: infinite loading on click

The second method involves adding additional articles to the same page. Typically, in such cases, there is no easy way to jump to a specific date or to the end of the content. Pages with articles do exist, but they are only for the benefit of the server that processes the loading requests, not for users. Therefore, it is important to optimize the loading process to improve user experience and increase the page's visibility in search engines.

To demonstrate the pagination method, we will scrape the Skillbox Media "Code" section, focusing exclusively on articles, excluding news. In this context, the question arises of how to apply the previously described data collection principles if there is no visible pagination on the page. We will follow the same steps as in the previous example to efficiently extract the required information.

In this situation, a different approach to analysis is necessary. To do this, open your browser's web development tools and go to the "Network" tab. This will allow you to monitor the site's requests. Then, on the category page, click the "Show More" button to load additional materials. This method will help you study the content and interact with the site more effectively.

Clicking the button makes a POST request to the Skillbox Media website. Screenshot: Evgeny Kolesnikov for Skillbox Media

The list of requests displays a POST request to the skillbox.by website to execute a PHP file called getArticlesIndex.php. The response to this request is returned in JSON format, which is widely used for transferring data. Request URL: https://skillbox.by/local/ajax/getArticlesIndex.php. In the "Request" tab, you can see the parameters with which this request is sent.

The "PAGE_NUM" parameter, set in this case to 2, specifies the current page number. The "SECTION_ID" parameter, equal to 10, corresponds to the "Code" category that we plan to parse. The "COUNT" parameter, equal to 7, determines the number of materials displayed on the page. These settings allow you to effectively organize the display of content according to the specified category and number of articles.

Additional articles are loaded via a POST request, not a GET request. Typically, a GET request is used to retrieve data from the server, while a POST request is intended for sending it. The reasons for this choice are a separate topic that is beyond the scope of this text. When developing a parser, it is important to consider the logic established by the site developers. However, for fun, you can conduct a small experiment and examine how this affects data processing.

If we copy the link above and follow it without specifying parameters, the site will return an error («status: error») – it simply won't know what information we are asking for. Here the browser will send a GET request, not a POST one, but the site still responds (the error message is also a message).

Error message when sending a GET request Screenshot: Evgeny Kolesnikov for Skillbox Media

When executing a direct request to the specified link with the correct parameters, the result of the GET request will be a JSON response containing the HTML code of additional articles and the status «ok». This allows you to effectively obtain relevant information and data for further processing.

Let's combine the base link with the specified parameters into a single string: https://skillbox.by/local/ajax/getArticlesIndex.php?params[SECTION_ID]=10&params[CODE_EXCLUDE]=news&params[FIRST_IS_FULL]=Y&params[COUNT]=7&params[PAGE_NUM]=2&params[FIELDS][]=PROPERTY_FAKE_COUNTER&params[CACHE_TYPE]=A&params[COMPONENT_TEMPLATE]=articles. By executing a GET request for this link, we will receive data in JSON format. This will represent the markup of the list of articles on the second page, which can be easily verified using the search by the article titles available on the site. This approach allows you to effectively extract information and adapt it to the needs of the user.

Response of the Skillbox Media website to a GET request for the second page of the "Code" section. Screenshot: Evgeny Kolesnikov for Skillbox Media

Now that we've covered the pagination structure, we need to clarify where exactly the parser should stop—where the articles end. Defining article endpoints is important for streamlining the data collection process and improving content quality. This will help avoid redundant information and focus on the most relevant and valuable materials. Properly setting parser stops ensures efficient information extraction and improves content performance.

The search query "Skillbox launched media" returns an article titled "A selection of Skillbox articles in honor of the media launch," published on July 8, 2018, on the Skillbox blog on the Medium platform. This suggests that content on the site began appearing around the first half of 2018.

To determine the last page number, we use the brute-force method of the «[PAGE_NUM]» parameter. If the requested page number is missing, the site automatically redirects to the first page. In this case, it is necessary to reduce the page number to display the content correctly.

At the time of writing, the last page of the "Code" section was number 101, with each page containing seven articles. This suggests that the total number of articles in this category is approximately 707. In reality, the number turned out to be 705, as the last page only contains five publications. The author could have verified the number of articles with the section editor, who would have confirmed the accuracy of the calculations, but this is not always possible. Judging by the information provided by the site, the first article in the section is "Which programming language should a beginner learn? Choosing JavaScript," dated May 3, 2018.

This is what the end of the section looks like if you scroll to it manually (however, you shouldn't do this) Screenshot: Evgeny Kolesnikov for Skillbox Media

Let's return to the first section page and use web developer tools to determine the location of the links to the articles. This will allow us to effectively indicate them to the parser and optimize the data collection process.

Location of the block with the pinned article Screenshot: Evgeny Kolesnikov for Skillbox Media

In There are no problems with the article when pinned - it is unique and presented in an element with the class «important-block__main-title».

Location of the block with a regular article Screenshot: Evgeny Kolesnikov for Skillbox Media

In this case, we are dealing with a block element with the class "media-catalog__tile-title" that is nested within a link. This may seem unusual, since the title only contains text, and the link has no class. However, we can solve this problem with proper navigation and HTML structure optimization. A proper approach to navigation will ensure user convenience and improve the site's indexing by search engines.

Create two files: skbx_code_articles_parser.js for the code and start.bat for running it. The batch file (.bat) will change only slightly—only the path and name of the script being run. Paste the following code into the JavaScript file:

Viewing code on Pastebin allows you to quickly share software and scripts with other users. This service provides a convenient interface for uploading and storing code that can be easily viewed and edited. Pastebin supports a variety of programming languages, making it an ideal tool for developers and students. Using Pastebin, you can create temporary or permanent links to your code, making collaboration and knowledge sharing easier. If you need to share a code snippet or get programming help, Pastebin is a great choice.

Our code has undergone changes, but remains similar to previous versions. Please note a few key nuances:

  • We are making a POST request, not a GET request, so instead of axios.get() we will use axios.post() (line 29).
  • We use the URLSearchParams interface to pass and read the network request parameters found above in a special format (lines 14–23, 27, and 62–63).
  • We briefly touch on obtaining data from JSON format, but only in one line (lines 32–33).
  • On each page, we first parse the pinned article separately, and then six regular ones, following the logic of the site layout.
Parsing result of the "Code" section of Skillbox Media Screenshot: Evgeny Kolesnikov for Skillbox Media

Run the parser by clicking on the start.bat file and wait for the result in about 17 minutes. As a result, you will receive an HTML file containing a list of 705 articles.

Nothing Lasts Forever

Your parser may also be useful. You can refer to this material either a day or a year after its publication. At the time of writing, the Skillbox Media website displayed seven articles per page: one pinned and six below. However, the developers soon unexpectedly increased the number of materials to 14, which are now arranged in the following order: one pinned, six below, then one pinned again, and six more at the bottom of the page.

It is important to keep in mind that when developing a scraper for collecting data from websites, you must understand that the design and structure of materials on the site may change. This means that your scraper will not work flawlessly on the same resource for a long time. Regularly updating and adapting your scraper to changes on the target site is an important part of its effective functioning. Be prepared to make changes to the code and test the scraper to ensure it matches the current data structure.

During testing, it was found that the code successfully processes requests for 14 materials instead of 7. This is due to the fact that the parameters determining the page number and the number of articles on it are interrelated. As a result, the server response is adapted to your request, even if it is based on an outdated model. This approach provides flexibility and allows users to obtain the information they need in the optimal format.

To optimize navigation and improve the logic of content display, we recommend changing the number of pages in the constant by specifying a value half that value. You should also reconsider the order in which elements are iterated on the pages. As the number of articles on each page increases, it is important to maintain the correct order. To do this, you need to define an algorithm that will look like this: first, place the first pinned element, then regular articles one through six, then the second pinned element, and finally regular articles seven through twelve. This change will help improve the perception of content and improve the ease of navigation for users.

Conclusion

We have studied two effective methods for automatically collecting materials from media websites. However, many other possibilities exist. For example, you can parse the list of materials into an Excel spreadsheet or save them to a bookmarks file for later import into a browser. It's also possible to create an attractive design for data presentation, automate sending results to a Telegram chat using a bot, sort materials, and conduct content analysis, including studying categories, keywords, and publication frequency. Additionally, you can implement flagging features to mark materials as read, and much more—the only limit is your imagination.

The code presented above may be far from perfect, as it was created not by a professional programmer, but by a journalist who uses programming in their work. This point underscores the relevance and accessibility of programming for everyone. In today's world, programming skills are becoming essential, and anyone can master them if they are willing to go beyond their usual methods and explore new approaches.

Read also:

  • How to add JavaScript without making your site fat
  • What is a framework and how to choose a front-end framework: tips from experienced developers
  • A beginner's guide to writing JavaScript