Our MBA students in Data Journalism produced a series of tutorials as their final project in the Low Code course: Transforming data into news stories without programming, taught by Professor Adriano Belisário. This month you can check out some of their work and... Enjoy the adventures with the tutorials they've created. Today you can check out the tutorial made by Carolina Timm.
Web Scraper: a tutorial with the beginning, middle, and end of a scraping exercise + practical tips
Hey everyone. Tired of spending hours copying and pasting information that could clearly be in a downloadable spreadsheet instead of scattered across a website?
Would you like a simple way to collect all the information you need at once? How about a table with structured data of your choice extracted at the end of the data collection process?
So relax, what you're looking for is called data scraping, a method of collecting information in an automated way. Here, we'll learn about one tool in particular: Web Scraper.
Why read this tutorial?
By the end of this reading, you will know how to install and use the Web Scraper to create something called Sitemap containing Selectors capable of navigating autonomously and extracting information as you determine. To guide us in this tutorial, we will perform a scraping of Rotten Tomatoes (a film and series review site), more specifically content titled 200 Best LGBTQ+ Movies of All Time.
Contextualizing the chosen website
The data relevant to this tutorial is in this... url.
Note that the ranking is distributed across four pages of results, in this order of presentation: 200-151; 150-101; 100-51; 50-1. On the homepage, there is basic information: movie poster, title, release year, rating (the site's classic "Tomatometer"), whether there is critical consensus, synopsis, cast, and ranking position.
It's possible to locate and read the content on the website, but the elements aren't organized in a structured way that allows for easy extraction, handling, and analysis of the available data. Without a scraper, it would be necessary to organize the elements manually, repeating the process of selecting each piece of information for each film 200 times (literally). In other words, it would be a repetitive and time-consuming task. Therefore, from here on, we'll work hand-in-hand with a tool called... Web Scraper.
Getting to know Web Scraper
webscraper.io It's a free extension that you can install in your browser in just a few minutes. At first glance, before installation, this is what it looks like:
Just click Add Add the extension to your browser and authorize access permission. To check if the installation was successful, go to Extensions and check if the icon and name of Web Scraper They're already showing up there.
Step by step
First, install Web Scraper in your browser.
Access Roten Tomatoes
Anywhere on the page, right-click and select Inspect.
If the section opens on the side of your browser, I recommend moving it to the bottom. To do this, simply click on the three dots in the upper right corner and select the icon that illustrates this viewing method.
Next, locate the tab. Web Scraper at the end of the first line.
When you click on it, the extension displays three pieces of content:
>Sitemaps, where the sitemaps created or imported in your browser are "stored" (at this moment, right after installation, this tab will be empty);
>Sitemap, the "current" Sitemap space;
>Create new sitemapwhich offers two options:
Create Sitemap This is the option that interests us at the moment. After clicking on it, you must fill in two fields and finish by clicking on... Create Sitemap.
>Sitemap nameThe name that will identify your Systemp. It must contain only lowercase letters, without accents or spaces. Here, it's called filmes_lgbtq
>Start URLCopy and paste the page URL: https://editorial.rottentomatoes.com/guide/best-lgbt-movies-of-all-time/
As soon as you click on Create SitemapYou will then be taken to the Sitemap filmes_lgbtq tab, which will then present you with the selector field.
Go Click on Add new selectorThe first selector we're going to create will gather all the information that accompanies each film: movie poster, title, release year, website rating, concise synopsis, cast, and ranking position. Here, it will be called "movie." TypeYou will be presented with several options:
Our selector is ElementBelow, check the square. MultipleBecause we want the selector to continue browsing through the other films and selecting the information pattern that we indicate to it:
To select, click on SelectThen, simply move your cursor across the page until all the elements of the first film are selected (poster, title, year, and so on). Next, scroll down the page and make the same selection for the film immediately below. That's it, at this point, the web scraper has understood your idea and will select the next ones on this page.
Confirm the action by clicking on Done selectingCheck if Multiple It's confirmed and saved. We already have our first selector!
From now on, we will create selectors within this main selector for each piece of information that interests us.
However, before that, a few words: if you click on Data Preview Upon reviewing the information, you'll notice that only the films on the first page were selected. Wait, so do I need to create a different Sitemap for each of the four pages and then combine everything in a spreadsheet editor after exporting? No, the good news is that you don't need to do all that work. Simply change the URL..
Click through the remaining pages of the ranking, one by one, and observe the... URL of each one. Did you notice how the change is the number at the end?
https://editorial.rottentomatoes.com/guide/best-lgbt-movies-of-all-time/2/
https://editorial.rottentomatoes.com/guide/best-lgbt-movies-of-all-time/3/
https://editorial.rottentomatoes.com/guide/best-lgbt-movies-of-all-time/4/
Em Sitemap films_lgbtqYou will have these options:
Choose Edit metadata.
Em Start URLLet's change it to https://editorial.rottentomatoes.com/guide/best-lgbt-movies-of-all-time/[1-4Therefore, the four pages will be scraped off.
After saving this change, click again. Sitemap films_lgbtq and return to SelectorsNow, instead of adding a new "main" selector, let's create selectors within the already created selector. To do this, click on movie:
Notice that it is now indicated next to _root at the top. Now that's better. Add a new selector. Shall we begin by selecting the title? The Type Now is Text and you no longer need to select the option MultipleUltimately, there's only one title within our main "movie" selector. Using the cursor, select only the movie title information and repeat this same selection with the movie below.
WebScraper has already understood what you want to select as the "title". To confirm, you can check the Data Preview again.
Save the selector ☺
With this tactic, we will continue creating selectors within the main "film" selector:
One selector for each piece of information: title, year, poster, ranking, cast, synopsis, and rating. All of them will be... TypeTextHmm, almost. The only exception is the poster, which will be... Image Type.
Have all the selectors already been created? Great!
Em Sitemap films_lgbtqNow look at the option. scrapeYes, this is the button that makes the scraping happen automatically. Without needing to change the speed settings, clicking on it will automatically start the process. Start ScrapingThe extension will open a new window and collect the data from the created selectors. Do not close the window with the Web Scraper icon.Just let it happen ☺
After notification that scraping is complete, in the tab Sitemap films_lgbtYou can extract the information in CSV format. Export data as CSV
After exporting, this is how your spreadsheet
In addition, you can also export your own. Sitemap, which will be generated as follows:
{“_id”:”filmes_lgbtq”,”startUrl”:[“https://editorial.rottentomatoes.com/guide/best-lgbt-movies-of-all-time/[1-4]”],”selectors”:[{“id”:”filme”,”type”:”SelectorElement”,”parentSelectors”:[“_root”],”selector”:”div.countdown-item:nth-of-type(n+2)”,”multiple”:true,”delay”:0},{“id”:”titulo”,”type”:”SelectorText”,”parentSelectors”:[“filme”],”selector”:”h2 a”,”multiple”:false,”regex”:””,”delay”:0},{“id”:”ano”,”type”:”SelectorText”,”parentSelectors”:[“filme”],”selector”:”span.subtle”,”multiple”:false,”regex”:””,”delay”:0},{“id”:”poster”,”type”:”SelectorImage”,”parentSelectors”:[“filme”],”selector”:”img”,”multiple”:false,”delay”:0},{“id”:”ranking”,”type”:”SelectorText”,”parentSelectors”:[“filme”],”selector”:”div.countdown-index”,”multiple”:false,”regex”:””,”delay”:0},{“id”:”elenco”,”type”:”SelectorText”,”parentSelectors”:[“filme”],”selector”:”div.cast”,”multiple”:false,”regex”:””,”delay”:0},{“id”:”sinopse”,”type”:”SelectorText”,”parentSelectors”:[“filme”],”selector”:”div.synopsis”,”multiple”:false,”regex”:””,”delay”:0},{“id”:”avaliação”,”type”:”SelectorText”,”parentSelectors”:[“filme”],”selector”:”span.tMeterScore”,”multiple”:false,”regex”:””,”delay”:0}]}
For your information, this is the content you paste into the option. Import Sitemap whenever you want to consult this Sitemap in another browser or share so that other users can access it.
This is our finish line. Best practices and long life with Web Scraper!
