Our MBA students in Data Journalism produced a series of tutorials as their final project in the Low Code course: Transforming data into news stories without programming, taught by Professor Adriano Belisário. This month you can check out some of their work and explore the tutorials they created. The first on the list is by student Beatriz Pinheiro.
Despite the recent shift in Brazilian women's football, with changes to the calendar, greater availability of game broadcasts, and increased media and public interest, there is still a certain lag in the professionalization of the sport, which was prohibited by decree for 40 years in Brazil. This delay is reflected in the historical records of women's football, which is quite detrimental to journalistic coverage, as it is difficult to find systematized data to support the production of news stories.
Considering this scenario, the objective of this tutorial is to present Web Scraper as a tool that can help explore and create more user-friendly databases about teams, players, and competitions, in order to not only facilitate journalistic work but also contribute to historical records for the development of Brazilian women's football.
Understanding the tool
Web Scraper is a Google Chrome extension that allows you to extract data using the HTML code of websites as a source. This code structures website information into elements that function like "boxes" in which the data is ordered. Web Scraper's role is to extract the data from these boxes and transform it into a structured spreadsheet.
The data source used in this tutorial will be Soccerway Women, a website that compiles statistics from women's soccer games worldwide and information such as: teams, athletes, championships, matches, results, etc. For this exercise, we will use the table from the Brazilian Women's Championship A1 – 2020 as an example, and extract information about all the athletes who competed in the competition.
The idea we will work with here is to select the following data for all players: club, name, position, age, games played, and goals scored in the championship. If this process were done manually, we would have to individually access the page of each of the 16 teams, enter each player's page, and copy and paste the desired information into a spreadsheet.
Besides being extremely laborious and tiring, this process would also be more prone to errors if done manually, which would jeopardize the entire analysis of the collected data. That's where Web Scraper comes in, allowing you to automate the steps mentioned above.
Hands-on
The first step to begin the scraping process is to install Web Scraper, which can be done via this link. Then, simply click on the extensions icon in the upper right corner of Google Chrome and select Web Scraper to activate it.
With the extension installed and the Brazilian Women's Championship standings open, we will right-click and select the option. InspectNotice that a tab opens at the bottom of the screen, displaying information about the...
Page codes. Let's pay attention to the tab. Web Scraper, the last one that appears in the menu, on the right.
With the Web Scraper tab open, click the button. Create new sitemap and select the option create sitemapTwo blank fields will appear – the first one, Sitemap nameThis field will be filled in with the name of your robot, which will scrape the information. In our case, we will name it "brasileirao-feminino-2020". Below, the field will appear. Start URLIn this section, we will define the starting page for scraping the data. In this example, the standings page for the Brazilian Women's Championship.
Next, we'll define the first parameter to be scraped by the robot. To do this, we'll click the button. Add new selector and work with the fields ID, Type, and SelectThe Id field is used to name the information we want to extract, and in this case, we want information on each of the teams in the Brazilian Women's Championship, so we'll call the selector "teams".
The Type field indicates the type of HTML code element to be scraped, which can be text, link, image, among other options that appear when clicking on the field. Looking at the Brazilian Women's Championship standings, we see that each team in the table is a link that directs to the team's individual page. Therefore, in this step, we will select the option link.
The next step is to activate the button. select and click on the name of each team. Note that the link is highlighted in a red box and, from the second click onwards, the tool itself recognizes the selection we want to make. Check if everything is correct and confirm by clicking the green button. done selecting, which appears above the inspection bar.
Don't forget to check the option. MultipleTo ensure that all selected elements, that is, all teams, will be recognized by the selector, simply click the button. save selector at the bottom of the page and there you have it, the first scraper.
Our goal here is to gather information about the athletes from each team, so we'll access the Corinthians page, the top-ranked team, as an example, and scroll down to the section with the athletes' information.
In the Web Scraper control bar, click on the already created "teams" selector and repeat the previous process, this time for each athlete on the team: create a new selector, enter the name "players" in the Id field, select the element type as link again to ensure the robot accesses each player's page, and click the select button. Then, simply select the name of each player and click done selecting, remembering to check the multiple option. Finally, save the selector.
The next step is to access one of the athletes' pages, click on the "players" selector in the Web Scraper control bar, and repeat the process for the information we are looking for. This time, we want to select the position each athlete plays, so let's name the selector "position". Now, the type of element we want to select is a textand we won't need to select the option MultipleSince we only have one block of information of interest, we just need to save the selector.
From here on, the process remains the same for the other athlete information we are seeking: age, games played, and goals scored this season.
Scraping the data
With that done, it's now time for data scraping. In the Web Scraper control bar, we'll click the button. sitemap brasileirao-feminino-2020, select the option scrape, and then click the button start scraping.
Now it's time to rest, because the robot is already working: notice that a new browser window opens, in which the tool accesses the pages of each team and each athlete in the 2020 Brazilian Women's Championship to scrape the data we specified.
When the process is complete, simply click the button. refreshThe web scraper will then show a preview of the organized table after scraping the data. Now, simply click the sitemap button again. Brazilian Women's Championship 2020 and select the option Export as CSV.
Done! Now we have the complete table, with information on all the players who played in the 2020 Brazilian Women's Championship.
