Hello, today we are going to create a first project for you, beginners in the data area, to be able to start creating a cool portfolio and with all the necessary tools to work with data!

This project shows that, even if you are a beginner in Python, you can always find libraries to perform more complex tasks that you don't yet know how to do from scratch (some things aren't even worth doing from scratch either) . First of all, it is important that you have initial knowledge of Python and SQL, as well as a little knowledge of Tableau to create the dashboard. It is not necessary for you to be an expert, but knowing the basics of these tools will help you follow the project more easily, but you can read the entire article and try to reproduce it as well because I will try to explain it in the simplest way possible so that Now you can start creating your first dashboard!

Shall we get started?

The first step is to have your development environment configured on your machine, the requirements for this project are:

• Python 3
• MySQL 9.1 (latest version on website)
• Tableau Public

I'm developing this project in a Windows 11 environment, so some things may vary depending on your OS or Windows version, but nothing that deviates too much from what I'm going to present here.

Let's start with Python. Go to https://www.python.org/downloads/ and download the latest version of the installer. After installation, restart your PC to avoid bugs (as happened to me hahah) and be able to use the language without problems on the command line.

Then, with MySQL, go to the website https://dev.mysql.com/downloads/mysql/ and download the MySQL Community Server installer. Just follow the standard installation and everything will go perfectly.

Now, with Tableau Public, go to https://www.tableau.com/pt-br/products/public/download and create your account to start the download. Account creation will also be necessary to publish your first dashboard and will also be very important for your portfolio!

Another tool that is not necessary, but is very good to have, is git and a github account. I put all my code with commits and comments here and it's great to use github as a portfolio of your code, but if you don't know git it's okay and your project will work the same way.

When you have everything configured, go to the directory where you will place your application, then let's make some more configurations. You will need some Python libraries to use in the project, I will explain what each one does and how to install them.

The first library we will use is BeautifulSoup. The data we will need for this project is on the internet and we will have to do a process called Web Scraping to collect it, BeautifulSoup will help us with this process by bringing us tools that facilitate this collection.
To install it, just go to the terminal and type

pip install beautifulsoup4

and... that's it! Installing dependencies in Python is very simple!

The second library we will use is requests. If we are going to work with web pages we need something that helps us perform CRUD actions with APIs, so this will be our choice. Again, just install in the terminal with

pip install requests

We will also implement good practices and use environment variables (so that no one discovers our passwords, usernames and other sensitive information in our code), so we will need os and dotenv. os must already be installed by default in python, while dotenv is not, so it's the usual process

pip install dotenv

And last but not least, we need a library to connect to our MySQL database, so let's use mysql.connector

pip install mysql-connector-python

Once we have the development environment configured, just move on to the most fun part of the process, PROGRAMMING!!

We are going to make a project that will be divided into two parts (in terms of code), web scraping and database manipulation, so we will start by creating the web scraping file, which will also be where the main code will go stay, and then we will create a file to place our database manipulation functions. This helps us not only in maintaining the code but also in its reuse.

Create a file called web_scrapper.py in the application directory.
Next, we will import our dependencies that we installed previously.

from bs4 import BeautifulSoup
import requests
import db_manager
import os
from dotenv import load_dotenv

From dotenv we will only need the load_dotenv function and therefore we will only import it.

First, let's think about the structure of our code and write what we want each thing to do, step by step, so it's more organized. We want our code to do the following actions:

1. Create the web scraper and save the data in variables
2. Populate the database with the data we got
3. Get the data from the database and put it in a csv file so we can analyze it in Tableau Public

Let's go in parts, the first part we want to create and test is creating the web scraper, so the best way is to start with that!
We are going to use a website made for this type of thing, https://www.scrapethissite.com/, there you will find several types of pages to practice web scraping. We're particularly interested in the beginner model, so let's make a request for that page:

pip install beautifulsoup4

Here we use the requests get method which would be equivalent to CRUD's read, it returns the web page and stores it in its entirety in the variable we created page_countries_area_population.
Then, we need BeautifulSoup to parse the page's HTML so that it can find the information we need. To do this, we will create a variable called soup and call BeaultifulSoup and pass the text of the variable we created to it

pip install requests

This will return the page with the parse and BeautifulSoup methods linked to it within the variable we created, thus making our work easier.
Now we need to identify the information we want to remove from the page, to do this we need to inspect the web page and identify the elements and their patterns within the html document. In this case we see that the country names are inside an h3 tag and with the country-name class, so let's use this to get the country names

pip install dotenv

Here we call the soup we created earlier and call the findAll function which will fetch all instances of country names for us. The first parameter is the html element we are looking for and the second would be its attributes, as they may have other h3 tags that we don't want it to select, in this case we pass the country-name class to identify the elements we want.
We repeat the process for the number of inhabitants and the area of ??each country

pip install mysql-connector-python

Before passing this data to the database, we will clean it and leave it in a format that prevents unwanted things from entering with it. To do this, I will create a list of tuples to store the data before passing it to the database, as this will make the process easier. Before adding them, however, we need to remove blanks from the country names as well.

from bs4 import BeautifulSoup
import requests
import db_manager
import os
from dotenv import load_dotenv

And with that we already have the data we need! We can cross that first task off our list!

In part two of this article I will teach you how to manipulate a database using Python and finish our project?

The above is the detailed content of How to create a beginner project in data analysis. For more information, please follow other related articles on the PHP Chinese website!