Data Analytics With Python

Working with Missing Data in Pandas Working with Missing Data in Pandas In this blog post, we will explore various techniques for handling missing data using the Pandas library in Python. Specifically, we will focus on removing rows or columns with NaN or None values using different methods provided by Pandas. Setup We begin by importing the required libraries and reading the CSV file containing the movie data. import pandas as pd import numpy as np loc = 'https://raw.githubusercontent.com/aew5044/Python---Public/main/movie.csv' m = pd.read_csv(loc) Creating a Subset and Introducing Missing Values We create a subset of the data, keeping only the columns we want to work with. Additionally, we introduce NaN values in specific rows and columns. m1 = m[['movie_title','director_name','actor_1_name']][0:5] m1.loc[0:1,'director_name'] = np.nan m1.loc[0:2,'actor_1_name...

Understanding Groupby and Agg in Pandas for Group-wise Analysis Hello, data enthusiasts! Today, we are delving into one of the most useful functionalities in Pandas: the groupby and agg methods. We will particularly look into counting IDs and calculating the percent of total for various groupings in our data. Grouping in Pandas Pandas' group_by method is highly powerful. It allows us to split our data into separate groups to perform computations for better analysis. Let's consider a DataFrame 'df' with columns: 'id', 'course', 'building', and 'room'. import pandas as pd # Suppose df is your DataFrame print(df.head()) You might see something like this: | | id | course | building | room | |---|----|--------|----------|------| | 0 | 1 | Math | A | 101 | | 1 | 2 | Math | B | 102 | | 2 | 3 | Bio | A | 101 | | 3 | 4 | Chem | B | 102 | | 4 | 5 | Bio | B | 103 | Now, say ...

CategoricalDtypes in Pandas CategoricalDtypes can also be used for filtering.  For example, if we have low, high, and medium survey responses, we can use CategoricalDtypes to filter all responses less than or equal to medium. There are three basic steps! Create the CategoricalDtype. Apply the CategoricalDtype to the pandas Series. Filter the data. It is just that simple.  Let's look at an example. When we print the filtered_df, we can see we have our desired output.  It is just that simple! Link to Google Colab with the code!

  The pd.api.types.CategoricalDtype is used to define a categorical data type for a pandas Series. It allows you to specify the categories and their order explicitly. This is a very zen-like application. Hence the local guru meditates with always properly sorted indexes!  In a customer satisfaction survey, an organization collects feedback from customers regarding their experience with the company's products or services. To analyze the survey data effectively, they can leverage the CategoricalDtype feature in pandas to handle the satisfaction levels expressed by customers. The satisfaction levels are categorized as "Low," "Medium," and "High," representing different levels of customer satisfaction. Let's explore how CategoricalDtype can be utilized in this scenario: Defining the Categorical Data Type: Using CategoricalDtype, the organization can define a categorical data type with the categories ["Low", "Medium", "High...

Decoding the Art of Importing CSVs with Pandas The purpose of this blog is to review how python's pandas can clean column names dynamically to  remove leading and trailing spaces, replace spaces with "_" remove all special characters  Today, we are going to unravel a line of pandas code that incorporates several sophisticated features, making it a powerful tool to automate data import and preprocessing. Let's take a look at the code snippet: At first glance, it seems complex, but once we break it down piece by piece, you will appreciate its functionality. Unraveling the Code The central function of this line is `pd.read_csv()`, which is pandas' built-in function to read comma-separated values (CSV) files into a DataFrame, a two-dimensional size-mutable, heterogeneous tabular data structure. The `read_csv` function takes in several parameters:  `loc`: This is the location or path where your CSV file is stored.  `sep=','`: This is the separator/delimiter whi...

Exploratory Data Analysis (EDA) Situation: I’m at a new analyst job, and I’m given access to a new database. What next? Do I wait for a data request?  Do I wait for a random task from my boss? Do I review established reports?  What do I do? I just started a new job in February of 2022. Every day I continually see new datasets, inherit established code, and update old code.  I like to ask open questions: What does each row represent? What are the main “filter” columns? What are the data types? How did the last analyst handle the data? Each list of questions and associated processes are unique to each person and the tasks at hand.  By now, this process is second nature. The purpose of this blog is to learn about Pandas. How  would I approach this in Pandas? Let's dig in! Assumptions, I have access to the table. In this example, m associated with movies as a DataFrame  First, I want to look at everything. The describe method is a method to quickly ...

  Goal: Filter DataFrames Filtering data is required for every data analysis project.  The majority of blogs I see only detail how to filter the DataFrame by the row index. It is rare I need to filter on the row index and I don't want to reset the index for every filter. For example, we only want data in the DataFrame where the budget is over $10,000,000, and the director's name is James Cameron.  Well, that is a very specific example, but you get the idea.   There are two general steps (example on Google Colab ) Define the filters/mask Reference the mask(s) between []  I always made the filtering process more difficult than reality.  After looking, and looking, and looking for ways to filter data in pandas, I found a method that meets my expectations.  It must be easy to remember, discuss, and explain to non-programmers.  Also, if another person not all that familiar we Python, they can update the filters as needed, add new ones, and continu...

Goal: Create a new column based on data from another column. Use Pandas.DataFrame and Numpy.select to create the new column.  You can see the full program at  Google Colaboratory There are three basic steps to accomplish the goal.   Define the conditions Define the values Use np.select to apply the conditions and values. First, create a list containing each "condition," then  create a list containing each return v alue.   The goal is to correlate each condition with each value.  What does that mean?  It means if the first condition in the "conditions" list evaluates as True, return the first value in the "values" list.   An error will occur when the lists have different index counts.  For example, if there are five conditions and 4 values.      This condition example below has two useful applications we would be remiss if we didn't take note:   The first is the use of .isnull() == True. The second is th...

This platform aims to systematically curate and preserve my favored Python techniques for future reference. This digital journal, accessible globally wherever an internet connection is available, is a convenient and effective learning tool.  Going down a wooded path at night, alone, with your trusted companion by your side, can be scary and daunting.  However, with support from your friends, documenting your path (curating a code base like a blog), and continuously working on the code base can make the process appear as a walk in the park!  In a world dominated by sophisticated language models like OpenAI's GPT, blogging may seem like a relic of the past. But here's a nuanced perspective: this blog serves as an innovative instrument for both teaching and mastering new programming languages. It's about immersing in syntax, applying it, and then translating the information into layman's terms for broader understanding. Each topic presented here is elaborated in a separate...