Data Analytics With Python

  The pd.api.types.CategoricalDtype is used to define a categorical data type for a pandas Series. It allows you to specify the categories and their order explicitly. This is a very zen-like application. Hence the local guru meditates with always properly sorted indexes!  In a customer satisfaction survey, an organization collects feedback from customers regarding their experience with the company's products or services. To analyze the survey data effectively, they can leverage the CategoricalDtype feature in pandas to handle the satisfaction levels expressed by customers. The satisfaction levels are categorized as "Low," "Medium," and "High," representing different levels of customer satisfaction. Let's explore how CategoricalDtype can be utilized in this scenario: Defining the Categorical Data Type: Using CategoricalDtype, the organization can define a categorical data type with the categories ["Low", "Medium", "High...

Decoding the Art of Importing CSVs with Pandas The purpose of this blog is to review how python's pandas can clean column names dynamically to  remove leading and trailing spaces, replace spaces with "_" remove all special characters  Today, we are going to unravel a line of pandas code that incorporates several sophisticated features, making it a powerful tool to automate data import and preprocessing. Let's take a look at the code snippet: At first glance, it seems complex, but once we break it down piece by piece, you will appreciate its functionality. Unraveling the Code The central function of this line is `pd.read_csv()`, which is pandas' built-in function to read comma-separated values (CSV) files into a DataFrame, a two-dimensional size-mutable, heterogeneous tabular data structure. The `read_csv` function takes in several parameters:  `loc`: This is the location or path where your CSV file is stored.  `sep=','`: This is the separator/delimiter whi...

Exploratory Data Analysis (EDA) Situation: I’m at a new analyst job, and I’m given access to a new database. What next? Do I wait for a data request?  Do I wait for a random task from my boss? Do I review established reports?  What do I do? I just started a new job in February of 2022. Every day I continually see new datasets, inherit established code, and update old code.  I like to ask open questions: What does each row represent? What are the main “filter” columns? What are the data types? How did the last analyst handle the data? Each list of questions and associated processes are unique to each person and the tasks at hand.  By now, this process is second nature. The purpose of this blog is to learn about Pandas. How  would I approach this in Pandas? Let's dig in! Assumptions, I have access to the table. In this example, m associated with movies as a DataFrame  First, I want to look at everything. The describe method is a method to quickly ...

  Goal: Filter DataFrames Filtering data is required for every data analysis project.  The majority of blogs I see only detail how to filter the DataFrame by the row index. It is rare I need to filter on the row index and I don't want to reset the index for every filter. For example, we only want data in the DataFrame where the budget is over $10,000,000, and the director's name is James Cameron.  Well, that is a very specific example, but you get the idea.   There are two general steps (example on Google Colab ) Define the filters/mask Reference the mask(s) between []  I always made the filtering process more difficult than reality.  After looking, and looking, and looking for ways to filter data in pandas, I found a method that meets my expectations.  It must be easy to remember, discuss, and explain to non-programmers.  Also, if another person not all that familiar we Python, they can update the filters as needed, add new ones, and continu...

Goal: Create a new column based on data from another column. Use Pandas.DataFrame and Numpy.select to create the new column.  You can see the full program at  Google Colaboratory There are three basic steps to accomplish the goal.   Define the conditions Define the values Use np.select to apply the conditions and values. First, create a list containing each "condition," then  create a list containing each return v alue.   The goal is to correlate each condition with each value.  What does that mean?  It means if the first condition in the "conditions" list evaluates as True, return the first value in the "values" list.   An error will occur when the lists have different index counts.  For example, if there are five conditions and 4 values.      This condition example below has two useful applications we would be remiss if we didn't take note:   The first is the use of .isnull() == True. The second is th...