Data Profiling with Python's Great Expectations

-

 

pip install great_expectations

As a Data Quality Analyst, I constantly seek tools to enhance my daily workflow, and Great Expectations is a recent discovery.

Before getting into what this Python library has to offer, let's address a fundamental question:

What is Data Profiling?

In order to effectively tackle any problem, one must first understand it. 

Data Profiling is the art of uncovering and investigating data quality issues, such as duplication, missing values, and inconsistency. 

It is essentially determining the baseline level of quality of a dataset in terms of the six data quality dimensions : Completeness, Uniqueness, Validity, Timeliness, Accuracy and Consistency. 


Great Expectations Data Profiling Example

Now, picture yourself as a pastry chef whose mission is to ensure that every currants roll emerging from your bakery is nothing short of delightful.

 Let's explore how data profiling in Great Expectations guarantees they meet your standards.

Step 1: Defining a Delicious Currants Roll

Similar to the standards for the perfect currants roll – a flaky, golden-brown crust and a filling of sweet currants – in data profiling, you define your expectations or rules to be followed for data quality.

Step 2: Finding the Ingredients 

It's crucial to know the precise locations of key ingredients such as butter, sugar, flour, and currants in your kitchen. Similarly, when working with Great Expectations for data profiling, you must specify where the data is located.

Step 3: Evaluating the Currants Roll

After your currants roll is baked, you taste-test it to confirm it meets your standards. With Great Expectations, you evaluate the results of your data analysis to ensure data quality aligns with your expectations.

Resources




Comments

Post a Comment

Popular posts from this blog

Prompt Engineering : An Introduction

-
Image
  A stereotypical engineer Despite concerns about job changes due to ChatGPT, something interesting is happening: the emergence of a new job title known as the Prompt Engineer . Still skeptical? Take a moment to browse professional platforms like LinkedIn or Indeed—see it for yourself! Now, you might wonder, 'What is Prompt Engineering?' Well, I'll give you a prompt answer (okay, sorry for the pun). Essentially, prompt engineering involves using special words or hints to instruct computers without saying everything explicitly. Think of it like when your mom says, "Bring me my bag from on top of the table." Even if your house has many tables, you know it's not in her car. She assumes you know what her bag looks like, so she doesn't need to describe it. But if she tells you which room to search in, you'd find her bag faster, right? Having said that, here are some tips that can help you write robust prompts: Understand the Task This is where your domain ...
Read more

Women In STEM : Challenges and Advantages

-
Image
  Women make up only 27% of the STEM workforce. Introduction Being a female in Technology, I can also identify as being a "Woman in STEM" and I can say being part of this group is much different from our male counterparts. When you think of an engineer, the immediate image that comes to mind is often that of a man earning a six-figure salary. Before directing any resentment towards men, it is crucial to pose this question:  "What caused this gender gap in the first place?"  Common Challenges Two terms frequently used to answer this question are stereotypes and gender roles. The stereotypical woman is caring, nurturing, and supportive. She is also expected to stay at home, to cook, clean, and raise children. However, in 2024, being a stay at home mom is rare, with more women actively participating in the workforce, some even serving as the primary breadwinners of their households. But, they are still expected to take on the responsibilities of a stay at home mom, whi...
Read more

5 Authentication Methods

-
Image
  This piece of plastic tells people who you are.  The verb authenticate originated from the Latin word authenticus , meaning genuine. Confirming that someone is really who they claim to be has always been a problem that we humans have tried to solve. You see, just because someone says they are Michael Jackson, it doesn't mean they are, because they could also be lying (who doesn't lie to get what they want?) . Before the advent of modern technology, various authentication methods were used to verify the identities of individuals before granting access to resources, from signatures to special passphrases. As a child, I can remember devising specific ways of knocking on the door with my siblings and cousins to confirm each other's identities before granting access to our top-secret room. However, knocking on digital doors requires more robust methods of accessing resources, such as your social media profile or your bank account. Today, I will mention some authentication meth...
Read more

Inductive and Deductive Reasoning

-
Image
Reasoning : the action of thinking about something in a logical, sensible way   Earlier this week, a DataCamp course sparked an interesting thought about how we reach conclusions. In data analysis, we ask key questions, gather information, and draw insights from what we find. This happens across many STEM fields. But something I recently noticed is how we actually get to those conclusions – there are two main methods: inductive and deductive reasoning. Let's break it down: Inductive Reasoning:  Imagine you see all your friends going into tech or medicine. Based on this limited view (your friends), you might guess (general conclusion) that young people aren't interested in plumbing careers. However, this might not be entirely true! Deductive Reasoning:  This approach is more like a detective story. We use established facts (like employment data) and analysis tools to reach specific conclusions. For example, we could analyze data over time to see if there's actually been a...
Read more

Don't Be Bland : Spice Up Your Personal Brand

-
Image
   If you don't define your brand, other people will.  In today’s fast-paced digital landscape, personal branding is no longer a luxury; it’s a necessity.  These are the sentiments echoed by Keron Rose on an episode of his podcast, Digipreneur (check out the links below).  Whether you’re an aspiring developer, a seasoned IT professional, or a tech entrepreneur, your personal brand can significantly impact your career trajectory.  But before I get into that, I wanted to touch on a key difference between personal brand and reputation, which was also mentioned on the podcast. Personal Brand  Your personal brand is a deliberately crafted identity that you strategically create for yourself. You have direct control over your personal brand. It’s the image you project intentionally through your online presence, interactions, and content. Building a personal brand allows you to showcase your expertise, values, and unique qualities. It’s an ongoing process tha...
Read more

3 Common Diseases Associated With Sitting All Day

-
Image
We check our phones every 12 minutes This week I officially became an owner of a smartwatch and I must say that this device has encouraged me to get up and reach my goal of 7500 steps per day more frequently.  However, even before this addition, I've been taking afternoon walks on my own for some time and I do believe that this is a habit that most people can develop. Now, as someone in the field of technology, we often find ourselves spending an unhealthy amount of time glued to screens and seats. Our bodies, with a quarter of our bones chilling in our feet (seriously, go ahead and count!), were clearly designed for more than just staying in one place.  This post serves as your friendly reminder to get up, get moving, and rediscover the joy of not feeling like a human pretzel at the end of the day.  With that being said here are some common diseases associated with sitting all day: Heart disease Sitting for long periods increases your risk of heart disease, the leading c...
Read more

Coding Best Practices : Error Messages Are Friends, Not Foes.

-
Image
  This error's name originated from Room 404 At first, I saw error messages as coding villains, ready to ruin my day with red underlines. As soon as I see one, I panic and do everything humanly possible to get rid of it. But now, I view error messages as superheroes here to help me debug my code. The inspiration for this week's post comes from a problem I encountered on the job with code. The Problem Errors were introduced into SQL queries that were needed to generate a crucial report due to updates I made.  However, I didn't know exactly where to start the debugging process, as all the scripts were being executed dynamically.  Whenever one of these queries failed, the entire process would throw a generic error message that did not specify where the error occurred. The brute force approach of combing through dozens of scripts to find a handful of syntactical errors would have been stressful and time-consuming. The Solution Instead of manually looking for the problem, I wa...
Read more

Upskilling: Certificates vs. Certifications

-
Image
  Fun Fact: Those LinkedIn courses you do are NOT  certifications. It's performance review time, and I had to differentiate between certificates and certifications for the first time.  As we approached our final pivotal meeting for the fiscal year, I was tasked with presenting the total count of certifications attained by our team throughout the year. Initially, I presumed this to be a straightforward task. I circulated a Microsoft Form among my colleagues, expecting a simple input of numbers.  However, upon aggregating the data, I found the resulting figure to be somewhat inflated. To gain a deeper understanding of this count, I requested my colleagues to provide a list of certifications they had completed. When I looked closely at the names of these "certifications," I found that about one out of every four was actually a certificate. This prompted a need for a clearer definition distinguishing between a certificate and a certification. So, today, I'm going to expl...
Read more

There Has Been a Data Breach: Now What?

-
Image
  Hacker : a person who uses computers to gain unauthorized access to data. They say that prevention is better than finding a cure, but in the case of the Telecommunication Services of Trinidad and Tobago (TSTT), that is no longer an option.  Six gigabytes of sensitive data are now up for grabs on the dark web (the internet's equivalent of the black market).  From internal administrative credentials to customer names and addresses, you can access it all using The Onion Router (Tor). RansomEXX has successfully bypassed all the controls that were meant to mitigate TSTT's risk of a cyberattack.  I deliberately italicized the word 'controls' since, based on what Alex Samm said in his post on this topic, passwords were not encrypted, and RansomEXX's modus operandi usually involves gaining access through phishing. Now, I'm not saying that TSTT did nothing to prevent this, but storing passwords as plain text in a text file does make me raise my eyebrows. Alex also mad...
Read more

Scheduling Algorithms

-
Image
Hickory Dickory Dock! Today's post is inspired by my sister (AKA Dr. Dan Dan), who recently asked me for help scheduling appointments for her patients. As a budding dentist, it's imperative for her to find patients and fit them into her already busy schedule. Sometimes, patients cancel, and she faces a swiftly approaching deadline. Just like a computer prioritizes tasks based on system resources, my sister must prioritize her patients based on the complexity of the procedure and the time available. For example, root canals require more time and expertise than a simple cleaning, much like certain programs demand more of a computer's memory and processing power. The Task Manager in your computer is a program that manages all the active processes, or tasks, running at any given time.  It prioritizes tasks based on importance, ensuring that the most immediate requests—like opening Google Chrome—are allocated more system resources. With that being said here are some common sched...
Read more