Tech Talk with Toni

  A matrix of dots that you did not bother to count Let's face it, working with exponentially expanding datasets can be both exciting and overwhelming. Imagine dealing with a dataset of 22 million rows - that's a lot of information to process!  The question is, can your ETL process handle it?  This is a problem that I faced this week, and I had to find a way to improve the performance of the process that updates a dashboard as it was taking a decade to update. The initial question that crossed my mind was, "What was the actual size of this dataset?" At first, I mistakenly assumed that the dataset contained between 1 and 6 million rows. A quick COUNT(*) query made it clear that my estimate was way off and also provided me with some clarity to the problem. That dataset was a behemoth, it probably had its own gravitational pull! The sad truth was that the process was not scalable, and it was clear that immediate improvements were necessary. Here are a three tips that I h...

 Here's Your Guide to Successful Data Collection Tom is going duck hunting Congratulations on embarking on the exciting journey of becoming a Data Analyst!  You’ve got the degree, you’ve got the skills, and now it’s time to enter the wild world of data. But, wait!  Before you head out with your data traps and bait, there are a few things you need to know.  Unlike the nice and tidy datasets you worked with in school from Kaggle, data in the real world is a wild beast.  It’s unpredictable, unruly, and sometimes downright dangerous. Think of yourself as a hunter.  You need to know exactly what you’re looking for, where to find it, and how to capture it.  And, just like hunting, the data collection process requires patience, strategy, and a willingness to learn from your mistakes. So, here are five steps to help you capture the right data: Step 1: Define the Problem Just like a hunter needs to know which animal to hunt, you need to define your problem befo...