Data Stacks: Google, Microsoft and Amazon

-
Data Stack Illustration

Stack: a pile of objects, typically one that is neatly arranged.

Conversations about tech preferences tend to get heated, whether it’s Apple vs Android, Windows vs macOS, or the eternal debate about pineapple on pizza.

But lately, I’ve found myself drawn to something : data stacks.

Specifically, why do organizations choose Microsoft, Amazon, or Google as their cloud platform?

Microsoft has quietly become the stack of choice for many enterprises, but why?

What’s the real difference between AWS and Azure, and is one easier to learn than the other?

More importantly, are we so locked into these proprietary tools that we’re missing out on better, open-source alternatives?

Learning a new tool takes time, and that time investment often keeps us tethered to one ecosystem. But with so much innovation happening, it’s worth asking: Are we keeping our minds open to what’s out there?

To answer these questions, it helps to first understand the components of a data stack. A data stack isn’t just one tool or service—it’s an ecosystem of interconnected technologies that help organizations collect, process, store, and analyze data. Let’s break it down.

Data Stack Components

Layer Purpose
Data Collection Gather raw data from APIs, IoT devices, CRMs, and analytics platforms.
Data Storage Store data (structured, unstructured, or semi-structured) for querying and analysis.
Data Processing Clean, transform, and prepare data for analysis.
Data Analysis Query and analyze data to derive insights.
Data Visualization Present insights via dashboards, reports, or custom apps.
Security & Governance Manage access, compliance, and data security across the stack.

Cloud Provider Ratings


Criteria AWS Azure Google Cloud
Price 4 ⭐
Flexible but can get expensive

 4 ⭐
Discounts for Microsoft users

 4.5 ⭐
Affordable for storage and analytics
Support

 3 ⭐ 
Strong docs, costly for small teams

 4 ⭐
Enterprise-grade, great integration

 4 ⭐
Good free resources, slightly lagging
Learning Curve

 2.5 ⭐
Steep due to service variety

 3 ⭐ 
 Moderate for Microsoft users

 4 ⭐ 
 Beginner-friendly, analytics-focused
Free Alternatives 4 ⭐ 
Free-tier includes S3, Lambda, RDS		 4 ⭐
Free-tier includes Azure Data Factory		 5 ⭐
Generous free-tier: BigQuery, Firebase


The Case for Open-Source Alternatives

While AWS, Azure, and Google Cloud dominate the cloud computing market, open-source tools provide a compelling alternative. Open-source options allow developers to avoid vendor lock-in, maintain control over their data, and reduce costs. Here are some excellent open-source tools for each layer of the data stack:

1. Data Collection

  • Apache Kafka: A distributed event-streaming platform for collecting and managing real-time data streams.
  • Fluentd: A lightweight log collector that works across multiple environments.

2. Data Storage

  • PostgreSQL: A powerful open-source relational database that handles structured data.
  • Apache Cassandra: A distributed NoSQL database for handling large volumes of unstructured data.

Data Processing

  • Apache Airflow: A popular workflow orchestration tool for building and managing data pipelines.
  • dbt (Data Build Tool): Simplifies data transformation in the ELT process.

4. Data Analysis

  • DuckDB: A lightweight SQL analytics engine, perfect for fast queries on local data.
  • ClickHouse: An open-source columnar database for high-performance analytical queries.

5. Data Visualization
  • Metabase: An open-source platform for creating dashboards and visualizations.
  • Redash: A lightweight BI tool for querying databases and building reports.

6. Security & Governance

  • HashiCorp Vault: An open-source tool for managing secrets, tokens, and encryption keys.
  • Apache Ranger: Ensures fine-grained security for big data environments.


The question isn't whether open-source can compete- it's whether you're ready to explore the possibilities. After all, the world of data is constantly evolving, and staying open to new tools is the best way to keep up.


Resources



Comments

Post a Comment

Popular posts from this blog

Prompt Engineering : An Introduction

-
Image
  A stereotypical engineer Despite concerns about job changes due to ChatGPT, something interesting is happening: the emergence of a new job title known as the Prompt Engineer . Still skeptical? Take a moment to browse professional platforms like LinkedIn or Indeed—see it for yourself! Now, you might wonder, 'What is Prompt Engineering?' Well, I'll give you a prompt answer (okay, sorry for the pun). Essentially, prompt engineering involves using special words or hints to instruct computers without saying everything explicitly. Think of it like when your mom says, "Bring me my bag from on top of the table." Even if your house has many tables, you know it's not in her car. She assumes you know what her bag looks like, so she doesn't need to describe it. But if she tells you which room to search in, you'd find her bag faster, right? Having said that, here are some tips that can help you write robust prompts: Understand the Task This is where your domain ...
Read more

Coding Best Practices : Error Messages Are Friends, Not Foes.

-
Image
  This error's name originated from Room 404 At first, I saw error messages as coding villains, ready to ruin my day with red underlines. As soon as I see one, I panic and do everything humanly possible to get rid of it. But now, I view error messages as superheroes here to help me debug my code. The inspiration for this week's post comes from a problem I encountered on the job with code. The Problem Errors were introduced into SQL queries that were needed to generate a crucial report due to updates I made.  However, I didn't know exactly where to start the debugging process, as all the scripts were being executed dynamically.  Whenever one of these queries failed, the entire process would throw a generic error message that did not specify where the error occurred. The brute force approach of combing through dozens of scripts to find a handful of syntactical errors would have been stressful and time-consuming. The Solution Instead of manually looking for the problem, I wa...
Read more

5 Authentication Methods

-
Image
  This piece of plastic tells people who you are.  The verb authenticate originated from the Latin word authenticus , meaning genuine. Confirming that someone is really who they claim to be has always been a problem that we humans have tried to solve. You see, just because someone says they are Michael Jackson, it doesn't mean they are, because they could also be lying (who doesn't lie to get what they want?) . Before the advent of modern technology, various authentication methods were used to verify the identities of individuals before granting access to resources, from signatures to special passphrases. As a child, I can remember devising specific ways of knocking on the door with my siblings and cousins to confirm each other's identities before granting access to our top-secret room. However, knocking on digital doors requires more robust methods of accessing resources, such as your social media profile or your bank account. Today, I will mention some authentication meth...
Read more

There Has Been a Data Breach: Now What?

-
Image
  Hacker : a person who uses computers to gain unauthorized access to data. They say that prevention is better than finding a cure, but in the case of the Telecommunication Services of Trinidad and Tobago (TSTT), that is no longer an option.  Six gigabytes of sensitive data are now up for grabs on the dark web (the internet's equivalent of the black market).  From internal administrative credentials to customer names and addresses, you can access it all using The Onion Router (Tor). RansomEXX has successfully bypassed all the controls that were meant to mitigate TSTT's risk of a cyberattack.  I deliberately italicized the word 'controls' since, based on what Alex Samm said in his post on this topic, passwords were not encrypted, and RansomEXX's modus operandi usually involves gaining access through phishing. Now, I'm not saying that TSTT did nothing to prevent this, but storing passwords as plain text in a text file does make me raise my eyebrows. Alex also mad...
Read more

Scheduling Algorithms

-
Image
Hickory Dickory Dock! Today's post is inspired by my sister (AKA Dr. Dan Dan), who recently asked me for help scheduling appointments for her patients. As a budding dentist, it's imperative for her to find patients and fit them into her already busy schedule. Sometimes, patients cancel, and she faces a swiftly approaching deadline. Just like a computer prioritizes tasks based on system resources, my sister must prioritize her patients based on the complexity of the procedure and the time available. For example, root canals require more time and expertise than a simple cleaning, much like certain programs demand more of a computer's memory and processing power. The Task Manager in your computer is a program that manages all the active processes, or tasks, running at any given time.  It prioritizes tasks based on importance, ensuring that the most immediate requests—like opening Google Chrome—are allocated more system resources. With that being said here are some common sched...
Read more

Value Creation

-
Image
Money grows on trees—the tree is a metaphor for your mind. Defining a problem is the first step to solving it, but have you ever considered the value your solution will create? Before jumping into problem-solving, it's crucial to ask: Is this the right problem to tackle, and is solving it worth the cost? As professionals—especially in tech—it’s easy to fall into the habit of building solutions without pausing to assess whether the effort aligns with the value it delivers. For example, imagine spending weeks building an automated self-watering system for a cactus. Sure, it’s a cool project, but was it really worth the time and resources? After all, a cactus only needs watering once a week, and it would take less than a minute to do it yourself. Sometimes, solving the problem manually is the most effective and resource-efficient option. Your salary is directly linked to the value you bring to a company, and time and energy are finite resources. How we choose to allocate them signif...
Read more

Data Profiling with Python's Great Expectations

-
Image
  pip install great_expectations As a Data Quality Analyst, I constantly seek tools to enhance my daily workflow, and Great Expectations is a recent discovery. Before getting into what this Python library has to offer, let's address a fundamental question: What is Data Profiling? In order to effectively tackle any problem, one must first understand it.  Data Profiling is the art of uncovering and investigating data quality issues, such as duplication, missing values, and inconsistency.  It is essentially determining the baseline level of quality of a dataset in terms of the six data quality dimensions : Completeness, Uniqueness, Validity, Timeliness, Accuracy and Consistency.  Great Expectations Data Profiling Example Now, picture yourself as a pastry chef whose mission is to ensure that every currants roll emerging from your bakery is nothing short of delightful.  Let's explore how data profiling in Great Expectations guarantees they meet your standards. Step 1...
Read more

The Algorithm : Musk's Mental Framework

-
Image
A Decision Tree Today’s post is inspired by Elon Musk, the captivating biography by Walter Isaacson. At 688 pages, it’s a deep dive into the mind and methods of one of the most transformative figures of our time. Over the past two weeks, I’ve spent my early mornings absorbing Musk’s journey—and it’s left me contemplating the driving force behind his success.  What is the “algorithm” that powers his extraordinary ventures? Though his leadership style is unconventional, it’s undeniably effective. Musk has co-founded revolutionary companies like PayPal, SpaceX, and Tesla, each of which has disrupted its industry and pushed boundaries.  But what makes him tick? What’s the underlying formula behind his ability to juggle ambitious goals, relentless innovation, and transformative leadership? The Algorithm: Musk's Mental Framework As detailed in Isaacson’s biography, Musk often speaks of a personal algorithm—a mental framework that guides his decision-making process.  The algorit...
Read more

Upskilling: Certificates vs. Certifications

-
Image
  Fun Fact: Those LinkedIn courses you do are NOT  certifications. It's performance review time, and I had to differentiate between certificates and certifications for the first time.  As we approached our final pivotal meeting for the fiscal year, I was tasked with presenting the total count of certifications attained by our team throughout the year. Initially, I presumed this to be a straightforward task. I circulated a Microsoft Form among my colleagues, expecting a simple input of numbers.  However, upon aggregating the data, I found the resulting figure to be somewhat inflated. To gain a deeper understanding of this count, I requested my colleagues to provide a list of certifications they had completed. When I looked closely at the names of these "certifications," I found that about one out of every four was actually a certificate. This prompted a need for a clearer definition distinguishing between a certificate and a certification. So, today, I'm going to expl...
Read more

0 to 100: A Reflection

-
Image
What's 10 x 10? I started this blog on February 4th, 2023—almost two years ago (wow, I just checked!)—and I must say, writing consistently is a challenge that I gladly took on. Writing has been a constant theme in my life. It got me through my rebellious teen years and straight into adulthood (which, to be honest, I still have a hard time accepting). Many people ask me, "How do you do it?" To that, I say: I enjoy it, and I have a great support system. Today, I want to take a moment to reflect and give thanks—to those who read my posts, those who point out my typos, and especially to my dad, who always asks, "Toni, did you post yet?" But let’s be real: doing 100 of anything is difficult—whether it's 100 pull-ups or developing 100 apps. Even when you’re passionate about something, there are days when you just don’t have the energy or inspiration. And that’s okay! You just have to keep pushing forward—through the valley of despair, even when no one is watchin...
Read more