Welcome to my personal blog. Here I share tutorials and updates on my research.
Integrate a data loading pipeline with Terraform and Kestra through Docker and a Makefile. It solves the common problem of manual configuration by using Terraform outputs to dynamically generate environment variables, ensuring a 'Single Source of Truth' for infrastructure and orchestration.
A step-by-step walkthrough for setting up Terraform to manage Google Cloud Platform (GCP) resources. It covers project creation, service account security, variable management in HCL (HashiCorp Configuration Language), and the lifecycle of a Google Cloud Storage (GCS) bucket.
This guide provides an intro to Google BigQuery. It explains the separation of storage and compute, the benefits of columnar storage, and practical optimization techniques like partitioning and clustering to manage large-scale datasets efficiently.
A comparative guide for data professionals navigating the structural differences between Google Cloud Platform (GCP) and Amazon Web Services (AWS). It focuses specifically on how these platforms organize data within Data Lakes and Data Warehouses, emphasizing the philosophical differences between GCP’s 'Project-centric' model and AWS’s 'Account-centric' model.
Explore the fundamentals of data architecture, comparing high-speed OLTP systems used for daily transactions with analytical OLAP environments like Data Warehouses, Data Marts, and Data Lakes. The guide explains the mechanical differences between row-oriented and column-oriented storage and evaluates ETL versus ELT plumbing strategies for moving and cleaning data.
Construct a robust, containerized ETL pipeline using Kestra to orchestrate the ingestion of Greek road traffic data into PostgreSQL. The guide details extracting API data, normalizing JSON with Python/Pandas, and ensuring data integrity through staging tables and SQL merge strategies
Lets explore three strategies for running Python in Kestra (runtime installation, custom Docker images, and custom Worker images) which help us balance speed, isolation, and maintenance.
This guide introduces Kestra, a declarative orchestration platform, explaining how to define Flows using YAML, manage tasks like logging and downloading, and automate execution with Triggers and Cron.
This article breaks down the containerization setup for a Python application which connects to a PostgreSQL database, specifically focusing on the build process (Dockerfile) and the application service configuration (docker-compose.yml).
Bridge the gap between your local Python environment and a Dockerized database using SQLAlchemy, while securing your credentials with environment variables.