What your plumber can teach you about Data Pipelines
Author: Sarah Usher
Created: 2/19/2023
Last updated: 2/19/2023
Specialist tools (The right tools for the job) For your average home-based plumbing situations, you can often get by with fairly common tools in the average tool box, but once you start considering multiple connected systems across multiple plumbed rooms, the correct tools are needed. Similar to data platforms. You can get by with some Python scripts and cron jobs but once you hit a certain level or scale or requirement, switching to tools designed for work is required. In Software Engineering we call this "Using the right tools for the job".
One Directional Flow (DAGs) Water only flows in one direction in plumbing systems. In modern data systems, we ruthlessly apply this principle.
Building plans (Documentation & Lineage) The same way your plumber will use and (hopefully!) update changes to the plumbing systems in a building plan, helping them understand how the water is moving through different phases in the system, understanding your data's lineage throughout the company can help surface inconsistencies, aid in data discovery and trust and allow of optimisations to be applied.