Senior Data Engineer
Upon joining a new team, I played a pivotal role in a company-wide initiative aimed at enhancing data ingestion processes from our clients, placing significant emphasis on data quality and observability. The overarching objective was to identify and address issues proactively before they reached end users.
In pursuit of these objectives, I spearheaded various initiatives including:
- Facilitating seamless integration between our customer's Snowflake platform and our Data Lake
- Implementing data quality metrics at different stages of our ingestion pipeline and establishing automated reporting mechanisms to DataDog
- Establishing the groundwork for an ingestion API tailored for our customers' use
Tools
PySpark, AWS, PostgreSQL, Snowflake, DBT, Docker, k8s, Helm