Posts

In this post, I share how to build a Reverse ETL pipeline to upsert data from Delta (Databricks) into Postgres to provide sub-second response times to our tables. The goal is to make warehouse data available to downstream systems that require millisecond response times. These systems could be front-end applications that need to consume this data, or online machine learning models that require additional data as input to generate predictions in real time. ...

The other day my team came across a peculiar situation with our Postgres database. We had this table partitioned by a primary key, which we’ll call order_id. We realised one of the queries was taking a very long time to execute, and like good SQL debuggers we looked at the query plan. Let’s pretend this was the slow query: EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 12345; Upon inspection, we observed that the query plan was scanning at every single partition! And in our case, there were a lot of them. ...

I recently needed to run load tests on a PostgreSQL database. Postgres is no question a reliable and scalable database ready for production use cases, but there might be times when you’ll need to confirm it can handle your specific use case. This might be necessary if you’re using a PostgreSQL database hosted by a third party or if you want to check if your current instance size and specs can manage your existing load. ...

We all know that person that always asks for unit tests on your Pull Request. You curse under your breath wondering why they have nothing better to do than ruining your day. Unit tests let you know if your codebase is behaving as expected. As long as you’ve written good tests, you can be happy your codebase is healthy if tests are showing green. Any time a test appears in red, you will know that you’ve introduced a bug and you can fix it before it hits production. ...

Indexes are like going to the airport and looking at the signs to see where your gate is located. The signs tell you how to get to your gate quickly. The alternative to not having signs is walking the whole airport randomly until you come across your gate. So by having the signs you can reach your gate a lot faster. Makes sense right? This is the idea behind indexes. ...

Learning To prepare for the exam I took the Udemy Kubernetes Certified Application Developer (CKAD) with Tests which I could access for free through my company. The course covers all the topics required for the exam with labs to practice each section which I found super helpful. The labs are hosted on KodeKloud and they are free to access as part of the course. I had some previous experience interacting with kubernetes as part of my job, but I lacked a good understanding of the basic concepts. ...

This is my story of how I went from non-tech into becoming a Machine Learning Engineer. I must have told this story multiple times in many different forms, but this will be the first time it goes into writing. Enjoy. I have a bachelor’s degree in Commerce. I found the last few years of school so exhausting that the thought of doing an engineering degree put me off. Four + years of no life? Oft, no thanks. I also thought I wasn’t smart enough to do engineering anyway so instead I chose Commerce - which “coincidentally” was the same degree my brother was doing. ...