What is the most exciting thing you’ve learned?
As a Data Engineer in a fast-growing big data and AI company, I focus on building a scalable, efficient and cost-effective data pipeline framework. Crayon extensively uses cloud solution for its analytics, and I have the opportunity to optimize maya.ai’s core data processing engine deployed on AWS EMR. During this process, I’ve learnt a lot more about Spark Internals, optimization techniques, and effective usage of SPOT instance in EMR cluster. As the result of optimization, overall processing time reduced by 70% and the AWS utilization cost reduced by 60%. This optimization technique was later widely used across Crayon’s data platforms, which saved us around $2K per month in AWS billings.
What are some of the tech and tools you get to play with in your role?
Distributed processing systems are my area of expertise. I work predominantly on Hadoop and Spark across distributions like Cloudera, Hortonworks, AWS EMR and Azure Cloud. I used to experiment new tech stacks for data engineering to add value to the platform. The widely used tech stacks in our data engineering team are Hadoop, Spark, Git, Airflow, DBT (Data Build Tool), Jenkins and Singer.io. I also extend my support on Hadoop Administration activities like cluster planning and building on both cloud and on-prem environments.
What is it like working with Crayons?
Crayons are so colourful in their thoughts, and each one is truly unique. I feel privileged to work with such young and vibrant minds. The team is full of positivity and enthusiasm and will always have your back. Who doesn’t enjoy having pleasant and helpful co-workers around? I hope to continue learning from them and appreciate all their encouragement and support.
What kind of support do you receive from your teammates / the company?
There is the freedom to wear different hats apart from your designated role, and I see that as great opportunity to acquire knowledge across multiple streams. Crayon hosts various tech forums for us to gain and share knowledge, and also to make you aware of the technical challenges faced across the data platform. I am grateful that I constantly receive constructive feedback from my mentor. It pushes me to experiment with new stuff and value add the company. As Crayons, we rise by lifting one another.
