Jigar Shah drives vision in big data engineering and this is the case in today's rapidly evolving world of data.

Jigar Shah drives vision in big data engineering and this is the case in today's rapidly evolving world of data. He has more than 15 years of work experience and has made significant strides in the art of transforming bulky and complex data systems into work-efficient systems. His career crosscuts various domains and technology stacks, thus positioning him as a leader, a mentor and an architect of very progressive data systems. Without wasting more time, I would like to take you through his story in which we try to understand his career path, how he thinks and his future plans in this regard in a question-and-answer format.

Question 1: Jigar, can you tell us about your work history and how you got interested in data engineering?

A: The journey started with my degree in Electrical Engineering from Nirma University in India. In the early stages of my work experiences, I was dealing with rich data in enterprise systems and that's when I realized how data can be important to organizations. It was this possibility of data that led me to data engineering. Slowly but surely, I am training myself to acquire the skills required in building a Big Data ecosystem in terms of developing and managing such systems.

Q2: Please tell us what was the most important project that defined you as a data engineer.

A: A major project that develops a system for a sales data mart that feeds sales data in real time. We needed to create a system, capable of securely ingesting large amounts of data into a system that could return accurate real-time queries. This was a very complex endeavor as we wanted to maintain data consistency and still optimize it for performance. All my technical skills were a matter of perspective and thinking outside the box to exploit the inefficiencies of existing technology. Completing that project gave me the confidence to pursue data engineering as a career and it helped me improve my performance in more complex tasks over time.

Q3: You have impressive experience with different big data tools and platforms. Please describe how you functioned in this diverse environment.

A: It has been very important in my career that I have been able to work well in different environments. I have come across many ecosystems, Hadoop, Spark, Snowflake, AWS, etc., each with different features. I'm doing the opposite: focusing on fundamentals and then moving to platform specifics, for example – data architecture and modeling first and platform optimization second. I also value education which has helped me to be flexible in the acquisition of new technologies which have presented me with effective means to overcome strengths and weaknesses in different environments.

Q4: At certain points in your career have you held technical and leadership positions simultaneously. What is the point of dividing your time in this way?

Answer: Such duality of roles is fulfilling in itself and challenging at the same time. For starters, I am constantly at work as I work on coding and conceptualizing data engineering solutions and keeping abreast of developments in the field of data engineering. It really helps me understand the physics of the systems I'm dealing with. In my case, I make mentoring and people development a top priority and surround myself with decision-making and goal-producing individuals. It's about talking: getting my hands dirty on the technical side and making sure the team is taking the lead in what they're doing.

Q5: Where do you feel the importance of continuing education and related certifications in your career?

A: Lifelong learning has played an indispensable role in my career. To stay in the current stream of programs I tried to get some certifications including special certifications about Big Data or Spark. Doing certifications validates my abilities and exposes me to new opportunities and areas that I can explore. This helped me develop significant technical skills which I could later use to perform more complex tasks and manage technical teams.

Q6: Can you describe a project where you had to deal with teams from different functions to achieve a common goal?

A: Working in an integrated manner with cross-functional teams is an integral aspect of my professional life. A particularly interesting project involved creating a product for media campaign performance analysis. It implies close interaction with data engineering, product management and business analytics to ensure that data architecture meets business needs. The described data manipulation fought through numerous challenges that included creating a data pipeline capable of executing hundreds of events in a matter of seconds and still ensuring that the resulting insights were actionable, killing the interest of many users. To accomplish that, all teams have established truly constructive communication, implemented agile practices, and coordinated their aspirations.

Q7: Can you describe the measures you have adopted to guarantee the reliability and fail-safe nature of your data systems?

A: To build a fault-tolerant and resilient system, one needs to understand the structural design in detail. I am a high-level system designer who tries to implement redundancy mechanisms, data validation and mitigation strategies. My exposure to distributed systems has exposed me to the worst; Thus, I implemented sharding, partitioning and load balancing in hopes that any scalability challenges would not crush the system being built. A common practice for me is to integrate CI/CD which is implemented early through testing and deployment to minimize any errors and maintain a stable system.

Question 8: Tell me about a time when you found yourself in a situation where you needed to make a very important decision within a critical time frame. How did this affect the project?

A: There was a scenario where we had issues with the integrity of touching data in a geo-complex multi-node system. This was an important point because mistakes would have led to wrong insights into the business. So I need to quickly assess the situation, find the problem and solve it within a limited period of time. The option I wanted to take was to change some areas in the data pipeline and add more layers of authentication. It was fraught with risks, but worked for the good of the project by assuring data quality throughout the project.

Question 9: What advice would you give to aspiring data engineers aiming to pursue a career in the field?

A: The stakes are really high, my suggestion would be to focus on the first principles- data structures and algorithms and system design. Then, use modern technologies like big data or cloud or machine learning. Don't stop learning new technologies and don't be afraid to take on challenging assignments. The field is very broad, but having a sense of adventure with a passion for finding solutions will be a differentiating factor. Such certifications can be useful as they provide a roadmap for learning and also represent validation of acquired skills.

Question 10: What are your expectations regarding career growth in data engineering field?

A: I look forward to uncovering new faces of data as I continue to make further progress in this direction. I want to build a system where more AI and machine learning models are added to data pipelines for effective trend prediction and optimization of business processes towards growth. Apart from that, it is also important to be a change in thought leadership where I mentor others and share my knowledge as well as data engineering speaking and writing.

Jigar Shah is a highly accomplished Principal Data Engineer with over 15 years of experience developing enterprise applications and working in start-up businesses. He is proficient in developing and maintaining large data processing systems, establishing reliable data frameworks and using Spark, AWS and Snowflake technologies. As an accomplished data engineer and data transformation expert, thanks to his experience with Python, SQL and various cloud-based technologies, Jigger has introduced and facilitated many projects across the data lifecycle. His experience in industry and projects allows him to manage cutting-edge initiatives and train people at the very center of the modern data engineering world.