Role Summary
We are looking for a strong principal engineer to build and scale our Data & AI capabilities with a strong focus on Databricks, Apache Spark, Iceberg, Azure Data Fabric, Amazon EMR, and Redshift. This role is pivotal in driving Lakehouse architecture adoption, cloud-native modernization, and ML engineering across industries such as Oil & Gas.
You will lead a cross-functional team of architects, engineers, and ML specialists to deliver high-impact solutions that combine technical excellence, cost efficiency, and governance maturity.
Key Responsibilities
Strategic Leadership & Practice Growth
• Define and execute the strategic roadmap for the Data & AI practice, aligning with evolving trends in Lakehouse, cloud-native platforms, and AI engineering.
• Build reusable solution accelerators (e.g., Iceberg ingestion templates, Spark ETL frameworks, MLflow pipelines).
• Establish and manage strategic partnerships with Databricks, AWS, Azure, and other ecosystem players.
• Represent the practice in industry forums, client workshops, and thought leadership initiatives.
Technical Architecture & Delivery Oversight
• Architect scalable Lakehouse solutions using Delta Lake or Iceberg, optimized for performance and cost.
• Lead the design of unified batch + streaming pipelines using Spark Structured Streaming and Scala.
• Oversee multi-cloud data platform modernization using Azure Data Fabric, EMR, and Redshift.
• Guide implementation of data governance frameworks using Unity Catalog, Azure Purview, and Iceberg metadata.
Team Leadership & Capability Building
• Mentor and grow a high-performing team of data engineers, ML engineers, and platform specialists.
• Define career paths, training plans, and certification goals aligned with Databricks, AWS, and Azure.
• Foster a culture of innovation, experimentation, and continuous improvement.
Client Engagement & Business Impact
• Engage with clients to understand business challenges and translate them into scalable data solutions.
• Lead solutioning, proposal development, and delivery for strategic engagements.
• Ensure delivery excellence through performance optimization (Photon, EMR Autoscaling) and cost engineering.
Required Skills & Experience
• 10+ years of experience in data engineering, architecture, and platform modernization.
• 3+ years of hands-on experience with Databricks, Apache Spark, Iceberg, and Scala.
• Deep expertise in cloud-native data platforms (Azure Data Fabric, Amazon EMR, Redshift).
• Has experience in process optimization data analytics, operator UI predictive analytics, time series data, open telemetry and anomaly detection in oil and gas domain.
• Proven track record in building ML pipelines, implementing MLOps (MLflow), and integrating Redshift ML.
• Strong understanding of data governance, observability, and schema evolution.
• Excellent leadership, communication, and stakeholder management skills.
Preferred Qualifications
• Certifications in Databricks, Azure Data Engineering, or AWS Big Data.
• Experience in vertical-specific data solutions (e.g., predictive maintenance in Oil & Gas, real-time analytics in Retail).
• Familiarity with data mesh architectures, time travel, and audit compliance frameworks.