Senior Data Engineer

Senior Data Engineer with a minimum of 5 years of professional data engineering experience to design, build, and maintain reliable data pipelines and platforms that support analytics, AI/ML, and enterprise applications. Should have strong experience with modern data architectures and be comfortable working with Data Scientists, AI/ML Engineers, Backend Developers, and business stakeholders. Key Responsibilities • Design, develop, and maintain scalable data pipelines and data processing solutions. • Build and manage ETL/ELT pipelines for structured and unstructured data. • Integrate data from enterprise applications, databases, APIs, files, and other data sources. • Develop data ingestion and transformation workflows to support AI/ML and analytics use cases. • Prepare, clean, transform, and validate data for use by Data Scientists and AI/ML Engineers. • Design and maintain data warehouses, data lakes, and lakehouse architectures. • Implement data quality, validation, monitoring, and error-handling mechanisms. • Ensure data pipelines are reliable, secure, scalable, and performant. • Work with Data Scientists and AI/ML Engineers to understand model data requirements, feature engineering needs, and data availability. • Support the integration of AI/ML models into production by ensuring reliable access to required data. • Develop data models and establish appropriate data structures for analytical and operational use cases. • Implement data lineage, metadata management, and data governance practices. • Optimize queries, pipelines, storage, and processing performance. • Troubleshoot data pipeline failures and resolve data quality and integration issues. • Implement appropriate security and access controls for enterprise data. • Develop automated tests and validation processes for data pipelines. • Document data architecture, pipelines, data models, and integration processes. • Participate in code reviews and contribute to data architecture and technical decisions. • Work within Agile/Scrum teams and manage development activities through Jira. Required Skills & Experience • 5+ years of professional Data Engineering experience. • Strong experience designing and implementing ETL/ELT pipelines. • Strong programming experience with Python and/or SQL. • Strong experience with relational databases such as PostgreSQL, SQL Server, Oracle, or MySQL. • Experience with data warehouses and/or data lake architectures. • Experience integrating data through REST APIs, databases, files, and enterprise systems. • Strong understanding of data modeling, data transformation, data quality, and data governance. • Experience working with large datasets and optimizing data processing workloads. • Experience with Git and modern software development practices. • Experience working in Agile/Scrum environments and using Jira. • Strong troubleshooting, analytical, and problem-solving skills. AI/ML Data Engineering Experience • Experience preparing and managing data for AI/ML model development and production. • Understanding of data requirements for machine learning workflows. • Experience supporting feature engineering and feature data pipelines. • Experience handling structured and unstructured data for AI/ML applications. • Experience building data pipelines that support ML training, inference, and model monitoring. • Experience with data preparation for Generative AI, LLM, RAG, or AI Agent applications is highly desirable. • Familiarity with embeddings, vector databases, and retrieval pipelines is a plus. Preferred Technologies Experience with some of the following technologies is preferred: • Python, SQL • Apache Spark / PySpark • Airflow or similar workflow orchestration tools • Kafka or other messaging/event-streaming platforms • Databricks • Snowflake • Azure Data Factory / AWS Glue / similar cloud data services • PostgreSQL, SQL Server, Oracle, MongoDB • Redis • Cloud platforms such as AWS, Azure, or GCP • Docker and containerized data workloads Data Quality & Governance • Implement data quality checks covering completeness, accuracy, consistency, validity, and timeliness. • Support data lineage and metadata management. • Work with data owners and stakeholders to establish appropriate data definitions and standards. • Identify and resolve data quality issues that may affect AI/ML model performance. • Apply appropriate security and access controls to sensitive enterprise data. Collaboration • Work closely with Data Scientists, AI/ML Engineers, Backend Developers, Business Analysts, QA, DevOps, and Product teams. • Translate AI/ML and business requirements into practical data engineering solutions. • Support Data Scientists in accessing and preparing data required for model development. • Participate in Agile ceremonies, sprint planning, backlog refinement, daily stand-ups, and retrospectives. • Track data engineering activities, issues, and deliverables through Jira. Ideal Candidate A senior data engineer who can build production-grade data pipelines and platforms and understands how enterprise data supports analytics and AI/ML applications. The candidate should have strong Python/SQL skills, solid data architecture fundamentals, and the ability to work across data, AI/ML, backend, and business teams to deliver reliable data solutions.