Lead Data Engineer

Job Summary

We are seeking an experienced Data Engineering Architect / Lead Data Engineer to design, build, and optimize scalable data platforms and pipelines. The ideal candidate will have extensive experience in ETL/ELT development, cloud-based data engineering, real-time and batch data processing, and modern data warehouse architectures. This role requires strong expertise in designing end-to-end data solutions that are scalable, secure, and aligned with enterprise standards.

Required Skills & Qualifications

• Advanced expertise in designing and developing ETL/ELT pipelines.
• Strong experience with batch data processing and near real-time/streaming data pipelines.
• Hands-on experience working with structured and semi-structured data.
• Strong knowledge of:
  • Incremental data loading
  • Change Data Capture (CDC)
  • Pipeline orchestration and dependency management
• Strong programming skills in Python (preferred), Scala, or Java.
• Experience optimizing large-scale data processing workloads for performance and cost.
• Solid understanding of data modeling concepts:
  • Star Schema
  • Snowflake Schema
  • Normalized and denormalized data models
• Hands-on experience with at least one major cloud platform:
  • Microsoft Azure
  • Amazon Web Services (AWS)
  • Google Cloud Platform (GCP)
• Strong experience with modern data warehouses such as:
  • Snowflake
  • Azure Synapse
  • Google BigQuery
  • Amazon Redshift

Key Responsibilities

Data Architecture & Solution Design

• Design end-to-end data engineering architectures for enterprise-scale solutions.
• Develop scalable architectures for:
  • Data Lakes and Lakehouse platforms
  • Enterprise Data Warehouses
  • Streaming and real-time data processing systems
• Ensure solutions align with enterprise architecture, security, governance, and compliance standards.
• Review and approve technical designs and implementation strategies.

Data Pipeline Development & Management

• Lead the design and development of scalable ETL/ELT pipelines.
• Build and manage data ingestion pipelines for both batch and real-time data.
• Process structured and semi-structured data efficiently.
• Optimize data pipelines for performance, reliability, scalability, and cost.
• Manage schema evolution, metadata, and pipeline dependencies.

Data Quality, Reliability & Operations

• Establish and enforce data quality standards and validation frameworks.
• Implement monitoring, alerting, logging, and observability for data pipelines.
• Perform root cause analysis and resolve data-related production issues.
• Drive operational excellence by improving system stability and reliability.

DevOps / DataOps

• Build and maintain CI/CD pipelines for data engineering workloads.
• Automate testing, deployment, and rollback processes.
• Improve platform reliability and deployment efficiency through automation and DevOps best practices.

Preferred Qualifications

• Experience with modern DataOps and CI/CD practices.
• Knowledge of data governance, security, and compliance frameworks.
• Experience designing enterprise-scale cloud-native data platforms.
• Strong analytical, troubleshooting, and communication skills.