Swanktek

Compensation

Full Description

12+ years in data engineering or architecture, with a strong focus on Databricks (at least 4-5 years) and AI/MLenablement.Deep hands-on experience with Apache Spark, Databricks (Azure/AWS), and Delta Lake.Proficiency in AI/ML pipeline integration using Databricks MLflow or custom model deployment strategies.Strong knowledge of Apache Airflow, Databricks Jobs, and cloud-native orchestration patterns.Experience with structured streaming, Kafka, and real-time analytics frameworks.Proven ability to design and implement cloud-native data architectures.Solid understanding of data modeling, Lakehouse design principles, and lineage/tracking with Unity Catalog.Excellent communication and stakeholder engagement skills. Preferred QualificationsCertification in Databricks Data Engineering Professional is highly desirable.Experience transitioning from in house data platforms to Databricks or cloud-native environments.Hands-on experience with Delta Lake, Unity Catalog, and performance tuning in Databricks.Expertise in Apache Airflow DAG design, dynamic workflows, and production troubleshooting.Experience with CI/CD pipelines, Infrastructure-as-Code (Terraform, ARM templates), and DevOps practices.Exposure to AI/ML model integration within real-time or batch data pipelines.Exposure to MLOps, MLflow, Feature Store, and model monitoring in production environments.Experience with LLM/GenAI enablement, vectorized data, embedding storage, and integration with Databricksis an added advantage.