Overview
This role focuses on building and maintaining the Azure Databricks Enterprise Data Platform to enable federated data product development and self-service data consumption. The position involves setting technical standards, enforcing governance, controlling costs, and ensuring reliable data delivery at scale.
Key Responsibilities
- Build & Maintain Azure Databricks Enterprise Data Platform.
- Develop Platform as Product solutions for self-service data consumption.
- Set standards, enforce governance, and control costs for the data platform.
- Design, build, and maintain a production-grade Lakehouse architecture on Azure Databricks.
- Administer and govern Unity Catalog.
- Own the configuration and lifecycle of Databricks SQL Warehouses, clusters, and compute policies.
- Establish and maintain dbt as the transformation framework.
- Rationalize legacy Data Platform into modern Data Management solutions.
- Monitor and optimize the cost of Databricks clusters, SQL Warehouses, and dbt workloads.
- Implement chargeback/showback tagging, usage dashboards, and budget guardrails.
- Define and enforce data governance best practices.
- Implement data quality frameworks.
- Participate in agile ceremonies and collaborate with data scientists, analysts, and product owners.
- Analyze and maintain documentation for data processes, pipelines, and systems.
- Design and manage secure cloud networking for the platform.
- Partner with cloud infrastructure and security teams.
- Own the operation of data platforms in production.
- Build automation and observability for platform operations.
- Lead root-cause analysis and drive continuous improvement of platform stability and performance.
- Provide technical leadership and mentorship to data platform engineers.
- Engage with data vendors and lead RFP processes.
- Serve as a trusted advisor to data engineering and analytics teams.
- Support development, testing, and deployment of AI/ML models and platform components.
- Implement and support monitoring, observability, and operational reliability practices.
- Collaborate with cross-functional teams and contribute to agile development processes.
Required Experience
- Typically requires a Master's degree with 5 years of experience or a Bachelors Degree with a minimum of 7 years of relevant experience, or equivalent combination of education and experience.
- 5-10 years of experience in building Data Platforms, Platform Engineering discipline.
- Experience with coding projects, hackathons, or open-source contributions is a plus.
Qualifications
- SAFe or other relevant Agile certifications.
- Databricks Certified Data Engineer Associate
- Microsoft Azure Fundamentals (AZ-900) or AWS Certified Cloud Practitioner
- Azure AI Fundamentals (AI-900) or equivalent
- Microsoft Certified: Fabric Data Engineer Associate, or AWS Certified Data Engineer
