Overview
Design, develop, and maintain data pipelines for ingestion, transformation, and serving across batch and streaming workloads, and support federated data pipeline architecture.
Key Responsibilities
- Design, develop, and maintain data pipelines for ingestion, transformation, and serving across batch and streaming workloads
- Build ETL/ELT workflows to integrate data from diverse sources into enterprise data platforms
- Build document ingestion and processing pipelines incorporating OCR, metadata extraction, content enrichment, classification, and indexing capabilities.
- Develop data transformation logic using Apache Spark, PySpark, SparkSQL, and SQL
- Implement change data capture (CDC) patterns for real-time and near-real-time data synchronization
- Build streaming data pipelines for real-time analytics and operational use cases
- Optimize pipeline performance, resource utilization, and cost efficiency
- Support federated data pipeline architecture that enables Line of Business (LOB) teams to own and manage their domain data
- Contribute to self-serve data infrastructure that abstracts complexity and allows domain teams to build pipelines independently
- Develop standardized pipeline deployment patterns that LOB teams can adopt while maintaining autonomy
- Support domain teams in building data products that are discoverable, interoperable, and compliant with enterprise standards
- Support domain teams in creating and managing structured and unstructured data products, ensuring metadata, discoverability, and interoperability across enterprise platforms.
- Enable distributed data processing across domains while ensuring consistency through federated governance
- Assist in establishing data contracts and interoperability standards that allow seamless data sharing across domains
- Support the balance between domain autonomy and enterprise-wide governance requirements
- Develop reusable pipeline templates and Infrastructure as Code (IaC) patterns for common data product types
- Create blueprints for data ingestion, transformation, quality validation, and serving that LOB teams can customize
- Build standardized patterns for batch pipelines, streaming pipelines, CDC implementations, and API-based integrations
- Develop reusable patterns and accelerators for document ingestion, OCR, metadata extraction, content enrichment, and indexing of unstructured content
- Contribute to a pattern library covering medallion architecture, dimensional modeling, and data product packaging
- Document best practices and reference architectures that guide LOB teams in building compliant, high-quality pipelines
- Develop starter kits and accelerators that reduce time-to-value for domain teams building new data products
- Create cookbooks and implementation guides that translate enterprise standards into actionable steps
- Support LOB teams in adopting templates while allowing appropriate customization for domain-specific needs
- Integrate data from multiple internal and external sources into unified data assets
- Build reusable data integration patterns and connectors for enterprise data sources
- Implement data ingestion using Auto Loader, COPY INTO, and other ingestion frameworks
- Develop API-based data integrations and file-based data processing workflows
- Ensure data consistency and reliability across integrated sources
- Support data migration efforts and legacy system integrations
- Implement medallion architecture patterns (bronze, silver, gold) for data organization and quality progression
- Develop dimensional models, fact tables, and aggregations for analytics use cases
- Build data transformation logic that ensures accuracy, consistency, and business alignment
- Create reusable transformation components and modular pipeline designs
- Optimize data models for query performance and consumption patterns
- Support schema evolution and data versioning requirements
- Implement data quality checks, validation rules, and automated testing within pipelines
- Develop data profiling and anomaly detection to identify quality issues
- Build data reconciliation processes to ensure accuracy across systems
- Implement unit testing, integration testing, and regression testing for pipelines
- Monitor data quality metrics and remediate issues proactively
- Document data quality rules and thresholds for pipeline outputs
- Implement logging, monitoring, and alerting for pipeline health and performance
- Build dashboards to track pipeline execution, data freshness, and quality metrics
- Develop automated error handling, retry logic, and failure notifications
- Support incident response and troubleshooting for pipeline failures
- Implement data lineage tracking to support auditability and impact analysis
- Implement monitoring and observability for AI and data products, including pipeline health, data quality, and retrieval effectiveness.
- Ensure pipelines meet SLAs for data availability and freshness
- Build data pipelines that enable analytics, reporting, and business intelligence use cases
- Build and maintain document processing pipelines for extraction, classification, chunking, enrichment, and indexing of unstructured content
- Develop content chunking, metadata extraction, indexing, and enrichment workflows to support enterprise search and Retrieval-Augmented Generation (RAG) solutions.
- Prepare and serve data for machine learning and AI workloads
- Develop feature engineering pipelines for ML model development
- Create semantic layers and curated datasets that enable self-service analytics
- Support integration with analytics tools including Power BI and Tableau
- Build data products with clear documentation and consumption guidance
- Partner with data architects to align pipeline development with architectural standards
- Collaborate with business analysts and data scientists to understand data requirements
- Work with platform engineers to leverage platform capabilities effectively
- Contribute to technical documentation, runbooks, and knowledge sharing
- Contribute to platform standards, reusable frameworks, and engineering best practices for data and AI solutions
- Support data consumers in understanding and accessing data assets
- Participate in code reviews and follow engineering best practices
Required Experience
- 5-10 years of experience in data platform operations, cloud operations, SRE, DevOps, DataOps, production support, or platform engineering disciplines.
- Demonstrated expertise in Data Engineering, including the design, development, and optimization of scalable data pipelines, data platforms, and data processing solutions
- Experience designing and implementing solutions that leverage both structured and unstructured data to support AI, enterprise search, knowledge management, and analytics use cases
- Advanced experience designing and implementing modern data lake architectures and leveraging Google Cloud Platform / Databricks to build and maintain data engineering solutions
- Hands-on experience implementing enterprise-scale document processing capabilities, including OCR, metadata extraction, content enrichment, indexing, and knowledge discovery solutions
- Familiarity with graph databases and knowledge graph concepts to support enterprise search, knowledge discovery, and AI solutions
- Strong knowledge of data modeling, data structures and algorithms, and data integration techniques to support efficient and reliable data management
- Proven experience applying DevOps principles and practices, including automation, deployment, monitoring, and continuous improvement of data products and platforms.
- Strong understanding of workflow management and orchestration tools to support complex data processing and integration workflows.
- Experience managing and supporting the Product Development Life Cycle (PDLC), from requirements gathering and solution design through deployment and operational support
- Demonstrated ability to leverage business intelligence concepts and tools to deliver actionable insights and support data-driven decision-making
- Strong business acumen with the ability to understand organizational priorities and translate business requirements into effective technical solutions
- Experience working within Agile environments, including the Scaled Agile Framework (SAFe), and collaborating effectively across cross-functional teams
- Excellent stakeholder management, communication, and influencing skills, with the ability to build consensus and drive outcomes across technical and non-technical audiences
Qualifications
- Typically requires a Master's degree with 5 years of experience or a Bachelors Degree with a minimum of 7 years of relevant experience, or equivalent combination of education and experience.
- SAFe or other relevant Agile certifications.
- Databricks Certified Data Engineer Associate or equivalent
