Software Engineer - ETL Platform, IS&T Ai & Data Platforms
21 days ago China
About us
Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.
- Size: 10,000+ employees
- Industry: Technology: Manufacturing
Why you should apply for a job to Apple:
Ranked as one of the Best Technology Companies in 2020
3.7/5 in overall job satisfaction
3.1/5 in supportive management
66% say women are treated fairly and equally to men
65% would recommend this company to other women
66% say the CEO supports gender diversity
Ratings are based on anonymous reviews by Fairygodboss members. See all reviews ›
AI & Data Platforms (AiDP) is IS&T's engine for AI-powered innovation. The team brings together data, application development, and machine learning - including generative AI - along with data services and customer success functions, to help IS&T build solutions more efficiently and streamline the adoption and embedding of generative AI across Apple.
We are hiring an SRE to own reliability, performance, data freshness, and operational readiness for our production ETL platform. The platform supports data ingestion, transformation, and loading workflows across Kubernetes-based environments - including Airflow-based loader jobs and Spark-on-EKS jobs that load data into a Datalake/Lakehouse.
Description
You will operate and triage production pipelines end-to-end: extractors, loaders, batch jobs, streaming ingestion (Kafka), Spark workloads, and Airflow DAGs. You will tune Kubernetes and Spark for stability, build observability tooling, drive root cause analysis, write automation to reduce toil, and manage configuration and secrets through GitOps-style processes. The ideal candidate troubleshoots distributed systems from logs, metrics, and infrastructure signals, and drives permanent fixes through engineering partnership.
Preferred Qualifications
Experience supporting metadata-driven ETL platforms or internal ETL frameworks.
Experience with Lakehouse technologies (Spark, Iceberg, IRC, Hive Metastore, Parquet) and data loading performance tuning.
Experience with GitOps or source-of-truth configuration management, and infrastructure-as-code tools (Terraform, Helm, Argo CD, Ansible).
Experience operating multi-region or region-specific data platforms.
Experience with certificate/PKI/TLS management, JKS/truststore, Kerberos, or key rotation processes.
Familiarity with Spark performance tuning at scale, and platform dependencies such as API gateways, RabbitMQ, Redis, or Cassandra.
Minimum Qualifications
4+ years of experience in SRE, DevOps, platform engineering, data infrastructure, or production operations with strong Linux troubleshooting skills.
Strong Kubernetes/EKS operations experience (kubectl, deployments, pods, resource limits, service accounts, secrets, workload debugging) and hands-on experience supporting Apache Spark on Kubernetes - including tuning, log analysis, memory issues, shuffle failures, and performance bottlenecks.
Production experience with Apache Airflow (DAG operations, task failures, retries, scheduling, SLA misses) and Kafka or similar streaming platforms (consumer groups, lag, offsets, partitions, secure client connectivity).
Familiarity with Datalake/Lakehouse architectures, object storage (S3 or equivalent), monitoring/logging tools (Splunk, Prometheus, Grafana, CloudWatch, ELK, Datadog), and solid SQL skills.
Scripting proficiency in Python and Bash, with strong incident management, RCA, change management, and operational documentation skills.
We are hiring an SRE to own reliability, performance, data freshness, and operational readiness for our production ETL platform. The platform supports data ingestion, transformation, and loading workflows across Kubernetes-based environments - including Airflow-based loader jobs and Spark-on-EKS jobs that load data into a Datalake/Lakehouse.
Description
You will operate and triage production pipelines end-to-end: extractors, loaders, batch jobs, streaming ingestion (Kafka), Spark workloads, and Airflow DAGs. You will tune Kubernetes and Spark for stability, build observability tooling, drive root cause analysis, write automation to reduce toil, and manage configuration and secrets through GitOps-style processes. The ideal candidate troubleshoots distributed systems from logs, metrics, and infrastructure signals, and drives permanent fixes through engineering partnership.
Preferred Qualifications
Experience supporting metadata-driven ETL platforms or internal ETL frameworks.
Experience with Lakehouse technologies (Spark, Iceberg, IRC, Hive Metastore, Parquet) and data loading performance tuning.
Experience with GitOps or source-of-truth configuration management, and infrastructure-as-code tools (Terraform, Helm, Argo CD, Ansible).
Experience operating multi-region or region-specific data platforms.
Experience with certificate/PKI/TLS management, JKS/truststore, Kerberos, or key rotation processes.
Familiarity with Spark performance tuning at scale, and platform dependencies such as API gateways, RabbitMQ, Redis, or Cassandra.
Minimum Qualifications
4+ years of experience in SRE, DevOps, platform engineering, data infrastructure, or production operations with strong Linux troubleshooting skills.
Strong Kubernetes/EKS operations experience (kubectl, deployments, pods, resource limits, service accounts, secrets, workload debugging) and hands-on experience supporting Apache Spark on Kubernetes - including tuning, log analysis, memory issues, shuffle failures, and performance bottlenecks.
Production experience with Apache Airflow (DAG operations, task failures, retries, scheduling, SLA misses) and Kafka or similar streaming platforms (consumer groups, lag, offsets, partitions, secure client connectivity).
Familiarity with Datalake/Lakehouse architectures, object storage (S3 or equivalent), monitoring/logging tools (Splunk, Prometheus, Grafana, CloudWatch, ELK, Datadog), and solid SQL skills.
Scripting proficiency in Python and Bash, with strong incident management, RCA, change management, and operational documentation skills.
Additional Information
- Posted: 21 days ago