Software Engineer III - Pyspark

roduces architecture and design artifacts for complex applications while being accountable for ensuring design constraints are met by software code development

• Gathers, analyzes, synthesizes, and develops visualizations and reporting from large, diverse data sets in service of continuous improvement of software applications and systems

• Proactively identifies hidden problems and patterns in data and uses these insights to drive improvements to coding hygiene and system architecture

• Contributes to software engineering communities of practice and events that explore new and emerging technologies

• Adds to team culture of diversity, equity, inclusion, and respect

Required qualifications, capabilities, and skills

• Formal training or certification in Software engineering concepts, with 3+ years of applied experience.

• Hands-on practical experience in in PySpark, Spark SQL, and spark optimization, along with experience in databases and Hadoop technologies

• Deep understanding of data pipelines and proficiency in working with at least one database.

• Experience in building complex distributed systems using Python/Poetry on AWS

• Experience with Python, Kafka, Spark, advanced SQL, Glue

• Technical mentoring experience and strong communication skills.

• Excellent understanding of modern engineering practices to leverage public cloud benefits, such as auto-scaling, with a focus on end-to-end engineering experience supported by tooling and automation.

Preferred qualifications, capabilities, and skills

• Preferred experience in designing, developing, and implementing software solutions, with a commitment to becoming an expert.
• Exposure to cloud technologies, specifically AWS.
• Familiar in Python coding.
• Familiar in PySpark optimization