Data Engineer II

customizations in one or two tools to generate a product at the business's or customer's request.

• Updates logical or physical data models based on new use cases.

• Frequently uses SQL and understands NoSQL databases and their niche in the marketplace.

• Contributes to the team's culture by promoting collaboration and innovation.

• Designs, develops, and maintains robust data pipelines to automate the extraction, transformation, and loading (ETL) of data from various sources into data warehouses or data lakes.

• Integrates data from multiple sources, ensuring consistency, accuracy, and reliability.

• Collaborates with product owners to understand data requirements and provide solutions.

• Documents data pipeline processes and workflows for future reference and training.

Required qualifications, capabilities, and skills

• Formal training or certification on data engineering disciplines and 2+ years applied experience

• Advanced proficiency in SQL (e.g., joins and aggregations)

• Working understanding of NoSQL databases

• Proficiency in programming languages such as Python/Java for data processing tasks

• Proficient in Object-Oriented Programming (OOP) concepts, with a strong ability to design and implement robust, reusable, and maintainable code structures across various programming languages

• Experience with cloud platforms, particularly Amazon Web Services (AWS), including EMR, Glue, Lambda, and ECS, to design, deploy, and manage scalable and efficient cloud-based solutions

• Hands-on experience with frameworks like Apache Spark, leveraging its capabilities for large-scale data processing and analytics to drive efficient and insightful data solutions

• Experience in utilizing Cucumber and Gherkin for behavior-driven development (BDD)

• Proficiency in Unix scripting, data structures, data serialization formats such as JSON, AVRO, or similar, and big-data storage formats such as Parquet

• Excellent communication skills to collaborate with cross-functional teams and stakeholders

• Understanding of data security best practices and compliance requirements.

Preferred qualifications, capabilities, and skills

• Java programming knowledge, including experience with Maven, CI/CD pipelines, and Docker
• Provisioned infrastructure using a high-level configuration language
• Utilized Splunk to monitor and analyze system performance
• Flexibility and eagerness to learn new technologies and skills