Data Engineer: Data Analytics and AI

EAI, SOA, CEP, HDFS, ETL

Required Technical and Professional Expertise

• Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.

• Minimum of 3 years of experience in data engineering, with a focus on implementing data platforms, AI, and ML solutions.

• Hands-on experience with at least one major cloud provider: Azure, AWS, or GCP.

• Proficiency in programming languages such as Python, Py-Spark, PL/SQL, Spark SQL, GO, Java

• Experience with data processing frameworks like Apache Spark, Hadoop, or similar.

• Strong knowledge of SQL and experience with relational and non-relational databases.

• Familiarity with containerization technologies (e.g., Docker, Kubernetes) and CI/CD pipelines.

• Solid understanding of data warehousing concepts and tools (e.g., DataBricks, Redshift, BigQuery).

• Knowledge of AI/ML concepts and experience in supporting AI/ML teams.

• Strong problem-solving skills, with the ability to troubleshoot complex data issues.

• Excellent communication skills, with the ability to explain technical concepts to non-technical stakeholders.

Preferred Technical and Professional Expertise

• Design, implement, and manage data platforms on cloud environments (Azure, AWS, or GCP) to support AI and ML workloads.
• Develop, test, and maintain robust data pipelines that ensure the smooth ingestion, processing, and storage of large datasets from diverse sources.
• Integrate data solutions with cloud-native services, ensuring optimal performance, scalability, and cost-effectiveness.
• Work closely with data scientists, data engineer, business analyst and AI/ML engineers to deploy models and algorithms into production, optimizing them for performance and scalability.
• Design and develop Extract, Transform, Load (ETL) processes to clean, enrich, and prepare data for analysis and machine learning tasks.
• Implement and manage data governance practices, ensuring data quality, security, and compliance with industry standards and regulations.
• Monitor and optimize the performance of data platforms and pipelines, identifying and resolving bottlenecks and inefficiencies.
• Create detailed documentation for data platforms and pipelines, and provide regular status updates to stakeholders.
• Stay updated with the latest trends and technologies in data engineering, AI, and cloud computing, and advocate for best practices within the team.