AWS Data Engineer

is heard, respected, and valued. We encourage candidates from all backgrounds to apply. IBM is an Equal Opportunity Employer, committed to fostering a diverse and inclusive environment.

In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology.

Your role and responsibilities

Phase 1: Job Assessment and Code Preparation

1. Job Selection and Inventory

• Select target jobs from EMR JIRA project for migration prioritization

• Identify and catalog all jobs eligible for EMR Serverless migration

2. Code Discovery and Validation

• Locate jobs in production environment and retrieve latest code versions

• Verify job status to ensure active (non-retired) jobs are migrated

• Audit code repositories to confirm production code alignment with master branch

3. Source Code Management

• Push latest production code to master branch if discrepancies exist

Phase 2: Infrastructure and Compatibility Updates

4. Storage and Query Migration

• Migrate Hadoop and HDFS commands to utilize Amazon S3 for data storage

• Update data processing workflows to leverage Amazon Athena for query execution

5. Runtime Environment Modernization

• Update Spark version to EMR Serverless compatible releases

• Upgrade Python version to supported EMR Serverless specifications

• Ensure all dependencies are compatible with target environment

6. Logging and Monitoring Integration

• Implement native EMR logging integration with AWS CloudWatch

• Configure AWS Log Groups for centralized log management

• Optimize scripts for enhanced observability in serverless environment

Phase 3: Testing and Orchestration

7. Documentation and Workflow Validation

• Create/update test documentation for migrated jobs

• Verify existing workflow configurations and dependencies

8. Job Orchestration Migration

• Identify current job triggers (Control-M, cron, etc.)

• Migrate Control-M scheduled jobs to AWS Step Functions

• Configure serverless-compatible job orchestration

Phase 4: Deployment and Validation

9. Non-Production Deployment

• Deploy migrated jobs to non-production environment

• Conduct internal testing and validation

• Coordinate handoff to AMI team for additional validation

10. Quality Assurance Testing

• Deploy to QA environment for comprehensive testing

• Conduct internal testing and validation

• Obtain AMI team sign-off on QA results

11. Production Release

• Create production release change following AMI team approval

• Execute controlled production deployment

• Implement post-deployment monitoring and validation for 1 week

Required education

Bachelor's Degree

Required technical and professional expertise

• Proficient in Spark job development and optimization, including version upgrades and compatibility with EMR Serverless.

• Strong understanding of AWS services such as EMR Serverless, S3, Athena, CloudWatch, and Step Functions.

• Advanced skills in Python, including upgrading and refactoring code to meet EMR Serverless runtime requirements.

• Experience with job scheduling tools (e.g., Control-M, cron) and migrating workflows to AWS Step Functions.

• Proficiency in Git-based workflows, including branch management, code auditing, and repository alignment.

• Ability to integrate and configure logging solutions using AWS CloudWatch and Log Groups for serverless environments.

• Skilled in creating test documentation, conducting QA, and validating job performance across environments (non-prod, QA, production).

ABOUT BUSINESS UNIT

IBM Consulting is IBM's consulting and global professional services business, with market leading capabilities in business and technology transformation. With deep expertise in many industries, we offer strategy, experience, technology, and operations services to many of the most innovative and valuable companies in the world. Our people are focused on accelerating our clients' businesses through the power of collaboration. We believe in the power of technology responsibly used to help people, partners and the planet.

YOUR LIFE @ IBM

In a world where technology never stands still, we understand that, dedication to our clients success, innovation that matters, and trust and personal responsibility in all our relationships, lives in what we do as IBMers as we strive to be the catalyst that makes the world work better.

Being an IBMer means you'll be able to learn and develop yourself and your career, you'll be encouraged to be courageous and experiment everyday, all whilst having continuous trust and support in an environment where everyone can thrive whatever their personal or professional background.

Our IBMers are growth minded, always staying curious, open to feedback and learning new information and skills to constantly transform themselves and our company. They are trusted to provide on-going feedback to help other IBMers grow, as well as collaborate with colleagues keeping in mind a team focused approach to include different perspectives to drive exceptional outcomes for our customers. The courage our IBMers have to make critical decisions everyday is essential to IBM becoming the catalyst for progress, always embracing challenges with resources they have to hand, a can-do attitude and always striving for an outcome focused approach within everything that they do.

Are you ready to be an IBMer?

ABOUT IBM

IBM's greatest invention is the IBMer. We believe that through the application of intelligence, reason and science, we can improve business, society and the human condition, bringing the power of an open hybrid cloud and AI strategy to life for our clients and partners around the world.

Restlessly reinventing since 1911, we are not only one of the largest corporate organizations in the world, we're also one of the biggest technology and consulting employers, with many of the Fortune 500 companies relying on the IBM Cloud to run their business.

At IBM, we pride ourselves on being an early adopter of artificial intelligence, quantum computing and blockchain. Now it's time for you to join us on our journey to being a responsible technology innovator and a force for good in the world.

IBM is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, neurodivergence, age, or other characteristics protected by the applicable law. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.

OTHER RELEVANT JOB DETAILS

All our positions are eligible for people with disabilities PwD or rehabilitated.