Sr. Lead Software Engineer, Reliability & Monitoring

In this role, the ideal candidate will work independently across platform, SDK, and application teams to gather requirements, build, offer support, and troubleshoot our observability pipelines.

Our Flexible Future model offers a healthy mix of working in person and virtually, strengthening key elements of the Chick-fil-A culture by fostering collaboration and community.

Responsibilities

• Own solution architecture decisions for the team's product

• Lead delivery and operations of the team's product, including both individual contribution and support as well as delegated tasks and support to your team's engineers.

• We desire our lead engineers to be both leads and engineers, spending about half of their time on leading others and half contributing engineering work themselves.

• Lead, mentor, and assess other staff engineers, exemplifying and teaching best practices, helping to solve complex problems, reviewing code, and sharing stories

• Interview, select, onboard, and oversee contract engineers

• Assist in the design and delivery of products outside of your team, develop tools and processes for use outside of your team, and/or mentor other engineers or especially lead engineers outside of your team

• Guide engineering team in adoption of Chick-fil-A software engineering standards

• Document learnings to share with the broader engineering team(s)

• Ensure clear communication around R+M objectives

• Collaborate broadly across the entire engineering organization

• Oversee other SRE teams to bring best practices or learnings from across the organization to them

• Build internal tooling around operational observability

• Bring a strong mindset of continual improvement

• An aversion to toil and automatable tasks

• Advocate for R+M as a part of engineering culture

• Act as a conduit for Architecture, Security, Tools, and DTT Framework

• Keep abreast of industry changes and evaluate for implementation

• Work independently with DevOps teams to refine running production systems

  • Building on-call processes
  • Creating Incident Management and Response procedures
  • Instrumenting for observability and coaching on best practices
  • Monitoring SLIs

• Work to varying degrees with DevOps teams

  • Provide consultation on SRE best practices
  • Give guidance on specific topics
  • Oversee groups of dedicated engineers
  • Embed directly with teams

• Work with teams to define SLOs and error budgets

• Ensure services and systems meet availability needs of customers

• Design and develop software solutions

• Serve as a model developer in programming languages like Java, Go, Python, and Python

• Exercise skills in infrastructure and deployment services like AWS and

• Kubernetes as well as areas like application security, data analytics, and machine learning

Note - Working in a DevOps model, this opportunity includes both building and running solutions that could require off hours support. This support is sharedamongst the team members to coverweekends and weeknights. The goal is to design for failure and, using cloud-native infrastructure patterns, automate responses to possible issues so they can be worked during normal hours.

Minimum Qualifications

• Bachelor's Degree or the equivalent combination of education, training and experience from which comparable skills can be acquired

• 4+ years of relevant work experience

• Experience designing complex software solutions

• Experience mentoring and leading a team, including good interpersonal and team collaboration skills

• Broad and deep programming experience in Java, JavaScript, Python, or other comparable languages

• Experience with SQL and data modeling

• Experience with source control systems like Git or Subversion

• Experience implementing application security, software design patterns, and the SDLC

• Proven ability to positively influence the engineering culture and practices in a professional environment

• Significant experience with:

• Building and supporting systems

• Enterprise cloud providers

• Production containerized environments

• Excellent written and verbal communication

• Experience with CI/CD pipelines

• Ability to build strong relationships, collaborate, and influence diverse groups of engineers and non-technical roles

• Ability to influence other engineers without organizational authority

Preferred Qualifications

• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or similar area of study

• 9+ years of relevant work experience, with:

  • 4+ years supporting a production system on a Devops team
  • 2+ years in cloud platforms such as amazon web services, google cloud, or Microsoft azure

• Experience with test-driven development, continuous integration and deployment, Scrum discipline, or comparable software development practices

• Deep understanding of AWS architecture

• Familiarity with version control systems and code merging/branching; specific experience with git desirable

• Experience working with an agile development methodology featuring sprints, points estimation, and daily standups

• Experience in design, data collection, and data analysis

• Experience with Unix/Linux

• Experience with Kubernetes

• Experience with OpenTelemetry, Fluentd, Grafana Alloy, or Datadog Agent (DDOT preferred)

Minimum Years of Experience

4

Travel Requirements

10%

Required Level of Education

Bachelor's degree or equivalent experience

Preferred Level of Education

Bachelor's Degree

Major/Concentration

Computer Engineering, Computer Science, or related Technical Field