Lead Business Intelligence Engineer (Observability Engineer - Splunk / Grafana)

dards and governance.

Pay and Benefits:

• Competitive compensation, including base pay and annual incentive

• Comprehensive health and life insurance and well-being benefits, based on location

• Pension / Retirement benefits

• Paid Time Off and Personal/Family Care, and other leaves of absence when needed to support your physical, financial, and emotional well-being.

• DTCC offers a flexible/hybrid model of 3 days onsite and 2 days remote (onsite Tuesdays, Wednesdays and a third day unique to each team or employee).

The Impact you will have in this role:

We are seeking a highly motivated Observability Engineer to join our Observability Engineering & Product Delivery team. This role is critical in enhancing our enterprise observability capabilities by designing, implementing, and maintaining monitoring solutions using tools such as Grafana, Splunk, and Dynatrace. The ideal candidate will have a strong background in telemetry (logs, metrics, traces, events), performance monitoring, and dashboard visualization.

This role will be in Observability Engineering & Product Delivery team. The team maintains the firm's monitoring and Observability tools and infrastructure, and this position is primarily for working on Splunk, Grafana and Observability.

Your Primary Responsibilities:

• Working on engineering and development focused projects from start to finish with minimal supervision

• Providing technical and operational support for our customer base as well as other technical areas within the company that utilize our tools

• Risk management functions such as reconciliation of vulnerabilities, security baselines as well as other risk and audit related objectives

• Administrative functions for our tools such as keeping the tool documentation current and handling service requests

• Design and implement observability solutions across distributed systems using Grafana, Splunk ITSI, and Dynatrace.

• Develop and maintain custom dashboards and visualizations tailored to business and operational needs.

• Integrate observability tools with various data sources (e.g., Prometheus, CloudWatch, Service Now, Snowflake).

• Collaborate with application and infrastructure teams to define SLIs/SLOs and improve system reliability.

• Troubleshoot and resolve issues related to monitoring gaps, alert noise, and data ingestion.

• 24x7 on-call L3 support on a rotational schedule with other team members

• Participating in user training to increase awareness of Splunk, Grafana & Observability

• Ensuring incidents, problems and change tickets are addressed in a timely fashion, as well as escalating technical and managerial issues

• Following DTCC's ITIL process for incident, change and problem resolution.

• Good knowledge of TCP/IP and networking fundamentals

• Good knowledge of engineering, configuring, deploying and supporting Splunk Enterprise, Splunk Cloud, ITSI, Grafana, and Observability

• Ability to create and optimize Big Data correlations as a Splunk search language (SPL) proficient

• Proficient in Grafana queries, dashboard creations, and other development and administration tasks.

• Optimize/Tune logging source streams

• Develop Splunk reports to meet requirements of key stakeholders.

• Good knowledge of Amazon AWS products and services such as EC2, Lambda, VPC, Route 53, Amazon FW, API Gateway, ELB, and CloudTrail.

Qualifications:

• Minimum of 05+ years of related experience

• Bachelor's degree preferred or equivalent experience

Talents Needed for Success:

• 5+ years' experience of Splunk/Grafana engineering/support in a production environment. This includes all phases of lifecycle management: planning, design, deployment, upkeep and retirement

• Should have developed competency with both Splunk & Grafana in a production environment

• Hands-on experience with Grafana, Splunk (including ITSI), and Dynatrace.

• Strong understanding of telemetry data types and observability architecture.

• Experience with scripting (Python, Bash, PowerShell) and automation tools.

• Familiarity with cloud platforms (AWS and Azure) and containerized environments (Kubernetes).

• Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.

• Strong communication skills.

• Working knowledge in Open Telemetry.

Preferred Qualifications:

• Experience with integrating observability tools into CI/CD pipelines.

• Knowledge of ITSM tools like ServiceNow and incident response platforms like PagerDuty.

• Exposure to AIOps, anomaly detection, and predictive analytics use cases.

Actual salary is determined based on the role, location, individual experience, skills, and other considerations. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation

About Us

With over 50 years of experience, DTCC is the premier post-trade market infrastructure for the global financial services industry. From 20 locations around the world, DTCC, through its subsidiaries, automates, centralizes, and standardizes the processing of financial transactions, mitigating risk, increasing transparency, enhancing performance and driving efficiency for thousands of broker/dealers, custodian banks and asset managers. Industry owned and governed, the firm innovates purposefully, simplifying the complexities of clearing, settlement, asset servicing, transaction processing, trade reporting and data services across asset classes, bringing enhanced resilience and soundness to existing financial markets while advancing the digital asset ecosystem. In 2024, DTCC's subsidiaries processed securities transactions valued at U.S. $3.7 quadrillion and its depository subsidiary provided custody and asset servicing for securities issues from over 150 countries and territories valued at U.S. $99 trillion. DTCC's Global Trade Repository service, through locally registered, licensed, or approved trade repositories, processes more than 25 billion messages annually. To learn more, please visit us at https://www.dtcc.com or connect with us on LinkedIn , X , YouTube , Facebook and Instagram .

DTCC proudly supports Flexible Work Arrangements favoring openness and gives people freedom to do their jobs well, by encouraging diverse opinions and emphasizing teamwork. When you join our team, you'll have an opportunity to make meaningful contributions at a company that is recognized as a thought leader in both the financial services and technology industries. A DTCC career is more than a good way to earn a living. It's the chance to make a difference at a company that's truly one of a kind.

Learn more about Clearance and Settlement by clicking here .

About the Team

The IT SIFMU Delivery Department supports core Clearing and Settlement application delivery for DTC, NSCC and FICC. The department also develops and supports Asset Services, Wealth Management & Insurance Services and Master Reference Data applications.