Sr. Data Engineer
About The Position
As the Sr. Data Engineer, you will play a critical role in building and architecting automation and observability for the EDAS platform, with a particular focus on the data consumption side in the near term. You will be responsible for emphasizing automating workflows, ensuring platform reliability by developing data-driven observability tools and collaborating with teams to enhance efficiency and performance. The platform to automate & observe includes Redshift, Spotfire and triggers, Dataiku project flows, and Talend. The role focuses on designing and implementing automation solutions to streamline workflows, enhance efficiency, and ensure platform reliability. It involves building observability frameworks for real-time monitoring, logging, and alerting, optimizing data pipelines for scalability, and ensuring compliance with data governance and security standards.
Requirements
- Bachelor’s degree from accredited university or college with minimum of 2 years of professional experience OR associate's degree with minimum of 5 years of professional experience OR High School Diploma with minimum of 7 years of professional experience
- Programming: Python, Bash scripting.
- Containerization: Docker.
- Cloud Services: AWS (Lambda, CloudTrail, X-Ray, CloudWatch, S3, IAM).
- Observability Tools: Prometheus, Grafana, Datadog, FluentBit, ELK stack, OpenSearch.
- Data Pipelines: Airflow, Glue, Step Functions.
- Standards: OpenTelemetry, E2E lineage strategy, CDC tooling (AWS DMS, Debezium).
- Databases: SQL, Elasticsearch/OpenSearch stacks.
- Infrastructure: AMI creation, deployment, CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation).
- Legal authorization to work in the U.S. is required.
- ability to prove that you meet the status of a U.S. Person as one of the following: U.S. lawful permanent resident, U.S. Citizen, have been granted asylee or refugee status (i.e., a protected individual under the Immigration and Naturalization Act, 8 U.S.C. 1324b(a)(3)).
Nice To Haves
- Web application stacks (Node.js, Vue, React).
- Security/compliance observability integration.
- Custom connectors/APIs for observability tools.
- OpenMetadata experience.
- Machine learning or AI-driven automation techniques.
- Kubernetes for container orchestration.
Responsibilities
- Build and maintain observability frameworks to monitor the health, performance, and reliability of data and analytic platforms.
- Implement tools for real-time monitoring, logging, and alerting to ensure proactive issue detection and resolution.
- Analyze system metrics and logs to identify trends, anomalies, and areas for improvement.
- Design and implement intelligent automation solutions to streamline data and analytic processing workflows and platform operations.
- Develop scripts, tools, and frameworks to automate repetitive tasks and improve system efficiency.
- Collaborate with cross-functional teams to identify automation opportunities and implement solutions.
- Ensure high availability and fault tolerance of the platform through robust monitoring and automation strategies.
Benefits
- Healthcare benefits include medical, dental, vision, and prescription drug coverage; access to a Health Coach, a 24/7 nurse-based resource; and access to the Employee Assistance Program, providing 24/7 confidential assessment, counseling and referral services.
- Retirement benefits include the GE Retirement Savings Plan, a tax-advantaged 401(k) savings opportunity with company matching contributions and company retirement contributions, as well as access to Fidelity resources and planning consultants.
- Other benefits include tuition assistance, adoption assistance, paid parental leave, disability insurance, life insurance, and paid time-off for vacation or illness.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
