Senior Data Engineer
About The Position
VivSoft is seeking a Senior Data Engineer to support a Department of War program building and operating a modern, scalable, and secure data platform. The role will lead the design and optimization of an enterprise lakehouse architecture on AWS, using Apache Iceberg, Apache Spark, and cloud-native technologies to support advanced analytics, AI/ML initiatives, and operational reporting. The successful candidate will bring deep expertise in large-scale data platforms, distributed processing, data governance, and cloud infrastructure, while providing technical leadership and mentoring across engineering teams. VivSoft works on complex federal problems using emerging and open technologies. Its teams build secure Software Factories based on DoD reference designs and NIST Frameworks for Cloud and DevSecOps, delivering AI/ML applications, data science platforms, blockchain, and microservices for DoD, healthcare, and civilian agencies.
Requirements
- Must possess an active Secret Clearance.
- At least 8 years of professional experience in Data Engineering or large-scale data platform development.
- Expertise in Apache Spark, including performance tuning, partitioning, memory optimization, and handling data skew.
- Strong proficiency in Python or Scala and advanced SQL development.
- Experience with open table formats such as Apache Iceberg or Delta Lake.
- Strong understanding of distributed query engines, including Athena, Trino, and Spark SQL.
- Hands-on experience with AWS services including S3, Glue, EMR, Athena, EC2, SQS, and event-driven architectures.
- Experience with Apache Airflow for workflow orchestration.
- Proficiency with Terraform and Infrastructure as Code (IaC).
- Experience working with Kubernetes environments.
- Experience developing dashboards and data products using Grafana or similar visualization platforms.
- Strong understanding of data quality, data governance, lineage, and access control frameworks.
- Excellent technical leadership, mentoring, and stakeholder communication skills.
- Ability to translate business, operational, and analytics requirements into scalable data platform solutions.
- Strong collaboration skills with Data Scientists, AI Engineers, Cloud Engineers, and business stakeholders.
- Proven ability to lead technical discussions and mentor junior engineers.
- Excellent written and verbal communication skills.
- Strong attention to detail regarding data quality, governance, lineage, security, and operational reliability.
Nice To Haves
- Experience operating and maintaining Apache Iceberg tables at enterprise scale.
- Experience with Trino administration and performance optimization.
- Knowledge of dbt or similar data transformation frameworks.
- Experience with Helm and GitOps deployment methodologies.
- Experience evaluating and implementing modern data visualization platforms beyond Grafana.
- Familiarity with data catalog, metadata management, lineage, and governance tools.
- AWS Data Analytics Certification and/or Certified Kubernetes Administrator (CKA).
- Prior DoD, USAF, or Federal Government data platform experience.
- Experience supporting AI/ML, data science, or advanced analytics workloads in cloud environments.
Responsibilities
- Architect, build, and manage a cloud-based lakehouse environment using Apache Iceberg, AWS S3, and AWS Glue Catalog.
- Develop and optimize Apache Spark data pipelines on EMR and Kubernetes environments.
- Design and implement event-driven data ingestion solutions using S3 events and Amazon SQS.
- Optimize query performance across Athena, Trino, and Spark SQL environments.
- Orchestrate data workflows and ETL pipelines using Apache Airflow.
- Manage infrastructure deployment and automation using Terraform.
- Develop, publish, monitor, and maintain data products and analytical dashboards.
- Implement data quality, governance, lineage, access controls, and cost management best practices.
- Create technical documentation, architectural designs, and operational runbooks.
- Provide technical leadership and mentorship to junior engineering team members.
Benefits
- Comprehensive medical, dental, and vision plans; healthcare benefits are 100% employer-paid for employees only.
- Life insurance.
- Flexible/Combined PTO, bereavement leave, and 11 company-paid holidays.
- 401K retirement plan with employer match.
- Professional development training reimbursement.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
