Consultant, Data Engineer
About The Position
As a Consultant, Data Engineer at Infinitive, you will build and maintain data pipelines on Databricks for Infinitive clients. You will work on project teams with senior engineers, architects, and business analysts, writing production Python, PySpark, and SQL code that moves data from source systems into tables used by client analysts and data scientists. The role is suited to an engineer with one to three years of experience who wants client-facing work and exposure to new data platforms and industries. This is a hybrid role that requires working on-site with clients. The client location is McLean, Virginia.
Requirements
- 1 to 3 years of professional experience in data engineering, software engineering, or a related technical role.
- Bachelor's degree in computer science, information systems, engineering, mathematics, or a related field, or equivalent experience.
- Hands-on experience with Databricks on a professional or substantial academic project.
- Working proficiency in Python for data processing.
- Experience building transformations with Apache Spark and PySpark, including DataFrames and Spark SQL.
- Strong SQL skills, including joins, window functions, CTEs, and aggregations.
- Understanding of ETL concepts such as incremental loads, slowly changing dimensions, dimensional modeling, and data quality checks.
- Familiarity with at least one cloud platform: AWS, Azure, or Google Cloud.
- Clear written and verbal communication, and comfort explaining technical work to non-technical stakeholders.
- Authorization to work in the United States.
Nice To Haves
- Databricks Certified Data Engineer Associate certification.
- Experience with Delta Lake features such as MERGE and time travel, or Lakeflow Declarative Pipelines (formerly Delta Live Tables).
- Exposure to Unity Catalog, dbt, Apache Airflow, or Azure Data Factory.
- Experience with Terraform or CI/CD tools such as GitHub Actions or Azure DevOps.
- Prior consulting or client-facing experience.
- Background in financial services, healthcare, or media data.
Responsibilities
- Build and maintain batch and incremental ETL pipelines in Databricks using Python, PySpark, and Spark SQL.
- Ingest data from relational databases, APIs, flat files, and cloud storage, including Amazon S3 and Azure Data Lake Storage, into Delta Lake tables.
- Implement bronze, silver, and gold layers following medallion architecture patterns.
- Write and tune SQL for transformations, data validation, and reporting tables.
- Add data quality checks, handle schema changes, and troubleshoot failed jobs.
- Schedule and monitor workloads with Databricks Workflows while working within Unity Catalog permissions.
- Tune Spark jobs for performance and cost, including partitioning, file sizing, and cluster configuration.
- Use Git for version control, take part in code reviews, and contribute to CI/CD pipelines.
- Document pipelines, source-to-target mappings, and runbooks so client teams can operate the solutions.
- Join client meetings to clarify requirements, report status, and walk through work.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Entry Level
