Data Scientist

Millenniumsoft • Indianapolis, IN

13d • Onsite

About The Position

This role supports the agency’s data strategy by designing, building, and maintaining secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics. It supports data integration, pipeline development, data modeling, governance, and documentation, while collaborating with cross-functional teams to deliver reliable data products. The position develops batch and streaming pipelines, optimizes data warehouse workloads, and applies data governance standards. It also contributes to requirements gathering, solution reviews, technical support, operational processes, and policy and procedure improvements. Mentorship or supervision may be provided based on agency needs.

The role works with solution leads, project managers, data architects, data scientists, and data analysts to translate business needs into technical designs and implement solutions across secure systems. Responsibilities include building and orchestrating data pipelines, developing and tuning dimensional models and SQL workloads, applying security controls for sensitive health data, and documenting solutions. Local candidates only will be considered. The position is a 12-month contract on W2, with a stated pay rate of $43.35 per hour.

Requirements

• Master’s degree in data science, analytics, informatics, computer science, mathematics, engineering, or a related field; or a bachelor’s degree in a related field plus at least 2 years of relevant experience; or at least 4 years of hands-on experience in a similar data engineering role. Relevant certifications and coursework may substitute for formal education or experience requirements.
• At least 4 years developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions.
• At least 4 years working with ACID transactions, MERGE upserts, schema enforcement and evolution, time travel, and Delta table maintenance such as OPTIMIZE, Z-order, and VACUUM.
• At least 4 years working with medallion (bronze/silver/gold) architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.
• At least 4 years using Unity Catalog for workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials.
• At least 4 years right-sizing clusters and SQL warehouses and diagnosing partitioning, shuffle, skew, and caching issues using Spark UI.
• At least 4 years scheduling and managing dependencies with ADF Pipelines and Databricks Workflows (Jobs), and using Git folders (Repos) for source control.
• At least 4 years designing MPP warehouse tables, including hash, round-robin, and replicated distribution; columnstore indexes; and partitioning.
• At least 4 years querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.
• At least 4 years building star-schema facts and dimensions, stored procedures, and views, including SCD Type 1 and Type 2.
• At least 4 years developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines using copy activities, data flows, and triggers.
• At least 4 years with workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling or pausing.
• At least 4 years with ADLS Gen2 folder and file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault.
• At least 4 years using Git and CI/CD in Azure DevOps or GitHub, including branching, pull requests, and deploying Synapse artifacts and Databricks Asset Bundles.
• At least 4 years monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs.
• Strong written and verbal communication skills.
• At least 4 years using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks.
• Local candidates only will be considered.
• Must meet the stated eligibility requirement: U.S. citizen, green card holder, green card EAD, OPT, EAD, or CPT.

Nice To Haves

• Databricks Certified Data Engineer Associate certification; Databricks Certified Data Engineer Professional is a plus.
• Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.
• At least 4 years of experience with Power BI connectivity to Synapse, including Import versus DirectQuery and how model design affects dashboard performance.

Responsibilities

• Assist with the data technology strategy, including resolving strategic data integration issues related to data quality, stewardship, real-time and event-based integration, and a universal vision for data integration.
• Provide direction and oversight for designing and implementing data solutions that integrate and synthesize information across multiple secure systems.
• Assist in defining and implementing data management policies and data engineering design standards, and support continued adherence to them.
• Provide advice, training, and technical support for projects.
• Participate in project activities such as requirements gathering and solution reviews, and explain technical complexities to non-technical personnel.
• Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation.
• Extract, transform, and load data from varied sources using data engineering tools and methods.
• Design, build, and maintain batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lakehouse architecture.
• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics dedicated and serverless SQL pools to support reporting and dashboards.
• Orchestrate and monitor workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery.
• Apply governance and security controls, such as Unity Catalog permissions, role-based access, row- and column-level security, and data masking, to protect PHI/PII in accordance with HIPAA and agency policy.
• Document and demonstrate solutions using documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code.
• Conduct systems analysis and recommend changes to policies and procedures.
• Perform other duties, responsibilities, and activities as assigned.

What This Job Offers

• Career Level: Mid Level