IN - IDOH Data Scientist (812974)
About The Position
The Data Scientist designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics. This role supports data integration, pipeline development, data modeling, governance, and documentation, collaborating with cross-functional teams to deliver reliable data products for the agency. It contributes to the agency’s data strategy through batch and streaming pipelines, optimized data warehouse workloads, and data governance standards. The position also supports requirements gathering, solution reviews, technical support, operational processes, and policy and procedure improvements, and may provide mentorship or supervision based on agency needs. The role is hybrid and limited to local candidates. The selected candidate will work with data architects, scientists, analysts, project managers, and solution leads to translate business needs into technical designs. The position’s stated engagement dates are November 2, 2026, through November 30, 2027.
Requirements
- Meet one of the following qualifications: a master’s degree in data science, analytics, informatics, computer science, mathematics, engineering, or a related field; a bachelor’s degree in a related field plus at least 2 years of relevant experience; or at least 4 years of hands-on experience in a similar data engineering role. Relevant certifications and coursework may substitute for formal education or experience requirements.
- Four years of experience developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions.
- Four years of experience with ACID transactions, MERGE upserts, schema enforcement and evolution, time travel, and table maintenance such as OPTIMIZE, Z-order, and VACUUM.
- Four years of experience with medallion architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.
- Four years of experience using Unity Catalog for workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials.
- Four years of experience right-sizing clusters and SQL warehouses and diagnosing partitioning, shuffle, skew, and caching issues using Spark UI.
- Four years of experience scheduling and managing dependencies with ADF Pipelines and Databricks Workflows (Jobs), and using Git folders (Repos) for source control.
- Four years of experience designing MPP warehouse tables, including distribution methods, columnstore indexes, and partitioning.
- Four years of experience querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.
- Four years of experience building star-schema facts and dimensions, stored procedures, and views, including SCD Type 1 and Type 2.
- Four years of experience developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines, including copy activities, data flows, and triggers.
- Four years of experience with workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling or pausing.
- Four years of experience with ADLS Gen2 folder and file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault.
- Must be a local candidate; local candidates only will be considered.
Responsibilities
- Assist with data technology strategy, addressing data integration issues involving data quality, stewardship, real-time and event-based integration, and a universal vision for integration.
- Provide direction and oversight for designing and implementing data solutions that integrate and synthesize data across secure systems.
- Assist in defining, implementing, and maintaining data management policies and data engineering design standards.
- Provide advice, training, and technical support for projects.
- Participate in requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel.
- Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation.
- Extract, transform, and load data from varied sources using data engineering tools and methods.
- Design, build, and maintain batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake in a medallion (bronze/silver/gold) lakehouse architecture.
- Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics dedicated and serverless SQL pools for reporting and dashboards.
- Orchestrate and monitor workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery.
- Apply data governance and security controls, including Unity Catalog permissions, role-based access, row- and column-level security, and data masking, to protect PHI/PII in accordance with HIPAA and agency policy.
- Document and demonstrate solutions using documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code.
- Conduct systems analysis and recommend changes to operational policies and procedures.
- Perform other duties and responsibilities as assigned.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
