Data Scientist

Technotopia Solutions•Indianapolis, IN

13d•Hybrid

About The Position

The Data Scientist designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics for the Indiana Department of Health. The role supports data integration, pipeline development, data modeling, governance, and documentation, collaborating with cross-functional teams to deliver reliable data products. It contributes to the agency’s data strategy through batch and streaming pipelines, data warehouse workload optimization, and enforcement of data governance standards. The position also supports requirements gathering, solution reviews, technical support, operational processes, and policy and procedure improvements, and may provide mentorship or supervision based on agency needs. This is a one-year contract role based in Indianapolis, Indiana, with a hybrid work arrangement. Local candidates only will be considered. The stated rate is $43.35 per hour.

Requirements

Master’s degree in Data Science, Analytics, Informatics, Computer Science, Mathematics, Engineering, or a related field; or a bachelor’s degree in a related field with at least 2 years of relevant experience; or at least 4 years of hands-on experience in a similar data engineering role. Relevant certifications and coursework may substitute for formal education or experience requirements.
At least 4 years of experience developing Databricks notebooks using PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions.
At least 4 years of experience with Delta Lake features, including ACID transactions, MERGE upserts, schema enforcement and evolution, time travel, and table maintenance such as OPTIMIZE, Z-order, and VACUUM.
At least 4 years of experience with medallion architecture (Bronze/Silver/Gold) and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.
At least 4 years of experience using Unity Catalog for workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials.
At least 4 years of experience optimizing Databricks clusters and SQL warehouses, diagnosing partitioning, shuffle, data skew, and caching issues using Spark UI.
At least 4 years of experience scheduling jobs and managing dependencies with Azure Data Factory pipelines and Databricks Workflows/Jobs, using Git folders (Repos) for source control.
At least 4 years of experience designing MPP warehouse tables using hash, round-robin, and replicated distributions, columnstore indexes, and partitioning.
At least 4 years of experience querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.
At least 4 years of experience building star-schema fact and dimension tables, stored procedures, and views, including Slowly Changing Dimensions Types 1 and 2.
At least 4 years of experience developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines using copy activities, data flows, and triggers.
At least 4 years of experience with workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling or pausing.
At least 4 years of experience with ADLS Gen2 folder and file-format design (Parquet/Delta), Entra ID access control, managed identities, and Azure Key Vault.
At least 4 years of experience with Git and CI/CD using Azure DevOps or GitHub, including branching, pull requests, and deployment of Synapse artifacts and Databricks Asset Bundles.
At least 4 years of experience monitoring pipelines and workloads using Azure Monitor and Log Analytics, configuring alerts, and troubleshooting failed jobs.
Strong written and verbal communication skills.
At least 4 years of experience implementing Microsoft lakehouse patterns, including interoperability and migration paths between Azure Synapse Analytics and Databricks.

Nice To Haves

Databricks Certified Data Engineer Associate certification.
Databricks Certified Data Engineer Professional certification is a plus.
Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.
At least 4 years of experience with Power BI connectivity to Synapse, including Import versus DirectQuery modes and understanding how data model design affects dashboard performance.

Responsibilities

Assist with the data technology strategy, addressing data integration issues involving data quality, stewardship, real-time and event-based integration, and a universal vision for integration.
Provide direction and oversight for designing and implementing data solutions that integrate and synthesize data across secure systems.
Assist in defining, implementing, and maintaining data management policies and data engineering design standards.
Provide advice, training, and technical support for projects.
Participate in requirements gathering and solution reviews, and explain technical complexities to non-technical personnel.
Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation.
Extract, transform, and load data from varied sources using data engineering tools and methods.
Design, build, and maintain batch and streaming Azure Databricks pipelines using PySpark, Spark SQL, and Delta Lake within a bronze/silver/gold medallion lakehouse architecture.
Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics dedicated and serverless SQL pools to support reporting and dashboards.
Orchestrate and monitor workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery.
Apply data governance and security controls, including Unity Catalog permissions, role-based access, row- and column-level security, and data masking, to protect PHI/PII in accordance with HIPAA and agency policy.
Document and demonstrate solutions through documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code.
Support operations by conducting systems analysis and recommending policy and procedure changes.
Perform other duties and responsibilities as assigned; responsibilities may change at any time.