Databricks Data Engineer / Business Analyst – Data Contracts & OCDS

Scicom Infrastructure Services•Atlanta, GA

3d•Hybrid

About The Position

Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. This hybrid role bridges business, contracting, data-governance, and engineering teams to define business requirements, construct data contracts, map source data, and develop reliable Databricks data pipelines. The role requires hands-on experience with the Open Contracting Data Standard (OCDS) and an understanding of how contracting and procurement information is structured across planning, tender, award, contract, and implementation stages. The successful candidate will work with business stakeholders, procurement subject-matter experts, data architects, Databricks engineers, governance teams, and program leadership to translate contracting requirements into technical specifications and production-ready data products. Responsibilities span business analysis and data-contract management, OCDS data mapping and validation, Databricks engineering, and data quality and governance. The role includes requirements gathering, pipeline development and support, source-to-target mapping, validation and reconciliation, documentation, stakeholder coordination, and production troubleshooting.

Requirements

Bachelor’s degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field.
At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience.
Hands-on experience implementing or working with the Open Contracting Data Standard (OCDS).
Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts.
Experience mapping procurement or contracting data to OCDS schemas.
Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices.
At least three years of hands-on experience with Databricks.
Strong experience with Apache Spark, PySpark, Python, SQL, Delta Lake, ETL and ELT pipelines, data modeling, JSON and JSON Schema, REST APIs, data-quality validation, and source-to-target mapping.
Experience gathering requirements and translating them into implementable engineering specifications.
Ability to communicate effectively with technical and nontechnical stakeholders.
Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation.
Strong analytical, troubleshooting, facilitation, and documentation skills.
Experience working within Agile delivery teams.

Nice To Haves

Experience with government procurement, public-sector contracting, grants, acquisition, supplier, or financial data.
Experience implementing OCDS extensions or tailoring OCDS for specific organizational requirements.
Familiarity with procurement classifications, organizational identifiers, tender processes, awards, amendments, milestones, transactions, and contract implementation.
Databricks Certified Data Engineer Associate or Professional certification.
Experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, or Structured Streaming.
Experience with Microsoft Azure, AWS, or Google Cloud.
Experience with Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Power BI.
Experience using JSON Schema validation tools and automated data-quality frameworks.
Knowledge of metadata management, master-data management, data lineage, and reference-data governance.
Experience with CI/CD, Git, Azure DevOps, GitHub Actions, Jenkins, or Terraform.
Experience supporting large federal, state, local-government, or regulated-enterprise data programs.
Familiarity with federal acquisition, procurement, reporting, transparency, or open-data requirements.

Responsibilities

Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders.
Define, construct, document, and maintain data contracts between data producers and consumers, covering ownership, systems, schemas, business rules, quality expectations, schedules, versioning, security, service levels, and change management.
Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements.
Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping; identify differences between existing procurement data and required OCDS structures.
Facilitate agreement among data owners, producers, consumers, architects, and governance teams, and maintain traceability from requirements through implementation, testing, and acceptance.
Evaluate requested changes for downstream impacts and support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance.
Map procurement and public-contracting data to OCDS fields and structures across planning, tender, award, contract, and implementation stages.
Develop and maintain OCDS mappings for releases, records, packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related elements.
Interpret OCDS schemas, codelists, validation rules, and implementation guidance; determine when documented extensions are needed.
Support construction of complete contracting records from multiple transactional releases and establish rules for amendments, updates, cancellations, corrections, and historical changes.
Validate transformed data against applicable OCDS JSON schemas; identify and help resolve missing, invalid, inconsistent, or nonconforming procurement data.
Document assumptions, business rules, mappings, extensions, and exceptions, and support standardized contracting-data publication, exchange, analysis, or internal use.
Design, develop, test, and maintain Databricks pipelines using Apache Spark, PySpark, Python, and SQL for structured and semi-structured procurement data.
Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources; implement Bronze, Silver, and Gold medallion data layers and normalized, dimensional, analytical, and OCDS-aligned models.
Use Delta Lake for schema enforcement, schema evolution, versioning, auditability, and reliable processing; build reusable frameworks for mapping source data to OCDS-compatible outputs.
Implement batch and incremental ingestion, Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities as appropriate, and REST API integrations.
Implement automated data-quality, reconciliation, completeness, and conformity checks; support Unity Catalog for metadata, ownership, lineage, access control, and data discovery.
Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts; participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support.
Investigate pipeline failures, data discrepancies, and performance issues.
Define measurable data-quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness, and develop validation controls for OCDS fields and contracting relationships.
Create dashboards or reports showing data-contract compliance and data-quality results, and establish processes to detect and manage schema drift.
Document lineage from original procurement systems through Databricks transformations and downstream products; work with governance teams on data ownership, stewardship, classification, retention, and access policies.
Ensure sensitive procurement and supplier information is handled according to security and privacy requirements, and support auditability through documented rules, version-controlled mappings, validation results, and change histories.