Sr. Data Engineer - Data & Intelligence

TekWissen•Frisco, TX

1d•Onsite

About The Position

TekWissen is a global workforce management provider headquartered in Ann Arbor, Michigan that offers strategic talent solutions to our clients world-wide. Our client provider of digital technology and transformation, information technology and services. This role involves architecting, designing, and overseeing the development of enterprise-scale ELT/ETL pipelines for finance and revenue data. The position requires defining and enforcing standards for various data ingestion patterns, ensuring scalable and fault-tolerant pipeline design, and providing technical leadership for complex data integration projects. The role also includes leading the architecture and adoption of Snowflake and Databricks platforms, defining best practices for these tools, and establishing orchestration frameworks. Additionally, the Sr. Data Engineer will architect and optimize cloud-native data platforms on Azure or AWS, define infrastructure-as-code standards, and drive cost optimization strategies. Providing deep technical leadership in advanced SQL and Python/PySpark, architecting real-time data processing solutions, establishing enterprise frameworks for data quality and testing, and supporting data modeling are key responsibilities. The role also involves defining and enforcing CI/CD standards, leading the implementation of security and governance controls, and leveraging deep understanding of finance and revenue data domains. Strong soft skills and collaboration abilities are essential for technical leadership and stakeholder partnership.

Requirements

Advanced SQL (query tuning, execution optimization, complex transformations)
Python / PySpark for distributed data processing
Deep understanding of finance and revenue data domains, including billing and revenue systems, GL structures and financial reporting, revenue recognition and reconciliation, and period-end close cycles.
Experience with Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency)
Experience with Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization)
Experience with dbt (modular design, testing frameworks, CI/CD integration, reusable components)
Experience with orchestration frameworks like Airflow / Azure Data Factory
Experience with cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents
Experience with infrastructure-as-code (Terraform, Bicep)
Experience with real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming
Experience with data quality, validation, and observability frameworks
Experience with automated testing (unit, integration, regression)
Experience with data validation (completeness, accuracy, consistency)
Experience with data quality tools (dbt tests, Great Expectations, custom frameworks)
Experience interpreting and implementing enterprise data models (star, snowflake, data vault)
Experience with SCD (Type 1/2) strategies, partitioning, clustering, and performance optimization
Experience with CI/CD standards for data engineering (GitHub Actions, Azure DevOps)
Experience with code quality, versioning, and deployment best practices
Experience with environment promotion (dev → QA → prod) and release management
Experience with enterprise-grade security and governance controls: RBAC, row/column-level security, PII and CPNI compliance (TISS-310)
Experience with secrets management and secure pipeline design
Strong communication skills to explain complex technical topics to both technical and non-technical audiences.

Nice To Haves

Scala (where applicable)
Support semantic layer enablement for analytics and reporting.
Drive adoption of engineering excellence practices including reusable frameworks and templates.

Responsibilities

Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).
Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).
Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.
Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.
Provide technical leadership for multi-source, high-volume data integration pipelines.
Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.
Define best practices for Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency) and Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization).
Define best practices for dbt (modular design, testing frameworks, CI/CD integration, reusable components).
Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring.
Evaluate and drive tooling strategy and platform standardization across teams.
Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents.
Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning.
Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation).
Ensure platforms are scalable, secure, and production-ready.
Provide deep technical leadership in Advanced SQL (query tuning, execution optimization, complex transformations) and Python / PySpark for distributed data processing.
Guide teams on best practices, reusable frameworks, and performance optimization.
Oversee development standards for Spark, Scala (where applicable), and automation scripting.
Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming.
Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance.
Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting.
Establish enterprise frameworks for data quality, validation, and observability.
Define standards for automated testing (unit, integration, regression), data validation (completeness, accuracy, consistency), and data quality tools (dbt tests, Great Expectations, custom frameworks).
Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines.
Drive proactive data quality and governance practices across teams.
Interpret and implement architect-defined enterprise data models (star, snowflake, data vault).
Provide guidance on SCD (Type 1/2) strategies, partitioning, clustering, and performance optimization.
Collaborate with architects to evolve scalable and reusable data models.
Support semantic layer enablement for analytics and reporting.
Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps).
Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines).
Standardize environment promotion (dev → QA → prod) and release management.
Drive adoption of engineering excellence practices including reusable frameworks and templates.
Lead implementation of enterprise-grade security and governance controls: RBAC, row/column-level security, PII and CPNI compliance (TISS-310).
Define standards for secrets management and secure pipeline design.
Ensure data lineage, auditability, and compliance readiness across platforms.
Guide engineering teams on accurate implementation of finance logic.
Ensure high data integrity standards for regulated financial data.
Act as a technical leader and escalation point across engineering teams.
Partner with architects, product managers, analysts, and business stakeholders.
Drive cross-team alignment and solution consistency.
Communicate complex technical topics clearly to both technical and non-technical audiences.
Lead incident reviews and ensure continuous improvement.