Senior Director, Software Engineering

Circana•Chicago, IL

9d•$120,000 - $165,000

About The Position

Circana is seeking a skilled and motivated data engineering professional to design, build, and maintain data pipelines and infrastructure, with a focus on cloud data processing, large-scale workloads, data quality, and workflow orchestration. The role involves leading data engineers, providing technical guidance and mentorship, and collaborating with clients and cross-functional teams to deliver scalable, maintainable products. The position may be located remotely or in a hybrid arrangement in Chicago.

Requirements

At least 8 years of data engineering experience, with expertise in Python, PySpark, Spark, and Airflow.
Strong Python and SQL programming skills and ability to write efficient, maintainable code.
Deep understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning.
Experience with Airflow DAGs, scheduling, and dependency management.
Knowledge of Git, Docker, Kubernetes, and Terraform, and DevOps best practices for CI/CD workflows.
Strong problem-solving skills and ability to optimize large-scale data processing.
Experience leading teams and working in Agile/Scrum environments.
Experience with common data warehouse and/or data lake platforms.
Knowledge of cloud technologies such as Azure, AWS, or Google Cloud.
Proven experience using generative AI tools to accelerate software development, data engineering, system design, testing, and troubleshooting.
Understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows.
Strong communication and collaboration skills, including the ability to work effectively with remote teams.

Nice To Haves

Machine learning engineering experience.

Responsibilities

Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
Implement real-time and batch data processing using Spark.
Apply data quality, governance, and security practices throughout the data lifecycle.
Monitor and automate data systems to ensure data availability, reliability, and performance.
Manage cloud infrastructure and optimize costs for data processing workloads.
Implement CI/CD pipelines for data workflows and reliable deployments.
Build and optimize large-scale data pipelines using Apache Spark and PySpark.
Apply partitioning, caching, and performance tuning to Spark workloads.
Work with structured and unstructured data formats for analytics and machine learning initiatives.
Design and maintain Airflow DAGs to automate complex data workflows.
Monitor, troubleshoot, and optimize job execution and dependencies.
Lead data engineers by providing technical guidance and mentorship.
Promote collaboration and best practices for coding standards, version control, and documentation.
Engage with clients to understand business needs and clearly communicate technical concepts.
Collaborate with engineering, product, and architecture peers to develop scalable, maintainable products.
Partner with Program Management to align delivery timelines and mitigate risks.
Identify opportunities to apply AI solutions that improve team productivity, code quality, and operational efficiency.