Senior Director, Software Engineering

Circana•,

9h•$145,000 - $190,000

About The Position

Circana is seeking a skilled and motivated data engineering leader to design, build, and maintain robust data pipelines and infrastructure on cloud platforms. The role focuses on scalable data processing and orchestration using Python, PySpark, Apache Spark, and Airflow, with an emphasis on data quality, efficiency, reliability, and scalability. The position also leads a team of data engineers, provides technical guidance and mentorship, and works with clients and cross-functional partners to deliver maintainable products and meet business needs. The position may be based remotely or in a hybrid arrangement in Chicago. The stated salary range is $145,000 to $190,000, and the role is eligible for bonus pay. Circana offers eligible employees paid time off and medical, dental, vision, and 401(k) benefits. The role is subject to AI-assisted resume screening.

Requirements

At least 8 years of data engineering experience with Python, PySpark, Spark, and Airflow.
Strong Python and SQL programming skills, including the ability to write efficient, maintainable code.
Deep understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning.
Experience with Airflow DAGs, scheduling, and dependency management.
Knowledge of Git, Docker, Kubernetes, and Terraform, and application of DevOps best practices for CI/CD workflows.
Strong problem-solving skills and ability to optimize large-scale data processing.
Experience leading teams and working in Agile/Scrum environments.
Experience with a common data warehouse and/or data lake platform.
Knowledge of cloud technologies such as Azure, AWS, or Google Cloud.
Proven experience using generative AI tools to accelerate software development, data engineering, system design, testing, and troubleshooting.
Understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows.
Strong communication and collaboration skills, including the ability to work effectively with remote teams.

Nice To Haves

Experience in machine learning engineering.

Responsibilities

Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
Implement real-time and batch data processing using Spark.
Enforce data quality, governance, and security best practices throughout the data lifecycle.
Monitor and automate data systems to ensure data availability, reliability, and performance.
Manage cloud infrastructure and optimize costs for data processing workloads.
Implement CI/CD pipelines for reliable deployment of data workflows.
Build and optimize large-scale data processing pipelines using Apache Spark and PySpark.
Apply data partitioning, caching, and performance tuning to Spark workloads.
Work with structured and unstructured data formats to support analytics and machine learning initiatives.
Design and maintain Airflow DAGs to automate complex data workflows.
Monitor, troubleshoot, and optimize job execution and workflow dependencies.
Lead a team of data engineers, providing technical guidance and mentorship.
Promote collaborative working practices and coding standards, version control, and documentation.
Engage directly with clients to understand business needs and communicate technical concepts clearly.
Collaborate with engineering, product, and architecture peers to develop scalable and maintainable products.
Partner with Program Management to align delivery timelines and mitigate risks.
Identify and apply AI solutions to improve team productivity, code quality, and operational efficiency.