Senior Director, Software Engineering

Circana Careers•Chicago, IL

8d•$120,000 - $165,000

About The Position

Circana is seeking a skilled and motivated data engineering leader to design, build, and maintain robust data pipelines and infrastructure on cloud platforms. The role focuses on large-scale data processing and orchestration using Python, PySpark, Apache Spark, and Apache Airflow, with attention to data quality, efficiency, scalability, governance, and security. The position leads and mentors data engineers, works directly with clients, and collaborates with engineering, product, architecture, and program management teams to deliver scalable, maintainable products and align delivery timelines. The position may be located remotely or in a hybrid arrangement in Chicago. The stated salary range is $120,000 to $165,000, and the role is eligible for bonus pay. Benefits for eligible employees include paid time off and medical, dental, vision, and 401(k) benefits. The role is subject to AI-assisted screening.

Requirements

At least 8 years of data engineering experience, including expertise in Python, PySpark, Spark, and Airflow.
Strong Python and SQL programming skills, including the ability to write efficient, maintainable code.
Deep understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning.
Experience with Airflow DAGs, scheduling, and dependency management.
Knowledge of Git, Docker, Kubernetes, and Terraform, and application of DevOps best practices for CI/CD workflows.
Strong problem-solving skills and ability to optimize large-scale data processing.
Experience leading teams and working in Agile/Scrum environments.
Experience with a common data warehouse and/or data lake platform.
Knowledge of cloud technologies such as Azure, AWS, or Google Cloud.
Proven experience using generative AI tools to accelerate software development, data engineering, system design, testing, and troubleshooting.
Understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows.
Ability to identify opportunities to apply AI solutions to improve team productivity, code quality, and operational efficiency.
Strong communication and collaboration skills, including ability to work effectively with remote teams.

Nice To Haves

Experience in ML Engineering.

Responsibilities

Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
Implement real-time and batch data processing using Spark.
Apply data quality, governance, and security best practices throughout the data lifecycle.
Maintain data availability, reliability, and performance through monitoring and automation.
Manage cloud infrastructure and optimize costs for data processing workloads.
Implement CI/CD pipelines for data workflows to support reliable deployments.
Build and optimize large-scale data pipelines using Apache Spark and PySpark.
Apply partitioning, caching, and performance tuning to Spark workloads.
Work with structured and unstructured data to support analytics and machine learning initiatives.
Design and maintain Airflow DAGs to automate complex workflows.
Monitor, troubleshoot, and optimize job execution and dependencies.
Lead and mentor a team of data engineers.
Promote coding standards, version control, documentation, and collaborative practices.
Engage with clients to understand business needs and communicate technical concepts.
Collaborate with engineering, product, and architecture peers on scalable and maintainable products.
Partner with Program Management to align delivery timelines and mitigate risks.