Senior Data Engineer

The Port Authority of NY & NJ•New York, NY

49d•$100,256 - $160,394•Hybrid

About The Position

The Aviation Department is seeking a Senior Data Engineer with a proven background in cloud-based enterprise data architecture, excellent troubleshooting and analytical abilities, and strong communication skills who is motivated to strengthen the technical foundation of the Department’s data environment and contribute to the continued evolution of the Aviation Data Analytics Warehouse (ADAW). The Data Analytics & Business Intelligence (DABI) team serves as the centralized Data Science Unit for the Department and manages ADAW, a governed data environment that integrates more than fifteen data streams and provides information to more than 500 users across the agency. Reporting to the Program Manager, Data Analytics and Business Intelligence, the Senior Data Engineer will support the development, maintenance, and modernization of the team’s data infrastructure; manage a team of data engineer contract staff; help strengthen the Department’s technical capabilities; and deliver high quality analytical and engineering solutions that support airport operations, planning, customer experience, and strategic decision- making. This role is the technical backbone of a decision-support system that moves millions through our world‑class airport system.

Requirements

Bachelor’s degree in Computer Science, Data Science, Data Engineering, Information Systems, Applied Mathematics, or a related quantitative field.
At least three (3) years of experience supporting data systems, analytics pipelines, technical workflows, or data engineering activities in an enterprise cloud environment (ideally Azure).
Strong understanding of data architecture, dimensional data modeling, structured data warehousing, integration, ETL processes and analytics.
A minimum of (3) years of experience building datasets, automating tasks through scripts, writing database queries, and debugging/ maintaining code.
A minimum of (4) year of experience with Python and SQL programming.
Minimum of (3) years of experience with relational databases to obtain and manipulate data.
Experience designing or supporting structured data solutions or analytical tools for cross-functional business teams, ideally in a large organization or public agency.
Ability to multitask, prioritize, and manage time effectively.
Strong written and verbal communication skills and the ability to work collaboratively with technical and nontechnical partners across the Department.

Nice To Haves

Master’s degree in a relevant technical or analytical field, ideally Data Science or Engineering, Information Systems, Software Engineering or Computer Science
A minimum of (3) years developing enterprise-level Power BI semantic models and managing Power BI workspaces and resource capacity
Experience in FinOps, usage monitoring, billing analysis, or creating automated diagnostic workflows in a Microsoft Azure environment
Experience working with enterprise data systems or operational datasets in transportation, infrastructure, or public sector environments
Experience developing or maintaining APIs, Webhooks or other data transfer protocols
Knowledge of MCP-optimized data warehousing best practices for integration with AI Agents
Familiarity with geospatial data concepts or experience contributing to map-based analytics
Competency in JavaScript, front end development, or web app development
Experience improving data quality, documentation, or engineering processes within a collaborative team environment

Responsibilities

Design, develop, and maintain ADAW’s Azure‑based data architecture, including Synapse ETL/ELT Pipelines, SQL pools, Azure Event Hub jobs, Databricks workflows, and blob storage accounts, ensuring data is reliable, well‑structured, and aligned with departmental needs.
Build, maintain and optimize shared Power BI semantic models in close collaboration with Senior Data Scientist and relevant stakeholders, ensuring efficient relationships, stable table structures, and performant DAX measures.
Monitor and troubleshoot Azure Synapse performance, including CPU utilization, job schedule behavior, failed pipeline runs, and compute scaling needs; configure alerts and automated diagnostics using Azure Monitor, Log Analytics, and shared engineering dashboards.
Develop and manage end‑to‑end data ingestion and integration processes in Azure, including high-volume ingestion of third‑party message queues, automated API calls, and structured batch transfers from vendor systems, ensuring quality, completeness, and adherence to governance standards.
Implement durable data validation, schema governance, and automated QA checks and ensure standardized structures, naming conventions, surrogate key patterns, and data‑type/format rules.
Drive innovation in data management practices, stay abreast of industry trends and emerging data warehousing technologies.
Manage team of contract data engineers to establish engineering best practices, manage task priorities, address questions around business logic, help with QAQC activities, and ensure on-time delivery of fixes and enhancements.
Serve as the primary point of contact with the agency Technology Department to maintain cloud infrastructure, align with cybersecurity and network access requirements, and resolve Azure‑related issues (e.g., compute throttling, storage permissions, service interruptions).
Document dataset structures, key logic, and processing steps to ensure clarity and long-term sustainability of Aviation’s data products.
Help develop enhanced data dissemination methods (API endpoints, progressive web applications) for improved collaboration with other mature internal data teams.
Support continuous improvement of DABI’s engineering standards, internal documentation, business logic, and process transparency.
Provide technical guidance to colleagues across the Department, including embedded analysts, to strengthen analytical and engineering capacity.
Coordinate with Senior Data Scientist and Program Manager to provide project updates and align on business rules and technical requirements with stakeholders from other business units, operational teams, and external partners.
Support collaborative efforts with units such as Traffic Engineering, Redevelopment, and airport operations groups, ensuring alignment in data assumptions and analytic practices.
Participate in recurring technical discussions, coordination meetings, and cross-team engagements to ensure ADAW data and analytical tools remain consistent, reliable, and responsive to departmental needs.