Principal Engineering Lead — Content & Data Pipeline

Wiley Global Technology • Hoboken, NJ

1d

About The Position

The Principal Engineering Lead owns Wiley’s Content & Data Pipeline platform end to end, spanning ingestion, transformation, enrichment, metadata management, rights-aware delivery, and vendor and supplier integration for scholarly publications and books. The platform processes content from diverse sources and delivers it to internal AI and RAG systems and external customers under applicable entitlements. The role expands the existing platform to cover new sources and delivery surfaces, with higher standards for quality, freshness, reliability, and observability.

This is a hands-on player-coach leadership role. The lead sets technical direction, remains involved in architecture and code, and manages and develops a team of engineers responsible for pipeline work. The position works alongside the existing development lead, coordinating ownership and roadmap execution, and partners with product, business, licensing partners, and vendors to turn needs into a prioritized platform roadmap.

The role owns delivery commitments and engineering practices, including production operation, delivery SLAs, data quality, CI/CD, code review, on-call, incident response, and postmortems. It also guides platform expansion through architectural decisions and communicates technical strategy and tradeoffs to engineering, product, business, vendor, and executive stakeholders.

Requirements

• Experience leading an engineering team while remaining hands-on with architecture and code.
• Ability to influence product, business, vendor, and executive stakeholders and communicate technical strategy in language appropriate to each audience.
• Experience managing vendor and partner relationships, including technical integration, contract execution, renewal input, and escalation.
• Ability to communicate scope, quality, and timeline tradeoffs concretely and honestly.
• Deep production experience designing and operating end-to-end content or data pipelines at scale, including ingestion, processing, and delivery.
• Production-level AWS experience with S3, Lambda, ECS or EKS, SQS/SNS, EventBridge, Step Functions, RDS, and DynamoDB, including IAM and cost fluency.
• Production-level Python and SQL skills.
• Production experience with workflow orchestration such as Airflow, Step Functions, Dagster, or equivalent, including idempotency, backfills, retries, and dependency management.
• Experience with content transformation, XML processing, schema-driven transformation, and structured content formats.
• Data quality experience with schema validation, defensive parsing, incremental correctness checks, freshness monitoring, and drift alerting.
• Distributed systems knowledge, including idempotency, retries, duplicate handling, partial failure, and replay.
• Production experience with API-based and bulk ingestion from external sources, including authentication, pagination, rate limiting, and handling silent schema changes.
• Pipeline testing and CI/CD discipline, including unit, integration, and data-quality tests in CI and safe deployment patterns.
• Observability experience with logs, metrics, lineage, and determining whether a pipeline is healthy rather than merely running.
• Active use of AI coding agents in day-to-day work, with review.

Nice To Haves

• Experience in scholarly publishing, including journals, articles, and books.
• Experience building or operating pipelines that ingest, process, and deliver structured content such as XML, HTML, EPUB, or PDF, with attention to structure and metadata.
• Experience handling content corrections, updates, and withdrawals after publication.
• Experience delivering the same content in different forms to feeds, APIs, search, and AI systems.
• Experience with content usage rights or access restrictions.
• Experience partnering with editorial, production, or content-operations teams.
• Knowledge of scholarly publishing technologies and standards such as JATS/BITS XML, DOI infrastructure, Crossref deposits, ORCID/ROR identity, and MeSH or similar controlled vocabularies.
• BigQuery or GCP experience.
• Experience with specialized publishing tooling, XSLT at scale, content-management systems, or custom XML processing frameworks.
• Streaming ingestion experience with Kafka, Kinesis, Pub/Sub, or similar tools.
• Experience with vector search and embedding pipelines.
• Experience partnering with ML, AI, or agent teams as a downstream platform consumer.
• Experience with data catalog and lineage tools such as DataHub, Amundsen, or equivalent.
• Existing relationships in the scholarly publishing ecosystem, including vendors, standards bodies, or peer engineering leaders at similar organizations.

Responsibilities

• Own the end-to-end architecture of the content and data pipeline, including ingestion from vendors and public sources, transformation and enrichment, metadata management, and delivery to internal AI/RAG systems and external customers.
• Operate the pipeline in production and manage content lifecycle events, including corrections, retractions, version updates, metadata deposits, registry syndication, and rights and entitlements enforcement.
• Set standards for reliability, data quality, freshness monitoring, testing, observability, and CI/CD across the team's services.
• Document architectural decisions in ADRs and drive shared patterns across ingestion, processing, and delivery.
• Remain hands-on by writing code, conducting architecture reviews, prototyping, and unblocking the team.
• Manage and coach engineers working across content ingestion, processing, and delivery.
• Partner with the existing development lead to align ownership boundaries and coordinate roadmap execution.
• Own delivery commitments, including scope, sequencing, dependencies, and stakeholder tradeoff discussions.
• Establish and run scalable engineering practices, including code review, on-call rotations, incident response, postmortems, and CI quality gates.
• Partner with product and business teams to prioritize platform capabilities, content sources, delivery surfaces, and quality standards.
• Own technical relationships with content vendors and licensing partners, including integrations, data contracts, escalations, and renewal input.
• Translate editorial operations, product, and licensing needs into a coherent platform roadmap and push back when requests do not align with platform direction.
• Communicate technical strategy and tradeoffs to engineering peers and product, business, and executive stakeholders.
• Own the platform end to end, including architecture, standards, delivery SLAs, and data quality guarantees for internal and external consumers.
• Own the team’s build and delivery work and the technical roadmap for adding sources, delivery surfaces, and higher engineering standards.

Benefits

• Meeting-free Friday afternoons, providing more time for focused work and professional development.
• Employee programs that foster community, learning, and growth.
• Comprehensive benefits package.
• Competitive compensation and commitment to fair, transparent pay.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed