Principal Engineering Lead — Content & Data Pipeline

Wiley•Hoboken, NJ

3d•$140,000 - $200,733•Hybrid

About The Position

This role leads Wiley's Content & Data Pipeline function, which is responsible for taking scholarly content (journals, articles, books) from various sources through ingestion, processing, and enrichment, and delivering it to internal AI systems, RAG pipelines, and external customers. The role is hybrid, requiring both technical leadership in architecture and hands-on coding, as well as management and growth of a team of engineers. The position has end-to-end ownership of the platform, expanding its capabilities to cover new source types, delivery surfaces (especially internal AI/RAG and licensed external delivery), and higher standards of quality, freshness, and observability. The mandate covers the full content lifecycle, including ingestion, transformation, enrichment, metadata management, rights-aware delivery, and vendor/supplier integration. Key stakeholders include product, business, licensing partners, and vendors, and a significant part of the role involves translating their needs into a prioritized platform roadmap. The ideal candidate will lead by doing and coaching, contributing to code, reviews, unblocking team members, and mentoring, while also owning delivery and stakeholder alignment.

Requirements

Player-coach engineering leadership at scale — you have led an engineering team while remaining genuinely hands-on with architecture and code.
Cross-team influence with product, business, vendor, and executive stakeholders. Able to communicate technical strategy in language each audience actually uses.
Vendor and partner relationship management — technical integration, contract execution, renewal input, escalation.
Honest tradeoff communication — you make scope, quality, and timeline conversations concrete rather than optimistic.
Deep production experience designing and operating content or data pipelines end-to-end — ingestion, processing, delivery — at scale.
AWS at production depth — S3, Lambda, ECS or EKS, SQS / SNS, EventBridge, Step Functions, RDS, DynamoDB. Real IAM and cost fluency.
Python and SQL at production depth.
Workflow orchestration in production — Airflow, Step Functions, Dagster, or equivalent — with real experience in idempotency, backfills, retries, dependency management.
Content transformation experience — XML processing, schema-driven transformation, structured content formats.
Data quality practice: schema validation, defensive parsing, incremental correctness checks, freshness monitoring, alerting on drift.
Distributed systems literacy: idempotency, retries, duplicate handling, partial failure, replay.
Real production experience with API-based and bulk ingestion from external sources — auth, pagination, rate limiting, silent schema changes.
Testing and CI/CD discipline for pipelines — unit, integration, and data-quality tests in CI; safe deployment patterns.
Observability practice — logs, metrics, lineage, and the discipline of knowing whether the pipeline is actually healthy vs. just running.
Active use of AI coding agents in your day-to-day workflow — always with review.

Nice To Haves

Scholarly publishing — journals, articles, books — is our domain and our first preference. But the core skill is building pipelines for content, not just data, and that experience carries over from any content-heavy environment.
You have built or run pipelines that ingest, process, and deliver content in structured formats — XML, HTML, EPUB, PDF — where structure and metadata matter.
You have dealt with content that gets corrected, updated, or pulled after it ships, and built pipelines that handle that cleanly.
You have delivered the same content to different consumers in different shapes — feeds, APIs, search, AI systems.
You have worked with content that has usage rights or access restrictions attached.
You have partnered with editorial, production, or content-operations teams and earned their trust.
Scholarly-specific knowledge — JATS, DOIs, Crossref, retraction handling — is a strong plus. We can teach the acronyms to someone who already has the pipeline instincts.
Scholarly publishing specifics — JATS / BITS XML, DOI infrastructure, Crossref deposits, ORCID / ROR identity, MeSH and similar controlled vocabularies.
BigQuery / GCP experience — useful for the secondary consumer surface, especially as our AI teams standardise there.
Specialised publishing tooling — XSLT at scale, existing content-management systems, or custom XML processing frameworks.
Streaming ingestion — Kafka, Kinesis, Pub/Sub — for near-real-time content updates.
Vector search and embedding pipelines — useful for the internal AI / RAG delivery surface.
Prior partnership with ML, AI, or agent teams as a downstream consumer of your platform.
Data catalog and lineage tooling: DataHub, Amundsen, or equivalent.
Existing relationships in the scholarly publishing ecosystem — vendors, standards bodies, or peer engineering leaders at similar organisations.

Responsibilities

Own end-to-end architecture for the content and data pipeline: ingestion from vendors and public sources, transformation and enrichment, metadata management, and delivery to internal AI / RAG systems and external customers.
Run and operate the pipeline in production and manage the content lifecycle — corrections, retractions, version updates, metadata deposits and registry syndication, and rights and entitlements enforced at every delivery channels.
Set standards for pipeline reliability, data quality, freshness monitoring, testing, observability, and CI/CD across the team's services.
Codify architectural decisions in ADRs; drive convergence on shared patterns across ingestion, processing, and delivery.
Stay hands-on — write code, run architecture reviews, prototype, and unblock the team where needed.
Manage and coach a team of engineers spanning content ingestion, processing, and delivery.
Partner closely with the existing dev lead on the current pipeline; align on ownership boundaries and coordinate roadmap execution.
Own delivery commitments — scope, sequencing, dependencies, and honest tradeoff conversations with stakeholders when reality does not match the plan.
Run engineering practices that scale: code review discipline, on-call rotation, incident response and postmortems, quality gates in CI.
Partner with product and business teams on capability priorities — what content the platform ingests next, what delivery surfaces get built, what quality standards matter.
Own vendor and licensing-partner relationships for content sources and delivery pipes — technical integration, data contracts, escalation, renewal input.
Translate stakeholder needs (editorial ops, product, licensing) into a coherent platform roadmap; push back honestly when requests do not fit the platform's direction.
Communicate technical strategy and tradeoffs upward and outward — not just to peer engineering leaders but to product, business, and executive audiences.