Technology Architect Python Big Data
About The Position
We are seeking a senior Technology Architect / Python Platform Engineer to design and build a containerized API layer that abstracts and governs interactions with Apache Spark through a well-defined API contract. This is a platform-building role focused on creating consistent, secure, and scalable access to Spark-based data pipelines—not simply consuming existing tools. You will develop production-grade APIs in Python, package them as reusable containerized services (Docker/Kubernetes), and operate them as platform capabilities. Strong hands-on experience with GCP, CI/CD automation (especially GitHub Actions), and solid software engineering practices around testing, versioning, and deployment are essential. The role also involves client coordination, delivery ownership, and managing offshore resources while collaborating with onsite teams.
Requirements
- 10+ years of IT experience
- Proven ability to interact and coordinate with clients
- Experience managing offshore teams and collaborating with onsite resources
- Strong software engineering fundamentals (testing, versioning, deployment)
- Python (strong production experience)
- PySpark
- API development & Microservices
- CI/CD (GitHub Actions preferred)
- GCP
- ETL
- Experience building control planes / platform services
- Containerization (Docker / Kubernetes)
Nice To Haves
- FastAPI, ARQ, Kubernetes, Pytest
- Design patterns: modular containerized services, asynchronous processing, job-based work queues, immutability & idempotency, metadata management, policy enforcement, semantic modeling, observability/telemetry
- Functional and behavioral testing
Responsibilities
- Lead delivery coordination and manage complex software application development & support projects
- Design and build a containerized API control plane that securely governs access to Spark-based data pipelines
- Develop production-grade Python APIs and platform services using modern design patterns
- Drive CI/CD automation, testing, versioning, and deployment practices
- Coordinate with clients, work closely with onsite team members, and manage offshore resources
- Ensure observability, policy enforcement, metadata management, and scalable job execution
Benefits
- Flexible work from home options available
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
