Senior Engineer
About The Position
Minimum 7 Years of experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL. Experience in design, build and deployment of Python based applications. Experience in writing complex SQL queries/procedures using Relational databases like SQL Server or Oracle Experience in version control system like Git and CI/CD pipeline is a must. Experience in Delta lake APIs is a plus Experience in Docker and Kubernetes is a plus Knowledge of AWS services like S3, Athena, Glue, Lambda, Redshift or Cloud platform is a plus PySpark, Delta Lake, Shell scripting, AWS Services (like Glue, Lambda, Athena, Step functions, Event Bridge, EC2, Batch, MSK etc.), PostgreSQL, RedShift, Dynamo DB, Jenkins, Udeploy, GitHub
Requirements
- Minimum 7 Years of experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
- Experience in design, build and deployment of Python based applications.
- Experience in writing complex SQL queries/procedures using Relational databases like SQL Server or Oracle
- Experience in version control system like Git and CI/CD pipeline is a must.
Nice To Haves
- Experience in Delta lake APIs is a plus
- Experience in Docker and Kubernetes is a plus
- Knowledge of AWS services like S3, Athena, Glue, Lambda, Redshift or Cloud platform is a plus
Responsibilities
- Designing, implementing, and maintaining jobs written in PySpark for extracting raw data, applying transformation, and writing into file/tables on Apache Spark cluster.
- Writing complex queries as per the business rules and data extraction logic
- Query tuning and Performance optimization of various components as and when required.
- Work with development leads, System Architects and other teams (as needed) to manage dependencies, risks and issues.
- Contributing in all phases of the development lifecycle
- Support testers and defect resolution
- Update status/risks/issues/impediments in daily scrum calls to the customer and track it to closure.
- Writing testable, scalable and efficient code
- Working on the necessary documentations as needed
Benefits
- Discretionary Annual Incentive.
- Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
- Family Support: Maternal & Parental Leaves.
- Insurance Options: Auto & Home Insurance, Identity Theft Protection.
- Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
- Time Off: Vacation, Time Off, Sick Leave & Holidays.
- Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Industry
Professional, Scientific, and Technical Services
Education Level
No Education Listed
Number of Employees
5,001-10,000 employees
