Data Engineer
About The Position
The Commonwealth of Pennsylvania is seeking an experienced Data Engineer & Insights Analyst to support the Pennsylvania Longitudinal Data System (PALDS). The selected candidate will design and maintain data pipelines, integrate and transform complex datasets, improve data quality, and develop analytics solutions that support program delivery, operational improvements, and executive decision-making across state government agencies. The role calls for technical expertise in AWS, SQL, programming, big data technologies, data modeling, and data integration, along with the ability to work with large, complex, and confidential datasets. The candidate will also support data conversions related to system modernization and replacement initiatives.
Requirements
- At least 5 years of hands-on experience with AWS cloud services, including S3, EC2, Glue, EMR, RDS, Redshift, Athena, Lambda, and DynamoDB.
- At least 5 years of programming or scripting experience with Python, R, Java, C++, or Scala.
- At least 5 years of experience with SQL and relational or NoSQL databases, including Oracle, SQL Server, PostgreSQL, and Cassandra.
- At least 5 years of experience with Git, version control, and CI/CD pipelines.
- Strong knowledge of data modeling, including dimensional modeling, Data Vault, and star and snowflake schemas.
- Experience with AWS Glue Data Catalog, metadata management, and data cataloging.
- Understanding of data governance, data lineage, and data quality frameworks.
- Experience designing and integrating REST APIs.
- Hands-on experience with big data technologies such as Hadoop, Spark, and Kafka.
- Experience with data pipeline orchestration tools such as Apache Airflow, Azkaban, or Luigi.
- Experience with data integration platforms such as Informatica, MuleSoft, Talend, or TIBCO.
- Experience with stream-processing technologies such as Storm or Spark Streaming.
- Advanced SQL knowledge and experience working with large, complex datasets.
- Strong analytical, problem-solving, project management, and organizational skills.
- Experience supporting cross-functional teams in a dynamic environment.
Responsibilities
- Design, build, and maintain scalable data pipeline architectures.
- Develop cloud-based data solutions using AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.
- Build and maintain data orchestration workflows using Apache Airflow, AWS Step Functions, or similar tools.
- Extract, transform, integrate, and load data from multiple sources using SQL and big data technologies.
- Develop and optimize data environments supporting statistical analysis, predictive modeling, machine learning, and research.
- Collaborate with data scientists, research analysts, and policy analysts to support reproducible research and data-driven decision-making.
- Identify and resolve data quality issues through validation, monitoring, and root-cause analysis.
- Automate manual processes and improve data delivery, infrastructure scalability, and operational efficiency.
- Develop dashboards, reports, analytical models, and decision-support tools.
- Build monitoring, alerting, and logging frameworks for data pipelines.
- Support data conversions associated with system modernization and replacement projects.
- Ensure compliance with data governance, privacy, security, and Commonwealth policies.
- Collaborate with executive, data, design, and support teams to address technical issues and improve data systems.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Career Level
Senior
Education Level
No Education Listed
