Data Engineer #00246

Virginia Information Technologies Agency•Richmond, VA

2d•$100,000 - $125,000•Onsite

About The Position

The Library of Virginia seeks a versatile and motivated Data Engineer to help reimagine, transform, and administer an improved data platform and information infrastructure, from ingest through access. This redesigned role will have the opportunity to collaboratively transform data practices at a 200-year-old cultural heritage organization and state agency with an eye towards the future. The work will create efficiencies; improve staff information gathering, efficiency, and productivity; and enhance the Library’s data storage capacity. It will also support the development of applications that allow LVA staff and users to contribute to and interact with data and databases. The incumbent will manage security, optimize performance, and guide the evolution of legacy systems to meet changing and more complex modern requirements. They will perform database administration on new and existing systems across various platforms from design through performance tuning, data security, backup/recovery, and user management. The ideal candidate will have broad, intermediate-to-advanced data engineering / database administration knowledge, with a strong background in data architecture and planning. They will have experience and comfort both leading and implementing technological change, stewarding projects from inception to completion. The incumbent will have a proven record in data pipeline processes and integration tools to collect, process, store, and deliver data from multiple formats in organized, accessible, user-friendly ways. They will have strong troubleshooting skills, the ability and desire to learn new technologies quickly, and experience collaborating with non-technical staff to translate their needs into actionable technical tasks. They will be comfortable supporting disparate applications and systems—multitasking on various initiatives— and will show a demonstrated ability to work collaboratively within a team.

Requirements

Strong experience with data architecture and planning.
Experience with data analysis with understanding of data pipeline processes.
Ability to quickly evaluate and understand legacy systems and unknown technologies.
Experience using structured, semi-structured and unstructured data together for analysis and presentation.
Experience tracking and managing changes through version control for enhanced collaboration, recovery, project management and traceability. Git knowledge a plus.
Experience with MS SQL, MySQL.
Understanding of MS Access as a front-end interface.
Proficiency in languages such as Python and familiarity with tools and frameworks like Apache Spark.
Experience with integration tools.
Familiarity with Microsoft data management suite (Purview, Azure, Dynamics, etc).
Experience with performance tuning.
Ability to recover lost or inaccessible data due to accidental deletion, file corruption, HW failure or malicious intent, restoring data integrity and ensuring usability.
Working knowledge of multimodal RDBMS.
Understanding of cloud platforms and storage methodologies.
Strong technical leadership and guidance.
Ability to formulate and explain a long-term tech stack goal.
Considerable general database and application software knowledge.
Software installation and troubleshooting ability.
Strong ability to work independently and resolve problems of a complex nature.
Customer service experience collaborating with internal and external stakeholders, including prioritization, balancing user needs with technical realities, managing conflicting user goals and translating technical concepts for non-technical audiences.
Ability to manage time and multiple priorities effectively.
Ability to communicate clearly both orally and in writing with non-technical users.
Considerable experience in data engineering or a comparable role.
Any applicable licensing or certifications.
In-depth knowledge of SQL.
Extensive experience with database technologies and architecture.
Practical knowledge of best practices in data engineering and data security.
Strong organizational skills and attention to detail.
Exceptional problem-solving and critical thinking skills.
Excellent collaboration and communication skills.
Demonstrated expertise in data engineering
Demonstrated expertise in data architecture
Demonstrated expertise in data planning
Demonstrated expertise in data pipeline processes
Demonstrated expertise in data security
Demonstrated expertise in SQL (particularly MS SQL, MySQL, or both)
Demonstrated expertise in python
Demonstrated experience using structured, semi-structured and unstructured data together for analysis and presentation
Demonstrated experience using version control for enhanced collaboration, recovery, project management and traceability
Demonstrated experience using data processing tools such as (but not necessarily) Apache Spark
Demonstrated experience using data integration tools such (but not necessarily) Azure Data Factory
Demonstrated experience using cloud platforms and storage methodologies
Demonstrated ability to quickly evaluate and understand legacy systems and unknown technologies
Demonstrated ability to recover lost or inaccessible data due to accidental deletion, file corruption, HW failure, or malicious intent
Demonstrated ability to formulate, explain, and follow through on implementing long-term tech stack goals
Demonstrated ability to work and resolve complex problems independently
Excellent writing, communication, and interpersonal skills required

Nice To Haves

Experience working in cultural heritage, higher education, research (especially humanities), or similar sectors is a plus.
Familiarity or expertise with the following are a plus: o PostgreSQL o Oracle o Data performance tuning o Microsoft data management tools
Some familiarity with MS Access as a front-end interface
Some familiarity with Microsoft data management suite (Purview, Azure, Dynamics, etc.)
Some familiarity with Multimodal RDBMS

Responsibilities

Perform initial and ongoing evaluations of existing data and database infrastructure with an eye towards modernization, sustainability, and customer service. Implement recommended changes as necessary.
Ingest, process, transform, store and securely and accessibly deliver data for information sharing and decision-making.
Analyze, design, develop, implement and maintain custom databases and other systems that collect, store, and process data efficiently.
Migrate and combine large datasets from multiple formats/locations.
Monitor databases and related systems to ensure optimized performance, data quality and security.
Implement processes to validate data, ensuring accuracy and reliability.
Document processes related to database design, configuration, and performance.
Work closely with project managers, stakeholders and other technical staff to understand data requirements and ensure that the data infrastructure meets those needs.
Review and analyze staff work procedures to identify strategies for automation.
Envision, design and implement new systems to meet immediate and long-term needs of Library stakeholders.
Serve as a database administrator, at present primarily MS SQL but with the possibility to change based on assessments.
Assist with data conversion efforts, which includes working with developers to convert legacy access applications into a modern web format and working with external government entities to ingest custodial record transfers.
Manage database upgrades for all SQL based systems (MS SQL 2016/2019/2022 [most prevalent], MySQL, PostgreSQL 15), as well as supporting system administration.
Maintain and ultimately convert legacy Microsoft Access applications.
Determine security measures to be taken to protect data integrity. This includes vulnerability remediation and working with internal/external auditors to ensure compliance with Virginia security standards.
Adhere to best practices in securely backing up and archiving data.
Work with ISO and system admins to ensure adequate data backups are made.
Keep abreast of developments and best practices in data engineering.