Data Engineer

Barnes & Thornburg LLP•Indianapolis, IN

About The Position

Summary: The Data Engineer will focus on backend data engineering tasks, developing and maintaining extract, transform, load (ETL) processes to integrate data from various sources into the firm’s data systems. This role will work extensively with Microsoft Fabric and collaborate with data architects, other engineers, and analysts to ensure data pipelines are efficient and reliable. The Data Engineer will also support teams delivering data dashboards and reporting, ensuring the provision of necessary information to the organization. This role is essential for developing and maintaining our overall data taxonomy and ensuring its consistency across multiple systems and platforms. Key Responsibilities: Design, construct, and maintain data pipelines (ETL processes) to integrate large volumes of legal data efficiently and securely from various sources. Develop data lake and data warehouse solutions, implementing best practices for data ingestion, storage, and retrieval. Establish and deploy comprehensive data governance and security frameworks including data access controls, compliance measures, and data classification. Collaborate with cross-functional teams, including analysts, attorneys, and IT professionals, to understand data requirements and design backend solutions that meet their needs. Assess and prioritize data initiatives based on business objectives, best practices, and partner/practice need to ensure alignment with strategic goals. Design and implement testing strategies for data pipelines, validating data integrity, accuracy, and performance throughout the workflow. Create and maintain thorough documentation for data architectures, pipelines, and transformations, and processes to ensure transparency and knowledge sharing. Build and optimize automated data workflows to streamline ingestion, transformation, and processing, reducing manual effort and improving efficiency. Optimize data storage and retrieval processes to improve performance and scalability, leveraging cloud-based technology such as Azure. Participate in and contribute to data quality enhancement planning and implementation for new projects. Stay current with the latest industry trends and best practices in data engineering and analytics, continuously evaluating and implementing new tools and techniques to enhance our data infrastructure. Provide technical support and troubleshooting assistance for data-related issues, working proactively to identify and resolve potential problems. Collaborate with cybersecurity and compliance teams to ensure data security and regulatory compliance.

Requirements

Bachelor’s degree in Computer Science, Engineering, or a technology-related field.
7+ years of related work experience, preferably in a legal or regulatory environment.
Proven experience leading data engineering projects, including requirements gathering, architecture design, development, testing, and deployment.
Experience with Microsoft Fabric or a combination of Azure Synapse Analytics, ADLS Gen 2, Synapse SQL Pools, Azure Data Factory (ADF) and/or other Azure data platforms.
Experience with the Microsoft Power Platform, such as Power BI, Power Automate, Power Apps, or other similar tools.
Strong SQL Experience including complex queries, stored procedures, SSIS, SSRS
Experience building data pipelines using Python/PySpark
Proven experience designing and implementing data architectures, including system requirements, technology selection, and governance.
Expertise in end-to-end multi-layered data architectures (e.g., bronze/silver/gold), managing data from raw ingestion to business-ready datasets while ensuring quality, lineage, and documentation.
Strong data modeling skills, including conceptual, logical, and physical models for optimized storage and retrieval.
Proven ability to test data pipelines for accuracy, consistency, and performance, including unit and integration testing.

Nice To Haves

Experience in migrating data from on-premises databases and servers to Azure Cloud environments is preferred.
Experience with version control; knowledge and experience with GitHub is a plus.
Ability to present technical ideas in business-friendly and user-friendly language.

Responsibilities

Design, construct, and maintain data pipelines (ETL processes) to integrate large volumes of legal data efficiently and securely from various sources.
Develop data lake and data warehouse solutions, implementing best practices for data ingestion, storage, and retrieval.
Establish and deploy comprehensive data governance and security frameworks including data access controls, compliance measures, and data classification.
Collaborate with cross-functional teams, including analysts, attorneys, and IT professionals, to understand data requirements and design backend solutions that meet their needs.
Assess and prioritize data initiatives based on business objectives, best practices, and partner/practice need to ensure alignment with strategic goals.
Design and implement testing strategies for data pipelines, validating data integrity, accuracy, and performance throughout the workflow.
Create and maintain thorough documentation for data architectures, pipelines, and transformations, and processes to ensure transparency and knowledge sharing.
Build and optimize automated data workflows to streamline ingestion, transformation, and processing, reducing manual effort and improving efficiency.
Optimize data storage and retrieval processes to improve performance and scalability, leveraging cloud-based technology such as Azure.
Participate in and contribute to data quality enhancement planning and implementation for new projects.
Stay current with the latest industry trends and best practices in data engineering and analytics, continuously evaluating and implementing new tools and techniques to enhance our data infrastructure.
Provide technical support and troubleshooting assistance for data-related issues, working proactively to identify and resolve potential problems.
Collaborate with cybersecurity and compliance teams to ensure data security and regulatory compliance.