AI Auditing Engineer - Penn Center for Media, Technology, and Democracy - Penn Engineering
About The Position
The Penn Center for Media, Technology, and Democracy (Penn MEDIATED) works to advance the scientific understanding of the information ecosystem and leverage that research to strengthen the foundations of democracy. Penn MEDIATED seeks a part-time AI Auditing Engineer to develop and productionize auditing and monitoring of large language models (LLMs). The AI Auditing Engineer will unify existing research prototypes from several codebases in different academic projects to create an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs. The tooling will be used to automatically and longitudinally monitor rates of refusal or soft censorship, political lean, sourcing information, and other features of LLM responses on political topics. We seek an AI Auditing Engineer experienced in developing efficient and scalable data pipelines in Python, designing and managing NoSQL databases, working with cloud computing platforms, and in working with LLM APIs. Experience with and knowledge with natural language processing methods for LLM evaluations is also beneficial. The AI Auditing Engineer will work closely with Danaé Metaxa, Raj & Neera Singh Assistant Professor in the Department of Computer and Information Sciences. The position will be jointly supervised by Danaé Metaxa and Alex Engler, Executive Director of the Penn Center on Media, Technology, and Democracy. This role can be fully remote or hybrid. This is a temporary role, and may be either part-time or full-time, depending on the applicants, with the expected term during summer 2026 at between 20-40 hours per week.
Requirements
- Bachelor’s degree in a relevant field of study
- Required proficiency in developing efficient and scalable data pipelines in Python.
- Required proficiency in designing and managing NoSQL databases.
- Required proficiency in working with cloud computing platforms.
Nice To Haves
- Preferred understanding or prior experience working with LLM APIs.
- Preferred understanding or experience with natural language processing methods for LLM evaluation.
- Preferred experience with data visualization in interactive dashboards.
- Interest in the political outputs and impacts of large language models (LLMs) is a bonus.
Responsibilities
- Develop and deploy an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs.
- Design and manage appropriate databases for LLM monitoring.
- Advance research methods for LLM monitoring of political topics.
Benefits
- Penn offers comprehensive medical, prescription, behavioral health, dental, vision, and life insurance benefits to protect you and your family’s health and welfare.
- Penn offers generous retirement plans to help you save for your future.
- Penn’s Basic, Matching, and Supplemental retirement plans allow you to save for retirement on a pre-tax or Roth basis.
- Choose from a wide variety of investment options through TIAA and Vanguard.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Part-time
Career Level
Entry Level
Number of Employees
11-50 employees
