Engineering Expert

Weekday AI

1d • Remote

About The Position

This short-term contractor role involves designing, developing, and validating complex engineering simulation problems used to train and evaluate advanced AI models and coding agents. The work combines engineering expertise, scientific computing, simulation-based design, and AI evaluation. Experts create realistic, multi-constraint problems with measurable objectives and verifiable outcomes, then analyze AI execution logs and refine evaluation benchmarks.

The role spans electrical, mechanical, and aerospace engineering. Work includes building simulation environments with relevant open-source tools, creating Python scripts and automated grading mechanisms, validating physical plausibility and numerical behavior, and investigating model reasoning and tool-use failures. The engagement may last up to 24 weeks and is remote. The expected commitment is approximately 40 hours per week, including at least four hours of daily overlap with US Pacific Time. Weekend on-call availability is required. Compensation is USD 500 per task, with an estimated 30–40 hours of work per task; task allocation and payment terms should be confirmed before engagement.

Requirements

• Master's degree or PhD in Electrical Engineering, Mechanical Engineering, Aerospace Engineering, or a closely related technical discipline.
• At least 10 years of practical engineering design experience.
• Strong Python programming and scripting skills.
• Hands-on proficiency with at least one relevant open-source engineering simulation or design tool, such as ngspice, PySpice, OpenFOAM, FEniCSx, CalculiX, python-control, OpenModelica, CadQuery, build123d, Cantera, or Gmsh.
• Experience developing, executing, debugging, and validating simulation-based engineering workflows.
• Familiarity with modern LLMs, AI coding agents, and AI evaluation methodologies.
• Ability to analyze model execution logs and identify reasoning, planning, and tool-use failures.
• Strong understanding of physical constraints, units, boundary conditions, numerical convergence, and engineering validation.
• Excellent analytical, problem-solving, and technical documentation skills.
• Ability to work independently and deliver technically accurate results within defined timelines.
• Approximately 40 hours per week of availability, with at least four hours of daily overlap with US Pacific Time.
• Weekend on-call availability.
• Personal desktop or laptop and a stable, high-speed internet connection.
• Strong expertise in at least one of Electrical Engineering, Mechanical Engineering, or Aerospace Engineering.

Nice To Haves

• Experience in AI model evaluation, benchmark development, automated grading, or coding-agent assessment.
• Familiarity with repeated-trial evaluation, pass@k metrics, failure taxonomies, and nondeterministic model behavior.
• Experience building automated testing frameworks and reproducible computational environments.
• Knowledge of optimization methods, numerical analysis, simulation automation, and engineering design validation.
• Experience collaborating with multidisciplinary engineering and AI research teams.

Responsibilities

• Develop original, self-contained engineering design problems with realistic technical requirements, competing constraints, and defined optimization objectives.
• Create tasks that require AI models to interpret specifications, assess design trade-offs, configure simulation environments, and iteratively improve engineering solutions.
• Establish validated reference solutions and objective automated grading mechanisms.
• Build, configure, execute, and validate simulation-based problem environments using relevant open-source engineering tools.
• Develop Python scripts, custom test benches, and utilities to automate simulations and verify results.
• Validate simulation outputs for physical plausibility, dimensional consistency, boundary conditions, convergence, and other domain-specific constraints.
• Troubleshoot simulation failures, numerical instability, dependency issues, and computational errors.
• Evaluate LLMs and coding agents across repeated engineering problem-solving trials.
• Analyze execution trajectories, tool interactions, simulator feedback, and intermediate outputs to identify systematic reasoning failures.
• Investigate errors such as specification misinterpretation, premature convergence, invalid geometries, inconsistent units, and physically impossible solutions.
• Apply evaluation concepts including pass@k, failure-mode classification, and analysis of nondeterministic model behavior.
• Calibrate task difficulty using empirical model performance and evaluation results.
• Refine problems to distinguish successful from unsuccessful AI solutions and ensure they are unambiguous, rigorous, reproducible, and objectively gradable.
• Collaborate with AI researchers, technical leads, and subject-matter experts to integrate validated benchmarks into AI evaluation workflows.
• Work independently and deliver technically accurate results within defined timelines.

What This Job Offers

• Career Level: Senior