Weekday (YC W21)

Compensation

Full Description

Job Description: • Design, refine, and iterate on rubrics, evaluation frameworks, and quality criteria for preference data. • Review, assess, and label data while identifying subtle inconsistencies, quality issues, and opportunities for improvement. • Build, maintain, and optimize pipelines and infrastructure supporting data generation, collection, processing, and evaluation workflows. • Synthesize findings from data analysis and evaluation activities into concise reports, updates, recommendations, and actionable summaries. • Collaborate with researchers and engineers to translate qualitative judgments into structured, repeatable, and scalable processes. • Understand, contribute to, and maintain complex software repositories while ensuring code quality, reliability, and performance. • Identify workflow inefficiencies and implement practical improvements to data quality, evaluation consistency, and engineering processes. • Complete application screening, an AI-based interview, a coding challenge, a technical and cultural interview, and final candidate evaluation. Requirements: • At least 5 years of professional software engineering experience with Python, particularly in building and maintaining pipelines and infrastructure, or with TypeScript, JavaScript, or Go. • Hands-on experience with preference data collection, rubric creation, or developing evaluation criteria. • Strong understanding of software engineering principles, code quality, debugging, and maintainable system design. • Excellent written and verbal communication skills, with the ability to explain complex technical concepts clearly and concisely. • Strong analytical skills and the ability to make consistent, well-reasoned quality judgments. • Demonstrated ability to explain why a particular solution, example, or implementation meets a high quality standard. • Comfort working independently in ambiguous environments while prioritizing execution and timely delivery. • Experience contributing to or navigating large-scale, complex open-source codebases, such as VS Code (preferred). • Experience building data pipelines, developer infrastructure, or automated evaluation systems (preferred). • Background in AI model evaluation, preference data workflows, data annotation, or quality assessment (preferred). • Experience collaborating with AI researchers, machine learning engineers, or multidisciplinary technical teams (preferred). • Education from a leading university or professional experience at a highly regarded technology company (preferred). • LinkedIn profile mandatory and must be included in the candidate's resume. • Must not currently be employed by Microsoft. • Must be able to commit to the required weekly hours and PST overlap for the full contract duration. Benefits: