Careerflow.ai

You would manage end-to-end software-engineering data and evaluation programs, including contributor operations, technical review, quality, costs, timelines, and delivery recovery. | The role requires direct experience running production data programs for frontier AI models, managing distributed contributors and reviewers, and independently owning program quality, throughput, costs, and recovery. | Careerflow Human Data Labs partners with AI companies to bring real-world professional expertise into their products. About the role: We are looking for a Strategic Project Lead with direct experience operating software-engineering data programs for frontier AI models. You will own production across coding data, RLHF or preference data, SWE benchmarks, model evaluations, agentic coding, code generation, and code repair. This includes contributor calibration, throughput, quality, rework, cost, timelines, and delivery recovery. The role requires strong technical judgment. You should be able to read code and tests, inspect final artifacts, identify invalid work, and make release or rejection decisions with minimal training. WHAT DOES DAY-TO-DAY LOOK LIKE? * Operate active software-engineering data and evaluation programs end to end. * Manage contributor sourcing, calibration, onboarding, reviewer capacity, and performance. * Track throughput, quality, rework, defects, cost, and delivery timelines. * Review code, tests, task metadata, and final artifacts for technical correctness. * Diagnose quality failures and improve task design, rubrics, instructions, and review processes. * Lead recovery plans when programs fall behind on quality, throughput, cost, or timelines. * Coordinate distributed software engineers, reviewers, and team leads. * Communicate progress, risks, and corrective actions to customer research and program teams. * Use Python, SQL, or similar tools for quality analysis, defect tracking, and operational reporting. REQUIREMENTS * Direct experience operating production software-engineering data programs for frontier AI models, preferably involving coding training, coding RLHF, SWE benchmarks, model evaluations, agentic coding, code generation, or code repair. * Proven ownership of the full production system, including contributor sourcing and calibration, task throughput, quality, rework, cost management, and delivery recovery. * Strong software-engineering judgment, with the ability to read code and tests, inspect final artifacts, identify invalid technical work, and make release or rejection decisions. * Experience managing distributed software-engineering contributors, reviewers, and team leads while maintaining consistent quality and delivery performance. * Ability to take over an active program quickly, diagnose operational problems independently, and communicate clear decisions and recovery plans to demanding customers. FREELANCING * Work remotely with leading global technology companies and AI labs. * Flexible engagement options based on project requirements and availability. * Opportunity to work on high-impact AI training, evaluation, and data-generation programs. * Collaborate with experienced technical, research, operations, and delivery teams. * Build expertise at the intersection of software engineering, AI, and large-scale operations. OFFER DETAILS: * Commitments Required: 8 hours per day with at least 4 hour overlap with PST.

You would allocate resources across an AI training program, monitor utilization and operational risks, improve tracking processes, and manage Supply Coordinators. | The role seeks large-scale operations experience, preferably in AI training, plus hands-on solutions architecture, systems integration, and workforce or resource management systems experience. | Careerflow Human Data Labs partners with AI companies to bring real-world professional expertise into their products. About the role Own resource planning and deployment for a large-scale AI training program, managing 1,800+ contributors across multiple projects. You will align capacity with client demand, oversee resource tracking systems, identify supply risks, and lead a team of Supply Coordinators to ensure smooth 24/7 operations. Key Responsibilities * Make data-driven resource allocation and deployment decisions. * Forecast capacity gaps and guide recruiting and fulfillment teams. * Manage and improve centralized resource and throughput tracking systems. * Integrate data from recruiting, onboarding, performance, and project delivery. * Monitor utilization, throughput, and operational trends. * Identify bottlenecks and lead process improvements. * Collaborate with clients, Project Leads, PMO, and cross-functional teams. * Lead 1–3 Supply Coordinators and ensure operating coverage. Required Qualifications * Experience in large-scale operations or managed services, preferably AI training. * Hands-on experience with solutions architecture, systems integration, and workforce or resource management systems. * Strong analytical, decision-making, and communication skills. * Experience working with technical and business stakeholders. * Ability to manage fast-moving operations and work US Pacific Time hours. * AI training industry experience is preferred. Success Measures * Accurate and timely resource deployment. * Reduced turnaround time and capacity gaps. * Improved resource utilization and cross-skilling. * Strong stakeholder alignment and operational efficiency. Offer Details: 1. This is a project-based role involving a defined number of evaluation tasks. 2. Each task will involve comparing multiple chatbots and providing a final assessment. 3. Work Schedule: Five working days per week, including weekends as required, with two weekdays off. 4. Work Hours - 7am-4pm PST 5. Duration: 16 weeks.