Head Of Product – Model Evaluation

Job Description

Applause is raising the bar for digital quality and employee experience. Recognized as a Top Workplace, Applause provides award-winning software testing and UX research solutions to top brands. Our fully managed services leverage a global team and the world’s largest independent testing community. We improve digital experiences for global innovators like Google, Microsoft, PayPal, Starbucks, Vodafone, and BMW.

The Role

As Head of Product – Model Evaluation at Applause, you will lead the development of a strategic new AI evaluation platform—and play a foundational role in bringing it to market from the ground up. You will define the product vision, identify target customer personas, and shape the go-to-market strategy, while building the systems that measure, monitor, and improve AI models in production. As the business matures, you will act as the product leader. This position reports directly to the CTO.

This is a rare opportunity to combine 0→1 product building with enterprise-scale impact. You’ll create a new category for Applause—extending our leadership in digital quality into AI—by developing capabilities such as LLM-as-a-judge systems, human-in-the-loop feedback pipelines, and model observability frameworks. You won’t just refine an existing product; you’ll define what this business becomes and how it wins in the market and then build the team that scales it.

Key Responsibilities:

·       Define the vision, positioning, and roadmap for Applause’s AI evaluation offering

·       Identify and develop target customer personas (e.g., AI platform teams, ML leaders, product orgs)

·       Design and execute the go-to-market strategy, including packaging, pricing, and initial sales motions

·       Partner with sales and marketing to validate demand, refine messaging, and drive early adoption

·       Translate market feedback into rapid product iteration and differentiation

LLM Evaluation Systems (in partnership with ML/DS)

·       Partner with data science and ML teams to design closed loop systems:  model → evaluation → insight → improvement

·       Bring LLM-as-a-judge systems into production use for grading, ranking, and preference modeling

·       Partner with ML teams on iteration strategies (prompting, fine-tuning, data collection)

·       Ensure evaluation outputs translate into actionable improvements for customers

Human-in-the-Loop Feedback Systems (in partnership with Community Ops)

·       Leverage Applause’s global testing community to design scalable human evaluation pipelines

·       Define workflows, annotation schemas, and quality controls to produce gold data sets

·       Balance quality, cost, and latency to meet customer requirements

Cross-Functional Leadership

·       Act as the bridge between product, data science, engineering, and go-to-market teams

·       Translate complex technical capabilities into clear, differentiated product offerings

·       Drive alignment on priorities, success metrics, and execution plans

Job Requirements and Preferred Skills:

Required

·       5+ years of product management experience, including work on AI/ML-driven or data products

·       Experience owning or contributing to 0→1 product development and go-to-market strategy

·       Demonstrated ability to take a product from concept to paying customers

·       Strong ability to define customer personas, value propositions, and product positioning

·       Experience working closely with data science or machine learning teams

·       Comfortable engaging at a technical level with ML engineers. You can discuss model architectures, evaluation metrics, and API design without needing a translator

·       Strong understanding of metrics, experimentation, and data quality

·       Experience with B2B/enterprise products and technical buyer personas

Preferred

·       Background in data-science, machine learning, or engineering

·       MBA or equivalent experience

·       Experience with model evaluation systems, including human and/or automated approaches

·       Familiarity with LLM-as-a-judge, pairwise ranking, or preference modeling

·       Understanding of the limitations and tradeoffs in evaluating generative AI systems

·       Experience at AI evaluation/tooling companies (eg Arize, Langsmith, Galileo, Scale AI, Weights & Biases, or Humanloop) or as a buyer/implementer of such tools at an enterprise

·       MBA or equivalent experience in high growth, product led organizations

Why Applause?

You’ll help build a new business line from the ground up — with the backing of an established company that already has enterprise relationships with the world’s biggest brands. Here’s what makes this opportunity unique:

·       Built-in moat: Applause’s global testing community gives you a human-in-the-loop evaluation infrastructure that no AI startup can replicate. You’re not starting from zero — you’re starting from a million evaluators.

·       CTO sponsorship: This role reports directly to the CTO with a clear mandate and executive air cover to build fast.

·       Enterprise access: Applause already sells to the Fortune 500. You’ll have warm introductions, existing contracts, and credibility that would take a startup years to build.

·       Category creation: You’re not optimizing an existing product. You’re defining a new business for a company that’s ready to invest in it.

Additional benefits:

·       Flexible work environment with top talent from across the globe.

·       International team of 450+ passionate, talented co-workers.

·       Hands-on projects providing exposure to well-known, global brands.

 

Applause Core Values:

As a global employee community, we strive to uphold the following core values, which are critical to business success and how we measure individual and team performance. Do you share our core values?

·       Be Accountable: You love to take ownership, and hold yourself and others accountable to increase empowerment and success.

·       Celebrate Authenticity: You love bringing your true self to work and creating genuine and trustful relationships within a diverse environment.

·       In It Together: You have a team-first mindset and love collaborating with your peers.

·       Create Value for Our Customers: You love delivering meaningful business impact and being a release partner for all aspects of digital quality.

·       Crush Your Goals: You always strive for excellence and constantly seek ways to be better, more effective and more efficient. 

Accommodations:

Applause is a place where everyone belongs and where we believe everyone deserves the exceptional. We continue to celebrate diversity and are committed to creating an inclusive, equitable environment for our employees. If you believe you require a reasonable accommodation und.er any of the legally protected characteristics, please click here to complete an accommodation request. Please note, Applause will only review requests for applications that have been submitted. We will review your qualifications and follow up with you regarding your request if your qualifications meet our current needs.

 

#LI-OB1