cycle guide gen ai chatbot

Role Senior Content Designer | Generative AI

Company Flo Health, the leading app in the Health & Fitness category, it is the #1 OB-GYN-recommended app for period and cycle tracking.

Context: The goal is to create a conversational platform that transforms how people who menstruate learn about their bodies and engage with their health. By leveraging the power of user-logged data, Flo can provide hyper-personalized insights and educational experiences, enabling them to gain deeper self-awareness and optimise their health.

Challenge Our initial LLM, while safe, delivered overly cautious and unhelpful responses (e.g., high rejection rates, excessive disclaimers, lack of actionable advice), leading to a poor user experience. Simultaneously, there was a challenge in ensuring our AI's tone of voice (TOV) aligned with Flo's empathetic "doctor friend" brand, avoiding a robotic or clinical feel.

Contribution I spearheaded the development and implementation of a Content usefulness metric, an evaulation framework that I co-built with an ML engineer. The framework is a comprehensive set of metrics and criteria that enables the quantitative assessment of our LLM's responses.

Deliveries In collaboration with my ML engineer, the usefulness metrics were initially manually labeled and ultimately formalized into AI automated judges.

Impact These metrics provide a benchmark and definitive method for measuring improvements on conversational models being built across the company. Long term, my work on the usefulness metric will be formalized into Flo's company-wide "AI Judges Platform," becoming a single source of truth for all AI scoring (legal, medical, TOV, etc.), drastically reducing duplicated effort.