Beyond the prompt box: using LLMs for creating labeling questions

What's Wayground?

Wayground (formerly Quizizz) is an education platform helping teachers drive student outcomes with tools to create content like assessments & lessons, track student understanding, content libraries and more.

Our assessments tool supports around 20+ question types, and our premium plan gives access to premium question types like Labeling, in addition to unlimited assessment creation.

Problem statement

We wanted to improve the creation and consumption experience of premium question types across the platform, so that teachers build trust in our premium offering.

We started with Labeling QT as,

1. It was commonly used by science, biology and social studies teachers (good chunk of our user base).
2. It had the highest drop off rates of all premium question type creation experience.

How Labeling QT works

Labeling is used to ask questions like "label the parts of a flower" or "Annotate the map with landmarks", and students have to drag the appropriate labels onto the blanks on the image.

Using vision models to accelerate question creation

The old creation experience was a WYSIWYG-style editor that mirrored what students would eventually see when the question was answered, but teachers found it quite confusing, something we learnt through hotjar recordings, conversations with teachers and analytics data.

We wanted to reduce the time taken to create a labeling question using LLMs but throwing a prompt box at the teacher didn't feel right, It's 2024 and they were still kinda getting used to prompting.

Also, image generation is expensive and teachers generally have an image in mind before they start creating a labeling question. Can we rather help them select an image and generate questions with labels?

We prototyped various flows before arriving at the following pipeline.

It's simple and effective.

1. Teachers select an image on Wayground.
2. We analyse the image and suggest questions that can be asked on the image
3. The teacher picks a question and we generate labels automatically.

Handling hallucinations and label positioning

Click to read more about this

The questions and labels generated by the model had to be,

1. Diverse, with varying levels of difficulty to capture a wide range of teacher intent.
2. Accurate, with the model generating correct labels for a given image.
3. Positioned correctly on the image (the most difficult part)

LLMs hallucinated a lot when asked to share coordinates of labels directly. I added some abstraction by chopping the image into a 5x5 grid and asking it to share (x,y) position in the grid instead, which was better but still unreliable.

[image of grid breakdown and ]

So for V1 (2024) we simply provided all labels asked the teacher to reposition them. A few months ago we updated the pipeline to use Nano Banana Pro, which gave accurate label positions.

Label creation experience

LLMs aren't perfect, teachers should be able to edit labels and make final adjustments, but our editor was rough. You can't add labels directly on the image, editing was tough and the labels didn't have a pointer, so it often covered the subject.

[bento of label problems]

So we revamped our editor and the label creation experience.

[bento of label creation interactions - click to add, change label orientation, onboarding]

Full flow

[Full flow should show flow diagram and screen studio video]

Impact

The redesign meaningfully improved the creation experience for Labeling.

• Question save rate increased from 48–52% to 75%.
• Average creation time reduced from around 8 mins to 5.5–6 mins.
• AI suggestions were accepted about 1 in 3 times.
• Labeling became a great sales demo of AI being deeply integrated into product workflow.

An in-depth teardown of Labeling is available on request.