Share and evaluate stereotypes to help mitigate LLMs’ biases.
April 9 - May 7 | max 1 point
Incentives:
- Every 10 stereotypes validated per team = 0.1 points
- You will have access to the collected data to align your LLM
Guide
This guide explains how to use EDIA’s Stereotypes Validator. This guide contains a basic description of the interface usage, for a better understanding of how to use the tool, it is strongly recommended to watch the explanatory video.
1. Personal Data and Informed Consent
When entering the validator, the first thing you will find is a form where you must fill in your personal data and accept the informed consent to continue with the activity.
The steps are as follows:
- Select the interface language, that is, the language in which the page and general instructions will be displayed.
- Mark the languages you understand (reading and writing). You can select several, keeping in mind that the activity data may appear in any of the languages you select.
- Enter your email address.
- Select one or more nationalities that represent your cultural, personal, or national identity. You can also optionally choose one or more specific regions or write them manually if they don’t appear in the list.
- Finally, read the informed consent about data usage of the EDIA tool and, if you agree, accept it to continue.
2. Using the Stereotypes Validator
Once the first part is completed, access to the main activity is enabled.
This tool has been designed to explore how stereotypes are perceived in different regions of the world. Each time you submit a response, you will receive new data.
In each round, a nationality and an attribute will be shown. Your task is to:
Indicate the degree of association between the nationality and the attribute in your region, using a scale from 1 to 5:
- 1: Strongly disagree.
- 5: Strongly agree.
(Optional) You can write another attribute that you relate to the given nationality. This attribute can be in any of the languages you selected at the beginning. You are encouraged to write in different languages if possible.
- (Optional) You can also add other nationalities or regions that, in your opinion, are associated with the presented attribute.
When you have completed the fields you consider necessary, click the Submit button to receive a new nationality-attribute pair, possibly in one of the languages you marked as known.
If you don’t understand the pair that appears, detect an error in the data, or don’t feel comfortable annotating about that particular pair, you can use the Skip button to move on to the next one.
Organization
Many thanks to the team: Luciana Benotti, Marcos Javier Gómez, Guido Ivetta, Sofía Martinelli, Nair Carolina Mazzeo, Beatriz Busaniche, Emilia Echeveste, and Pietro Palombini. The team prepared this challenge for the SomosNLP 2025 Hackathon with an improved version of the workshop they taught at KHIPU 2025. The original workshop was developed in collaboration with Sunipa Dev, Vinodkumar Prabhakaran, and Fundación Vía Libre.