The Technology Innovation Institute (TII), the applied research pillar of Abu Dhabi’s Advanced Technology Research Council (ATRC), has launched The CrowdLabel Challenge: Crowdsourcing LLM Feedback, an initiative focused on building new innovative methods for passive human labelling.
Large language models (LLMs) utilise Reinforcement Learning from Human Feedback (RLHF) to align intelligent agents with human preferences; however, employing human labellers can be expensive. The goal of this challenge is to identify a passive human labelling approach where the act of labelling a data sample is seamlessly integrated into another process or operation, allowing
the human operator to perform the task without consciously realising they are labelling data. Passive alignment methods can lower costs and be built into other tasks or daily activities like bot detectors.
TII has created an alignment samples database for use by LLM experts and researchers to give scalable and reliable feedback to their models. In this challenge, TII is looking for approaches to present these alignment datasets to broad and diverse audiences to label hundreds of thousands of data samples at scale and at a low cost.
The proposed solutions will be competitive if (i) they can show adaptability to multiple languages, but must be feasible in at least two languages, and (ii) have multimodal content of varying label types, (iii) on varying topics. The submissions will be evaluated based on the novelty of the approach, cost-effectiveness, and quantity and quality of labels crowdsourced during an experimental launch. TII welcomes individuals, teams, startups, research institutes, and university students from around the world to propose and develop passive human labelling methods. These methods should enable human users to annotate data samples seamlessly, without necessarily realising they are completing a labelling task.
The total award pool is $50,000, including $30,000 prize for first place. Submissions to the Developing Innovative Passive Human Labeling Methods for LLM Alignment Challenge must be received by February 28th, 2025.
For more information, please visit https://tii.community.wazoku.com/challenge/d960068fccf7422aaf2824d3e120b289
Discussion about this post