Auto-J 4bits version, see https://github.com/GAIR-NLP/auto-j for more details.
We develop Auto-J, a new open-source generative judge that can effectively evaluate different LLMs on how they align to human preference. It is featured with:
Generality: Auto-J is trained on data from real-world user queries and responses from various LLMs, covering a wide range of 58 real-world scenarios.
Flexibility: Auto-J supports both pairwise response comparison and single-response evaluation by just switching to corresponding prompts.
Interpretability: Auto-J provides detailed natural language critiques that enhance the reliability of its evaluation outcomes and facilitate humans’ involvement in the evaluation loop.
评论