Ranking the members of an LLM council

Andrej Karpathy’s llm-council sends one question to several LLMs. The models then review each other’s answers with the authors anonymized, so none can play favorites, and a chairman model writes the final answer. I built on this idea and expanded it into my own version, llm-council-ranking-members, which adds a few things:

• Ranking. Every participant, model or human, is a ranking member. Their rankings, including pairwise or partial ones, are combined with a Plackett-Luce model (via the choix library), which works much like Elo ratings. It estimates a strength for each model and the odds that one answer beats another. That holds up better than counting votes or averaging positions. The chairman then weights each answer by its strength instead of treating all answers equally, so the better answers count for more in the final one.
• A human seat. You can add your own answer in the first stage and rank the anonymized answers, yours included, in the second, so you can compare your judgment with the models'.
• Similarity views. Response embeddings feed a heatmap, a chord diagram and a network graph. They show which models answer alike and which stand apart, so you can tell whether the council really offers different views or just repeats itself, and which members add something new.
• UX polish. Choosing the council and chair in the UI, adding models after seeing the first answers, and light and dark themes.