Show HN: Open-Source Image Model Leaderboard with Public Preference Data

Show HN: Open-Source Image Model Leaderboard with Public Preference Data(imgsys.org)

4 points by burkaygur 2 years ago | 7 comments

We recently open sourced this public leaderboard and dataset with 50K+ preference data points across 20+ image models. Hoping that this is a useful resource for fine-tuning / preference optimization.

Preference data is updated every day.

Code for frontend: https://github.com/fal-ai/imgsys-public

Preference Data: https://huggingface.co/datasets/fal-ai/imgsys-results

s1k3s 2 years ago |

I hope you're not training models based on the "which one is better" question, because that's incredibly subjective.

treesciencebot 2 years ago | |

we considered which one adheres to the prompt more, which one has overall best aesthetics etc but ended up with a simple which one is overall better type question. it is easier for people to vote and decide one and still applicable as preference data at a larger scope (trading volume for simplicity).

the dataset is open source and we plan to train an aesthetics picker on it but obviously have to do proper evals (with at least 1M data) to come to a reasonable conclusion.

s1k3s 2 years ago | | |

As feedback I can tell you it's not easy at all for me to decide which one is better. Maybe if the prompts would include the art style I would be able to clearly identify the better ones, but they don't. Style is where I see most of the differences.

Disclaimer: I only clicked on Surprise me.