Pelicans on a bicycle

For more on this benchmark check out my talk The last six months in LLMs, illustrated by pelicans on bicycles or review the pelican-riding-a-bicycle tag on my blog.

Prompt:

Generate an SVG of a pelican riding a bicycle

Cerebras Llama models

Cerebras Llama 3.1 70B

Cerebras Llama 3.1 8B

Claude models

Claude 3.5 Sonnet (2024-06-20)

Claude 3.5 Sonnet (2024-10-22)

Claude 3 Haiku

Claude 3 Opus

Gemini models

Gemini 1.5 Flash 001

Gemini 1.5 Flash 002

Gemini 1.5 Flash 8B 001

Gemini 1.5 Pro 001

Gemini 1.5 Pro 002

gemini-exp-1114

gemini-exp-1121

gemini-exp-1206

OpenAI models

GPT-3.5 Turbo

GPT-4o mini

GPT-4o

o1-mini

o1-preview

AWS models

Amazon Nova Micro

Amazon Nova Lite

Amazon Nova Pro