This repository contains the benchmark dataset of MLR-Bench. We collect 201 tasks from ICLR/NeurIPS/ICML workshops over the past three years. The followings record the metadata of our collection.

Workshop without an official website or deleted

• icml2024_fminwild
• neurips2024_attrib_late
• neurips2024_gsai
• neurips2024_rlfm
• iclr2023_ai4abm
• iclr2023_ml4iot
• iclr2023_mldd
• iclr2023_NeSy_GeMs
• neurips2023_new_in_ml
• neurips2023_ai4mat
• icml2023_esfomo

Non-general workshops

• neurips2024_queerinai
• iclr2024_africanlp
• iclr2023_africanlp
• iclr2023_IndabaX_Rwanda
• icml2023_lxai

Repeated workshops

• iclr2023_dl4c
• iclr2023_mefomo
• iclr2023_re_align and iclr2024_realign (TODO: check the details and delete one)
• icml2024_fminwild potentially repeat with other workshops

Missing workshop for unknown reason

• ICML 2023 The Second Workshop on New Frontiers in Adversarial Machine Learning (https://advml-frontier.github.io/)

Workshop information to be updated

• iclr2025_nfam (too many citations without link, should we keep or delete them? https://nfam.vizhub.ai/)

Workshop Links

• ICML 2024 (33 workshops) https://openreview.net/group?id=ICML.cc/2024/Workshop

  • We keeped 24 workshops
  • Workshops not included
    • without an official website or deleted
      • Agentic Markets Workshop at ICML 2024
      • Workshop on Efficient Systems for Foundation Models II @ ICML2024
    • not included with unknown reason
      • ICML 2024 AI for Science Workshop (website: https://ai4sciencecommunity.github.io/icml24.html)
      • First Workshop on Controllable Video Generation @ICML24 (https://openreview.net/group?id=ICML.cc/2024/Workshop/CVG#tab-recent-activity / https://sites.google.com/view/cvgicml2024/home)
      • ICML 2024 Workshop Differentiable Almost Everything (https://openreview.net/group?id=ICML.cc/2024/Workshop/Differentiable_Almost_Everything#tab-accept / https://differentiable.xyz/)
      • ICML 2024 Workshop on Models of Human Feedback for AI Alignment (https://openreview.net/group?id=ICML.cc/2024/Workshop/MFHAIA#tab-accept-oral / https://sites.google.com/view/mhf-icml2024)
      • ICML 2024 Workshop on Mechanistic Interpretability (https://openreview.net/group?id=ICML.cc/2024/Workshop/MI#tab-accept-oral)
    • repeated page
      • Challenge of ICML 2024 TiFA Workshop
      • MLLM Attack Challenge of ICML 2024 TiFA Workshop
    • Non-general workshops
      • LatinX in AI (LXAI) Research
      • ICML 2024 Joint Workshop Queer in AI and {Dis}Ability in AI
  • Workshop out of nowhere
    • imcl2024_modelofhf??.md

• NeurIPS 2023 Workshop (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop)

  • In total 53 workshops

  • 42 workshops have been kepped

  • Not included

    • not included with unknown reason
      • AI for Accelerated Materials Design - NeurIPS 2023 Workshop (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/AI4Mat#tab-accept-spotlight)
      • Associative Memory & Hopfield Networks in 2023 (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/AMHN#tab-accept-oral)
      • Socially Responsible Language Modelling Research (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/SoLaR#tab-accept-spotlight)
    • without an official website or deleted
      • NeurIPS Workshop on Attributing Model Behavior at Scale
      • NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly
      • Intrinsically-Motivated and Open-Ended Learning Workshop @NeurIPS2023
      • Multi-Agent Security Workshop @ NeurIPS'23
    • Not general
      • 2nd Annual Global South in AI Meetup
      • Muslims in Machine Learning Workshop @ NeurIPS2023
      • New in Machine Learning Workshop, NeurIPS 2023
      • NeurIPS 2023 Queer in AI Workshop

  • Things to notice

    • neurips2023_mlncp.md has no topics -> maybe too general for the LLMs to generate ideas?
    • neurips2023_want.md has been repeated but not deleted

• NeurIPS 2024 Workshop

  • 54 workshops, 51 are included
  • delete
    • 3rd Annual Global South in AI Meetup 2024
    • Latinx in AI @ NeurIPS 2024
    • Muslims in ML Workshop co-located with NeurIPS 2024
  • No website or delted
    • Workshop on Generalization and Safety from Reinforcement Learning to Foundation Models
  • Things to notice: without an official website or deleted
    • NeurIPS 2024 Workshop Machine Learning with new Compute Paradigms (still with an abstract here)
    • NeurIPS 2024 Third Table Representation Learning Workshop
    • UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models