LIMA ERP data (LimaRP)

Following the principles highlighted in arXiv:2305.11206 by Zhou et al. and replicated in some aspects by Kaiokendev with SuperHOT, the archive in this repository contains about 2000 manually selected and curated 1-on-1 human-human roleplaying conversations and associated LLM-generated persona and scenario data. The RP conversations all feature only two human participants, although occasionally the participants may play the role of more than one character.

The conversation data is in the form of source files in .yaml format + basic Python script for building the dataset, intended to be finetuned in "completion" format (similar to unsupervised finetuning).

Having reached the minimum number of examples suggested in the LIMA paper and after putting overall probably more than 500 hours of work on manually gathering and curating the data, LimaRP can be considered a finished project at this point in time. Future work (cleaning, trimming, expansion) would require more resources and community help.

Notes

• Be aware that although retrieved solely from age-restricted (18+) internet forums, the data contains roleplaying elements and topics that may be considered extreme, distasteful, shocking, inappropriate and disturbing. Do not download it if you're not sure of the legal ramifications of possessing fictional written content of any kind in your country.
• The first ~500 samples were designed to be trained with a 2048 tokens context size; the following 500 with a 4096 tokens context size or greater. The later training samples (data-long) were designed for an 8192 tokens context size. Note that while the 8k samples can be reduced to 4k size, this can confuse the model to some extent, as scenario and persona data may end up referring to events removed from the context.
• Please note that no support will be provided for the dataset and building script here on HuggingFace.
• A persona–scenario summarizer/generator made with LimaRP data also exists.
• 🔑 The archive is password-protected. The password is LimaRP

Known issues

LimaRP has a few notable issues, here in subjective decreasing order of severity.

• Grammar and typos. Although care has been put to reduce the amount of typos and grammatical errors, they are still present to some extent (punctuation issues in particular). Automated AI-based grammar checking with language models like CoEdit could be performed, but the results would then have to be manually validated since these models often tend to correct more than necessary, which can be undesirable in dialogues, as well as being avoidable manual work. Some data sources (threads) show a larger amount of grammatical issues than others, and for those this could be an acceptable tradeoff if they're worth saving.
• Dullness. Overall the conversations may feel too polite or even dull in some aspects. This might be due to various reasons, but the main one is probably that most are from generally moderately well-written "vanilla" ERP where people try to be respectful with each other. More noncon and/or extreme content may be needed to reduce the general "politeness" of the conversational data, spice it up.
• Compiling errors. While the provided script performs a certain amount of validation checks, there may still be instances where due to human error utterances have been assigned the wrong label, or placeholder names been assigned to the wrong character. The former issue is more likely to have happened in the first (4k-context) ~1000 training samples (data-short). The data needs to be carefully checked to make sure that no issue in this regard exists.
• Repetitive and inaccurate descriptions. While conversations are almost entirely human-generated, character information and scenario exhibit gpt-4-isms and can be repetitive, lack depth and miss certain character traits; manual editing will be needed to make them more human-like and respond to more specialized personality traits and keywords—as a result, LimaRP-generated text may appear to ignore certain character traits. A more powerful personality summarizer capable of being both accurate while generating sufficiently long descriptions could be conceived for solving this issue.
• Lack of instructions. No instruction data whatsoever is present in the dataset. While the initial plan was only making it focused on conversations, in retrospect a minimal amount of instruction-oriented roleplay data could be beneficial in making the dataset able to better stand up on its own feet, without the need for merging the data with smarter models or mixing it with external instruction datasets.
• Name biases. Character names may need to be diversified to remove potentially undesirable bias. In other words, certain names may have ended up getting associated with certain personalities since they have been used more frequently than others.
• Lack of diversity. In general, more focus needs to be put on improving conversation diversity. The total number of conversations may have been excessively padded up, as several long conversations that couldn't fit within the 4k/8k tokens target have been split into multiple ones (on the other hand, Persona and Scenario data was never reused).
• Poor dataset building script. The Python script for building the dataset, although working, is not great quality-wise and not particularly efficient.
• Possible sources of impersonation. Several of the conversations in the 8k set feature participants consistently playing the role of two characters at the same time. Character names in these files (which include the suffix _MULTI or _GROUP in the filename) have been assigned a name with the format Char1&Char2. Testing didn't reveal issues with this, but it's something to keep in mind if more severe impersonation problems occur compared to the initial release of LimaRP. Furthermore, in a few conversations additional characters (roleplayed by either of the two users) may also temporarily participate to the story. These have often (but not always) been assigned a _BAD tag in the filename.
• Gender confusion sources. Some conversations feature "futanari" or "transgender" content. These have been found to confuse small-scale models to a certain extent. All source files have a content field and in most cases they contain keywords like shemale, futa, futanari, trans, transgender when relevant to assist filtering.

Conversation data form

Only one format has been used: forum/novel-style. This includes:

• Quotation marks for dialogues;
• Narration in third person, simple past form, without delimiters;

Other RP styles have been excluded, and messages showing them have been fixed when possible and feasible.

Format details

• Narration does not have any delimiter.
  • Jessica looked at Mark with disdain.
• Dialogues are enclosed with ASCII double quotation marks.
  • "I say this."
• Onomatopoeias are enclosed with asterisks.
  • *thud*
• Inner thoughts are enclosed with underscores.
  • _What is he doing?_
• Non-dialogue quotes are enclosed with two apostrophes on each side (caveat: not all have been converted in this way).
  • ''The Jungle Book''
• Punctuation has been normalized. Fancy quotes has been converted to the ASCII equivalent, ellipses always turned into a standard format (... with a trailing space when a word follows) and em-dashes always converted to three consecutive dashes (---) without any surrounding space.
  • For stylistic reasons, when building the dataset em-dash surrogates get converted to their UTF-8 symbol (—).
• Placeholder names have been used for the characters, even within the messages, whenever possible. <FIRST> is always assumed to be the bot/model, and <SECOND> always assumed to be the human/user. All conversations terminate with a message by <FIRST>.
  • When building the dataset, placeholder names currently get converted to the ones actually used in the RP conversations.

Data sources

Weights are naively calculated in terms of bytes for the entire conversation files as of 2023-11-10.

Note that users are required to be 18+ to write in the listed ERP forums or forum subsections.

Usernames, OOC and other personal information have not been included in the training data, only the names of the roleplayed characters as used in the conversations (or sometimes with minor changes).

Some lessons learned while making LimaRP

• There is indeed no real need for a large amount of data to give models a strong bias towards producing roleplaying text with good reliability in respecting user/bot turns. The 2000 training rows could even be trimmed in number with likely similar results.
• Incidentally, roughly 2000 training examples are also about the limit that can still be finetuned in reasonable amounts of time on a single 24GB consumer GPU at a decent context length (in the case of 13B models).
• Data quality beats quantity, but ensuring good quality is very difficult without the help of unrestricted powerful LLMs and/or extensive human intervention.
• Remaining focused on a strict set of rules with a clear goal and consistently adding a relatively small number of training examples on a daily basis were a driving force for completing the dataset.
• In retrospect, the RP conversation processing pipeline could have been improved and better planned, or made more efficient.
• Finetuning the dataset in "completion" mode (on the entire chat) may have been part of the reasons why it was so effective in altering model outputs, whether finetuned from scratch or applied as a LoRA.
• The initial GPT-3.5/4 "seed" data for generating Persona and Scenario was crucial in the initial stages of the project, but once a dedicated summarizer was trained from that data, access to more powerful LLMs became for the most part unnecessary.
• Compiling and verifying the data manually took much more work than anticipated. This should have been a group effort.
• In the end, advances in base model performance, increases in context size (so far from 2k tokens of the original LLaMA to 8k tokens or more of Mistral and other recent models) and improvements in ICL (in-context learning) capabilities may eventually render finetuning on datasets like LimaRP unnecessary except for zero-shot RP performance or cases where models have a strong built-in alignment preventing ERP.

---

Appendix

Additional points of improvement

Ideas in random order that could be applied for improving the dataset. Some have been already mentioned earlier.

• Recheck old conversations and consolidate them if short
  • Carefully watch for wrongly assigned character names and labels
• Rewrite scenario and personality to be more human-like and diverse
• Include the instruction at various locations in the conversation (e.g. at the end), or multiple times
  • May be performed just for "Scenario" to make the model learn to change the direction of the conversation
• Classify message "safety" (sexual/non-sexual, etc)
  • Perhaps using some external LLM and over a continuous range rather than just two discrete classifications
• Add more "Exaggerated" personalities
  • Anime/Japanese character tropes, etc.
• Include other information during finetuning
  • Inferred character motivations, psychology, notes, etc.
  • Could also include summarized (word list) traits in "Personality", making Persona as a general "Description"
  • These could go in the system instruction in a way to boost the model's reasoning capabilities.
  • Chain-of-Thought-like indications in long-form descriptions: "Since the character is/does X, he is Y", etc.
• Add focused RP-oriented small instruction dataset
  • Low amount of turns on very specific instructions
  • Greentext, logic, puzzles, etc.
• Truly randomized, gender-neutral names in the training data
  • Could avoid avoid name bias
  • Could be implemented as an "Alternative" name field in the source files
  • An easy alternative would be instead simply using standardized names like USER and CHAR—never tried with the full dataset.
• Use when finetuning message length hints in tokens/10
  • Could be more precise and easier to use than fuzzy lengths
• Include human training data from books in the form of "Questions and Answers" or interviews
• Make a greater use of what Markdown formatting offers, while maintaining the novel-style narration