MTG-AI-Generator: Autogenerate MTG cards end-to-end using AIs to write fields and create art.

Project Overview

scripts/generate_cards.py is the main entry point. This will

• sample AIs
• process + decode their raw data
• render the final cards as image files
• and finally save outputs in the specified folder

4 separate AIs contribute to the card design, so that certain attributes can be lifted out of scope from the main text training data.

• First an AI generates an arbitrary card name. Then the name is whispered to a second AI which generates the main text and all in play properties of the card. The name and type are prompted to a third AI which generates flavor text for the card. Finally, the name, type, and flavor text are prompted to a fourth AI to create visual art for the card.
• nns/names* is an LSTM created for this project and trained to generate card names. This was trained both on all exisitng MTG card names as well as on a smattering of other words, phrases, and name-like strings. This increases the diversity of names outside of normal MTG card names.
• nns/main_text* is an LSTM created for this project and trained to generate all functional attributes and text of the cards. This was trained on select fields from all MTG cards (at time of training). This AI is whispered the names generated by the above AI during sampling.
• Flavor text is written by an LLM, prompted by card name, card type, and some static descriptors. The model was originally created by Meta, then leaked to the public, and further trained and refined by many individuals. This is wrapped by an open-source web-UI at llm.
• Card art is generated by stable diffusion, prompted by the card name, type, flavor text, and some static descriptors. The model was originally created by the CompVis open source project to create images from text-based descriptions, and then further trained and refined by many individuals. This is wrapped by an open-source web-UI at A1SD.
• The AI's are trained independently, and sampling is wrapped by generate_cards.py, which pulls all the ingredients together to create new cards.

Workflow / Getting Started

• Run through the environment setup section below
• The main entry point for the project is scripts/generate_cards.py
  • example command: python generate_cards.py --lstm_gpu 1 --names_nn ../nns/names_3 --main_text_nn ../nns/main_text_13 --flavor_nn timdettmers_guanaco-65b-merged --gpu-memory 23 --cpu-memory 250 --sd_nn "nov_mtg_art_v2_3.ckpt [76fcbf0ef5]" --outdir ../outputs --num_cards 10 --hr_upscale 2 --verbosity=9 --to_pdf
    • This command takes about 2 hours to execute on the dev's machine
    • see python generate_cards.py --help for more info on arguments
  • AI samples are cached in subdirectories (*_cache) under the auto-generated output folder (eg ../outputs/00003_481992436), so the generator can be stopped and resumed without losing much via the --resume_folder ... argument. Also useful if it crashes for some inconsistent reason.
  • Statistics about generated cards are output to stats.yaml in the output folder
  • Full card text for all cards is output to card_data.yaml in the output folder
  • A printable pdf is generated (printable_cards.pdf) if you specify the --to_pdf arg

• Overall timing (using the nns called out in setup; using other nns may change timing dramatically)

• other run modes
  • generate basic lands with unique art: python generate_cards.py --sd_nn "nov_mtg_art_v2_3.ckpt [76fcbf0ef5]" --verbosity=9 --outdir ../outputs --hr_upscale 2 --basic_lands "{'Plains':51, 'Island':51, 'Swamp':51, 'Mountain':51, 'Forest':51, 'Snow-Covered Plains':25, 'Snow-Covered Island':25, 'Snow-Covered Swamp':25, 'Snow-Covered Mountain':25, 'Snow-Covered Forest':25, 'Wastes':36}" --to_pdf
  • Quickly create card text, which can be hand-modified before rendering
    • Run generate_cards.py like normal plus args --no_flavor --no_art --no_render, so that only the text is generated
    • Open the generated card_data.yaml, and hand-modify the contents to your heart's desire. You can also delete cards entirely from the file.
    • Run generate_cards.py like normal with --finish_yaml <yaml path>
  • resume an interupted generation with --resume_folder <folder path> (and specify the same args)

Environment Setup

Install NVidia CUDA toolkit version 11.8. (Trying newer versions may require manual customization of subtrees, especially torch-rnn which is no longer developed.)

A1SD

• download this custom model from civitai to A1SD/models/ldm/stable-diffusion-v1/. Git does not support large files (several GB), so these files are not committed to the repo.
• set install_dir in webui.sh
• update COMMANDLINE_ARGS in webui-user.sh based on your amount of ram, see docs
• launch bash webui.sh to finish setup
  • the first time it runs, it will download a bunch of dependancies (several GB)
  • it's ready once it launches the webserver (eg it prints Running on local URL: http://127.0.0.1:7860)
  • and you can then ctrl+c it to close the process for now

llm

• cd llm
• bash setup.sh
  • might be able to unpin deepspeed, there was a bug (and this) in main branch when I installed it
• conda run -n llm python download-model.py timdettmers/guanaco-65b-merged (~160 GB)
  • May need to copy tokenizer configs from here (transformers RecursionError)
• launch server.py to finish setup
  • the first time it runs, it will download a bunch of dependancies (several GB)
  • it's ready once it launches the webserver (eg it prints Running on local URL: )
  • and you can then ctrl+c it to close the process for now

torch-rnn

• Setup torch dev environment. Conda doesn't handle lua / torch very well. Lua-torch is no longer maintained, and we can't use an old cuda installation on newer cards, so just install torch globally to ~/torch and fiddle until it works. The order of these steps is critical. If you screw up, its often easier to rm -rf ~/torch and start over than try to recover.
• install libhdf5-dev
  • add deb [trusted=yes check-valid-until=no] http://dk.archive.ubuntu.com/ubuntu/ trusty main universe to /etc/apt/sources.list
  • sudo apt update
  • sudo apt-get install libhdf5-dev==1.8.11*
  • sudo apt-mark hold libhdf5-dev to pin version
• conda env create -f environment-python.yaml. Use this enviropnment only for the preprocessing script
• install the nvidia cuda toolkit, version 11.8
• install gcc-6 and g++-6, since the older torch repo + cuda combination only works with this version
  • add deb [trusted=yes] http://dk.archive.ubuntu.com/ubuntu/ bionic main universe to /etc/apt/sources.list
  • sudo apt update
  • sudo apt install gcc-6 g++-6
• soft link cuda to gcc-6 and g++-6
  • sudo ln -s /usr/bin/gcc-6 /usr/local/cuda/bin/gcc
  • sudo ln -s /usr/bin/g++-6 /usr/local/cuda/bin/g++
• link missing cmake input sudo ln -s -T /usr/local/cuda-11.8/lib64/libcublas.so /usr/lib/x86_64-linux-gnu/libcublas_device.so
• add repo for outdated software dependancies sudo add-apt-repository ppa:ubuntuhandbook1/ppa and sudo apt-get update
• fix luarockspeck using outdated (unsupported) URLs, by forcing git to correct them on the fly
  • git config --global url."https://github.com/".insteadOf [email protected]
  • git config --global url."https://".insteadOf git://
• pip install ipython
• purge and install latest cmake
  • sudo apt-get purge cmake
  • cd ~
  • git clone https://github.com/Kitware/CMake.git
  • cd CMake
  • ./bootstrap; make; sudo make install
• install torch using bash install_torch.sh |& tee log-torch-install.txt. There will be several prompts.

main repo

• sudo apt install expect to get unbuffer command
• Install miniconda
• conda env create -f environment.yaml
• download nltk language files
  • cd ~/nltk_data
  • python -c "import nltk; nltk.download('punkt')"
• Pretrained AIs are provided for names and main text, but you can optionally train your own (eg with updated data inputs).
  • Build a dataset to train the AIs upon
    • Download AllPrintings.json from mtgjson website to raw_data_sources/.
    • optionally update raw_data_sources/names.yaml and raw_data_sources/flavor.yaml manually with additional training data
    • run bash rebuild_data_sources.sh |& tee log-data-build.txt in scripts/
      • use printed Average chunk length for each AI to update constants in generate_cards.py -> LSTM_LEN_PER_MAIN_TEXT, LSTM_LEN_PER_NAME, and LSTM_LEN_PER_FLAVOR
      • use printed Longest chunk length for each AI to set minimum -seq_length argument to train.lua
      • use printed Total vocabulary size for each AI to set -wordvec_size argument to train.lua?
    • The dataset format is considered an implementation detail, so for more information on the format see scripts/encode.py
  • Train AIs with commands similar to these.
    • th train.lua -gpu 0 -input_h5 ../encoded_data_sources/names.h5 -input_json ../encoded_data_sources/names.json -checkpoint_name ../nns/names_3/checkpoint -rand_chunks_n_epochs 5 -checkpoint_n_epochs 100 -validate_n_epochs 10 -print_every 1 -num_layers 3 -rnn_sizes "200, 200, 200" -max_epochs 100000000 -batch_size 2000 -seq_length 150 -dropout 0.5 -learning_rate 0.02 -lr_decay_n_epochs 30 -lr_decay_factor 0.985 -wordvec_size 74
    • th train.lua -gpu 0 -input_h5 ../encoded_data_sources/main_text.h5 -input_json ../encoded_data_sources/main_text.json -checkpoint_name ../nns/main_text_13/checkpoint -rand_chunks_n_epochs 1 -checkpoint_n_epochs 30 -validate_n_epochs 1 -print_every 1 -num_layers 4 -rnn_sizes "600, 600, 600, 600" -max_epochs 100000000 -batch_size 260 -seq_length 900 -dropout 0.5 -learning_rate 0.001 -lr_decay_n_epochs 10 -lr_decay_factor 0.99 -wordvec_size 122
    • th train.lua -input_h5 ../encoded_data_sources/flavor.h5 -input_json ../encoded_data_sources/flavor.json -checkpoint_name ../nns/flavor_0/checkpoint -rand_chunks_n_epochs 1 -checkpoint_n_epochs 100 -validate_n_epochs 1 -print_every 1 -num_layers 3 -rnn_size 256 -max_epochs 100000000 -batch_size 200 -seq_length 500 -dropout 0.5 -learning_rate 0.002 -lr_decay_n_epochs 50 -lr_decay_factor 0.99
    • CTRL+c to stop training when the AI is ready. Use scripts/plot_nn_loss.py to assess progress.
    • Check that the trained AIs work with th sample.lua -checkpoint ../nns/names_0/checkpoint_1001.000000.t7 -length 50. The main generator will use a similar command to sample the AIs when generating cards.

Repo Data Structure

• raw_data_sources include user inputs for AI training data. These are processed into encoded_data_sources via rebuild_data_sources.sh, which utilizes scripts/encode.py and torch-rnn.
• nns contains trained text-based neural networks
• torch-rnn contains code for training and sampling the text neural networks
• A1SD contains image generating neural networks and associated code.
• llm contains text generating neural networks and associated code.
• scripts contains the main generator entry point generate_cards.py as well as intermediary and utility scripts
• outputs contains rendered card images, card sheets, full text and stats yaml files, and cached intermediate steps
• image_templates contains template images for rendering the generated cards

Subtree List and their Customizations

torch-rnn, with some modifications inspired by mtg-rnn

• Created environment.yaml for python portion of the environment
• Implemented whispering during sampling
• Removed test fraction loading from DataLoader, which is unused, so that it can accept an empty test fraction
• Updated preprocessor to
  • partition input data on specified delimeter (eg between encoded cards)
  • randomize the chunk order
  • and assign a fraction of those chunks to training, validation, and testing; instead of assigning a fraction of raw data
  • store the data as processed chunks, which can be order randomized during batching
• stabalized shuffle order in preprocessor
• Updated DataLoader to
  • accept data chunks instead of raw data from the new proprocessing script
  • dynamically randomize the order and batch locality of the chunks each epoch
• Added option to DataLoader to dynamically randomize the order of structured content in encoded mtg cards in each batch
  • symbols in mana costs
  • card field order (other than the card name field, which is always the first field and treated as defining for the AI)
• Added option to trainer to set validation / checkpoint at a whole number of epochs, to avoid resetting the neural network in the middle of an arbitrarily segmented stream
• Updated trainer to load history and learning rate from checkpoints
• Updated trainer to print learning rate each time its updated
• Updated trainer to decouple checkpoint, validation, and learning rate decay frequencies from epochs / each other, and have CLI params for all
• Updated trainer to not clear optim state each time the learning rate is changed, for smoother loss curves
• Added seed input option to sampler for repeatable sampling

A1SD (aka AUTOMATIC1111 webui for stable diffusion)

• customized install dir, API, and vram usage
• forced unbuffering to python call for reading server state when called as a subprocess

llm (aka oobabooga webui for large language models)

• added setup.py

Util

• watch -n1 nvidia-smi to see GPU resource utilization
• torch docs
• batch convert svg images to png find . -name "*.svg" | xargs inkscape --export-type=png --export-width=1000 --export-height=1000 --export-png-color-mode=RGBA_8 --batch-process
• python plot_nn_loss.py --json_path nns/names_1/checkpoint_21.000000.json
• after changing environment.yaml update the environment with conda deactivate && conda env remove --name mtg-ai-main && conda env create -f environment.yaml && conda activate mtg-ai-main
• working with unicode in lua: http://lua-users.org/wiki/LuaUnicode
• each subtree has a remote under the same name as the directory
  • create remote: git remote add -f <name> <url>
  • add subtree: git subtree add --prefix <dir> <remote> <branch> --squash
  • pull subtree: git fetch <remote> <branch> and then git subtree pull --prefix <dir> <remote> <branch> --squash
• Stable Diffusion prompt development tooling
  • Lexica and OpenArt provide generated images and their prompts
  • img2prompt and BLIP predict the prompts for uploaded images
  • etc: see reddit stable diffusion tooling catalog

🔴 TODOs

• render sagas
• implement a lower font size limit for readability after printing
  • maybe dynamically expand the text box to accomodate the smallest text?
  • And if there's a smallest font size, we could just always use that to increase rendering speed? - probably not a meaningful speed improvement tho
  • make text box more opaque (or all the way opaque, to the art underneath it)
• implement tight 2D fitting around stats box which juts into the main text area
  • in addition to max width and height passed to the renderer, also pass a single lr_lockout_zone parameter, which is the width and height of a box protruding into the lower right corner of hte text area, overwhich text should not be rendered. Implement "not rendering text in that area" by checking currently rendered line height and prospective width against the new constraints every time a rendered-text image is considered for addition to the current line.
  • This can eliminate the whole line width allotted to these elements (eg the easy way out we did to begine with).
• fine tune LLMs to produce main_text and names instead of LSTMs
• move flavor text and its nn name out of card_data.yaml
  • keep that in the cache files similar to how the sd nn name is embeded in the SD image cache file
  • can dynamically add it back to the card_data.yaml when its defined at time of file writing, and omit when its not
  • then update the renderer to make flavor text optional if its defined
  • remove nns_names attribute from the yaml, and repalce it with individual network name attributes (eg nn_names)
• create to_tts_asset.py to format cards into sheets for upload to TTS
  • limit file size to something appropriate for TTS assets (40MB??)
  • Try card naming script here: https://steamcommunity.com/app/286160/discussions/0/364040166684390460/
  • Try making the entire deck-asset + room + mod files from scratch at generation time?, so that all we'd need to do manually ot use hte asset is load from the autogenerated mod files
• low priorty (ie probably never)
  • retrain names_3 after removal of ë character from training data...
  • add stats for mana value and types by color
  • add bypass to lstm parser where if max resamples are exceeded, instead of raising the error, it raises a unique error which is caught by generate_cards.py who loudly removes that card from the processing and finishes the rest of the cards
  • render.py
    • decrease save file resolution to limit file size?
    • refine txt2img args to furthr dissuade creating art images resembling mtg cards ?
  • encode.py
  • update torch-rnn to handle rand_mtg_fields argument given new field sep, card sep, and mana formats from encode.py ?
    • This would enable whispering arbitrary field definitions, in arbitrary orders, to generate sets or cards with highly customized parameters, instead of defining fields only from left to right in the standard order