z-waif

Fully local & open source AI Waifu. VTube Studio, Discord, Minecraft, custom made RAG (long term memory), alarm, and plenty more! Has a WebUI and hotkey shortcuts. All software is free (or extremely cheap)!

Reccomended Windows 10/11 and a CUDA (NVidia) GPU with atleast 16GB+ of VRAM. Can now support Mac and Linux! Any brand (AMD, NVidia, Intel) GPU with 8GB+ VRAM bare mininum requirement! Uses Oobabooga, RVC, and Whisper to run AI systems locally. Works as a front end to tie many programs together into one cohesive whole.

The goal of the project is less about giving an "all in one package", and moreso to give you the tools and knowledge for you to create your own AI Waifu!

Features

• 🎙️ Quality Conversation       ( /・0・)

  • Speak back and forth, using Whisper text to speech.
  • Configure your own waifu's voice with thousands of possible models.
  • Imperial-tons of quality of life tweaks.

• 🍄 Vtuber Integration         ღゝ◡╹ )ノ♡

  • Uses VTube Studio, and any compatible models!
  • Ability to send emotes to the model, based on their actions.
  • Idle / Speaking animation.

• 💾 Enhanced Memory           (ー_ーゞ

  • Add Lorebook entries, for your waifu to remember a wide array of info as needed.
  • Enable the custom RAG, giving your them knowledge of older conversations.
    • Import old logs and conversations, keeping your same AI waifu from another software!

• 🎮 Modularity               ⌌⌈ ╹므╹⌉⌏

  • Enable various built in modules;
    • Discord, for messaging.
    • Vision, to enable multimodal, and allow them to see!
    • Alarm, so your waifu can wake you up in the morning.
    • Minecraft, allowing your waifu to control the game using Baritone, Wurst, and other command based mods.
  • All the options and modularity from any external software used. Oobabogoa, RVC Voice, ect.
  • Open-source, meaning you can edit it as you please.

YouTube Showcase

Install & Links

Here is some documentation that you can look at. It will show you how to install, how to use the program, and what options you have. Please also take a look at the Youtube videos for the install.

If you need help / assistance, please submit a GitHub issue, or feel free to email me for this project at zwaif77@gmail.com

Z-Waif has a basic website that you can visit. I have also set up a small Discord for community members to chat as well.

Diaspora

The Original:

TumblerWarren/Virtual_Avatar_ChatBot, this is the original project that this code is spun-off of. Full credit to that project - it provided the skeleton for the many advancements now in place. It has more of a focus on non-local AI, if that is what you need.

Branches & Versions:

Pre-Release Branch, test the next update, all the new features and help prepare it before release!

Drakkadakka/z-waif-experimental-, offers a few upgrades; namely Twitch chat & streaming support, as well as a few other enhancements.

MaolinkLife/z-waif-ru-adaptation, offers Russian language support, lighter-weight edge-tts, lorebook enhancements, ect.

Recent Changelog

v1.15

• Added rolling summaries.

  • Will automatically generate every 10 message pairs, spaced back by 10 message pairs.
    • No need to summarize things still in live memory, so it is spaced back.
  • Toggleable in env.
  • Can configure the total messages summarized per summary chunk (SEARCH_POINT_SIZE).
  • Can configure the max size of these summaries (SUMMARY_TOKENS_MAX_COUNT).
  • RAG should probably be on, might break if not! So turn on after a few days of use.

• Added LMStudio API access option.

  • Works with image access, too.
  • Can change model configs in Configurables.
  • Pretty much the same config style as the Ollama.
  • Best new loader, I personally think.

• Added an option to remove quotation marks from being saved to memory, so the bot doesn't loop.

• Backups are now saved to their own folder. There are also now two backups - the ususal one on boot, and one that runs every 8 messages.

  • There is also a backup option for the new rolling summaries, too.

• New Discord API commands.

  • /help will list all commands
  • /allow_newlines and /disallow_newlines can turn on and off the cuts for newline.
  • Still have the /reroll command, by the way.

• Enhanced detection for blank messages in the case autochat is left on to be skipped.

• Note: For anyone using the image API, it can help to turn on "Multiple-access" in windows camera settings.

  • This will allow you to keep a preview of what is being taken by opening the Camera app, Obs, or anything that lets you see.
  • Leaving another app open also improves image quality, I notice.

• Fixed a bug where the mic would stay recording during semi-auto chat improperly.

• Fixed a bug where toggling on newline cut would leave it on, even if turned off.

To-Do

📶 Enhancements

• Make the RAG/Long Term Memory be multiprocessed for better performance
• Make the LLM input and TTS output streaming, to lower the "processing time"
• Figure out how to load LLAMA 3.2 Vision, for better multimodal, and no needed loader

🤖 Improvements

• Give internal dialoguing for chain of thought / reasoning
• Emotional / Tone understanding
• Automatic gaming & real world interaction
• Use an integrated voice generation system, with the ability to modify the tone
• Long term experience-based summarizations of ideas and history (pull form experience)

🦄 Imperium

• Create more Youtube tutorials and other related content
• Look more into optimal LLMs and configs
• Set up better Git and contribution methods
• Create a way for users to auto-update the program without having to hack files together
• Evangelize AI Waifus to the world!

State of Development

The project could be considered in an "early access state". Some parts may be mildly buggy, janky, or obtuse. The project as a whole, however, is stable and reasonably effective.

The goal of the project is pretty simple; make AI waifus. The extents of this project are intended to stay within the bounds of helping people create a singular, locally hosted AI waifu, who's partnership can benefit both you and them. In short, symbiosis.