[!NOTE] Lobe Vidol is currently in early development and is now open for Beta testing. We welcome you to join us and contribute!

Table of Contents

TOC

• 👋🏻 Getting Started & Communication
• ✨ Features Overview
  • 1. Text Chat Mode
  • 2. Video Conversation Mode
  • 3. Multi-Model Provider Support
  • 4. Character and Dance Marketplace
  • 5. TTS & STT Voice Conversations
  • 6. Progressive Web App (PWA)
• 📦 Ecosystem
• ⌨️ Local Development
• 🤝 Contributing
• 🩷 Community Sponsorship
• 🔗 More Tools
  • More Projects
  • Related Links

👋🏻 Getting Started & Communication

Lobe Vidol is actively under development. If you have any requests or questions, feel free to submit [issues][issues-link].

[!IMPORTANT]

Star the project to receive all release notifications from GitHub without delay!～⭐️

Star History

✨ Feature Overview

1. Text Chat Mode

Streamed responses provide a smooth conversational experience, allowing you to create an immersive dialogue by setting up character dialogue scenarios.

2. Video Chat Mode

With the video button in the message box, you can engage in richer interactions with character models, such as face-to-face chats, character performances, and more. You can also set the background, stage, actions, and dances for the characters.

You can enhance the dialogue's context by setting background images:

With a built-in library of Mixamo character motions and poses, you can have characters strike the desired poses or perform actions during the conversation:

Using the character editor, you can create your own virtual idol, set touch responses, upload VRM models to the site, and interact with them.

With the support of mmd-parser, you can dance with your favorite idols using vmd files; Just Dance!

https://github.com/user-attachments/assets/c017be86-dbac-4ce1-9f00-a10248b58621

We have added stage loading features and built-in a variety of stage options, allowing characters to dance with different stage styles:

By clicking on different body parts of the character, they will respond in various ways. You can also edit the character's touch responses to create unique interactive effects:

https://github.com/user-attachments/assets/a283bca0-222c-4ac8-8757-8c56ce3873c2

3. Multi-Model Provider Support

The diversity of model providers is crucial for meeting community needs when providing AI conversation services. Therefore, we have expanded our support to multiple model providers, rather than being limited to a single one, to offer users a richer and more diverse conversation selection. This way, LobeVidol can flexibly adapt to different user needs while providing developers with a broader range of options.

Supported Model Providers

We have implemented support for the following model providers:

• AWS Bedrock: Integrated with AWS Bedrock services, supporting models like Claude / LLama2, providing powerful natural language processing capabilities. Learn more
• Google AI (Gemini Pro, Gemini Vision): Accessed Google's Gemini series models, including Gemini and Gemini Pro, to support advanced language understanding and generation. Learn more
• Anthropic (Claude): Integrated with Anthropic's Claude series models, including Claude 3 and Claude 2, featuring multimodal breakthroughs and ultra-long context, setting new industry benchmarks. Learn more
• ChatGLM: Added Zhipu's ChatGLM series models (GLM-4/GLM-4-vision/GLM-3-turbo), providing users with another efficient conversation model option. Learn more
• Moonshot AI: Integrated with the Moonshot series models, an innovative AI startup from China, aiming to provide deeper conversation understanding. Learn more
• Together.ai: Integrated hundreds of open-source models and vector models, accessible without local deployment. Learn more
• 01.AI: Integrated with the 01.AI model, featuring APIs with fast inference speeds, reducing processing time while maintaining excellent model performance. Learn more
• Groq: Accessed Groq's AI models, efficiently processing message sequences and generating responses, suitable for multi-turn dialogues and single interaction tasks. Learn more
• OpenRouter: Supports routing for models including Claude 3, Gemma, Mistral, Llama2, and Cohere, enabling smart routing optimization to enhance usage efficiency, open and flexible. Learn more
• Minimax: Integrated with Minimax's AI models, including the MoE model abab6, providing more options. Learn more
• DeepSeek: Integrated with DeepSeek's AI models, including the latest DeepSeek-V2, offering models that balance performance and price. Learn more
• Qwen: Integrated with Qwen's AI models, including the latest qwen-turbo, qwen-plus, and qwen-max models. Learn more
• Novita AI: The most cost-effective open-source model provider, supporting cutting-edge open-source models like Llama and Mistral series. Excelling in emotional companionship scenarios, with no content restrictions or censorship. Learn more

We are also planning to support more model providers to further enrich our provider library. If you would like LobeVidol to support your favorite provider, feel free to join our community discussion.

4. Character and Dance Market

The character market at LobeVidol brings together a variety of meticulously designed characters, allowing you to experience different scenarios and interactions, providing you with a unique companionship experience. Our market is not just a showcase platform; it is also a collaborative space where everyone can contribute their imagination and share their personally designed characters.

[!TIP]

With the character creation feature, you can easily submit your character creations to our platform. We emphasize that LobeVidol has established a sophisticated automated internationalization (i18n) workflow, which seamlessly transforms your characters into multiple language versions. This means that regardless of the language your users speak, they can experience your characters without barriers.

The dance market at LobeVidol is enriched with a variety of MMD dance resources, creating a rich visual experience. You can combine different characters, stages, music, and dances to create a unique viewing experience.

5. TTS & STT Voice Conversations

LobeVidol supports Text-to-Speech (TTS) and Speech-to-Text (STT) technologies, enabling our application to convert text information into clear voice output. Users can interact with our conversational assistant as if they were talking to a real person. Users can choose from a variety of voices to match their assistant with the appropriate sound source. Additionally, for those who prefer auditory learning or want to gather information while busy, TTS offers an excellent solution.

At LobeVidol, we have carefully selected a range of high-quality voice options (OpenAI Audio, Microsoft Edge Speech) to meet the needs of users from different regions and cultural backgrounds. Users can choose the appropriate voice based on personal preferences or specific scenarios, thus obtaining a personalized communication experience.

6. Progressive Web Application (PWA)

We understand the importance of providing a seamless experience for users in today's multi-device environment. To this end, we have adopted Progressive Web Application PWA technology, which elevates web applications to a near-native app experience. Through PWA, LobeVidol can deliver a highly optimized user experience on both desktop and mobile devices while maintaining lightweight and high-performance characteristics.

Visually and functionally, we have also designed it carefully to ensure that its interface is indistinguishable from native applications, providing smooth animations, responsive layouts, and adapting to different device screen resolutions.

[!NOTE]

If you are unfamiliar with the PWA installation process, you can follow these steps to add LobeVidol as your desktop application (also applicable to mobile devices):

• Run Chrome or Edge browser on your computer.
• Visit the LobeVidol webpage.
• Click the Install icon in the upper right corner of the address bar.
• Follow the on-screen instructions to complete the PWA installation.

📦 Ecosystem

• Vidol market - This is the Market Index of Vidol Chat. Vidol accesses index.json from this repo to show users the list of available agents and dances.
• Vidol agent sample - Template for Vidol character data.
• Vidol dance sample - Template for Vidol dance data.

⌨️ Local Development

You can use GitHub Codespaces for online development:

Or use the following commands for local development:

$ git clone https://github.com/lobehub/lobe-vidol.git
$ cd lobe-vidol
$ bun install
$ bun dev

🤝 Contributing

We warmly welcome contributions in various forms. If you're interested in contributing code, please check out our GitHub [Issues][github-issues-link] and [Projects][github-project-link] to showcase your creativity and ideas.

[![][pr-welcome-shield]][pr-welcome-link]

🩷 Community Sponsorship

Every bit of support is incredibly valuable, coming together to form the brilliant galaxy we support! You are like a shooting star that lights up our path forward. Thank you for your trust — your support acts like a guiding star, illuminating the way for the project time and again.

🔗 More Tools

More Projects

• [🤖 Lobe Chat][lobe-chat] : An open-source, extensible (Function Calling) plugin system, modern design ChatGPT/LLMs chat application and development framework. Supports one-click free deployment of your private ChatGPT/LLMs application.
• [🅰️ Lobe SD Theme][lobe-theme]: A modern theme for Stable Diffusion WebUI, featuring exquisite interface design, highly customizable UI, and efficiency-enhancing features.
• [⛵️ Lobe Midjourney WebUI][lobe-midjourney-webui]: Midjourney WebUI, capable of quickly generating a rich variety of images based on text prompts, inspiring creativity and enhancing dialogue.
• [🌏 Lobe i18n][lobe-i18n]: Lobe i18n is an automation tool for the i18n (internationalization) translation process powered by ChatGPT. It supports automatic splitting of large files, incremental updates, and customizable options for OpenAI models, API proxies, and temperature.

Related Links

• mmd-parser - https://github.com/takahirox/mmd-parser
• three-vrm - https://github.com/pixiv/three-vrm
• tts-vue - https://github.com/LokerL/tts-vue

---

📝 License

Copyright © 2024 [lobehub][profile-link].
This project is licensed under the Apache 2.0.