MiloMusic 🎵

🦙 AI-Powered Music Creation for Everyone

MiloMusic is an innovative platform that leverages multiple AI models to democratize music creation. Whether you're a seasoned musician or have zero musical training, MiloMusic enables you to create high-quality, lyrics-focused music through natural language conversation.

A platform for everyone - regardless of musical training at the intersection of AI and creative expression.

🌐 Demo

Try MiloMusic online: Hugging Face Space

The demo is deployed on Nvidia 1 ✖ L40S with 62GB RAM and 48GB VRAM.

Note: Local deployment references deepbeepmeep/YuEGP, while the Hugging Face Space deployment references multimodal-art-projection/YuE.

Important: The files requirements_space.txt, README_SPACES.md, app_space.py, and packages.txt are configuration files specifically for YuEGP's Hugging Face Space deployment and are not used in this project. They are also not applicable to HugMilo/MiloMusic since it is based on YuE rather than YuEGP.

🚀 Features

• Natural Language Interface - Just start talking to generate song lyrics
• Genre & Mood Selection - Customize your music with different genres and moods
• Iterative Creation Process - Refine your lyrics through conversation
• High-Quality Music Generation - Transform lyrics into professional-sounding music
• User-Friendly Interface - Intuitive UI built with Gradio

🔧 Architecture

MiloMusic employs a sophisticated multi-model pipeline to deliver a seamless music creation experience:

Phase 1: Lyrics Generation

1. Speech-to-Text - User voice input is transcribed using whisper-large-v3-turbo(STT)
2. Conversation & Refinement - llama-4-scout-17b-16e-instruct handles the creative conversation, generates lyrics based on user requests, and allows for iterative refinement

Phase 2: Music Generation

1. Lyrics Structuring - Gemini flash 2.0 processes the conversation history and structures the final lyrics for music generation
2. Music Synthesis - YuE (乐) transforms the structured lyrics into complete songs with vocals and instrumentation

💻 Technical Stack

• LLM Models:
  • whisper-large-v3-turbo(STT) - For speech-to-text conversion
  • llama-4-scout-17b-16e-instruct - For creative conversation and lyrics generation
  • Gemini flash 2.0 - For lyrics structuring
  • YuE - For music generation
• UI: Gradio
• Deployment: Support for H100 GPU with 80GB VRAM (full model) or RTX 4090 with 24GB VRAM (quantized model)

📋 Requirements

• Python 3.10+
• CUDA-compatible GPU (24GB+ VRAM recommended)
• UV package manager (install here)
• Dependencies listed in pyproject.toml

🔍 Installation & Usage

# Clone the repository
git clone https://github.com/futurespyhi/MiloMusic.git
cd MiloMusic

# Install dependencies using uv
uv sync --index-strategy unsafe-best-match

# Install flash-attn separately (due to ABI compatibility issues with PyTorch)
uv pip install --no-build-isolation --no-cache-dir flash-attn

# Run the application
uv run python app.py

Using the Interface:

1. Select your genre, mood, and theme preferences
2. Start talking about your song ideas
3. The assistant will create lyrics based on your selections
4. Give feedback to refine the lyrics
5. When you're happy with the lyrics, click "Generate Music from Lyrics"
6. Listen to your generated song!

🔬 Performance

MiloMusic can run in two configurations:

• Full Model (H100 GPU with 80GB VRAM): Generates music in ~9 minutes
• Quantized Model (RTX 4090 with 24GB VRAM): Generates music in ~13 minutes

🛠️ Development Challenges & Solutions

🔮 Future Improvements

• Implement a wait queue system to improve UX during music generation
• Explore faster model alternatives for reduced generation time
• Add more customization options for music style and composition
• Implement batch processing for multiple song generation

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

👥 Team

• Norton Gu
• Anakin Huang
• Erik Wasmosy

📝 License

This project is licensed under the BSD 3-Clause License - see the LICENSE file for details.

🙏 Credits

Special thanks to the following projects that made MiloMusic possible:

• multimodal-art-projection/YuE - For the YuE music generation model
• deepbeepmeep/YuEGP - For local deployment inspiration

---

Made with ❤️ and 🦙 (LLaMA)