higgs-gguf

• run it with gguf-connector; simply execute the command below in console/terminal

ggc h6

GGUF file(s) available. Select which one to use:

1. higgs-audio-3b-q5_1.gguf
2. higgs-audio-3b-q6_k.gguf
3. higgs-audio-3b-q8_0.gguf

Enter your choice (1 to 3): _

• opt a gguf file in your current directory to interact with; nothing else

connector h2 (alternative)

ggc h2

• note: differ from ggc h6, for ggc h2, you don't need gguf files; model file(s) will be pulled to local cache automatically during the first launch; then opt to run it entirely offline; i.e., from local URL: http://127.0.0.1:7860 with lazy webui
• as higgs audio model is very sensitive to quant type loss; would recommend to pick higher tier quants, i.e., q6, q8, etc. of course, you could test them one by one and pick the one suits you
• one more thing, you might need to downgrade tranformers to 4.46.3 since transformers.models.llama.modeling_llama was removed from the new version but higgs need it to encode text; just remember to revert it back if you need the updated transformers for other models

reference

• base model from bosonai
• gguf-connector (pypi)