xinyanghuang/Basic-Visual-Language-Model

Model

1

stars

8

commits

4

linked in READMEs

Jun 8, 2024

updated

visual-question-answering

README

Building Your Own Multimodal Large Model from Scratch

For the Chinese version of the README, please refer to 中文文档.

Model Architecture 🤖

In the VLM (Visual Language Model), the visual component utilizes the CLIP or SIGLIP models, which have already achieved preliminary semantic alignment. A two-layer MLP is used for feature mapping. By overriding the forward method of the QWenModel, the corresponding image tokens are replaced with visual features.

GitHub Repository 🏠

The code for running the model can be found at Basic-Visual-Language-Model.

References 📚

Special thanks to the following projects for their great work 🙌:

Contact ✉

If you have any questions or ideas, feel free to reach out to me 😊:

hsinyanghuang7@gmail.com

I will respond as soon as I see your email!

Contributors

xinyanghuang

8 commits

xinyanghuang/Basic-Visual-Language-Model

Model

1

stars

8

commits

4

linked in READMEs

Jun 8, 2024

updated

visual-question-answering

README

Building Your Own Multimodal Large Model from Scratch

For the Chinese version of the README, please refer to 中文文档.

Model Architecture 🤖

In the VLM (Visual Language Model), the visual component utilizes the CLIP or SIGLIP models, which have already achieved preliminary semantic alignment. A two-layer MLP is used for feature mapping. By overriding the forward method of the QWenModel, the corresponding image tokens are replaced with visual features.

GitHub Repository 🏠

The code for running the model can be found at Basic-Visual-Language-Model.

References 📚

Special thanks to the following projects for their great work 🙌:

Contact ✉

If you have any questions or ideas, feel free to reach out to me 😊:

hsinyanghuang7@gmail.com

I will respond as soon as I see your email!

Contributors

xinyanghuang

8 commits