OneEncoder: A Lightweight Framework for Multimodal Training

OneEncoder is a streamlined framework for aligning multiple modalities (text, image, audio, video) using a progressive training approach. It reduces training costs by aligning new modalities without retraining the entire system, achieving strong results even on small datasets.

🚀 Key Features

• Progressive Modality Alignment:
  • Step 1: Align image and text with a Universal Projection (UP) module.
  • Step 2: Freeze UP, train an Alignment Layer to integrate audio, video, and more.
• Efficient and Cost-Effective: Works well on small paired datasets, outperforming large-scale models with specialized encoders.
• Flexible Backbone Choices: Supports various image and text encoders (e.g., ALBERT, BERT, RoBERTa, ViT, DeiT, BeiT).

🏁 Quickstart

Installation

conda create -n OneEncoder python=3.9 pip
conda activate OneEncoder
conda install pytorch=2.1.1 torchvision=0.16.1 cudatoolkit=12.1 -c pytorch
pip install -r requirements.txt

Datasets

Update dataset paths in the config file or class CFG.

• Image-Text Alignment: COCO Captions, Flickr30k, TextCaps
• Audio Integration: ibriSpeech SRA
• Video Integration: MSR-VTT
• X-ray Integration: ROCO Dataset
• Visual QA: DAQUAR

📘 Usage

Train the UP for image-text alignment:

cd "contrastive learning/text-image/addition"
python text_image.py

Freeze UP, train the Alignment Layer for new modalities:

mv "contrastive learning/text-image/addition/best.pt" "contrastive learning/audio-image/addition/text_image.pt"
cd "contrastive learning/audio-image/addition"
python audio_image.py

Run Visual QA:

cd "visual question answering/albert and beit"
python albert_beit.py

🛠️ Fusion Operations

• For Alignment: Addition, Multiplication, Concatenation, Attention
• For VQA: Addition, Scaled Dot Product Attention

🧠 Demos

• Colab Demo
• Hugging Face Spaces Application Demo
• Pretrained Models:
  • Text & Image → HF Model
  • Text, Image & Audio → HF Model
  • Text, Image & Video → HF Model
  • Text, Image & X-Ray Image → HF Model

---

🔧 Default Training Config: temperature = 2.5, fusion via addition