EmojiNotion: An Image-to-Emoji Search and Prediction Engine Integrated into a Text Editor 🔍

Project overview

This project implements a web-based text editor with semantic emoji search and prediction, natively supporting an unlimited number of custom and animated emojis, crawled and captioned automatically via a multimodal retrieval pipeline, and rendered as normal inline text characters. It integrates key information retrieval techniques, including crawling, results ranking, and the use of Large Language Models.

The system is deployed using Docker for containerization, with MongoDB as the database (2 tables – Emojis and Tags), Python as the backend for running 2 Large Language Models — all-MiniLM-L6-v2 sentence transformer and the fine-tuned on Twitter data RoBERTa text classification model, and Node.js based web application serving as the single entry point for users and the system's hub, centralizing all API endpoints (plus also running Word2Vec model). Emojis search and prediction are both based on cosine similarity ranking; the only difference lies in the model used to convert the input query vector into an embedding vector.

Architecture diagram

Key components:

• Data Generation – pipeline to automatically build a database of image-based and animated emojis, crawling emoji data from the web and generating descriptions for efficient image semantic search and prediction by refining the name metadata and using BLIP for multimodal image captioning.
• Semantic Emoji Search and Prediction – utilizes Word2Vec and sentence transformer MiniLM embeddings to enable emoji retrieval and sentence-aware emoji prediction.
• Text Editor – integrates the engine into a full-featured editor that supports inline animated emojis and multimedia file format export.

Folders structure:

• aux_docs – auxiliary project files
• backend – web application server
• frontend – web application UI
• models – LLM models server
• scripts – files for initializing tables

Result

Manual

First time initialization
chmod +x init.sh (to make init.sh script executable)
./init.sh – automated setup
python scripts/crawl-emojis.py – populate Emoji table (manually terminate the script once you have the desirable number of emojis)
python scripts/fill-in-tags.py – populate Tags table

To start the text editor for subsequent launches
docker compose up

Accessing the Web UI
http://localhost:3000 – web application

The emoji database can be connected to and managed using MongoDB Compass. To connect, create a new connection and use the following URI: mongodb://localhost:27017/.