Welcome to "Mustashhed," our open-source project submitted as a contribution to Arabicthon 2023 (https://arabicthon.ksaa.gov.sa/). Mustashhed is a Flask web application accessible at https://mustashhed.ngrok.dev/.
Our project leverages a set of large datasets of text containing a substantial number of Arabic texts. We utilize pySBD (Python Sentence Boundary Disambiguation) to split the text from the corpora into sentences, forming the examples.
Next, we employ the aubmindlab/bert-base-arabertv02, aubmindlab/bert-large-arabertv02, and medmediani/Arabic-KW-Mdel models to obtain embeddings for these examples. When a user enters a word along with its meaning and type, or uses the Alriyadh dictionary, we utilize camel_tools to retrieve all forms of the word based on its type (verb, noun, preposition).
For each word form, it is associated with the meaning to create a query. This query is then passed to the one of the embedding models mentioned above. Subsequently, we utilize Faiss, a library for efficient similarity search and clustering of dense vectors, to find the most similar examples to the user's query, by comparing the embeddings generated by the same model that embedded the user's query.
To run and host this project locally, follow these steps:
pip install -r requirements.txt
pip install faiss-cpu
python main.py
Note: The first time you run the application, it may take some time to download the selected model. Subsequent runs will not require this download.
Welcome to "Mustashhed," our open-source project submitted as a contribution to Arabicthon 2023 (https://arabicthon.ksaa.gov.sa/). Mustashhed is a Flask web application accessible at https://mustashhed.ngrok.dev/.
Our project leverages a set of large datasets of text containing a substantial number of Arabic texts. We utilize pySBD (Python Sentence Boundary Disambiguation) to split the text from the corpora into sentences, forming the examples.
Next, we employ the aubmindlab/bert-base-arabertv02, aubmindlab/bert-large-arabertv02, and medmediani/Arabic-KW-Mdel models to obtain embeddings for these examples. When a user enters a word along with its meaning and type, or uses the Alriyadh dictionary, we utilize camel_tools to retrieve all forms of the word based on its type (verb, noun, preposition).
For each word form, it is associated with the meaning to create a query. This query is then passed to the one of the embedding models mentioned above. Subsequently, we utilize Faiss, a library for efficient similarity search and clustering of dense vectors, to find the most similar examples to the user's query, by comparing the embeddings generated by the same model that embedded the user's query.
To run and host this project locally, follow these steps:
pip install -r requirements.txt
pip install faiss-cpu
python main.py
Note: The first time you run the application, it may take some time to download the selected model. Subsequent runs will not require this download.