Idiom Identification Project

This project identifies idioms in text for Italian and Turkish languages using transformer-based models.

This project folder contains the code of our main implementation (Seperate models per Language).

The 'Other Models and Files' folder contains implementation of BERT-base with CRF along with the files provided for the competition.

Notes:

• We reccomend you to run the codes in Google Colab or a PC with CUDA for faster performance utilizing GPU.
• By default, the 'models' folder is empty. Because of size limits in Ninova, we expect you to download our pretrained weights or train the models.
• If you want to download the pretrained weights provided by us, you have to download near 500 MBs per model (2 models, for Italian and Turkish).
• When this code runs, it saves checkpoints for each epoch which takes up around 1.3GB of space. So you should check the available space on your machine before running.
• In the competition, we had submitted a model with very slight better performance than this one but due to it's code being messy we decided to upload this one for the submission. We have removed the messy code model from the leaderboard and currently the output of this is sitting in the leaderboard. (The performance boost from the other model was not worth the hassle)

Setup

1. Create a Conda environment (optional but recommended):

   conda create -n idiom-env python=3.11
   conda activate idiom-env
   

2. Install dependencies:

   pip install -r requirements.txt
   

   (Ensure you are in the submission directory when running this command, or adjust the path to requirements.txt)

Running the Code

The main script src/main.py (run from the submission directory) handles training, inference, evaluation, and model downloading.

Interactive Mode:

Run the script without arguments to enter interactive mode:

python src/main.py

The script will guide you through the options:

• Choose between training, inference, or evaluation.
• If training, specify the language ('it' for Italian, 'tr' for Turkish).
• If performing inference with a model not found locally, you'll be asked if you want to download the pre-trained models.
• Provide paths for input data, model saving/loading, and output predictions. Paths should be relative to the submission directory (e.g., data/my_input.csv, models/it/final).

Command-Line Arguments:

• Training:

  python src/main.py train --language <lang_code> --input_file <path_to_train_csv> [--eval_file <path_to_eval_csv>] [--model_path <path_to_save_model_dir>] [other_training_options]
  

• Inference:

  python src/main.py inference --input_file <path_to_combined_input_csv> --output_file <path_to_merged_output_csv> [--model_path <path_to_load_models_base_dir_or_specific>]
  

• Evaluation:

  python src/main.py evaluate --prediction_file <path_to_prediction_csv> --ground_truth_file <path_to_ground_truth_csv>
  

Common Arguments:

• action: train, inference, or evaluate.
• --input_file <path>: Path to the input CSV file (relative to submission/).
  • For training: This CSV can be language-specific or a combined file containing an 'lang' column (which will be filtered by --language). Expected columns: id, text, tokenized_sentence, indices (and lang if combined).
  • For inference: This CSV must contain data for both 'it' and 'tr' languages, with a lang column to distinguish them. Expected columns: id, lang, text, tokenized_sentence.
• --model_path <path> (optional):
  • For training: Directory to save the trained model (e.g., models/it_custom_trained/final). Defaults to models/<language>/final.
  • For inference: Can be a base directory (e.g., models/) where it/final and tr/final subdirectories are expected, or a specific path to a language model if you want to override one (less common for the combined inference). Defaults to looking for models/it/final and models/tr/final.
• --batch_size <int>: Batch size (default: 16).

Training-Specific Arguments:

• --language <lang_code>: it or tr. Required for training.
• --eval_file <path> (optional): Path to an evaluation CSV file for validation during training.
• --train_epochs <int>: Number of training epochs (default: 3).
• --learning_rate <float>: Learning rate (default: 5e-5).
• --weight_decay <float>: Weight decay (default: 0.01).
• --warmup_steps <int>: Warmup steps (default: 100).

Inference-Specific Arguments:

• --output_file <path>: Path to save the merged predictions CSV file (e.g., data/predictions.csv). Required for inference.

Evaluate-Specific Arguments:

• --prediction_file <path>: Path to the prediction CSV file (e.g., data/my_predictions.csv). Required for evaluation.
• --ground_truth_file <path>: Path to the ground truth CSV file (e.g., data/actual_labels.csv). Required for evaluation.

Example Usage (run from submission/ directory):

• Interactive Mode:

  python src/main.py
  

  Then follow the prompts.

• Training Italian Model:

  python src/main.py train --language it --input_file data/train_italian.csv --eval_file data/dev_italian.csv --model_path models/my_italian_model/final --train_epochs 3
  

  (Assuming train_italian.csv and dev_italian.csv are in submission/data/)

• Training Turkish Model (using a combined training file):

  python src/main.py train --language tr --input_file data/train_combined.csv --model_path models/my_turkish_model/final
  

  (The script will filter train_combined.csv for lang == 'tr')

• Inference on Combined Data:

  python src/main.py inference --input_file data/test_combined.csv --output_file data/predictions_merged.csv
  

  (This will use default model paths models/it/final and models/tr/final. If they don't exist, it will offer to download them. test_combined.csv must have id, lang, text, tokenized_sentence columns.)

• Inference using custom model locations:

  python src/main.py inference --input_file data/test_combined.csv --output_file data/predictions_merged.csv --model_path my_trained_models/
  

  (This expects my_trained_models/it/final and my_trained_models/tr/final relative to submission/)

• Evaluating a Prediction File against a Ground Truth File:

  python src/main.py evaluate --prediction_file data/my_predictions.csv --ground_truth_file data/actual_labels.csv
  

  (This will calculate F1 scores for 'it', 'tr', and their average, based on the provided files.)

Input and Output File Format

Input CSV (for training): Columns:

• id: Unique identifier for the sentence.
• sentence: The raw sentence text.
• tokenized_sentence: A string representation of a list of tokens (e.g., "['Questo', 'è', 'un', 'test', '.']"). Correctly pre-parsed lists in the DataFrame are also accepted.
• indices: A string representation of a list of integer indices indicating the words forming an idiom (e.g., "[2, 3]"). Use "[-1]" if no idiom is present. Correctly pre-parsed lists are also accepted.
• language (optional): Language code ('it' or 'tr'). If present, used to filter data when training for a specific language. If not present, all rows are used for the specified training language.
• expression (optional): The idiomatic expression itself.
• category (optional): A category for the idiom.

Input CSV (for inference): Columns:

• id: Unique identifier for the sentence.
• language: Language code ('it' or 'tr'). Mandatory for inference to route data to the correct model.
• sentence: The raw sentence text.
• tokenized_sentence: A string representation of a list of tokens. Correctly pre-parsed lists are also accepted.
• expression (optional): The idiomatic expression. Will be carried over to the output if present.
• category (optional): A category for the idiom. Will be carried over to the output if present.

Output CSV (from inference): The script will produce a single CSV file with predictions from both languages, merged and sorted by id.

• id: Copied from input.
• language: Copied from input.
• sentence: Copied from input.
• tokenized_sentence: Copied from input (as string list).
• expression (if present in input): Copied from input.
• category (if present in input): Copied from input.
• indices: A string representation of the list of predicted idiom word indices (e.g., "[2, 3]" or "[-1]").

Input CSVs (for evaluate mode):

• Prediction CSV (--prediction_file):
  • Should be in the same format as the output of the inference mode.
  • Required columns: id, language, indices (string representation of a list of predicted integers).
• Ground Truth CSV (--ground_truth_file):
  • Should contain the true labels.
  • Required columns: id, language, indices (string representation of a list of true integers).

Data Assumptions

• Input data is provided in CSV format.
• tokenized_sentence and indices columns, if provided as strings, are Python list representations evaluatable by ast.literal_eval().
• For inference, the input CSV contains a lang column.
• The task is token classification for idiom identification.

Reproducibility

• Random seeds are set for numpy, torch, and Python's random module to ensure deterministic output during training and model-related operations.
• Running inference with the provided pre-trained models (or your consistently trained models) and the same test data should yield consistent results.

Model Weights

Pre-trained model weights are available for download (the script will offer to download them if needed):

• Italian (dbmdz/bert-base-italian-xxl-cased based): Google Drive Link
• Turkish (dbmdz/convbert-base-turkish-mc4-cased based): Google Drive Link

These will be downloaded into models/<language>/final/ by default if not found. Each model folder contains: config.json, model.safetensors, special_tokens_map.json, tokenizer_config.json, tokenizer.json, training_args.bin, vocab.txt.

Modular Structure

The code is organized as follows (within the submission directory):

• src/main.py: Main script for user interaction, training, and inference orchestration.
• src/utils.py: Contains utility functions for data loading (IdiomDataset), F1 score calculation, mapping token labels to word indices, saving predictions, and model downloading.
• src/model_handler.py: Contains functions for training (train_model) and prediction (predict_idioms) for a single language model.

This structure allows for clear separation of concerns and easier maintenance.