OffresCertificationsFormacodes Project

This project focuses on matching job offers (offres) to certifications (certifications) and the Formacode hierarchy using various machine learning and natural language processing techniques, particularly for the French labor market as described in our paper (in French) "Affinés pour la réussite : Évaluation des représentations pour graphes attribués textuels dans le domaine de la formation professionnelle".

Citation

You can find our paper at https://talnarchives.atala.org/TALN/TALN-2026/10.pdf

If you find our work useful, please also cite the paper:

@inproceedings{Breidenstein-Leang-Labeau:CORIA-TALN:2026,
    author = "Breidenstein, Alicia and Leang, Marguerite and Labeau, Matthieu",
    title = "Affinés pour la réussite : Évaluation des représentations dans le domaine de la formation professionnelle",
    booktitle = "Actes de CORIA-TALN 2026. Actes des 33ème Conférence sur le Traitement Automatique des Langues Naturelles.  Volume 1 : articles scientifiques originaux",
    month = "6",
    year = "2026",
    address = "Nantes, France",
    publisher = "Association pour le Traitement Automatique des Langues",
    pages = "65-90",
    note = "",
    url = "https://talnarchives.atala.org/TALN/TALN-2026/10.pdf"
}

Project Structure

• src/: Source code directory

  • anomaly_histogram.py: functions to generate histograms for anomaly detection
  • classification.py: Main script for classification tasks
  • embeddings.py: Functions for generating text embeddings
  • ERPCamemBERT_utils.py: Functions to help training CamemBERT with the ERP
  • filterDataset.py: to filter the test set and the train set and avoid contamination and redundancies
  • fineTuningClassif.py: Fine-tuning scripts for classification models
  • fineTuningClassif_LoRA_multilabel.py: Fine-tuning scripts for classification models when applying Lora for training Qwen in multilabel mode
  • fineTuningClassif_LoRA_singlelabel.py: Fine-tuning scripts for classification models when applying Lora for training Qwen in multilabel mode
  • fineTuningSim.py: Fine-tuning scripts for similarity models
  • generateDetectAnomalies.py: Scripts for generating and detecting anomalies
  • preprocessing.py : to prepare the data at the beginning
  • simOffreCertif.py: Similarity computation between offers and certifications
  • trainFT.py: Functions to train FastText and W2V on the data
  • utils.py: Utility functions for text processing

• llm_classif_zero_shot_few_shot_anomaly_detection/: Code directory with scripts for in-context learning using Llama3.1:8B and Mistral:7B for zero-shot and few-shot classification, and anomaly detection

• escoxlmr/: Code directory corresponding to pre-training ERP-CamemBERT with Masked Language Modeling (MLM) objective (predicting masked words in the text) and the Entity Relation Prediction (ERP) ob- jective (predicting node relationships).

• data/: Data files

Dependencies

• Python 3.x
• Key libraries: scikit-learn, transformers, sentence-transformers, gensim, nltk, pandas, numpy
• See environment.yml for full conda environment

Installation

1. Clone the repository
2. Create conda environment: conda env create -f environment.yml
3. Activate environment: conda activate OCFenv

Downloading the data

The following data files are available on HuggingFace ([https://huggingface.co/datasets/ABreidenstein/OCFproject]) and should be downloaded into the ./data folder:

• certifsFormacodeFull.csv: contains the codes to the certifications and the formacodes, domains and broad domains linked to them.
• offreCertifFormacode.csv: original data with training offers, certifications and formacode information, without any additional preprocessing
• gd_domaines.csv: contains all the codes for broad domains
• domaines.csv: contains all the codes for domains
• formacodes.csv: contains all the codes for formacodes
• offreCertifFormacode.pkl: this file can be obtained with preprocessing.py. In comparison to certifsFormacodeFull.csv, the text is clean.
• offreCertifFormacodeFiltered_multilab_False.pkl: this file contains our fully cleaned dataset in simple label mode after having been filtered for duplicates and can be obtained with filterDataset.py
• offreCertifFormacodeFiltered_multilab_True.pkl: this file contains our fully cleaned dataset in multilabel mode after having been filtered for duplicates and can be obtained with filterDataset.py

Usage

Here are some examples of how the functions may be used, but other configurations can be obtained with different parameters (refer to the corresponding files to see them).

Preprocessing

Run preprocessing file to generate label encoders (necessary for classification):

cd src
python preprocessing.py

To use FastText trained on the data, this needs to be run before using it on the similarity and classification tasks:

python python trainFT.py --results_file "test.txt" --embeddings "FastTextDT" --epochs 1

This training can take a lot of time.

Classification

Run classification with different embeddings (to use embeddings necessitating fine-tuning, the corresponding fine-tuning needs to be run before):

python classification.py --attribute_level code_grand_domaine --multilabel False --embeddings BOW --results_file test.txt

Similarity Computation

Compute similarities between offers and certifications (to use embeddings necessitating fine-tuning, the corresponding fine-tuning needs to be run before):

python simOffreCertif.py --embeddings BOW --results_file test.txt
python simOffreCertif.py --embeddings SentenceCamemBert --results_file test.txt --finetuningloss multipleNegCosSim --finetuninglevel all

Fine-tuning

Fine-tune models for classification:

python fineTuningClassif.py --results_file test.txt --finetuninglevel code_formacode --multilabel True

Fine-tune models for similarity:

python fineTuningSim.py --results_file test.txt --lossname multipleNegCosSim

Anomaly Detection

Generate and detect anomalies: For anomalies on the certification:

python generateDetectAnomalies.py --anomaly_pct 0.5 --attribute_level code_grand_domaine --multilabel False --embeddings BOW --anomaly_type mauvaiseCertif

For anomalies on the content:

python generateDetectAnomalies.py --anomaly_pct 0.5 --attribute_level code_domaine --multilabel_anomalies True --embeddings W2V --anomaly_type mauvaisContenuEtObjectifs

Launch eval for pretrained ERP-CamemBERT-base (MLM + ERP)on similarity task

srun python launcher_sim.py --embeddings "['EscoCamemBertBase']"

which is equivalent to:

python simOffreCertif.py --embeddings EscoCamemBertBase --results_file ../log/log_eval_erpcamembert_sim.txt

Launch eval for pretrained CamemBert-base (MLM+ERP) for classification task

• Multilabel: False

cd src/
python launcher_classif.py --attribute_level "[code_formacode,code_domaine,code_grand_domaine]" --multilabel "[False]" --embeddings "[EscoCamemBertBase]" --results_dir "results_eval_EscoCamemBertBase"

• Multilabel: True

cd src/
python launcher_classif.py --attribute_level "[code_formacode,code_domaine,code_grand_domaine]" --embeddings "[EscoCamemBertBase]" --results_dir "results_eval_EscoCamemBertBase" --multilabel True

Fine-tune pretrained ERP-CamemBERT-base (MLM+ERP) for classification + evaluate

• Multilabel: False / True, at 3 levels (gran_domaine, domaine, formacode)

cd src/
python launcher_fineTune.py --finetuningalgo fineTuningClassif --finetuninglevel "[code_grand_domaine, code_domaine, code_formacode]" --dataset_filtered "[fullTrainingSet]" --model_name "[EscoCamemBertBase]" --multilabel "[False, True]"

Fine-tune pretrained ERP-CamemBERT-base (MLM+ERP) for similarity + evaluate

python launcher_fineTune.py --finetuningalgo fineTuningSim --finetuninglevel "[all]" --lossname "[multipleNegCosSim]" --dataset_filtered "[fullTrainingSet]" --model_name "[EscoCamemBertBase]"

which is equivalent to

python fineTuningSim.py --results_file ../log/test_general_esco/resultsLogs_esco/2025-09-30_14-19-39.txt --finetuninglevel all --lossname multipleNegCosSim --model_name EscoCamemBertBase --dataset_filtered fullTrainingSet

Fine-tuning Qwen3 Embedding 0.6B for classification with PEFT Low-Rank Adaptation (LoRA) at grand_domaine, domaine, formacode level

We used 1 GPU A100 for 24 hours to make the fine-tuning.

Fine-tuning for single label classification during 10 epochs

python launcher_fineTune_20260415_relaunch_qwenFTclassif_singlelabel.py \
    --finetuningalgo fineTuningClassif_LoRA \
    --finetuninglevel "[code_grand_domaine, code_domaine, formacode]" \
    --dataset_filtered "[trainingSetFilteredPerClass]" \
    --model_name "[Qwen3-Embedding-0.6B]" \
    --use_lora "[True]" \
    --debug_mode "[False]"

Fine-tuning for multi label classification during 8, 8 and 5 epochs for grand_domaine, domaine, formacode level respectively

python launcher_fineTune_20260415_relaunch_qwenFTclassif_multilabel.py \
    --finetuningalgo fineTuningClassif_LoRA_multilabel_pos_weight_wo_log \
    --finetuninglevel "[code_grand_domaine, code_domaine, formacode]" \
    --dataset_filtered "[trainingSetFilteredPerClass]" \
    --model_name "[Qwen3-Embedding-0.6B]" \
    --multilabel "[True]" \
    --max_pos_weight "[500]" \
    --use_lora "[True]" \
    --debug_mode "[False]"

In-Context learning with Llama3.1:8B and Mistral:7B

For this section, all the codes are in src/llm_classif_zero_shot_few_shot_anomaly_detection/.

1. Setup Ollama and download Llama3.1:8B and Mistral:7B

Download Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Then download llama3.1:8b and [mistral:7b].

2. Zero-shot learning - single / multi label classification for grand_domaine and domaine

Example of single label classification for grand_domaine with Llama3.1:8B using English prompts.

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
python llm_classification_zero_shot.py \
          --attribute_level "grand_domaine" \ # "domaine"
          --model_name "llama3.1:8b" \ # "mistral:7b"
          --output_dir "./results/" \
          --shot_type "zsc" \
          --multilabel False \ # True
          --temperature 0 \
          --text_column "textFormation" \
          --language "en" \ # "fr"
          --prefix "exp_llama_multilab_false_zsc_" \
          --verbose False \
          --debugging_mode False

3. Few-shot learning - single label classification for grand_domaine with Llama3.1:8B

For few shot learning, there are 2 options to selection of the examples:

• Option 1: select one sample per class one_per_class

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
python llm_classification_few_shot_clean.py
    --pickle_file_dir "../../data/" \
    --attribute_level "grand_domaine" \
    --model_name "llama3.1:8b" \
    --output_dir "./results/" \
    --shot_type "fsc" \
    --temperature 0 \
    --text_column "textFormation" \
    --multilabel False \
    --language "en" \ # "fr"
    --prefix "exp_llama_multilab_false_fsc_fr_oneperclass_" \ 
    --num_max_retries 2 \
    --is_representative True \ 
    --fsc_select_opt "one_per_class" \ #"random_n"
    --fsc_balance_opt "by_freq" \ 
    --fsc_n_examples 3 \ 
    --fsc_random_seed 42 \ 
    --verbose False \  
    --debugging_mode False

• Option 2: select 3 random samples (with 3 different classes, no collision) random_n

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
python llm_classification_few_shot_clean.py
    --pickle_file_dir "../../data/" \
    --attribute_level "grand_domaine" \
    --model_name "llama3.1:8b" \
    --output_dir "./results/" \
    --shot_type "fsc" \
    --temperature 0 \
    --text_column "textFormation" \
    --multilabel False \
    --language "en" \
    --prefix "exp_llama_multilab_false_fsc_fr_oneperclass_" \
    --num_max_retries 2 \
    --is_representative True \ 
    --fsc_select_opt "random_n" \
    --fsc_balance_opt "by_freq" \ 
    --fsc_n_examples 3 \ 
    --fsc_random_seed 42 \ 
    --verbose False \  
    --debugging_mode False

4. Anomaly detection

Anomalies on certifications (AUROC = 61.2%, AUPR = 56.5%)

The experiment consists of using Llama3.1:8b with English as prompt language to detect anomalies on certifications. The data is dfAnomaliesMauvaiseCertif.csv.

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
ollama serve & bash run_anomaly_detection_with_llm_20260216_anomalycertif.sh

which is equivalent to launch ollama serve in a terminal and then, to open another one to launch :

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
python llm_anomaly_detection.py \
    --anomalies_data_dir "./data/" \
    --anomalies_type "anomalies_certif" \
    --model_name "llama3.1:8b" \
    --output_dir "./results/" \
    --temperature 0 \
    --certif_column "textCertification" \
    --text_column "textFormation" \
    --language "en" \
    --prefix "LLM_AD_certif_" \
    --num_max_retries 2 \
    --verbose False \
    --debugging_mode True

Anomalies on content (AUROC = 60.4%, AUPR = 55.8%)

The experiment consists of using Llama3.1:8b with English as prompt language to detect anomalies on content. The data is dfAnomaliesdfANomaliesMauvaisContenuEtObjectifs.csv.

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
ollama serve & bash run_anomaly_detection_with_llm_20260211_anomalycontent.sh

which is equivalent to launch ollama serve in a terminal and then, to open another one to launch :

cd src/llm_classif_zero_shot_few_shot_anomaly_detection
python llm_anomaly_detection.py \
    --anomalies_data_dir "./data/" \
    --anomalies_type "anomalies_content" \
    --model_name "llama3.1:8b" \
    --output_dir "./results/" \
    --temperature 0 \
    --certif_column "textCertification" \
    --text_column "textFormation" \
    --language "en" \
    --prefix "LLM_AD_content_" \
    --num_max_retries 2 \
    --verbose False \
    --debugging_mode True

🎓 Citation

TALN 2026

📜 License

Apache 2.0 ? Other one ?

🙏 Acknowledgments

This work was supported by Hi! PARIS and by the ANR/France 2030 program (ANR-23-IACL-0005). We would like to acknowledge Mike Zhang and Tim Luka Horstmann for their precious help !

📧 Contacts

For questions, open an issue or contact: