This project focuses on medical domain adaptation and evaluation for cross-lingual medical knowledge understanding between English and Japanese. The codebase consists of three main components:
The data generation component creates both training and evaluation datasets for medical domain language models.
The training data pipeline (datasets/generate_train/) creates instruction datasets from medical abstracts through multiple stages, including regex-based instruction creation, LLM-based QA generation, romanization, and instruction formatting.
Please refer to datasets/generate_train/README.md for details of training data generation.
The component (datasets/generate_eval/) creates benchmarks for assessing cross-lingual medical knowledge understanding and distractor quality in multiple-choice questions, based on the proposal of AdaXEval.
AdaXEval follows four steps to generate multi-choice questions given the medical documents, including fact detection, queries crafting, distractor generation and quality filtering. The annotation for generated evaluation are conducted by human, while AI (GPT APIs) are used for assiting annotator understanding the medical knowledge and context. The AI-based evaluation references are only used for evaluation of distractor quality and cross-linugal factuality. The GPT-based annotation used for reference of human annotation are included in ANNOTATIONS.
The training settings component manages model training configurations and execution.
Training configurations are defined in YAML files that specify dataset collections, model parameters, parallelism settings, and hyperparameters.
The training pipeline consists of three stages executed via shell scripts:
The actual training is performed using Megatron-LM with distributed optimization across multiple GPUs.
The project uses a flexible configuration system where experiments can inherit from a default configuration and override specific parameters. This module supports various dataset combinations including monolingual medical corpora, cross-lingual transfer tasks, and instruction tuning datasets. Furthermore, while our paper focus on full-parameter continual training, this module enables both next-word-prediction training, instruction tuning and other specified training paradigm.
The evaluation/ component performs loss-based evaluation using the multiple-choice QA pairs generated by the AdaXEval pipeline.
This module is adapted from the implementation in
nii-nlp/med-eval, with additional modifications to support loss-based comparison and our custom evaluation datasets.
The evaluation scripts and notebook for analysis are all included in evaluation/dataset folder.
Please note that, due to licensing restrictions associated with J-STAGE, the source corpus used to construct our evaluation dataset, we are unable to include the evaluation data in this repository.
This project focuses on medical domain adaptation and evaluation for cross-lingual medical knowledge understanding between English and Japanese. The codebase consists of three main components:
The data generation component creates both training and evaluation datasets for medical domain language models.
The training data pipeline (datasets/generate_train/) creates instruction datasets from medical abstracts through multiple stages, including regex-based instruction creation, LLM-based QA generation, romanization, and instruction formatting.
Please refer to datasets/generate_train/README.md for details of training data generation.
The component (datasets/generate_eval/) creates benchmarks for assessing cross-lingual medical knowledge understanding and distractor quality in multiple-choice questions, based on the proposal of AdaXEval.
AdaXEval follows four steps to generate multi-choice questions given the medical documents, including fact detection, queries crafting, distractor generation and quality filtering. The annotation for generated evaluation are conducted by human, while AI (GPT APIs) are used for assiting annotator understanding the medical knowledge and context. The AI-based evaluation references are only used for evaluation of distractor quality and cross-linugal factuality. The GPT-based annotation used for reference of human annotation are included in ANNOTATIONS.
The training settings component manages model training configurations and execution.
Training configurations are defined in YAML files that specify dataset collections, model parameters, parallelism settings, and hyperparameters.
The training pipeline consists of three stages executed via shell scripts:
The actual training is performed using Megatron-LM with distributed optimization across multiple GPUs.
The project uses a flexible configuration system where experiments can inherit from a default configuration and override specific parameters. This module supports various dataset combinations including monolingual medical corpora, cross-lingual transfer tasks, and instruction tuning datasets. Furthermore, while our paper focus on full-parameter continual training, this module enables both next-word-prediction training, instruction tuning and other specified training paradigm.
The evaluation/ component performs loss-based evaluation using the multiple-choice QA pairs generated by the AdaXEval pipeline.
This module is adapted from the implementation in
nii-nlp/med-eval, with additional modifications to support loss-based comparison and our custom evaluation datasets.
The evaluation scripts and notebook for analysis are all included in evaluation/dataset folder.
Please note that, due to licensing restrictions associated with J-STAGE, the source corpus used to construct our evaluation dataset, we are unable to include the evaluation data in this repository.
