README
FinalThesis_12Aug
Upload Again Attempt
File Structure
Adapters
Code for Using PEFT type as part of the framework
Cleaned Datasets
Core Datasets in format of sentence and label for easier processing. Raw datasets are available via Huggingface.
HPO
Ray Cluster Test
MetaDataCreation
Relevant to training the surrogate. Contains code to make the performance matrix, training data, train, optimise and predict with the model.
RayCode
Test stuff for Ray, trying the library and available optimisers of Ray Tune.
BoHBCode
HPO:
Adjust bohb_ray_slurm_template.sh as needed and execute it. It starts a RayCluster before starting the BoHB runner. The BoHB runner, bohb_runner.py starts multiple processes for HPO and results are stored. There is no Multiprocessing clash due to Ray having a separate environment. Parameter space is defined in bohb_ray_cluster.py
Search Space
| Name | Type | Range/Value |
|---|---|---|
| max seq length | categorical | 128, 256, 512 |
| train batch size | categorical | 4, 8, 16 |
| model name | categorical | Bert base uncased, Bert base multilingual cased |
| Bert base german cased oldvocab,gottbert base | ||
| TinyBERT General 4L de, Alberti bert base multilingual cased | ||
| Distilbert base german cased | ||
| optimizer | categorical | Adam, AdamW, SGD , RAdam |
| learning rate | float | [2e-5 7e-5] log |
| scheduler | categorical | linear with warmup , cosine with warmup |
| cosine with hard restarts with warmup, polynomial decay with warmup | ||
| weight decay | categorical | [1e-5, 1e-3] log |
| warmup steps | categorical | 10, 100, 500 |
| gradient accumulation steps | categorical | 1, 4, 8, 16 |
| adam epsilon | float | [1e-8, 1e-6] log |
Surrogate Training
Run create_training_data.py to make the cost matrix, create_training_data.py to make. the training data for the model.
The files to train (metamodel_train.py) and optimise the model (metamodel_train.py) are provided. metamodel_predict.py is used to test.
TinyBERT
Code for distillation of TinyBERT. Consists of 2 stage distillation:
General for the Embedded layers
Finetuning for the predicted layers.
Workflow
- Run the search for the incubent pipelines using
bohb_ray_slurm_template.sh - Create the performance matrix based on the results
- Create the Training data using
create_training_data.py - Train, optimise and predict using
metamodel_train.py,metamodel_train.pyandmetamodel_predict.py.
Extension for new Datasets
- Add the dataset processing to
data_modules.py.
Extension for new Pipeline options
- Add the changes to the parameter space in
bohb_ray_cluster.py
Contributors
6 commits
Similar repos
Languages
Python
96.3%
Shell
3.7%
README
FinalThesis_12Aug
Upload Again Attempt
File Structure
Adapters
Code for Using PEFT type as part of the framework
Cleaned Datasets
Core Datasets in format of sentence and label for easier processing. Raw datasets are available via Huggingface.
HPO
Ray Cluster Test
MetaDataCreation
Relevant to training the surrogate. Contains code to make the performance matrix, training data, train, optimise and predict with the model.
RayCode
Test stuff for Ray, trying the library and available optimisers of Ray Tune.
BoHBCode
HPO:
Adjust bohb_ray_slurm_template.sh as needed and execute it. It starts a RayCluster before starting the BoHB runner. The BoHB runner, bohb_runner.py starts multiple processes for HPO and results are stored. There is no Multiprocessing clash due to Ray having a separate environment. Parameter space is defined in bohb_ray_cluster.py
Search Space
| Name | Type | Range/Value |
|---|---|---|
| max seq length | categorical | 128, 256, 512 |
| train batch size | categorical | 4, 8, 16 |
| model name | categorical | Bert base uncased, Bert base multilingual cased |
| Bert base german cased oldvocab,gottbert base | ||
| TinyBERT General 4L de, Alberti bert base multilingual cased | ||
| Distilbert base german cased | ||
| optimizer | categorical | Adam, AdamW, SGD , RAdam |
| learning rate | float | [2e-5 7e-5] log |
| scheduler | categorical | linear with warmup , cosine with warmup |
| cosine with hard restarts with warmup, polynomial decay with warmup | ||
| weight decay | categorical | [1e-5, 1e-3] log |
| warmup steps | categorical | 10, 100, 500 |
| gradient accumulation steps | categorical | 1, 4, 8, 16 |
| adam epsilon | float | [1e-8, 1e-6] log |
Surrogate Training
Run create_training_data.py to make the cost matrix, create_training_data.py to make. the training data for the model.
The files to train (metamodel_train.py) and optimise the model (metamodel_train.py) are provided. metamodel_predict.py is used to test.
TinyBERT
Code for distillation of TinyBERT. Consists of 2 stage distillation:
General for the Embedded layers
Finetuning for the predicted layers.
Workflow
- Run the search for the incubent pipelines using
bohb_ray_slurm_template.sh - Create the performance matrix based on the results
- Create the Training data using
create_training_data.py - Train, optimise and predict using
metamodel_train.py,metamodel_train.pyandmetamodel_predict.py.
Extension for new Datasets
- Add the dataset processing to
data_modules.py.
Extension for new Pipeline options
- Add the changes to the parameter space in
bohb_ray_cluster.py
Similar repos
Contributors
6 commits
Languages
Python
96.3%
Shell
3.7%