LLM Converter Project

Directories

• adversarial: Contains modified boundary values of standoff annotations using create_adversarial.ipynb.

• analysis: Folder containing generated results and original CoNLL documents.

  • analysis/generated: Raw GPT-3.5 generated results.
    • analysis/generated/false: Zero-shot results.
      • Example: analysis/generated/false/s800/separated: Zero-shot results for S800 separated into individual documents.
    • analysis/generated/true: Few-shot results.
  • analysis/generated_adversarial_gpt3_5: Raw GPT-3.5 generated results with adversarial values. ※ Results generated by GPT-3.5 usually conform to the aforementioned structures
  • analysis/llama3: Raw results generated by Llama3.
    • analysis/llama3/modified: Manually adjusted results from Llama3.
  • analysis/llama3_adversarial: Raw results from adversarial inputs generated by Llama3. ※ Results generated by Llama3 usually conform to the aforementioned structures
  • analysis/original: Original separated CoNLL documents.
  • gpt-3.5-turbo-0125: Raw GPT-3.5 results using new script.
  • gpt-4o-2024-05-13: Raw GPT-4o results using new script.
  • analysis/llama3_new: Raw results generated by Llama3 with new prompt.
  • analysis/temp: Directory containing manually adjusted results generated by GPT-3.5, GPT-4o, and Llama3. Folders are verbosely named, indicating whether the results were from few-shot (true), or zero-shot (false) setting. Also, this naming convention shows whether the results were generated using adversarial inputs or not.
  • analysis/temp2: Same as temp but with new prompt.

• archive: Archive folder.

• cache: Cache folder containing results generated by all models with all settings.

• corpora_sandbox: Folder containing all documents generated by rule-based converters.

• example: Example folder containing example documents that are used in the few-shot approach.

• generated: Raw results generated by GPT-3.5 (deprecated).

• json_corpora: Inputs formated into json files that for Llama3.

• json_corpora_adversarial: Inputs formated into json files containing adversarial values that for Llama3.

• json_corpora_new: Inputs formated into json files that for Llama3 with new prompt.

• levenshtein_distance: Unmodified datasets that were tuned for the calculation of Levenshtein distance with the ones in temp.

• log: Log files from sbatch.

• unmodified_corpora: Untouched datasets.

Python Scripts

• gptconverter.py: GPT-3.5/4o converter.

  • Example: $ python gptconverter.py --input=corpora_sandbox/copious_ascii/dev/ --output_dir=generated/false/copious_ascii --with_example=False

• conllseparator.py: Separated GPT-3.5 results.

  • Example: $ python conllseparator.py --input=analysis/generated_adversarial_gpt3_5/true/bb_ascii/output.bio --output=analysis/gpt3_5_adversarial_bb_ascii_true/

• llama3converter.py: Llama3 converter.

  • Example: $ python llama3converter.py --input=bb_ascii --with_example=true

• convert_tojson.ipynb: Converts datasets into json format suitable for Llama3.

• create_adversarial.ipynb: Augments some dataset files to have incorrect boundary values.

Shell Scripts

• diff_check.sh: Calculates difference for each file of two directories. 1. File name, 2. number of different lines, 3. number of lines in the file from the first directory, percentage of difference (2.)/(3.). The results are copied directly to Apple Numbers for further analysis.

  • Example:

  $ ./diff_check.sh analysis/original/copious_ascii/dev/ analysis/temp/llama3_adversarial_copious_ascii_true/
  107004_English_78294_34354169_1912.bio	56	531 	10.546%
  107004_English_78294_34354176_1912.bio	114	443 	25.734%
  107004_English_78294_34354179_1912.bio	59	516 	11.434%
  107004_English_78294_34354180_1912.bio	105	529 	19.849%
  107004_English_78294_34354214_1912.bio	101	624 	16.186%
  110038_English_49236_35517643_2000.bio	49	681 	7.195%
  11358225.bio	44	753 	5.843%
  11364520.bio	39	394 	9.898%
  28831271.bio	1031	1259 	81.890%
  37190476.bio	55	790 	6.962%
  

• gptrunner.sh: Run experiments via sbatch for GPT-3.5/4o with all datasets and settings.

  • Example: $ ./gptrunner.sh

• llama3runner.sh: Run experiments via sbatch for Llama3 with all datasets and settings.

  • Example: $ ./llama3runner.sh