Copilo.sh

Your personal assistant directly integrated into your terminal.

Description

Copilo.sh is a wrapper function to add to your .bashrc or .zshrc file. Accompanied by a FastAPI local server, running a LM on CPU. It will catch all the errors you make in your terminal (non-zero exit code) and call the LM to generate a response to help you solve the error.

Pre-requisites

Set up the environnements:

Create a conda environnement:

conda create -n copilosh_env python=3.8 poetry=1.4.0 --y
conda activate copilosh_env

poetry lock --no-update # TO AVOID AUTO-UPDATE
poetry install

Create a cache directory, to store your models:

mkdir cache

Installation

Add the copilosh wrapper function to your shell profile file (e.g. ~/.bashrc, ~/.zshrc, etc.):

Ubuntu/Unix:

cat copilosh.sh >> ~/.bashrc
source ~/.bashrc .

MacOS:

cat copilosh.sh >> ~/.zshrc
source ~/.zshrc .

Run the assistant server:

uvicorn app:app --host 0.0.0.0 --port 8082

Usage

Use any command in your terminal, if an error occurs, the assistant will suggest a solution.

Evaluation

├── create_response_dataset.py      # creates a yaml file with all the generated responses for each model, for each
│                                   # system prompt, each 'pre_prompt' and each error
├── create_test_suite.py            # runs the error scripts and creates the test suite in a yaml file
├── evaluation
│   ├── evaluation.ipynb            # notebook for statistical analysis of the evaluation results
│   ├── evaluation.py               # script to evaluate the models (BLEU, ROUGE, BERTScore, response time, rank)
│   ├── expected_responses.yml      # contains baseline responses for the 24 common errors by ChatGPT
│   ├── model_ranker.py             # tool to ease the model ranking
│   ├── preprompt_ranker.py         # tool to ease the pre-prompt ranking
│   └── system_prompt_ranker.py     # tool to ease the system prompt ranking
├── models
│   └── slm.py                      # contains the models to generate responses (11)
├── pre_prompts.yml                 # contains the pre-prompts (few-shot, CoT) (2)
├── system_prompts.yml              # contains the system prompts (6)
├── test_models.py                  # pytest to one-time test the models in the models folder
└── test_scripts                    # contains the scripts to get the 24 common errors 
    └──...

Test procedure

1. Go into tests folder

cd tests

2. Create the test suite

python create_test_suite.py

A file test_suite_results.yml will be created in the tests folder. Containing a list of yaml object:

• command: the command that has been run (content of the script)
• id: the id of the error (created by the script)
• return_code: the exit code of the command
• script: the script that has been run
• stderr: the error output of the command
• stdout: the output of the command

3. Generate all the responses

python create_response_dataset.py

A file response_dataset.yml will be created in the tests folder. Containing a list of yaml object:

• error_id: the id of the error
• model_name: the model that has been used
• parsed_response: the response generated by the model
• preprompt_id: the id of the pre-prompt used
• response_id: the id of the response
• system_prompt_id: the id of the system prompt used
• time: the time taken to generate the response

4. Go to the evaluation folder

cd evaluation

5. Run the evaluation

python evaluation.py

A csv file will be created in the evaluation folder.

6. Ranking 6.1. Model ranking

python model_ranker.py

• A number of groups (1 group: error_id, system_prompt_id, preprompt_id) will be asked at the beginning.
• 2 responses will be shown, you have to choose the one you prefer. Type 1 or 2 and press enter.

A csv file will be created in the evaluation folder.

6.2. System prompt ranking

python system_prompt_ranker.py

Same as the model ranking, but you will have to first input the name of the model you prefer.

A csv file will be created in the evaluation folder.

6.3. Pre-prompt ranking

python preprompt_ranker.py

Same as the model ranking, but you will have to first input the name of the model you prefer and the system prompt you prefer.

A csv file will be created in the evaluation folder.

7. Notebook

You can look at the notebook to see the statistical analysis of the evaluation results.