Data Copilot

🎯 What is Data Copilot ?

Data Copilot is AI assistant for data scientist and data analyst for suggest snippets of codes using implementation of Intent and Entity classifier. Data Copilot use DIET for classify user requests (intent and entity) and provide text output in the form of code suggestions. DIET its have means Dual Intent and Entity Transformer so the model can predict the intent and entity simultaneously from the input of text.

🚀 How to Use

Inference using Google Colab :

1. Clone our repository

% git clone https://github.com/rahmadai/githubcopilot.git

2. Install requirements

% cd datacopilot
% pip install -r requirements.txt

3. Download model

%  wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1QstKji0PimR9w0TJ_0HR1E9xcTSs9lJ7' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1QstKji0PimR9w0TJ_0HR1E9xcTSs9lJ7" -O epoch9.ckpt && rm -rf /tmp/cookies.txt

4. Inference
Example code for inference

import sys
import pprint

sys.path.append("datacopilot")
from DIET import Inferencer
from DIET.postprocessing import post_processing

inferencer = Inferencer(
        "model/checkpoints/epoch=9.ckpt"
    )

text = "put the latest dataset.csv on gdrive using pandas into my program"
result = post_processing(inferencer.inference(text, intent_topk=1))
pprint.pprint(result)

Output

{'entities': [{'end': 19,
               'entity': 'datasate_name',
               'start': 15,
               'value': 'dataset.csv'},
              {'end': 31,
               'entity': 'directory',
               'start': 30,
               'value': 'gdrive'},
              {'end': 48,
               'entity': 'lib_name',
               'start': 43,
               'value': 'pandas'}],
 'intent': {'confidence': 0.9954274892807007, 'name': 'dataset/load'},
 'intent_ranking': [{'confidence': 0.9954274892807007, 'name': 'dataset/load'}],
 'text': 'put the latest dataset.csv on gdrive using pandas into my program'}

📊 Model Performance

Here is the model performance by fine tuning it using BERT pretrained model and custom dataset. Evalution metrics is calculated use weighted avg.

⚙️ WIP List

✔️ Load and save dataset
✔️ Install, import library
✔️ Dataframe manipulation (rename)
✔️ Dataframe manipulation (update value)
✔️ Handling missing value
❌ Cleaning outlier
❌ AutoVisualize data
❌ Recommender Transformation (Normalization)
❌ Deploy on server
❌ Implement MLOps for insert new dataset

📄 License

MIT License
Pull Request are open
Written by Rahmad Kurniawan, 2022

Many thanks to Ilham Fazri (my co-workers at widya wicara) for the very helpful discussion

🖤 Acknowledgements

• DIET-pytorch
• BERT pretrained
• DIET: Lightweight Language Understanding for Dialogue Systems