Small-Models-Big-Tasks

This repository contains the code for running inference with various language models mentioned in the paper and evaluating their performance on the XLAM dataset for function calling tasks.

Project Structure

The project consists of the following main files and folders:

• model_edge_inferences - Inference responses for edge experiments
• model_server_inferences - Inference responses for server experiments
• salesforce_xlam - Dataset used for experiments
• finetuned_models - Finetuned model adapters
• finetuned_inference_scripts: Files needed for inferencing from the finetuned models.
• finetuning_script: Files needed to finetuned the models
• edge_inference.py: Run inference using server-based models (e.g., llama.cpp)
• evaluate.py: Evaluate model outputs against ground truth
• inference.py: Run inference using Hugging Face models on GPU
• prompt_utils.py: Contains prompt construction utilities
• prompt_utils_prompt_injection.py: Contains prompt injection-based utilities
• env.yml: Packages required to be installed

Setup and Requirements

Prerequisites

1. Hugging Face account with API token
2. To install the required packages, run the following command

conda env create -f env.yml

3. For edge_inference.py, a running llama.cpp server is required

Running Inference

Using Hugging Face models (inference.py)

This script loads models from Hugging Face and runs inference on the XLAM dataset:

# Basic zero-shot inference
python inference.py

# Few-shot inference
python inference.py --few_shot

# With prompt injection
python inference.py --prompt_injection

# Combining flags
python inference.py --few_shot --prompt_injection

Using edge models (edge_inference.py)

The GGUF models were built using the ggml-org/gguf-my-repo space on hugging face (https://huggingface.co/spaces/ggml-org/gguf-my-repo). Upload the model (merged with adapter in case of finetuned models) to a hugging face repository and provide that repository and the quantization type (Q4_K_M) on gguf-my-repo to get the gguf model. The script connects to a locally running llama.cpp server to run inference:

# First start a llama.cpp server (example)
# ./path/to/llama-server -m models/7B/ggml-model.gguf -c 4096 --host 127.0.0.1 --port 8080 --no-webui

# Then run inference
python edge_inference.py

# Support same flags as inference.py
python edge_inference.py --few_shot
python edge_inference.py --prompt_injection

You can modify the model name and test size directly in the script at the top:

model_name = "deepseek"  # Change to your model name
test_size = 100          # Change test size

Running Evaluation

After running inference, you can evaluate the results:

python evaluate.py --test_output_file [path_to_output_file] --output_dir [directory_for_results]

# Example
python evaluate.py --test_output_file stablecode3b_xlam/stablecode3b_xlam_zeroshot__100.json --output_dir stablecode3b_xlam/

Inference through different models

• To use different models, modify the model loading section in inference.py
• To change the dataset, modify the dataset loading section
• To modify prompts, edit the prompt_utils.py or prompt_utils_prompt_injection.py files