Project_baseon_LMsEvaluator

An experimental web platform based on LMsEvaluator for evaluating language-model security and privacy risks. The repository combines a Python evaluation backend, a Vue/Ant Design admin frontend, and report/dashboard assets for recording attacks, defenses, logs, and generated reports.

Features

• User registration, login, token refresh, profile management, and role-based user administration.
• Attack task creation and execution through a Flask API.
• Real-time log streaming with Flask-SocketIO.
• Attack record management, favorites, and report generation/reading.
• LMsEvaluator-based model evaluation for tasks such as text classification, question answering, multiple choice, pretraining, and Chinese NER.
• Security and privacy evaluation modules including adversarial attacks, backdoor attacks, poisoning attacks, gradient inversion, model inversion, and model stealing.
• Vue 3 frontend built from Stepin Template with pages for login, workplace, users, attack tables, records, and personal settings.

Repository Structure

.
├── LMsEvaluator/              # Python backend and model-evaluation code
│   ├── main.py                # Flask + SocketIO web API entrypoint
│   ├── run_multi.py           # Background attack execution
│   ├── attack/                # Attack and defense implementations
│   ├── model/                 # Basic BERT and downstream-task models
│   ├── tasks/                 # Training/evaluation task wrappers
│   ├── user_config/           # Generated per-user attack configs
│   ├── web_databse/           # SQLite helpers and sample data
│   └── README.md              # Detailed LMsEvaluator module notes
├── stepin-template-main/      # Vue 3 + Ant Design frontend
├── material-dashboard/        # Static dashboard pages and generated reports
├── new_image/                 # Image assets
└── package.json               # Root Node dependency for md-to-pdf

Backend Setup

The backend is a Python 3.9 project. A full dependency snapshot is available in LMsEvaluator/requirement.txt, and a Conda environment export is available in LMsEvaluator/environment.yaml.

cd LMsEvaluator
conda create -n LMsEvaluation python=3.9
conda activate LMsEvaluation
pip install -r requirement.txt
python main.py

By default, LMsEvaluator/main.py starts the SocketIO server on 0.0.0.0:5999.

Frontend Setup

The frontend lives in stepin-template-main and uses Vue 3, Vite, Pinia, Ant Design Vue, and Stepin.

cd stepin-template-main
yarn install
yarn dev

The current frontend source contains hard-coded API calls to http://127.0.0.1:46666/api/.... Before local testing, align the frontend API address with the backend port, or update the backend startup port to match the frontend.

Running Evaluation Tasks

Core evaluation behavior is controlled by YAML files under LMsEvaluator/, especially:

• RelatedConfig.yaml for the main model, task, dataset, and attack list configuration.
• config4msa.yaml for model-stealing attack experiments.
• user_config/*.yaml for per-user generated web-task configurations.

Typical backend execution flow:

1. The frontend calls /api/attack_list and /api/execute_attack.
2. The backend creates or updates user-specific YAML config files.
3. run_multi.py launches test4transformers.run_pipeline.
4. Logs are written under LMsEvaluator/logs/.
5. Results are extracted and stored in the SQLite database.
6. Reports can be generated and read through /api/generate_report and /api/read_report.

Data and Model Preparation

Large datasets and pretrained models are not fully documented at the repository root. See LMsEvaluator/README.md for the original LMsEvaluator data/model notes, including BERT model paths and supported downstream datasets.

Common expected assets include:

• pretrained models under LMsEvaluator/LMs/
• datasets under LMsEvaluator/datasets/
• generated outputs under LMsEvaluator/modelOutput/, LMsEvaluator/evaluationResult/, or configured output paths

Notes

• This project is research-oriented and may require manual path, dataset, model, CUDA, and port configuration before running on a new machine.
• Before public deployment, move mail credentials, JWT secrets, API keys, and other sensitive configuration into environment variables or a private config file.
• Some bundled frontend and dashboard files come from template projects; the project-specific backend/evaluation logic is mainly under LMsEvaluator/ and stepin-template-main/src/pages/.