CakeCN/hsa
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
CakeCN
2024-12-03 16:21:19 +08:00
commit 4198ca63b1
975 changed files with 333413 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

313
AlgoriAgent/scripts/README.md Normal file
View File

@@ -0,0 +1,313 @@
# Set up Local Model API Serving
AgentScope supports developers to build their local model API serving with different inference engines/libraries.
This document will introduce how to fast build their local API serving with provided scripts.
Table of Contents
=================
- [Set up Local Model API Serving](#set-up-local-model-api-serving)
- [Table of Contents](#table-of-contents)
- [Local Model API Serving](#local-model-api-serving)
- [ollama](#ollama)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving)
- [How to use in AgentScope](#how-to-use-in-agentscope)
- [Flask-based Model API Serving](#flask-based-model-api-serving)
- [With Transformers Library](#with-transformers-library)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-1)
- [How to use in AgentScope](#how-to-use-in-agentscope-1)
- [Note](#note)
- [With ModelScope Library](#with-modelscope-library)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-2)
- [How to use in AgentScope](#how-to-use-in-agentscope-2)
- [Note](#note-1)
- [FastChat](#fastchat)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-3)
- [Supported Models](#supported-models)
- [How to use in AgentScope](#how-to-use-in-agentscope-3)
- [vllm](#vllm)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-4)
- [Supported models](#supported-models-1)
- [How to use in AgentScope](#how-to-use-in-agentscope-4)
- [Model Inference API](#model-inference-api)
## Local Model API Serving
### ollama
[ollama](https://github.com/ollama/ollama) is a CPU inference engine for LLMs. With ollama, developers can build their local model API serving without GPU requirements.
#### Install Libraries and Set up Serving
- First, install ollama in its [official repository](https://github.com/ollama/ollama) based on your system (e.g. macOS, windows or linux).
- Follow ollama's [guidance](https://github.com/ollama/ollama) to pull or create a model and start its serving. Taking llama2 as an example, you can run the following command to pull the model files.
```bash
ollama pull llama2
```
#### How to use in AgentScope
In AgentScope, you can use the following model configurations to load the model.
- For ollama Chat API:
```python
{
"config_name": "my_ollama_chat_config",
"model_type": "ollama_chat",
# Required parameters
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
# Optional parameters
"options": { # Parameters passed to the model when calling
# e.g. "temperature": 0., "seed": 123,
},
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
}
```
- For ollama generate API:
```python
{
"config_name": "my_ollama_generate_config",
"model_type": "ollama_generate",
# Required parameters
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
# Optional parameters
"options": { # Parameters passed to the model when calling
# "temperature": 0., "seed": 123,
},
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
}
```
- For ollama embedding API:
```python
{
"config_name": "my_ollama_embedding_config",
"model_type": "ollama_embedding",
# Required parameters
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
# Optional parameters
"options": { # Parameters passed to the model when calling
# "temperature": 0., "seed": 123,
},
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
}
```
### Flask-based Model API Serving
[Flask](https://github.com/pallets/flask) is a lightweight web application
framework. It is easy to build a local model API serving with Flask.
Here we provide two Flask examples with Transformers and ModelScope library,
respectively. You can build your own model API serving with few modifications.
#### With Transformers Library
##### Install Libraries and Set up Serving
Install Flask and Transformers by following command.
```bash
pip install flask torch transformers accelerate
```
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
set up the model API serving by running the following command.
```shell
python flask_transformers/setup_hf_service.py \
--model_name_or_path meta-llama/Llama-2-7b-chat-hf \
--device "cuda:0" \
--port 8000
```
You can replace `meta-llama/Llama-2-7b-chat-hf` with any model card in
huggingface model hub.
##### How to use in AgentScope
In AgentScope, you can load the model with the following model configs: `./flask_transformers/model_config.json`.
```json
{
"model_type": "post_api_chat",
"config_name": "flask_llama2-7b-chat-hf",
"api_url": "http://127.0.0.1:8000/llm/",
"json_args": {
"max_length": 4096,
"temperature": 0.5
}
}
```
##### Note
In this model serving, the messages from post requests should be in **STRING
format**. You can use [templates for chat model](https://huggingface.co/docs/transformers/main/chat_templating) in
transformers with a little modification in `./flask_transformers/setup_hf_service.py`.
#### With ModelScope Library
##### Install Libraries and Set up Serving
Install Flask and modelscope by following command.
```bash
pip install flask torch modelscope
```
Taking model `modelscope/Llama-2-7b-chat-ms` and port `8000` as an example,
to set up the model API serving, run the following command.
```bash
python flask_modelscope/setup_ms_service.py \
--model_name_or_path modelscope/Llama-2-7b-chat-ms \
--device "cuda:0" \
--port 8000
```
You can replace `modelscope/Llama-2-7b-chat-ms` with any model card in
modelscope model hub.
##### How to use in AgentScope
In AgentScope, you can load the model with the following model configs:
`flask_modelscope/model_config.json`.
```json
{
"model_type": "post_api_chat",
"config_name": "flask_llama2-7b-chat-ms",
"api_url": "http://127.0.0.1:8000/llm/",
"json_args": {
"max_length": 4096,
"temperature": 0.5
}
}
```
##### Note
Similar with the example of transformers, the messages from post requests
should be in **STRING format**.
### FastChat
[FastChat](https://github.com/lm-sys/FastChat) is an open platform that
provides quick setup for model serving with OpenAI-compatible RESTful APIs.
#### Install Libraries and Set up Serving
To install FastChat, run
```bash
pip install "fschat[model_worker,webui]"
```
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
to set up model API serving, run the following command to set up model serving.
```bash
bash fastchat/fastchat_setup.sh -m meta-llama/Llama-2-7b-chat-hf -p 8000
```
#### Supported Models
Refer to
[supported model list](https://github.com/lm-sys/FastChat/blob/main/docs/model_support.md#supported-models)
of FastChat.
#### How to use in AgentScope
Now you can load the model in AgentScope by the following model config: `fastchat/model_config.json`.
```json
{
"model_type": "openai_chat",
"config_name": "fastchat_llama2-7b-chat-hf",
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8000/v1/"
},
"generate_args": {
"temperature": 0.5
}
}
```
### vllm
[vllm](https://github.com/vllm-project/vllm) is a high-throughput inference
and serving engine for LLMs.
#### Install Libraries and Set up Serving
To install vllm, run
```bash
pip install vllm
```
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
to set up model API serving, run
```bash
./vllm/vllm_setup.sh -m meta-llama/Llama-2-7b-chat-hf -p 8000
```
#### Supported models
Please refer to the
[supported models list](https://docs.vllm.ai/en/latest/models/supported_models.html)
of vllm.
#### How to use in AgentScope
Now you can load the model in AgentScope by the following model config: `vllm/model_config.json`.
```json
{
"model_type": "openai_chat",
"config_name": "vllm_llama2-7b-chat-hf",
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8000/v1/"
},
"generate_args": {
"temperature": 0.5
}
}
```
## Model Inference API
Both [Huggingface](https://huggingface.co/docs/api-inference/index) and
[ModelScope](https://www.modelscope.cn) provide model inference API,
which can be used with AgentScope post api model wrapper.
Taking `gpt2` in HuggingFace inference API as an example, you can use the
following model config in AgentScope.
```json
{
"model_type": "post_api_chat",
"config_name": "gpt2",
"headers": {
"Authorization": "Bearer {YOUR_API_TOKEN}"
},
"api_url": "https://api-inference.huggingface.co/models/gpt2"
}
```