314 lines
9.4 KiB
Markdown
314 lines
9.4 KiB
Markdown
# Set up Local Model API Serving
|
|
|
|
AgentScope supports developers to build their local model API serving with different inference engines/libraries.
|
|
This document will introduce how to fast build their local API serving with provided scripts.
|
|
|
|
Table of Contents
|
|
=================
|
|
|
|
- [Set up Local Model API Serving](#set-up-local-model-api-serving)
|
|
- [Table of Contents](#table-of-contents)
|
|
- [Local Model API Serving](#local-model-api-serving)
|
|
- [ollama](#ollama)
|
|
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving)
|
|
- [How to use in AgentScope](#how-to-use-in-agentscope)
|
|
- [Flask-based Model API Serving](#flask-based-model-api-serving)
|
|
- [With Transformers Library](#with-transformers-library)
|
|
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-1)
|
|
- [How to use in AgentScope](#how-to-use-in-agentscope-1)
|
|
- [Note](#note)
|
|
- [With ModelScope Library](#with-modelscope-library)
|
|
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-2)
|
|
- [How to use in AgentScope](#how-to-use-in-agentscope-2)
|
|
- [Note](#note-1)
|
|
- [FastChat](#fastchat)
|
|
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-3)
|
|
- [Supported Models](#supported-models)
|
|
- [How to use in AgentScope](#how-to-use-in-agentscope-3)
|
|
- [vllm](#vllm)
|
|
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-4)
|
|
- [Supported models](#supported-models-1)
|
|
- [How to use in AgentScope](#how-to-use-in-agentscope-4)
|
|
- [Model Inference API](#model-inference-api)
|
|
|
|
## Local Model API Serving
|
|
|
|
### ollama
|
|
|
|
[ollama](https://github.com/ollama/ollama) is a CPU inference engine for LLMs. With ollama, developers can build their local model API serving without GPU requirements.
|
|
|
|
#### Install Libraries and Set up Serving
|
|
|
|
- First, install ollama in its [official repository](https://github.com/ollama/ollama) based on your system (e.g. macOS, windows or linux).
|
|
|
|
- Follow ollama's [guidance](https://github.com/ollama/ollama) to pull or create a model and start its serving. Taking llama2 as an example, you can run the following command to pull the model files.
|
|
|
|
```bash
|
|
ollama pull llama2
|
|
```
|
|
|
|
#### How to use in AgentScope
|
|
|
|
In AgentScope, you can use the following model configurations to load the model.
|
|
|
|
- For ollama Chat API:
|
|
|
|
```python
|
|
{
|
|
"config_name": "my_ollama_chat_config",
|
|
"model_type": "ollama_chat",
|
|
|
|
# Required parameters
|
|
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
|
|
|
|
# Optional parameters
|
|
"options": { # Parameters passed to the model when calling
|
|
# e.g. "temperature": 0., "seed": 123,
|
|
},
|
|
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
|
|
}
|
|
```
|
|
|
|
- For ollama generate API:
|
|
|
|
```python
|
|
{
|
|
"config_name": "my_ollama_generate_config",
|
|
"model_type": "ollama_generate",
|
|
|
|
# Required parameters
|
|
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
|
|
|
|
# Optional parameters
|
|
"options": { # Parameters passed to the model when calling
|
|
# "temperature": 0., "seed": 123,
|
|
},
|
|
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
|
|
}
|
|
```
|
|
|
|
- For ollama embedding API:
|
|
|
|
```python
|
|
{
|
|
"config_name": "my_ollama_embedding_config",
|
|
"model_type": "ollama_embedding",
|
|
|
|
# Required parameters
|
|
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
|
|
|
|
# Optional parameters
|
|
"options": { # Parameters passed to the model when calling
|
|
# "temperature": 0., "seed": 123,
|
|
},
|
|
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
|
|
}
|
|
```
|
|
|
|
### Flask-based Model API Serving
|
|
|
|
[Flask](https://github.com/pallets/flask) is a lightweight web application
|
|
framework. It is easy to build a local model API serving with Flask.
|
|
|
|
Here we provide two Flask examples with Transformers and ModelScope library,
|
|
respectively. You can build your own model API serving with few modifications.
|
|
|
|
#### With Transformers Library
|
|
|
|
##### Install Libraries and Set up Serving
|
|
|
|
Install Flask and Transformers by following command.
|
|
|
|
```bash
|
|
pip install flask torch transformers accelerate
|
|
```
|
|
|
|
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
|
|
set up the model API serving by running the following command.
|
|
|
|
```shell
|
|
python flask_transformers/setup_hf_service.py \
|
|
--model_name_or_path meta-llama/Llama-2-7b-chat-hf \
|
|
--device "cuda:0" \
|
|
--port 8000
|
|
```
|
|
|
|
You can replace `meta-llama/Llama-2-7b-chat-hf` with any model card in
|
|
huggingface model hub.
|
|
|
|
##### How to use in AgentScope
|
|
|
|
In AgentScope, you can load the model with the following model configs: `./flask_transformers/model_config.json`.
|
|
|
|
```json
|
|
{
|
|
"model_type": "post_api_chat",
|
|
"config_name": "flask_llama2-7b-chat-hf",
|
|
"api_url": "http://127.0.0.1:8000/llm/",
|
|
"json_args": {
|
|
"max_length": 4096,
|
|
"temperature": 0.5
|
|
}
|
|
}
|
|
```
|
|
|
|
##### Note
|
|
|
|
In this model serving, the messages from post requests should be in **STRING
|
|
format**. You can use [templates for chat model](https://huggingface.co/docs/transformers/main/chat_templating) in
|
|
transformers with a little modification in `./flask_transformers/setup_hf_service.py`.
|
|
|
|
#### With ModelScope Library
|
|
|
|
##### Install Libraries and Set up Serving
|
|
|
|
Install Flask and modelscope by following command.
|
|
|
|
```bash
|
|
pip install flask torch modelscope
|
|
```
|
|
|
|
Taking model `modelscope/Llama-2-7b-chat-ms` and port `8000` as an example,
|
|
to set up the model API serving, run the following command.
|
|
|
|
```bash
|
|
python flask_modelscope/setup_ms_service.py \
|
|
--model_name_or_path modelscope/Llama-2-7b-chat-ms \
|
|
--device "cuda:0" \
|
|
--port 8000
|
|
```
|
|
|
|
You can replace `modelscope/Llama-2-7b-chat-ms` with any model card in
|
|
modelscope model hub.
|
|
|
|
##### How to use in AgentScope
|
|
|
|
In AgentScope, you can load the model with the following model configs:
|
|
`flask_modelscope/model_config.json`.
|
|
|
|
```json
|
|
{
|
|
"model_type": "post_api_chat",
|
|
"config_name": "flask_llama2-7b-chat-ms",
|
|
"api_url": "http://127.0.0.1:8000/llm/",
|
|
"json_args": {
|
|
"max_length": 4096,
|
|
"temperature": 0.5
|
|
}
|
|
}
|
|
```
|
|
|
|
##### Note
|
|
|
|
Similar with the example of transformers, the messages from post requests
|
|
should be in **STRING format**.
|
|
|
|
### FastChat
|
|
|
|
[FastChat](https://github.com/lm-sys/FastChat) is an open platform that
|
|
provides quick setup for model serving with OpenAI-compatible RESTful APIs.
|
|
|
|
#### Install Libraries and Set up Serving
|
|
|
|
To install FastChat, run
|
|
|
|
```bash
|
|
pip install "fschat[model_worker,webui]"
|
|
```
|
|
|
|
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
|
|
to set up model API serving, run the following command to set up model serving.
|
|
|
|
```bash
|
|
bash fastchat/fastchat_setup.sh -m meta-llama/Llama-2-7b-chat-hf -p 8000
|
|
```
|
|
|
|
#### Supported Models
|
|
|
|
Refer to
|
|
[supported model list](https://github.com/lm-sys/FastChat/blob/main/docs/model_support.md#supported-models)
|
|
of FastChat.
|
|
|
|
#### How to use in AgentScope
|
|
|
|
Now you can load the model in AgentScope by the following model config: `fastchat/model_config.json`.
|
|
|
|
```json
|
|
{
|
|
"model_type": "openai_chat",
|
|
"config_name": "fastchat_llama2-7b-chat-hf",
|
|
"model_name": "meta-llama/Llama-2-7b-chat-hf",
|
|
"api_key": "EMPTY",
|
|
"client_args": {
|
|
"base_url": "http://127.0.0.1:8000/v1/"
|
|
},
|
|
"generate_args": {
|
|
"temperature": 0.5
|
|
}
|
|
}
|
|
```
|
|
|
|
### vllm
|
|
|
|
[vllm](https://github.com/vllm-project/vllm) is a high-throughput inference
|
|
and serving engine for LLMs.
|
|
|
|
#### Install Libraries and Set up Serving
|
|
|
|
To install vllm, run
|
|
|
|
```bash
|
|
pip install vllm
|
|
```
|
|
|
|
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
|
|
to set up model API serving, run
|
|
|
|
```bash
|
|
./vllm/vllm_setup.sh -m meta-llama/Llama-2-7b-chat-hf -p 8000
|
|
```
|
|
|
|
#### Supported models
|
|
|
|
Please refer to the
|
|
[supported models list](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
|
of vllm.
|
|
|
|
#### How to use in AgentScope
|
|
|
|
Now you can load the model in AgentScope by the following model config: `vllm/model_config.json`.
|
|
|
|
```json
|
|
{
|
|
"model_type": "openai_chat",
|
|
"config_name": "vllm_llama2-7b-chat-hf",
|
|
"model_name": "meta-llama/Llama-2-7b-chat-hf",
|
|
"api_key": "EMPTY",
|
|
"client_args": {
|
|
"base_url": "http://127.0.0.1:8000/v1/"
|
|
},
|
|
"generate_args": {
|
|
"temperature": 0.5
|
|
}
|
|
}
|
|
```
|
|
|
|
## Model Inference API
|
|
|
|
Both [Huggingface](https://huggingface.co/docs/api-inference/index) and
|
|
[ModelScope](https://www.modelscope.cn) provide model inference API,
|
|
which can be used with AgentScope post api model wrapper.
|
|
Taking `gpt2` in HuggingFace inference API as an example, you can use the
|
|
following model config in AgentScope.
|
|
|
|
```json
|
|
{
|
|
"model_type": "post_api_chat",
|
|
"config_name": "gpt2",
|
|
"headers": {
|
|
"Authorization": "Bearer {YOUR_API_TOKEN}"
|
|
},
|
|
"api_url": "https://api-inference.huggingface.co/models/gpt2"
|
|
}
|
|
```
|