CakeCN/hsa
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
CakeCN
2024-12-03 16:21:19 +08:00
commit 4198ca63b1
975 changed files with 333413 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

313
AlgoriAgent/scripts/README.md Normal file
View File

@@ -0,0 +1,313 @@
# Set up Local Model API Serving
AgentScope supports developers to build their local model API serving with different inference engines/libraries.
This document will introduce how to fast build their local API serving with provided scripts.
Table of Contents
=================
- [Set up Local Model API Serving](#set-up-local-model-api-serving)
- [Table of Contents](#table-of-contents)
- [Local Model API Serving](#local-model-api-serving)
- [ollama](#ollama)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving)
- [How to use in AgentScope](#how-to-use-in-agentscope)
- [Flask-based Model API Serving](#flask-based-model-api-serving)
- [With Transformers Library](#with-transformers-library)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-1)
- [How to use in AgentScope](#how-to-use-in-agentscope-1)
- [Note](#note)
- [With ModelScope Library](#with-modelscope-library)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-2)
- [How to use in AgentScope](#how-to-use-in-agentscope-2)
- [Note](#note-1)
- [FastChat](#fastchat)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-3)
- [Supported Models](#supported-models)
- [How to use in AgentScope](#how-to-use-in-agentscope-3)
- [vllm](#vllm)
- [Install Libraries and Set up Serving](#install-libraries-and-set-up-serving-4)
- [Supported models](#supported-models-1)
- [How to use in AgentScope](#how-to-use-in-agentscope-4)
- [Model Inference API](#model-inference-api)
## Local Model API Serving
### ollama
[ollama](https://github.com/ollama/ollama) is a CPU inference engine for LLMs. With ollama, developers can build their local model API serving without GPU requirements.
#### Install Libraries and Set up Serving
- First, install ollama in its [official repository](https://github.com/ollama/ollama) based on your system (e.g. macOS, windows or linux).
- Follow ollama's [guidance](https://github.com/ollama/ollama) to pull or create a model and start its serving. Taking llama2 as an example, you can run the following command to pull the model files.
```bash
ollama pull llama2
```
#### How to use in AgentScope
In AgentScope, you can use the following model configurations to load the model.
- For ollama Chat API:
```python
{
"config_name": "my_ollama_chat_config",
"model_type": "ollama_chat",
# Required parameters
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
# Optional parameters
"options": { # Parameters passed to the model when calling
# e.g. "temperature": 0., "seed": 123,
},
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
}
```
- For ollama generate API:
```python
{
"config_name": "my_ollama_generate_config",
"model_type": "ollama_generate",
# Required parameters
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
# Optional parameters
"options": { # Parameters passed to the model when calling
# "temperature": 0., "seed": 123,
},
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
}
```
- For ollama embedding API:
```python
{
"config_name": "my_ollama_embedding_config",
"model_type": "ollama_embedding",
# Required parameters
"model_name": "{model_name}", # The model name used in ollama API, e.g. llama2
# Optional parameters
"options": { # Parameters passed to the model when calling
# "temperature": 0., "seed": 123,
},
"keep_alive": "5m", # Controls how long the model will stay loaded into memory
}
```
### Flask-based Model API Serving
[Flask](https://github.com/pallets/flask) is a lightweight web application
framework. It is easy to build a local model API serving with Flask.
Here we provide two Flask examples with Transformers and ModelScope library,
respectively. You can build your own model API serving with few modifications.
#### With Transformers Library
##### Install Libraries and Set up Serving
Install Flask and Transformers by following command.
```bash
pip install flask torch transformers accelerate
```
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
set up the model API serving by running the following command.
```shell
python flask_transformers/setup_hf_service.py \
--model_name_or_path meta-llama/Llama-2-7b-chat-hf \
--device "cuda:0" \
--port 8000
```
You can replace `meta-llama/Llama-2-7b-chat-hf` with any model card in
huggingface model hub.
##### How to use in AgentScope
In AgentScope, you can load the model with the following model configs: `./flask_transformers/model_config.json`.
```json
{
"model_type": "post_api_chat",
"config_name": "flask_llama2-7b-chat-hf",
"api_url": "http://127.0.0.1:8000/llm/",
"json_args": {
"max_length": 4096,
"temperature": 0.5
}
}
```
##### Note
In this model serving, the messages from post requests should be in **STRING
format**. You can use [templates for chat model](https://huggingface.co/docs/transformers/main/chat_templating) in
transformers with a little modification in `./flask_transformers/setup_hf_service.py`.
#### With ModelScope Library
##### Install Libraries and Set up Serving
Install Flask and modelscope by following command.
```bash
pip install flask torch modelscope
```
Taking model `modelscope/Llama-2-7b-chat-ms` and port `8000` as an example,
to set up the model API serving, run the following command.
```bash
python flask_modelscope/setup_ms_service.py \
--model_name_or_path modelscope/Llama-2-7b-chat-ms \
--device "cuda:0" \
--port 8000
```
You can replace `modelscope/Llama-2-7b-chat-ms` with any model card in
modelscope model hub.
##### How to use in AgentScope
In AgentScope, you can load the model with the following model configs:
`flask_modelscope/model_config.json`.
```json
{
"model_type": "post_api_chat",
"config_name": "flask_llama2-7b-chat-ms",
"api_url": "http://127.0.0.1:8000/llm/",
"json_args": {
"max_length": 4096,
"temperature": 0.5
}
}
```
##### Note
Similar with the example of transformers, the messages from post requests
should be in **STRING format**.
### FastChat
[FastChat](https://github.com/lm-sys/FastChat) is an open platform that
provides quick setup for model serving with OpenAI-compatible RESTful APIs.
#### Install Libraries and Set up Serving
To install FastChat, run
```bash
pip install "fschat[model_worker,webui]"
```
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
to set up model API serving, run the following command to set up model serving.
```bash
bash fastchat/fastchat_setup.sh -m meta-llama/Llama-2-7b-chat-hf -p 8000
```
#### Supported Models
Refer to
[supported model list](https://github.com/lm-sys/FastChat/blob/main/docs/model_support.md#supported-models)
of FastChat.
#### How to use in AgentScope
Now you can load the model in AgentScope by the following model config: `fastchat/model_config.json`.
```json
{
"model_type": "openai_chat",
"config_name": "fastchat_llama2-7b-chat-hf",
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8000/v1/"
},
"generate_args": {
"temperature": 0.5
}
}
```
### vllm
[vllm](https://github.com/vllm-project/vllm) is a high-throughput inference
and serving engine for LLMs.
#### Install Libraries and Set up Serving
To install vllm, run
```bash
pip install vllm
```
Taking model `meta-llama/Llama-2-7b-chat-hf` and port `8000` as an example,
to set up model API serving, run
```bash
./vllm/vllm_setup.sh -m meta-llama/Llama-2-7b-chat-hf -p 8000
```
#### Supported models
Please refer to the
[supported models list](https://docs.vllm.ai/en/latest/models/supported_models.html)
of vllm.
#### How to use in AgentScope
Now you can load the model in AgentScope by the following model config: `vllm/model_config.json`.
```json
{
"model_type": "openai_chat",
"config_name": "vllm_llama2-7b-chat-hf",
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8000/v1/"
},
"generate_args": {
"temperature": 0.5
}
}
```
## Model Inference API
Both [Huggingface](https://huggingface.co/docs/api-inference/index) and
[ModelScope](https://www.modelscope.cn) provide model inference API,
which can be used with AgentScope post api model wrapper.
Taking `gpt2` in HuggingFace inference API as an example, you can use the
following model config in AgentScope.
```json
{
"model_type": "post_api_chat",
"config_name": "gpt2",
"headers": {
"Authorization": "Bearer {YOUR_API_TOKEN}"
},
"api_url": "https://api-inference.huggingface.co/models/gpt2"
}
```

20
AlgoriAgent/scripts/fastchat/fastchat_setup.sh Normal file
View File

@@ -0,0 +1,20 @@
#!/bin/bash
model_name_or_path="meta-llama/Llama-2-7b-chat-hf"
port=8000
while getopts "m:p:" flag
do
# shellcheck disable=SC2220
case "${flag}" in
m) model_name_or_path=${OPTARG};;
p) port=${OPTARG};;
esac
done
python3 -m fastchat.serve.controller &
sleep 1
python3 -m fastchat.serve.model_worker --model-path "${model_name_or_path}" &
sleep 1
python3 -m fastchat.serve.openai_api_server --host localhost --port "${port}" &

12
AlgoriAgent/scripts/fastchat/model_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"model_type": "openai_chat",
"config_name": "fastchat_llama2-7b-chat-hf",
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://localhost:8000/v1/"
},
"generate_args": {
"temperature": 0.5
}
}

9
AlgoriAgent/scripts/flask_modelscope/model_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"model_type": "post_api_chat",
"config_name": "flask_llama2-7b-chat-ms",
"api_url": "http://127.0.0.1:8000/llm/",
"json_args": {
"max_length": 4096,
"temperature": 0.5
}
}

121
AlgoriAgent/scripts/flask_modelscope/setup_ms_service.py Normal file
View File

@@ -0,0 +1,121 @@
# -*- coding: utf-8 -*-
"""Set up a local language model service."""
import datetime
import argparse
from flask import Flask
from flask import request
import modelscope
from agentscope.utils.tools import reform_dialogue
def create_timestamp(format_: str = "%Y-%m-%d %H:%M:%S") -> str:
"""Get current timestamp."""
return datetime.datetime.now().strftime(format_)
app = Flask(__name__)
@app.route("/llm/", methods=["POST"])
def get_response() -> dict:
"""Receive post request and return response"""
json = request.get_json()
inputs = json.pop("inputs")
inputs = reform_dialogue(inputs)
global model, tokenizer
if hasattr(tokenizer, "apply_chat_template"):
prompt = tokenizer.apply_chat_template(
inputs,
tokenize=False,
add_generation_prompt=True,
)
else:
prompt = ""
for msg in inputs:
prompt += (
f"{msg.get('name', msg.get('role', 'system'))}: "
f"{msg.get('content', '')}\n"
)
print("=" * 80)
print(f"[PROMPT]:\n{prompt}")
prompt_tokenized = tokenizer(prompt, return_tensors="pt").to(model.device)
prompt_tokens_input_ids = prompt_tokenized.input_ids[0]
response_ids = model.generate(
prompt_tokenized.input_ids,
**json,
)
new_response_ids = response_ids[:, len(prompt_tokens_input_ids) :]
response = tokenizer.batch_decode(
new_response_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(f"[RESPONSE]:\n{response}")
print("=" * 80)
return {
"data": {
"completion_tokens": len(response_ids[0]),
"messages": {},
"prompt_tokens": len(prompt_tokens_input_ids),
"response": {
"choices": [
{
"message": {
"content": response,
},
},
],
"created": "",
"id": create_timestamp(),
"model": "flask_model",
"object": "text_completion",
"usage": {
"completion_tokens": len(response_ids[0]),
"prompt_tokens": len(prompt_tokens_input_ids),
"total_tokens": len(response_ids[0])
+ len(
prompt_tokens_input_ids,
),
},
},
"total_tokens": len(response_ids[0])
+ len(
prompt_tokens_input_ids,
),
"username": "",
},
}
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--model_name_or_path", type=str, required=True)
parser.add_argument("--device", type=str, default="auto")
parser.add_argument("--port", type=int, default=8000)
args = parser.parse_args()
global model, tokenizer
model = modelscope.AutoModelForCausalLM.from_pretrained(
args.model_name_or_path,
device_map=args.device,
)
tokenizer = modelscope.AutoTokenizer.from_pretrained(
args.model_name_or_path,
use_fast=False,
)
app.run(port=args.port)

9
AlgoriAgent/scripts/flask_transformers/model_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"model_type": "post_api_chat",
"config_name": "flask_llama-2-chat-7b-hf",
"api_url": "http://127.0.0.1:8000/llm/",
"json_args": {
"max_length": 4096,
"temperature": 0.5
}
}

121
AlgoriAgent/scripts/flask_transformers/setup_hf_service.py Normal file
View File

@@ -0,0 +1,121 @@
# -*- coding: utf-8 -*-
"""Set up a local language model service."""
import datetime
import argparse
from flask import Flask
from flask import request
import transformers
from agentscope.utils.tools import reform_dialogue
def create_timestamp(format_: str = "%Y-%m-%d %H:%M:%S") -> str:
"""Get current timestamp."""
return datetime.datetime.now().strftime(format_)
app = Flask(__name__)
@app.route("/llm/", methods=["POST"])
def get_response() -> dict:
"""Receive post request and return response"""
json = request.get_json()
inputs = json.pop("inputs")
global model, tokenizer
inputs = reform_dialogue(inputs)
if hasattr(tokenizer, "apply_chat_template"):
prompt = tokenizer.apply_chat_template(
inputs,
tokenize=False,
add_generation_prompt=True,
)
else:
prompt = ""
for msg in inputs:
prompt += (
f"{msg.get('name', msg.get('role', 'system'))}: "
f"{msg.get('content', '')}\n"
)
print("=" * 80)
print(f"[PROMPT]:\n{prompt}")
prompt_tokenized = tokenizer(prompt, return_tensors="pt").to(model.device)
prompt_tokens_input_ids = prompt_tokenized.input_ids[0]
response_ids = model.generate(
prompt_tokenized.input_ids,
**json,
)
new_response_ids = response_ids[:, len(prompt_tokens_input_ids) :]
response = tokenizer.batch_decode(
new_response_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(f"[RESPONSE]:\n{response}")
print("=" * 80)
return {
"data": {
"completion_tokens": len(response_ids[0]),
"messages": {},
"prompt_tokens": len(prompt_tokens_input_ids),
"response": {
"choices": [
{
"message": {
"content": response,
},
},
],
"created": "",
"id": create_timestamp(),
"model": "flask_model",
"object": "text_completion",
"usage": {
"completion_tokens": len(response_ids[0]),
"prompt_tokens": len(prompt_tokens_input_ids),
"total_tokens": len(response_ids[0])
+ len(
prompt_tokens_input_ids,
),
},
},
"total_tokens": len(response_ids[0])
+ len(
prompt_tokens_input_ids,
),
"username": "",
},
}
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--model_name_or_path", type=str, required=True)
parser.add_argument("--device", type=str, default="auto")
parser.add_argument("--port", type=int, default=8000)
args = parser.parse_args()
global model, tokenizer
model = transformers.AutoModelForCausalLM.from_pretrained(
args.model_name_or_path,
device_map=args.device,
)
tokenizer = transformers.AutoTokenizer.from_pretrained(
args.model_name_or_path,
use_fast=False,
)
app.run(port=args.port)

32
AlgoriAgent/scripts/ollama/model_config.json Normal file
View File

@@ -0,0 +1,32 @@
[
{
"config_name": "my_ollama_chat_config",
"model_type": "ollama_chat",
"model_name": "{model_name}",
"options": {
"temperature": 0.5,
"seed": 123
},
"keep_alive": "5m"
},
{
"config_name": "my_ollama_generate_config",
"model_type": "ollama_generate",
"model_name": "{model_name}",
"options": {
"temperature": 0.5,
"seed": 123
},
"keep_alive": "5m"
},
{
"config_name": "my_ollama_embedding_config",
"model_type": "ollama_embedding",
"model_name": "{model_name}",
"options": {
"temperature": 0.5,
"seed": 123
},
"keep_alive": "5m"
}
]

3
AlgoriAgent/scripts/ollama/ollama.sh Normal file
View File

@@ -0,0 +1,3 @@
#!/bin/bash
ollama pull llama2

12
AlgoriAgent/scripts/vllm/model_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"model_type": "openai_chat",
"config_name": "vllm_llama2-7b-chat-hf",
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://localhost:8000/v1/"
},
"generate_args": {
"temperature": 0.5
}
}

18
AlgoriAgent/scripts/vllm/vllm_setup.sh Normal file
View File

@@ -0,0 +1,18 @@
#!/bin/bash
# pip3 install vllm
model_name_or_path="meta-llama/Llama-2-7b-chat-hf"
port=8000
while getopts "m:p:" flag
do
# shellcheck disable=SC2220
case "${flag}" in
m) model_name_or_path=${OPTARG};;
p) port=${OPTARG};;
esac
done
python -m vllm.entrypoints.openai.api_server --model "${model_name_or_path}" \
--port "${port}" --enforce-eager