CakeCN/hsa
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
CakeCN
2024-12-03 16:21:19 +08:00
commit 4198ca63b1
975 changed files with 333413 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
AlgoriAgent/examples/distributed_simulation/README.md Normal file
View File

@@ -0,0 +1,103 @@
# Distributed Large Scale Simulation
> **WARNING:**
> **This example will consume a huge amount of tokens.**
> **Using paid model API with this example can introduce a high cost.**
> **Users with powerful GPUs (A800 or better) can use local inference services (such as vLLM) to run this example,**
> **while CPU inference services such as ollama is not recommended.**
This example is a large scale simulation to demonstrate the scalability of AgentScope's distributed mode. From this example, you can learn:
- How to run a large number of agent servers in a GPU cluster.
- How to connect to those agent servers and run a huge number of agents in them.
> Based on this example, we deploy 64,000 agents evenly on 4 machines, and each machine has 64 CPU cores and 8 A100 GPUs. The running time is about 30s (excluding initialization time).
## Background
This example simulates the following scenario:
A large number of people participate in a game in which the moderator asks each participant to provide a number between 0 and N. The moderator will calculate the average of all numbers and announce it. The person closest to the average will win.
## Tested Models
Only vLLM local inference service is tested for this example.
This example will consume a huge amount of tokens. Please do not use model API that requires payment.
## Prerequisites
- The distribute version of AgentScope is installed
- Use MacOS or Linux (Windows requires some modifiations to the scripts)
- [Optional] Have multiple machines with powerful GPUs (A800 or better) and install [vLLM](https://github.com/vllm-project/vllm)
## How to Run
### Step 1: start local inference service
> If you only have one machine and don't have a powerful GPU (A800 or better), you can ignore this step.
You can use `start_vllm.sh` to start vllm inference services on each of your machines.
Before running the script, please set `gpu_num`, `model_path` and `base_port` properly.
- `gpu_num`: number of GPUs for this machine.
- `model_path`: the model checkpoint path.
- `base_port`: The starting point of the port number used by the local inference services.
For example, if `base_port` is `8010` and `gpu_num` is `4`, 4 inference services will be started, and the port numbers are `8010`, `8011`, `8012` and `8013` respectively.
vLLM inference services start slowly, so you need to wait for these servers to actually start before proceeding to the next step.
> The above configuration requires that the model checkpoint can be loaded by a single GPU.
> If you need to use a model that must be loaded by multiple GPUs, you need to modify the script.
### Step 2: start agent server
> If you only have one machine and don't have a powerful GPU, you can just use the default setting of the scripts.
You can use `start_all_server.sh` to start multiple agent servers on each of your machine.
Before running the script, please set `base_port`, `host_name` and `moderator_num` properly.
- `base_port`: The starting point of the port number used by the agent servers.
- `host_name`: The hostname of this machine, and must be accessible to other machines in the cluster (The default value `localhost` is only used for single machine scenario).
- `moderator_num`: Number of moderators. When the number of participants is large, this value needs to be expanded to avoid bottlenecks.
After setting the above values correctly, you can use the script to start multiple agent server on your machine. The following command will start 10 agent servers on your machine with port numbers starting from `base_port` to `base_port + 9`, and will also start `moderator_num` agent servers for moderators with port numbers starting from `base_port + 10` to `base_port + moderator_num + 9`.
```shell
#./start_all_server.sh <number_of_server_per_host>
./start_all_server.sh 10
```
If you have multiple machines, please make sure the `base_port` and `moderator_num` parameters are exactly the same on all machines, and start the same number of agent servers.
### Step 3: run simulation
You can use `run_simulation.sh` to start the simulation.
Before running the script, please set the following setting correctly:
- `base_port`: the base port for agent servers, must be the same as used in Step 2.
- `hosts`: hostnames of all machines. If you only have one machine, use the default value `localhost`.
- `moderator_per_host`: Consistent with `moderator_num` in Step 2.
- `agent_type`: `random` or `llm`. Please use `random` if you don't have local inference service.
- `max_value`: The upper bound of numbers generated in the game.
The command below will run a simulation with 1000 participant agents and evenly distributed those agents to the 10 agent servers started in Step 2.
```shell
#./run_simulation.sh <number_of_server_per_host> <total_number_of_participant>
./run_simulation.sh 10 1000
```
The following is sample output from a single-machine (16 CPU cores) simulation scenario:
```log
2024-04-16 10:31:53.786 | INFO | agentscope.models:read_model_configs:178 - Load configs for model wrapper: model_1, model_2, model_3, model_4, model_5, model_6, model_7, model_8
2024-04-16 10:31:53.822 | INFO | agentscope.utils.monitor:_create_monitor_table:343 - Init [monitor_metrics] as the monitor table
2024-04-16 10:31:53.822 | INFO | agentscope.utils.monitor:_create_monitor_table:344 - Init [monitor_metrics_quota_exceeded] as the monitor trigger
2024-04-16 10:31:53.822 | INFO | agentscope.utils.monitor:__init__:313 - SqliteMonitor initialization completed at [./runs/run_20240416-103153_h0xuo5/agentscope.db]
2024-04-16 10:31:53.829 | INFO | __main__:run_main_process_new:106 - init 1000 random participant agents...
2024-04-16 10:31:53.829 | INFO | __main__:run_main_process_new:139 - init 4 moderator agents...
2024-04-16 10:31:54.211 | INFO | __main__:run_main_process_new:163 - [init takes 0.38274645805358887 s]
Moderator: The average value is 49.561 [takes 4.197571277618408 s]
```

98
AlgoriAgent/examples/distributed_simulation/configs/model_configs.json Normal file
View File

@@ -0,0 +1,98 @@
[
{
"model_type": "openai_chat",
"config_name": "model_1",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8010/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_2",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8011/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_3",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8012/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_4",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8013/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_5",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8014/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_6",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8015/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_7",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8016/v1/"
},
"generate_args": {
"temperature": 1.0
}
},
{
"model_type": "openai_chat",
"config_name": "model_8",
"model_name": "path-to-your-model-dir",
"api_key": "EMPTY",
"client_args": {
"base_url": "http://127.0.0.1:8017/v1/"
},
"generate_args": {
"temperature": 1.0
}
}
]

216
AlgoriAgent/examples/distributed_simulation/main.py Normal file
View File

@@ -0,0 +1,216 @@
# -*- coding: utf-8 -*-
""" A large-scale social simulation experiment """
import argparse
import time
from concurrent import futures
from concurrent.futures import as_completed
from loguru import logger
from participant import Moderator, RandomParticipant, LLMParticipant
import agentscope
from agentscope.agents import AgentBase
from agentscope.server import RpcAgentServerLauncher
from agentscope.message import Msg
def parse_args() -> argparse.Namespace:
"""Parse arguments"""
parser = argparse.ArgumentParser()
parser.add_argument(
"--role",
choices=["participant", "main"],
default="main",
)
parser.add_argument(
"--agent-type",
choices=["random", "llm"],
default="random",
)
parser.add_argument("--max-value", type=int, default=100)
parser.add_argument("--sleep-time", type=float, default=1.0)
parser.add_argument(
"--hosts",
type=str,
nargs="+",
default=["localhost"],
)
parser.add_argument("--participant-num", type=int, default=100)
parser.add_argument("--base-port", type=int, default=12010)
parser.add_argument(
"--server-per-host",
type=int,
)
parser.add_argument("--model-per-host", type=int, default=1)
parser.add_argument("--moderator-per-host", type=int, default=1)
return parser.parse_args()
def setup_participant_agent_server(host: str, port: int) -> None:
"""Set up agent server"""
agentscope.init(
project="simulation",
name="server",
runtime_id=str(port),
save_code=False,
save_api_invoke=False,
model_configs="configs/model_configs.json",
use_monitor=False,
)
assistant_server_launcher = RpcAgentServerLauncher(
host=host,
port=port,
max_pool_size=16384,
custom_agents=[Moderator, RandomParticipant, LLMParticipant],
)
assistant_server_launcher.launch(in_subprocess=False)
assistant_server_launcher.wait_until_terminate()
def init_moderator(
name: str,
configs: list[dict],
host: str,
port: int,
agent_type: str,
max_value: int,
sleep_time: float,
) -> AgentBase:
"""Init moderator"""
return Moderator( # pylint: disable=E1123
name=name,
part_configs=configs,
agent_type=agent_type,
max_value=max_value,
sleep_time=sleep_time,
to_dist={
"host": host,
"port": port,
},
)
def run_main_process(
hosts: list[str],
base_port: int,
server_per_host: int,
model_per_host: int,
participant_num: int,
moderator_per_host: int = 10,
agent_type: str = "random",
max_value: int = 100,
sleep_time: float = 1.0,
) -> None:
"""Run main process"""
agentscope.init(
project="simulation",
name="main",
save_code=False,
save_api_invoke=False,
model_configs="configs/model_configs.json",
use_monitor=False,
)
host_num = len(hosts)
total_agent_server_num = server_per_host * host_num
participant_per_agent_server = participant_num // total_agent_server_num
ist = time.time()
configs = []
logger.info(f"init {participant_num} {agent_type} participant agents...")
# build init configs of participants
for i in range(participant_num):
idx = i // participant_per_agent_server
host_id = idx // server_per_host
port_id = idx % server_per_host
model_id = i % model_per_host
host = hosts[host_id]
port = base_port + port_id
config_name = f"model_{model_id + 1}"
if agent_type == "random":
configs.append(
{
"name": f"P{i}",
"host": host,
"port": port,
},
)
else:
configs.append(
{
"name": f"P{i}",
"model_config_name": config_name,
"host": host,
"port": port,
},
)
mods = []
moderator_num = moderator_per_host * host_num
participant_per_moderator = participant_num // moderator_num
tasks = []
logger.info(f"init {moderator_num} moderator agents...")
# init moderators
with futures.ThreadPoolExecutor(max_workers=None) as executor:
for i in range(moderator_num):
tasks.append(
executor.submit(
init_moderator,
name=f"mod_{i}",
configs=configs[
i
* participant_per_moderator : (i + 1) # noqa
* participant_per_moderator
],
host=hosts[i // moderator_per_host],
port=base_port + server_per_host + i % moderator_per_host,
agent_type=agent_type,
max_value=max_value,
sleep_time=sleep_time,
),
)
for task in as_completed(tasks):
mods.append(task.result())
iet = time.time()
logger.info(f"[init takes {iet - ist} s]")
# run te
st = time.time()
results = []
for p in mods:
results.append(p())
summ = 0
cnt = 0
for r in results:
try:
summ += int(r["content"]["sum"])
cnt += int(r["content"]["cnt"])
except Exception:
logger.error(r["content"])
et = time.time()
logger.chat(
Msg(
name="Moderator",
role="assistant",
content=f"The average value is {summ/cnt} [takes {et-st} s]",
),
)
if __name__ == "__main__":
args = parse_args()
if args.role == "participant":
setup_participant_agent_server(args.hosts[0], args.base_port)
elif args.role == "main":
run_main_process(
hosts=args.hosts,
base_port=args.base_port,
participant_num=args.participant_num,
server_per_host=args.server_per_host,
model_per_host=args.model_per_host,
moderator_per_host=args.moderator_per_host,
agent_type=args.agent_type,
sleep_time=args.sleep_time,
max_value=args.max_value,
)

158
AlgoriAgent/examples/distributed_simulation/participant.py Normal file
View File

@@ -0,0 +1,158 @@
# -*- coding: utf-8 -*-
"""A general dialog agent."""
import random
import time
import re
from typing import Optional, Union, Sequence
from loguru import logger
from agentscope.message import Msg
from agentscope.agents import AgentBase
class RandomParticipant(AgentBase):
"""A fake participant who generates number randomly."""
def __init__(
self,
name: str,
max_value: int = 100,
sleep_time: float = 1.0,
) -> None:
"""Initialize the participant."""
super().__init__(
name=name,
)
self.max_value = max_value
self.sleep_time = sleep_time
def generate_random_response(self) -> str:
"""generate a random int"""
time.sleep(self.sleep_time)
return str(random.randint(0, self.max_value))
def reply(self, x: Optional[Union[Msg, Sequence[Msg]]] = None) -> Msg:
"""Generate a random value"""
# generate a response in content
response = self.generate_random_response()
msg = Msg(self.name, content=response)
return msg
class LLMParticipant(AgentBase):
"""A participant agent who generates number using LLM."""
def __init__(
self,
name: str,
model_config_name: str,
max_value: int = 100,
) -> None:
"""Initialize the participant."""
super().__init__(
name=name,
model_config_name=model_config_name,
use_memory=True,
)
self.max_value = max_value
self.prompt = Msg(
name="system",
role="system",
content="You are participating in a game where everyone "
f"provides a number between 0 and {max_value}. The person "
"closest to the average will win.",
)
def parse_value(self, txt: str) -> str:
"""Parse the number from the response."""
numbers = re.findall(r"\d+", txt)
if len(numbers) == 0:
logger.warning(
f"Fail to parse value from [{txt}], use "
f"{self.max_value // 2} instead.",
)
return str(self.max_value // 2)
else:
return numbers[-1]
def reply(self, x: Optional[Union[Msg, Sequence[Msg]]] = None) -> Msg:
"""Generate a value by LLM"""
if self.memory:
self.memory.add(x)
# prepare prompt
prompt = self.model.format(self.prompt, self.memory.get_memory())
# call llm and generate response
response = self.model(prompt).text
response = self.parse_value(response)
msg = Msg(self.name, response, role="assistant")
# Record the message in memory
if self.memory:
self.memory.add(msg)
return msg
class Moderator(AgentBase):
"""A Moderator to collect values from participants."""
def __init__(
self,
name: str,
part_configs: list[dict],
agent_type: str = "random",
max_value: int = 100,
sleep_time: float = 1.0,
) -> None:
super().__init__(name)
self.max_value = max_value
if agent_type == "llm":
self.participants = [
LLMParticipant(
name=config["name"],
model_config_name=config["model_config_name"],
max_value=max_value,
).to_dist(
host=config["host"],
port=config["port"],
)
for config in part_configs
]
else:
self.participants = [
RandomParticipant(
name=config["name"],
max_value=max_value,
sleep_time=sleep_time,
).to_dist(
host=config["host"],
port=config["port"],
)
for config in part_configs
]
def reply(self, x: Optional[Union[Msg, Sequence[Msg]]] = None) -> Msg:
results = []
msg = Msg(
name="moderator",
role="user",
content=f"Now give a number between 0 and {self.max_value}.",
)
for p in self.participants:
results.append(p(msg))
summ = 0
for r in results:
try:
summ += int(r["content"])
except Exception as e:
print(e)
return Msg(
name=self.name,
role="assistant",
content={"sum": summ, "cnt": len(self.participants)},
)

25
AlgoriAgent/examples/distributed_simulation/run_simulation.sh Normal file
View File

@@ -0,0 +1,25 @@
#!/bin/bash
# default values
base_port=12330
hosts="localhost" # or "server1 server2 server3 ..."
moderator_per_host=4
model_per_host=8
agent_type="random" # or "llm"
max_value=100
# check server-per-host
if ! [[ "$1" =~ ^[0-9]+$ ]]; then
echo "Usage: $0 <server-per-host> <participant-num>"
exit 1
fi
# check participant-num
if ! [[ "$2" =~ ^[0-9]+$ ]]; then
echo "Usage: $0 <server-per-host> <participant-num>"
exit 1
fi
mkdir -p log
python main.py --role main --hosts ${hosts} --base-port ${base_port} --participant-num $2 --server-per-host $1 --model-per-host ${model_per_host} --moderator-per-host ${moderator_per_host} --agent-type ${agent_type} --max-value ${max_value}

29
AlgoriAgent/examples/distributed_simulation/start_all_server.sh Normal file
View File

@@ -0,0 +1,29 @@
#!/bin/bash
# default values
base_port=12330
host_name="localhost"
moderator_num=4
# get number of server
if ! [[ "$1" =~ ^[0-9]+$ ]]; then
echo "Usage: $0 <number-of-server-for-participant>"
exit 1
fi
participant_server_num=$1
# create files for pid
> .pid
# create log dir
mkdir -p log
# start all agent servers
for ((i=0; i<(participant_server_num + moderator_num); i++)); do
port=$((base_port + i))
python main.py --role participant --hosts ${host_name} --base-port ${port} > log/${port}.log 2>&1 &
echo $! >> .pid
echo "Started agent server on ${host_name}:${port} with PID $!"
done
echo "All servers started"

19
AlgoriAgent/examples/distributed_simulation/start_vllm.sh Normal file
View File

@@ -0,0 +1,19 @@
#!/bin/bash
# default values
gpu_num=8
model_path="path-to-your-model-dir"
base_port=8010
> .vllm_pid
mkdir -p log
for ((i=0; i<8; i++)); do
port=$((base_port + i))
export CUDA_VISIBLE_DEVICES=$i
python -m vllm.entrypoints.openai.api_server --model "${model_path}" --port ${port} --enforce-eager > log/vllm-${port}.log 2>&1 &
echo $! >> .vllm_pid
echo "Started vllm server on port ${port} with PID $!"
done
echo "All vllm server started"

19
AlgoriAgent/examples/distributed_simulation/stop_all_server.sh Normal file
View File

@@ -0,0 +1,19 @@
#!/bin/bash
if [ ! -f .pid ]; then
echo "PID file not found. Are the servers running?"
exit 1
fi
while read pid; do
kill -9 $pid
if [ $? -eq 0 ]; then
echo "Killed server with PID $pid"
else
echo "Failed to kill server with PID $pid"
fi
done < .pid
rm .pid
echo "All servers stopped."

19
AlgoriAgent/examples/distributed_simulation/stop_vllm.sh Normal file
View File

@@ -0,0 +1,19 @@
#!/bin/bash
if [ ! -f .vllm_pid ]; then
echo "PID file not found. Are the servers running?"
exit 1
fi
while read pid; do
kill -9 $pid
if [ $? -eq 0 ]; then
echo "Killed vllm server with PID $pid"
else
echo "Failed to kill vllm server with PID $pid"
fi
done < .vllm_pid
rm .vllm_pid
echo "All vllm servers stopped."