[Core] Upgrade vllm to v0.6.3.post1 #69

ZeldaHuang · 2024-11-11T08:32:08Z

support pipeline parallelism.
support multi-step scheduling.
support chunked prefill.

xixingya · 2024-11-20T10:11:07Z

Does it work properly in version 0.6.3.post1 now?

s5u13b · 2024-11-21T01:42:02Z

Does it work properly in version 0.6.3.post1 now?

Not yet, we will merge it into main once it work properly.

xixingya · 2024-11-21T09:28:27Z

Does it work properly in version 0.6.3.post1 now?

Not yet, we will merge it into main once it work properly.

Thank you, I look forward to it, and it is very helpful to me.

github-actions · 2024-12-19T10:45:48Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	11197.03	63520.64	129841.45	175147.71	179215.11	70996.51

decode	p25	p50	p75	p95	p99	mean
latency(ms)	52.27	56.92	73.23	111.80	206.68	66.02

github-actions · 2024-12-19T11:01:44Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB
rayrpc_speed(GB/s)	0.88	1.35	1.64	1.84	1.90	2.02	2.10	2.16	2.18	2.27	2.27	2.26	2.26	2.13	2.21	2.32

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB
gloo_speed(GB/s)	0.90	1.45	1.84	2.06	2.29	2.58	2.49	2.58	2.67	2.76	2.83	2.81	2.45	3.05	3.05	2.74

github-actions · 2024-12-20T10:06:21Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	3948.71	85906.31	124175.78	176010.55	188518.26	75108.50

decode	p25	p50	p75	p95	p99	mean
latency(ms)	49.34	53.94	62.80	98.12	192.23	63.98

github-actions · 2024-12-20T10:43:23Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB
rayrpc_speed(GB/s)	0.88	1.37	1.64	1.84	1.95	2.06	2.12	2.17	2.26	2.34	2.29	2.35	2.23	2.07	2.34	2.31

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB
nccl_speed(GB/s)	0.29	0.49	0.75	0.92	1.25	1.44	1.51	1.61	1.68	2.13	1.56	2.18	2.01	2.15	2.08	2.07

github-actions · 2024-12-21T05:08:16Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB
rayrpc_speed(GB/s)	0.89	1.33	1.62	1.77	1.87	2.06	2.12	2.11	2.24	2.25	2.24	2.25	2.08	2.19	2.30	2.41

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB
nccl_speed(GB/s)	0.27	0.52	0.86	0.97	1.05	1.29	1.49	1.46	1.65	1.56	2.10	1.66	1.77	2.23	2.25	2.24

github-actions · 2024-12-21T05:12:01Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	3723.53	62014.10	148679.22	210168.42	211821.17	82627.49

decode	p25	p50	p75	p95	p99	mean
latency(ms)	43.46	53.77	67.40	109.24	279.27	63.64

github-actions · 2025-01-15T07:38:28Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	11769.43	62208.46	108368.70	160949.35	164400.94	68337.35

decode	p25	p50	p75	p95	p99	mean
latency(ms)	51.92	57.14	70.00	105.05	299.16	65.94

github-actions · 2025-01-16T08:38:47Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	224.00 MB	232.00 MB	248.00 MB	272.00 MB	320.00 MB	472.00 MB	544.00 MB
rayrpc_speed(GB/s)	0.89	1.38	1.68	1.84	1.93	2.03	2.13	2.17	2.21	2.35	2.30	2.39	2.33	2.33	2.43	2.43	2.44	2.49	2.49	2.59	2.64	2.66	2.61	2.53	2.63	2.70	2.73	2.91	2.64	2.81	3.06	3.22	3.30

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	192.00 MB	200.00 MB	216.00 MB	224.00 MB	312.00 MB
gloo_speed(GB/s)	0.90	1.54	1.85	2.06	2.27	2.36	2.45	2.61	2.71	2.62	2.97	2.82	2.44	2.59	2.52	2.62	1.29	2.37	2.52	2.27	2.04	2.31	2.36	1.73	3.29	2.58	2.52

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	208.00 MB	216.00 MB	232.00 MB	248.00 MB	256.00 MB	296.00 MB	312.00 MB	488.00 MB
nccl_speed(GB/s)	0.21	0.42	0.61	0.83	1.08	1.10	1.37	1.42	1.71	1.87	1.95	2.10	2.12	2.04	2.41	2.19	2.72	2.47	3.64	2.82	3.38	4.07	3.06	2.10	1.17	3.82	2.53	4.33	4.25	2.90	0.81	4.02

github-actions · 2025-01-16T08:52:34Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	8514.14	73022.60	132145.10	169363.06	171401.27	76795.01

decode	p25	p50	p75	p95	p99	mean
latency(ms)	49.90	54.27	66.65	103.31	261.64	66.93

github-actions · 2025-01-16T13:35:16Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	4635.00	61853.15	121386.89	163098.32	164838.71	70405.77

decode	p25	p50	p75	p95	p99	mean
latency(ms)	49.96	56.36	66.98	103.40	199.65	64.91

github-actions · 2025-01-16T13:39:56Z

migration_size	1.59 GB	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	224.00 MB	232.00 MB	288.00 MB	296.00 MB	320.00 MB	328.00 MB	360.00 MB	472.00 MB	568.00 MB
rayrpc_speed(GB/s)	3.55	0.89	1.36	1.65	1.81	1.96	2.07	2.13	2.16	2.16	2.34	2.31	2.33	2.31	2.32	2.37	2.35	2.47	2.41	2.47	2.60	2.56	2.66	2.62	2.46	2.46	2.71	2.39	2.69	2.60	2.82	2.83	2.91	2.97	2.75	3.07	3.28

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	192.00 MB	216.00 MB	224.00 MB	240.00 MB	248.00 MB	312.00 MB	336.00 MB	416.00 MB
gloo_speed(GB/s)	0.90	1.47	1.81	2.07	2.25	2.22	2.38	2.49	2.60	2.78	2.53	2.33	2.85	2.73	2.56	2.56	2.44	2.54	2.45	2.56	1.93	2.61	2.96	2.99	2.47	3.23	0.57	2.74	0.70	1.96

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	224.00 MB	232.00 MB	240.00 MB	296.00 MB	360.00 MB	376.00 MB
nccl_speed(GB/s)	0.20	0.44	0.62	0.86	1.09	1.25	1.29	1.59	1.68	1.69	1.92	2.10	1.97	2.24	2.28	2.43	2.71	2.85	2.74	2.73	1.86	3.15	2.70	3.35	3.44	2.17	3.87	4.24	3.93	2.13	2.97	1.22	3.26

.github/workflows/offline_inference.yml

.github/workflows/pylint.yml

examlpes/offline_inference.py

llumnix/backends/vllm/executor.py

llumnix/backends/vllm/llm_engine.py

llumnix/backends/vllm/sequence.py

s5u13b · 2025-01-21T05:45:37Z

And we can add news or some other places in readme for upgrading vllm to 0.6.3.post1.

github-actions · 2025-01-21T11:31:24Z

migration_size	1.59 GB	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	224.00 MB	232.00 MB	288.00 MB	312.00 MB	320.00 MB	360.00 MB	472.00 MB
rayrpc_speed(GB/s)	3.84	0.89	1.39	1.65	1.85	1.94	2.07	2.17	2.22	2.25	2.29	2.34	2.35	2.25	2.37	2.28	2.40	2.52	2.56	2.54	2.60	2.61	2.59	2.76	2.60	2.64	2.81	2.79	2.93	2.82	3.00	2.96	3.11	3.24	3.18

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	192.00 MB	200.00 MB	224.00 MB	232.00 MB	248.00 MB	360.00 MB	480.00 MB
gloo_speed(GB/s)	0.89	1.46	1.78	2.01	2.22	2.37	2.48	2.50	2.48	2.50	2.53	2.63	2.76	2.74	2.99	2.60	2.60	2.39	2.79	2.37	2.97	1.66	3.03	3.21	1.78	2.47	3.84	1.65	2.61

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	208.00 MB	224.00 MB	232.00 MB	248.00 MB
nccl_speed(GB/s)	0.20	0.45	0.62	0.83	1.01	1.17	1.29	1.42	1.69	1.73	2.08	2.07	1.92	1.91	2.09	2.45	2.34	2.28	3.37	2.71	3.23	3.30	4.10	0.72	4.32	3.10	3.20

github-actions · 2025-01-21T11:31:41Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	3784.87	71725.87	116747.06	154311.61	176412.35	67903.84

decode	p25	p50	p75	p95	p99	mean
latency(ms)	52.66	55.72	67.57	110.71	229.87	69.02

ZeldaHuang · 2025-01-22T01:49:17Z

And we can add news or some other places in readme for upgrading vllm to 0.6.3.post1.

update news in readme

requirements/requirements_vllm.txt

llumnix/entrypoints/vllm/client.py

llumnix/manager.py

llumnix/backends/vllm/llm_engine.py

github-actions · 2025-01-23T08:29:50Z

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	208.00 MB	216.00 MB	232.00 MB	248.00 MB	272.00 MB	280.00 MB	312.00 MB	336.00 MB	360.00 MB	384.00 MB	544.00 MB
rayrpc_speed(GB/s)	0.88	1.35	1.64	1.85	1.95	2.06	2.09	2.21	2.21	2.28	2.31	2.39	2.31	2.33	2.42	2.45	2.35	2.53	2.60	2.64	2.70	2.69	2.61	2.61	2.85	2.86	2.75	2.71	2.64	2.91	3.01	3.04	2.88	3.02	3.18	3.24

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	176.00 MB	184.00 MB	192.00 MB	200.00 MB	232.00 MB	280.00 MB	312.00 MB	336.00 MB	360.00 MB
gloo_speed(GB/s)	0.90	1.49	1.86	2.09	2.18	2.38	2.47	2.55	2.51	2.73	2.37	2.78	2.70	2.81	2.80	2.77	2.45	2.98	2.75	1.93	2.50	1.56	1.42	2.84	2.14	2.63	1.91	0.46	1.92

migration_size	8.00 MB	16.00 MB	24.00 MB	32.00 MB	40.00 MB	48.00 MB	56.00 MB	64.00 MB	72.00 MB	80.00 MB	88.00 MB	96.00 MB	104.00 MB	112.00 MB	120.00 MB	128.00 MB	136.00 MB	144.00 MB	152.00 MB	160.00 MB	168.00 MB	176.00 MB	184.00 MB	192.00 MB	208.00 MB	224.00 MB	232.00 MB	248.00 MB	256.00 MB	272.00 MB
nccl_speed(GB/s)	0.20	0.44	0.67	0.84	1.06	1.36	1.35	1.57	1.47	1.84	2.04	1.92	2.08	2.13	3.12	2.32	2.84	2.41	3.46	2.31	2.47	2.96	3.00	1.10	3.57	2.01	3.61	1.14	4.21	2.46

github-actions · 2025-01-23T08:43:32Z

prefill	p25	p50	p75	p95	p99	mean
latency(ms)	10800.51	67904.27	117441.41	172847.96	173101.26	70835.29

decode	p25	p50	p75	p95	p99	mean
latency(ms)	51.85	57.21	69.34	106.86	145.81	64.30

ZeldaHuang added 4 commits November 4, 2024 07:43

add node id to BackendSim

273d920

update unitests

40952a4

update

a26ead8

update

1d9c438

ZeldaHuang added 11 commits December 11, 2024 08:52

Merge branch 'main' into vllm_upgrade

6044d19

update

a1bd218

update

05b5499

Merge branch 'main' into vllm_upgrade

c44810e

update

8ddd0a5

update

f607c90

update

b23214d

Merge branch 'main' into vllm_upgrade

ffc42a8

update

e29b97a

update

e9d1eff

update

fc73288

ZeldaHuang added 5 commits December 19, 2024 11:06

update

1b82ce1

update

fc18695

update

abb36ab

update

66f729f

update

f1b0592

update

80101cc

ZeldaHuang added 2 commits January 15, 2025 06:59

update

8d0401c

update

9ea2359

ZeldaHuang changed the title ~~[WIP] upgrade vllm to v0.6.3.post1~~ [Core] upgrade vllm to v0.6.3.post1 Jan 15, 2025

ZeldaHuang changed the title ~~[Core] upgrade vllm to v0.6.3.post1~~ [Core] Upgrade vllm to v0.6.3.post1 Jan 15, 2025

ZeldaHuang added 2 commits January 16, 2025 08:02

update

d1f221f

Merge branch 'main' into vllm_upgrade

a93165e

update

18489f2

s5u13b reviewed Jan 20, 2025

View reviewed changes

ZeldaHuang added 4 commits January 20, 2025 09:07

update

fdbfc75

update

623fdf4

Merge branch 'main' into vllm_upgrade

c4b165f

update

1b6811e

update readme

b36301c

s5u13b approved these changes Jan 22, 2025

View reviewed changes

KuilongCui approved these changes Jan 23, 2025

View reviewed changes

requirements/requirements_vllm.txt Show resolved Hide resolved

llumnix/entrypoints/vllm/client.py Show resolved Hide resolved

llumnix/manager.py Show resolved Hide resolved

llumnix/backends/vllm/llm_engine.py Show resolved Hide resolved

ZeldaHuang added 2 commits January 23, 2025 06:35

update

6c3b23c

update

9a73da9

KuilongCui merged commit 2a4822a into main Jan 23, 2025
14 checks passed

KuilongCui deleted the vllm_upgrade branch January 23, 2025 08:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Upgrade vllm to v0.6.3.post1 #69

[Core] Upgrade vllm to v0.6.3.post1 #69

ZeldaHuang commented Nov 11, 2024

xixingya commented Nov 20, 2024

s5u13b commented Nov 21, 2024

xixingya commented Nov 21, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 20, 2024

github-actions bot commented Dec 20, 2024

github-actions bot commented Dec 21, 2024

github-actions bot commented Dec 21, 2024

github-actions bot commented Jan 15, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

s5u13b commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

ZeldaHuang commented Jan 22, 2025

github-actions bot commented Jan 23, 2025

github-actions bot commented Jan 23, 2025

[Core] Upgrade vllm to v0.6.3.post1 #69

[Core] Upgrade vllm to v0.6.3.post1 #69

Conversation

ZeldaHuang commented Nov 11, 2024

xixingya commented Nov 20, 2024

s5u13b commented Nov 21, 2024

xixingya commented Nov 21, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 20, 2024

github-actions bot commented Dec 20, 2024

github-actions bot commented Dec 21, 2024

github-actions bot commented Dec 21, 2024

github-actions bot commented Jan 15, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

s5u13b commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

ZeldaHuang commented Jan 22, 2025

github-actions bot commented Jan 23, 2025

github-actions bot commented Jan 23, 2025