Symptoms Observed #

The model started normally, but the engine invariably crashed several minutes to tens of minutes after sending a chat message. The following errors were repeatedly logged.

AttributeError: '_TorchTensorAcceleratorChannel' object
has no attribute '_accelerator_group'.
Did you mean: '_accelerator_group_id'?

ray.exceptions.ActorUnavailableError: The actor is temporarily unavailable:
  RpcError: RPC error: Socket closed rpc_code: 14.

This error is a Ray 2.54.0 bug that occurs in cleanup (teardown) after a crash, not the direct cause. The true root cause lies upstream: initialization failure of the GPU accelerator communication channel between pipeline_parallel stages.

Root Cause: V100 Does Not Support pipeline_parallel P2P GPU Transfers #

vLLM’s pipeline_parallel (pipeline parallelism) uses _TorchTensorAcceleratorChannel (direct P2P transfer between GPUs) for activation transfer between stages. This requires NVLink or high-speed GPU P2P, which is not available on Tesla V100-PCIE (compute capability 7.0).

Solution: Complete Fix with Configuration Changes Only #

Changing the model configuration in the GPUStack UI as follows solved the problem completely. No infrastructure changes required.

--tensor-parallel-size 2
--pipeline-parallel-size 2
--distributed-executor-backend ray
--max-model-len 32768
--generation-config vllm

--tensor-parallel-size 4
--pipeline-parallel-size 1
--distributed-executor-backend ray
--max-model-len 65536
--generation-config vllm
--enable-auto-tool-choice
--tool-call-parser hermes

Additionally, RAY_CGRAPH_get_timeout is a configuration for inter-stage communication timeout in pipeline_parallel, so it becomes unnecessary when pipeline_parallel=1. It can be safely removed from environment variables.

Comparison of Effects from Changes #

# Before change (handled by 2 GPUs)
Model loading took 15-17 GiB / GPU

# After change (handled by 4 GPUs)
Model loading took 6.95 GiB / GPU

# Before change
KV cache: Limited (VRAM constrained)

# After change
Available KV cache memory: 20.97 GiB
GPU KV cache size: 458,

Extending Context Length to 64K #

With tensor_parallel=4 providing VRAM headroom, we attempted to extend context length.

Add to Environment Variables:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

Backend Parameters:

--max-model-len 65536

Startup verification:

INFO: Using max model len 65536
INFO: Available KV cache memory: 20.97 GiB
INFO: GPU KV cache size: 458,
INFO: Maximum concurrency for 32, per request: 13.98x
INFO: Starting vLLM API server ← Normal startup

Final Stable Operation Configuration Summary #

--distributed-executor-backend ray
--tensor-parallel-size 4
--pipeline-parallel-size 1
--max-model-len 65536
--generation-config vllm
--enable-auto-tool-choice
--tool-call-parser hermes

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

--ipc=host
--shm-size=64g

Checklist #

• In V100 environments, use --pipeline-parallel-size 1 (pipeline_parallel uses P2P GPU communication, which V100 does not support)
• When using 4 GPUs in multi-node configuration, unify to --tensor-parallel-size 4
• RAY_CGRAPH_get_timeout is unnecessary with pipeline_parallel=1 and can be removed
• For 64K context, add VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 to Environment Variables
• Start the gpustack-worker container with --ipc=host --shm-size=64g options
• Perform load testing with long text and concurrent requests before production deployment to verify stability