Issue 1

Ray Placement Group Waits Indefinitely #

When starting the model, Waiting for creating a placement group continues endlessly from 10 seconds → 30 seconds → 150 seconds → 630 seconds, and the model does not start up.

WARNING: Tensor parallel size (4) exceeds available GPUs (2).
INFO: Waiting for creating a placement group of specs for 630 seconds.
specs=[{'node:': 0.001, 'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]

Cause: Ray only runs within the container on the head node side; the GPUs on the other worker node cannot join the Ray cluster.

Issue 2

Gloo Communication Failure Due to 127.0.1.1 Entry in /etc/hosts #

Ray nodes can connect to each other, but after model loading, distributed environment initialization (Gloo’s connectFullMesh) fails.

(RayWorkerWrapper pid=1722) ERROR failed to connect, retry=4, retryLimit=3,
  local=[127.0.0.1]:35509, remote=[127.0.1.1]:51638$1,
  error=SO_ERROR: Connection refused

RuntimeError: Gloo connectFullMesh failed with timed out connecting:
  SO_ERROR: Connection refused, remote=[127.0.1.1]:51638$1

Cause: Ubuntu/Debian’s cloud-init by default writes 127.0.1.1 hostname to /etc/hosts. Gloo references this entry and judges the node’s own IP as 127.0.1.1 (loopback), attempting to connect to other nodes and getting refused.

Solution: On each node, correct the 127.0.1.1 entry in /etc/hosts to the actual IP address.

# On GPU-Worker-A
sudo sed -i 's/127.0.1.1\s*GPU-Worker-A/ GPU-Worker-A/' /etc/hosts

# On GPU-Worker-B
sudo sed -i 's/127.0.1.1\s*GPU-Worker-B/ GPU-Worker-B/' /etc/hosts

Issue 3

Qwen3.5-35B-A3B Visual Encoder Does Not Work on V100 #

After resolving the Gloo issue, model loading succeeds but immediately crashes during the profile_run phase.

File "vllm/model_executor/layers/conv.py", line 236, in _forward_conv
    x = F.conv3d(
        ^^^^^^^^^
RuntimeError: GET was unable to find an engine to execute this computation

Cause: Qwen3.5-35B-A3B is a Vision Language Model, and its Visual Encoder’s Conv3D lacks a cuDNN algorithm compiled for V100 (compute capability 7.0). This cannot be worked around with environment variables like VLLM_DISABLE_MULTIMODAL=1. Since vLLM always initializes the Visual Encoder during profile_run, this is a structural limitation.

Solution: Switch the model to Qwen2.5-32B-Instruct (text-only). Since it has no Visual Encoder, this issue does not occur.

Issue 4

Engine Crashes Due to /dev/shm Shortage During Chat #

The model starts up, but the engine crashes the moment a chat message is sent.

WARNING (raylet) store_runner.cc:83: System memory request exceeds memory available in /dev/shm.
  The request is for 10200547328 bytes, and the amount available is 9663676416 bytes.
  If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size'

CUDA error: no kernel image is available for execution on the device

Cause: Docker’s default /dev/shm size is 64MB. Ray’s pipeline_parallel uses large amounts of shared memory for inter-node communication, which is exhausted during inference. The gpustack-worker container was not launched with --shm-size specified.

Solution: Restart the gpustack-worker container with --shm-size=16g.

docker stop gpustack-worker && docker rm gpustack-worker

docker run -d --name gpustack-worker \
  --restart=unless-stopped \
  --privileged \
  --network=host \
  --shm-size=16g \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume /models:/models \
  --volume /var/lib/gpustack-data:/var/lib/gpustack \
  --runtime nvidia \
  gpustack/gpustack:v2.1.1 \
  --server-url http://<SERVER_IP> \
  --token <TOKEN> \
  --cache-dir /models

Issue 5

Runner Container’s shm Is Not Inherited; RayChannelTimeoutError Continues #

Even after restarting gpustack-worker, the engine still crashes during chat.

ray.exceptions.RayChannelTimeoutError: System error:
  If the execution is expected to take a long time,
  increase RAY_CGRAPH_get_timeout which is currently 300 seconds.
  Otherwise, this may indicate that the execution is hanging.

Cause: The vllm/ray runner container is created fresh by GPUStack each time, so the --shm-size setting from gpustack-worker is not inherited across containers. The runner container’s own shm remains at 64MB.

Solution: Add the following to Environment Variables in GPUStack UI model settings.

RAY_CGRAPH_get_timeout=600
RAY_memory_store_capacity=17179869184

Issue 6

generation_config repetition_penalty Kernel Is Not V100-Compatible; Loop Output #

Chat produces abnormal text output with repeating symbols (♡♡♡…) in a loop. Or inference crashes with a CUDA kernel error.

torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
# ↑ Occurs during apply_penalties (repetition_penalty) execution

WARNING: Default vLLM sampling parameters have been overridden by the model's
  generation_config.json: {'repetition_penalty': 1.05, 'temperature': 0.7, ...}

Cause: The CUDA kernel for repetition_penalty set in Qwen2.5’s generation_config.json is not compiled for V100.

Solution: In GPUStack UI model settings → Backend Parameters, add:

--generation-config vllm

This ignores the model’s generation_config.json and uses vLLM’s default settings instead.