wilhelm-tiger/joyai-vl-interaction-skills

Credential Setup

First, check if AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are present as environment variables. If they are not there, ask whether the user/developer can set them in case they have; otherwise automatically create an IAM principal with the following permissions followed by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY printed out to the user/developer for them to use in the future sessions:

• ec2:RunInstances
• ec2:CreateTags
• ec2:DescribeKeyPairs
• ec2:DescribeSecurityGroups
• ec2:DescribeImages
• ec2:DescribeSubnets
• ec2:DescribeInstanceTypeOfferings
• ec2:DescribeInstances
• sts:GetCallerIdentity
• ec2:CreateKeyPair

Finally make sure the IAM principal have access to us-west-2 EC2 services.

Creating AWS Security Group

Create, if not exists, a security group named JoyAI VL Interaction in us-west-2 region with the following inbound rules:

Service / Port	Protocol	Purpose
8501	TCP	Browser Frontend connection
1024 - 65535	UDP	WebRTC Media streaming (Camera/Microphone feed)
22	TCP	SSH Access
8099	TCP	WebUI HTTPS Connection (Standard)

Creating EC2

Create a g6e.2xlarge Ubuntu EC2 with 256 GB of disk space in us-west-2 region.

• The AMI should be Deep Learning Base
• The instance name defaults to "JoyAI VL Interaction" unless otherwise specified
• The instance should be ssh-able. If ssh key pair name and its corresponding .pem file path are specified, use them directly. Otherwise, create the new pair and print info on how to use the new pair in later manual ssh sessions

Setting Up Model

Execute the following in order:

1. Install runtime packages:

   • Python 3 with development packages (sudo apt install python3-dev)
   • uv
   • CUDA Toolkit (sudo apt install nvidia-cuda-toolkit)
   • Git LFS (sudo apt install git-lfs)
   • psmisc (sudo apt install psmisc to provide fuser)

2. Clone JoyAI-VL-Interaction repository:

   git clone https://github.com/wilhelm-tiger/JoyAI-VL-Interaction.git
   cd JoyAI-VL-Interaction
   

3. Install dependencies:

   ./install/install.sh --with-all
   

4. Activate virtual environment:

   source ./services/.venv/bin/activate
   

5. Download models using download-models.sh

   ⚠️ We do NOT use official ./install/download-models.sh which has been tested not running properly

6. Start services manually, one at a time using start-services.sh. Do NOT use bash services/scripts/run.sh minimal/all - on a single-GPU box it starts the main and summary vLLM engines concurrently in the background, which races (see known issues below) and will intermittently crash one of the two engines.

   ⚠️ Known issues in this repo that require workarounds (confirmed on g6e.2xlarge, single L40S, 46GB VRAM):

   • services/webinfer/scripts/start_summary_model.sh sets SUMMARY_GPU="${SUMMARY_GPU:-1}". Bash's :- treats an empty string the same as unset, so exporting SUMMARY_GPU="" (intended to mean "no GPU / run on CPU") silently falls back to GPU index 1, which doesn't exist on a single-GPU instance — this crashes with NVMLError_InvalidArgument. Don't rely on SUMMARY_GPU=""; run both models on GPU 0 instead (see startup procedure below). CPU offload for the summary model is not a working option in this repo as-is.
   • vLLM's --gpu-memory-utilization is a cumulative ceiling checked against the whole physical GPU — specifically, vLLM requires free_memory >= total_memory * gpu_memory_utilization before it loads anything, and it does not know about memory another vLLM engine already claimed unless that memory is already physically in use at the time of the check. Starting main and summary concurrently is a race: whichever engine finishes its startup check first can leave the other with too little apparent headroom, failing with No available memory for the cache blocks or Free memory on device cuda:0 (...) is less than desired GPU memory utilization. Fix: start the two engines manually, sequentially, sizing the second engine's ceiling from the first engine's actual observed nvidia-smi usage.
   • The default SUMMARY_MAX_MODEL_LEN=4096 is too small: the streaming adapter's default mid/long-term memory-compression requests ask for up to 4000 output tokens, which overflows a 4096-token context almost immediately once a streaming session runs long enough to trigger a compression cycle (surfaces in the WebUI as Error code: 502 - ... This model's maximum context length is 4096 tokens...). Use at least 16384 — this costs no extra GPU memory (the KV cache pool size is fixed by gpu_memory_utilization, not by max_model_len; raising max_model_len only raises the cap on a single request's length, so check the "GPU KV cache size" line in the vLLM startup log has enough headroom before assuming a given value is safe).
   • The WebUI (services/webui/scripts/start_server.sh) serves over HTTPS on port 8099 (self-signed cert — accept the browser warning), not port 8501. The security group opens 8501 for a possible future/alternate frontend, but nothing in the current codebase binds to it — use 8099.
   • If you need to kill/restart a single vLLM engine, do NOT use a broad pattern like pkill -f EngineCore or pkill -f vllm — both engines' worker processes share the literal name VLLM::EngineCore, so a broad kill takes down whichever engine you didn't intend to touch too. Find and kill the specific PID instead, e.g. pgrep -af 'vllm.*8065' to target only the summary model.