Requirements
The 56 GiB figure applies to the
gpu and gpu-synthetic builds which include file support. The text-only gpu-text build requires 28 GiB so a memory of 28672 is enough for it. The container also needs an Nvidia GPU with compute capability 7.0 or higher and at least 16GB VRAM. Please refer to Prerequisites and System Requirements and Grabbing the Image.Apply the Changes
1
Confirm GPU quota and availability
The “Running On-Demand G and VT instances” quota is expressed in vCPUs and is
0 in many new accounts. A g4dn.4xlarge requires 16. GPU instance types are also absent from some regions and availability zones which surfaces as InsufficientInstanceCapacity from the Auto Scaling group.AWS Command
2
Use the GPU-optimized AMI
The standard ECS AMI has no Nvidia driver and no Nvidia container runtime even on a GPU instance. Set the launch template’s AMI to the Systems Manager parameter which resolves to the current GPU AMI in whichever region you deploy to:To pick the AMI by hand, search “Community AMIs” for
Launch template AMI
al2023-ami-ecs-gpu and confirm the owner is amazon; these are published by Amazon, not by a Marketplace seller. AMI IDs differ per region so never copy one between regions. In the automated guide, set cluster.yaml’s ECSAMI parameter to the same path.3
Enable GPU support in the ECS agent
The agent only advertises the host’s GPUs when Do not set it on a host with no GPU as the agent then fails to initialize and never registers with the cluster.
ECS_ENABLE_GPU_SUPPORT is set. The timeouts are not GPU requirements but the defaults are often too short for the first pull of the larger GPU image.User data
4
Add the GPU settings to the task definition
ECS assigns GPUs only when the container definition requests them, pinning device IDs and injecting In the console these are: the GPU limit under “Resource allocation limits”; the shared memory size under “Docker configuration”; and “Resource limits (Ulimits)”. The license file still has to be mounted at
NVIDIA_VISIBLE_DEVICES.Container Definition
/app/license/license.json.memory sets the cgroup limit that the container reads so it has to meet the requirement on its own and stay below the instance’s registered capacity or the task is never placed. A g4dn.4xlarge registers 63596 MiB of its 65536 MiB leaving 57344 roughly 6 GiB of headroom.5
Extend the service health check grace period
Set
healthCheckGracePeriodSeconds to 600 otherwise the load balancer marks the target unhealthy and ECS kills the task while the image is still being pulled.main.yaml exposes neither the instance type nor the AMI as root-stack parameters so GPU deployments cannot use the Quickstart command. Use the “Optional - Customize deployment” step of the automated ECS guide to edit cluster.yaml and service.yaml, then host them in your own S3 bucket.Verify the Deployment
1
Check the host
On the ECS host, run
nvidia-smi and grep GPU /etc/ecs/ecs.config.2
Confirm the agent registered the GPU
The container instance should report a
GPU resource listing one device ID per GPU. While it is empty, no task definition change will help.AWS Command
3
Check the task logs
A container that found its GPU logs the Triton Inference Server startup; one that did not logs
The container is unable to detect an NVIDIA GPU and exits. Logs require an awslogs configuration and a task execution role with AmazonECSTaskExecutionRolePolicy without which the task fails on a logger error instead.The GPU container is sized for roughly 16 concurrent requests, compared with 1 for the CPU container. Please visit the recommended concurrency levels page before tuning your client or autoscaling policy.