Skip to main content
The recommended level of concurrency, i.e. the optimal number of simultaneous requests to make to the container is covered below for the CPU and GPU containers. The recommended concurrency level is driven primarily by the compute requirement of Limina’s Neural Network models, such as for PII detection. For an example of how to make concurrent requests, please visit our examples repository.
The recommendations below apply to text de-identification (e.g., the process/text route). Audio de-identification does not benefit from higher concurrency — see Audio.

​
CPU

For Neural Network inference workloads, CPUs don’t require inputs to be batched together to achieve good hardware utilization. In practice, due to network overhead and pre/post-processing code it is best to use a low level of concurrency such as 1 per container instance.

​
GPU

Unlike CPUs, GPUs require inputs to be batched together and processed as a single large input to achieve optimal hardware utilization. This means that there is a tradeoff between latency and throughput. A concurrency level of 32 per container instance is a good tradeoff between latency and throughput, however 16 is the recommended number of concurrent connections to optimize for latency.

​
Audio

Audio requests are processed in a fixed sequence — speech-to-text transcription, de-identification of the transcript, and bleeping or distorting the result — and the transcription step is by far the dominant cost. Transcription is served by a single model instance per container, so audio requests are effectively processed one at a time, on both CPU and GPU. This means that raising client concurrency will not increase audio throughput.
  • For audio, start at a concurrency of 1 per container instance and measure.
  • To increase audio throughput, scale horizontally by adding container instances rather than by raising concurrency.
  • In a mixed text + audio deployment on a single GPU container, keep audio concurrency low (1–2) and reserve the 16–32 concurrency range for the text path. Since the transcription step is serialized per instance, an extra in-flight audio request simply waits in its queue — a concurrency of 2 can still let the de-identification and post-processing stages of one request overlap with the transcription of another, while anything higher only adds queuing without a throughput gain.
Regardless of the workload, we recommend confirming the performance with your own data and hardware in your deployment environment before going to production. For sizing the number of instances required for a given daily audio volume, see Audio capacity planning. The guidance above is a recommended starting point for most installation types. For both CPU and GPU instance types it is recommended that you tune the volume of simultaneous requests based on your unique traffic patterns and use case specific volumes. Acceptable latency, desired throughput, and tolerance for variability greatly influence how you manage the overall load and performance on your Limina installation.