Reading files from remote storage

By mounting external storage systems to the Private AI container teams can take advantage of a wide array of different technologies to source data for input. This can enable simplification of the overall design for data pipelines and additionally allow teams to take advantage of the feature sets present in those underlying storage providers.

S3 Object storage

When working with s3 buckets it can be useful to mount the bucket and process files directly without the need to encode the file and transfer data over the network. The mini guide below demonstrates how to do this using the Amazon mountpoint application. There are two primary steps required to process files directly from object storage:

Configure the container / host machine for S3 access
Process files using the /process/files/uri endpoint

Prerequisites and assumptions:

The /process/files/uri endpoint assumes that all files are available in local storage in the running container. At time of writing there are no connectors or drivers deployed with the Private AI container that enable communication with any additional protocols. As such any storage volume that is mounted to docker is available via the /process/files/uri endpoint. For S3 object storage this mini-guide assumes that you have:

Active AWS account
An S3 bucket
A running Private AI instance
AWS CLI installed on the host
Appropriate software installed on the host to enable mounting an S3 bucket as local storage.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html

Step 1 - Setup S3 mount point

Ensure you have connected your AWS account. Follow the documentation here to assist with authentication: https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials The format of the command to mount the S3 bucket is fairly straight forward:

mount-s3 s3://MY_BUCKETNAME /PATH/TO/FOLDER/ON/HOST --allow-other --uid $(id -u) --gid $(id -g)

Take special note of the uid and gid, setting the appropriate user and groups for your configuration. The command above sets the logged in user/group to have permission to access the mount.

Step 2 - Modify the docker run / container start parameters to mount the local folder

The docker run command will now need to be modified to include a new volume linked to the mount point. This is done by utilizing the parameter -v on the command line. Example below:

docker run --rm -v /home/some/local/folder/mounted-s3-input:/home/ubuntu/s3-mount-input \
-v /home/some/local/folder/mounted-s3-output:/home/ubuntu/mounted-s3-output \
-e PAI_OUTPUT_FILE_DIR=/home/ubuntu/mounted-s3-output \
-e PAI_FILE_SUPPORT_ENABLED=True -v //home/some/local/folder/license.json:/app/license/license.json \
-p 8080:8080 -it  crprivateaiprod.azurecr.io/deid:4.2.2-cpu

With the steps above complete the container will start and requests can be issued using the following payload format:

curl

curl -i -X POST \
  http://localhostname/process/files/uri \
  -H 'Content-Type: application/json' \
  -d '{
    "uri": "/home/ubuntu/s3-mount-input/example.txt",
    "entity_detection": {
      "return_entity": true
    }
  }'

S3 File Support (New in v4.4.0)

The DEID container can now process files stored directly in Amazon S3 without requiring any local file mounts or volume bindings. You specify input and output locations using s3:// URIs, and the container handles downloading and uploading automatically.

Required Environment Variables

To use S3 URIs for input or output, you must set three environment variables:

Variable	Description
`PAI_AWS_ACCESS_KEY_ID`	Your AWS access key ID
`PAI_AWS_SECRET_ACCESS_KEY`	Your AWS secret access key
`PAI_AWS_DEFAULT_REGION`	The target S3 region

These three variables are required whenever an s3:// URI is used for input or output. Note that PAI_OUTPUT_FILE_DIR is always needed for the /process/files/uri endpoint, but can be set to either a local path or an s3:// URI — it is not S3-specific.

Processing Files from S3 (Input)

This is a new way to specify file paths for the /process/files/uri endpoint — instead of local mount paths, you can now pass s3:// URIs directly. For details on the /process/files/uri endpoint itself, see Processing Files with the URI Endpoint. To process files from S3 buckets, you can use either the raw HTTP API or our Python client:

Python

import requests
url = "http://localhost:8080/process/files/uri"

# Input from S3 — output follows PAI_OUTPUT_FILE_DIR
payload = {
    "uri": "s3://<your_input_bucket>/path/to/document.pdf",
    "entity_detection": {"return_entity": True},
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
results = response.json()
print(results)

This works for all supported file types.

Writing Output to S3 (Output)

Set PAI_OUTPUT_FILE_DIR to an s3:// URI when starting the container. The redacted file will be written to that path with .redacted appended to the original filename (e.g., s3://my-output-bucket/redacted/document.redacted.pdf).

Specifying S3 as Output Destination — Bucket Name Requirement

The bucket name is mandatory and must be the first segment after s3://:

s3://[bucket-name]/[optional-path/]
       ^^^^^^^^^^^^
       Required

A valid example: s3://my-output-bucket/redacted/ An invalid example (missing bucket): s3:///redacted/

Mixing Input from Local Mount + Output to S3

You can use a mounted directory for input while writing output to S3:

Python

import requests
url = "http://localhost:8080/process/files/uri"
payload = {
    "uri": "/mounted/path/document.pdf",
    "entity_detection": {"return_entity": True},
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
print(response.json())

The input is read from the local mount, and the output goes to PAI_OUTPUT_FILE_DIR (S3 in this case). Container startup example

Docker Command

docker run --rm -v <full path to your license.json file>:/app/license/license.json \
-v <full path to files>:<path in container> \
-e PAI_OUTPUT_FILE_DIR=s3://my-output-bucket/redacted/ \
-e PAI_AWS_ACCESS_KEY_ID=<your-access-key-id> \
-e PAI_AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
-e PAI_AWS_DEFAULT_REGION=ap-northeast-1 \
-p 8080:8080 -it crprivateaiprod.azurecr.io/deid:<version>

Mixing Input from S3 + Output to Local Mount

You can also use an s3:// URI for input while writing output to a locally mounted directory:

Python

import requests
url = "http://localhost:8080/process/files/uri"
payload = {
    "uri": "s3://<your_input_bucket>/path/to/document.pdf",
    "entity_detection": {"return_entity": True},
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
print(response.json())

The input is fetched from S3, and the output is written to PAI_OUTPUT_FILE_DIR (a local mount in this case). Container startup example

Docker Command

docker run --rm -v <full path to your license.json file>:/app/license/license.json \
-e PAI_OUTPUT_FILE_DIR=<path to mounted folder in container> \
-e PAI_AWS_ACCESS_KEY_ID=<your-access-key-id> \
-e PAI_AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
-e PAI_AWS_DEFAULT_REGION=ap-northeast-1 \
-p 8080:8080 -it crprivateaiprod.azurecr.io/deid:<version>

How It Works

When an s3:// URI is provided as input, the container:

Validates that the S3 bucket exists and is accessible
Fetches the file for processing. Nothing is held permanently inside the container.

Writing to S3

The container processes files synchronously. When writing to different S3 keys at the same time, each request uses its own client and target path, so there are no conflicts. If multiple requests write to the same key, the last one wins — this is standard S3 behavior. To avoid unexpected overwrites, use unique paths per user or session (e.g., include a user ID or timestamp).

Unsupported Features

Pre-signed URLs

Pre-signed S3 URLs are not supported.

Container Quickstart

Demo

Installation

Configuration & Operations

Product Guides

Entities & Languages

Translations

Using remote storage

Reading files from remote storage

S3 Object storage

Prerequisites and assumptions:

Step 1 - Setup S3 mount point

Step 2 - Modify the docker run / container start parameters to mount the local folder

S3 File Support (New in v4.4.0)

Required Environment Variables

Processing Files from S3 (Input)

Writing Output to S3 (Output)

Specifying S3 as Output Destination — Bucket Name Requirement

Mixing Input from Local Mount + Output to S3

Mixing Input from S3 + Output to Local Mount

How It Works

Writing to S3

Unsupported Features

Pre-signed URLs

​Reading files from remote storage

​S3 Object storage

​Prerequisites and assumptions:

​Step 1 - Setup S3 mount point

​Step 2 - Modify the docker run / container start parameters to mount the local folder

​S3 File Support (New in v4.4.0)

​Required Environment Variables

​Processing Files from S3 (Input)

​Writing Output to S3 (Output)

​Specifying S3 as Output Destination — Bucket Name Requirement

​Mixing Input from Local Mount + Output to S3

​Mixing Input from S3 + Output to Local Mount

​How It Works

​Writing to S3

​Unsupported Features

​Pre-signed URLs

Reading files from remote storage

S3 Object storage

Prerequisites and assumptions:

Step 1 - Setup S3 mount point

Step 2 - Modify the docker run / container start parameters to mount the local folder

S3 File Support (New in v4.4.0)

Required Environment Variables

Processing Files from S3 (Input)

Writing Output to S3 (Output)

Specifying S3 as Output Destination — Bucket Name Requirement

Mixing Input from Local Mount + Output to S3

Mixing Input from S3 + Output to Local Mount

How It Works

Writing to S3

Unsupported Features

Pre-signed URLs