> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getlimina.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Using remote storage

> Using remote storage

# Reading files from remote storage

By mounting external storage systems to the Private AI container teams can take advantage of a wide array of different technologies to source data for input. This can enable simplification of the overall design for data pipelines and additionally allow teams to take advantage of the feature sets present in those underlying storage providers.

## S3 Object storage

When working with s3 buckets it can be useful to mount the bucket and process files directly without the need to encode the file and transfer data over the network.
The mini guide below demonstrates how to do this using the Amazon mountpoint application.

There are two primary steps required to process files directly from object storage:

1. Configure the container / host machine for S3 access
2. Process files using the `/process/files/uri` endpoint

## Prerequisites and assumptions:

The `/process/files/uri` endpoint assumes that all files are available in local storage in the running container.
At time of writing there are no connectors or drivers deployed with the Private AI container that enable communication with any additional protocols.
As such any storage volume that is mounted to docker is available via the `/process/files/uri` endpoint.

For S3 object storage this mini-guide assumes that you have:

* Active AWS account
* An S3 bucket
* A running Private AI instance
* AWS CLI installed on the host
* Appropriate software installed on the host to enable mounting an S3 bucket as local storage.

[https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html)

## Step 1 - Setup S3 mount point

Ensure you have connected your AWS account. Follow the documentation here to assist with authentication: [https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials)

The format of the command to mount the S3 bucket is fairly straight forward:

```shell theme={"theme":"poimandres"}
mount-s3 s3://MY_BUCKETNAME /PATH/TO/FOLDER/ON/HOST --allow-other --uid $(id -u) --gid $(id -g)
```

Take special note of the uid and gid, setting the appropriate user and groups for your configuration. The command above sets the logged in user/group to have permission to access the mount.

## Step 2 - Modify the docker run / container start parameters to mount the local folder

The docker run command will now need to be modified to include a new volume linked to the mount point.

This is done by utilizing the parameter -v on the command line.

Example below:

```shell theme={"theme":"poimandres"}
docker run --rm -v /home/some/local/folder/mounted-s3-input:/home/ubuntu/s3-mount-input \
-v /home/some/local/folder/mounted-s3-output:/home/ubuntu/mounted-s3-output \
-e PAI_OUTPUT_FILE_DIR=/home/ubuntu/mounted-s3-output \
-e PAI_FILE_SUPPORT_ENABLED=True -v //home/some/local/folder/license.json:/app/license/license.json \
-p 8080:8080 -it  crprivateaiprod.azurecr.io/deid:4.2.2-cpu
```

With the steps above complete the container will start and requests can be issued using the following payload format:

```shell curl theme={"theme":"poimandres"}
curl -i -X POST \
  http://localhostname/process/files/uri \
  -H 'Content-Type: application/json' \
  -d '{
    "uri": "/home/ubuntu/s3-mount-input/example.txt",
    "entity_detection": {
      "return_entity": true
    }
  }'
```

## S3 File Support (New in v4.4.0)

The DEID container can now process files stored directly in Amazon S3 without requiring any local file mounts or volume bindings. You specify input and output locations using `s3://` URIs, and the container handles downloading and uploading automatically.

### Required Environment Variables

To use S3 URIs for input or output, you must set three environment variables:

| Variable                    | Description                |
| --------------------------- | -------------------------- |
| `PAI_AWS_ACCESS_KEY_ID`     | Your AWS access key ID     |
| `PAI_AWS_SECRET_ACCESS_KEY` | Your AWS secret access key |
| `PAI_AWS_DEFAULT_REGION`    | The target S3 region       |

These three variables are required whenever an `s3://` URI is used for input or output. Note that `PAI_OUTPUT_FILE_DIR` is always needed for the `/process/files/uri` endpoint, but can be set to either a local path or an `s3://` URI — it is not S3-specific.

### Processing Files from S3 (Input)

This is a new way to specify file paths for the `/process/files/uri` endpoint — instead of local mount paths, you can now pass `s3://` URIs directly. For details on the `/process/files/uri` endpoint itself, see [Processing Files with the URI Endpoint](/configuration-and-operations/working-with-files/processing-files/#processing-files-with-the-uri-endpoint).

To process files from S3 buckets, you can use either the raw HTTP API or our Python client:

```python Python wrap lines theme={"theme":"poimandres"}
import requests
url = "http://localhost:8080/process/files/uri"

# Input from S3 — output follows PAI_OUTPUT_FILE_DIR
payload = {
    "uri": "s3://<your_input_bucket>/path/to/document.pdf",
    "entity_detection": {"return_entity": True},
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
results = response.json()
print(results)
```

This works for all [supported file types](/configuration-and-operations/working-with-files/supported-file-types).

### Writing Output to S3 (Output)

Set `PAI_OUTPUT_FILE_DIR` to an `s3://` URI when starting the container. The redacted file will be written to that path with `.redacted` appended to the original filename (e.g., `s3://my-output-bucket/redacted/document.redacted.pdf`).

#### Specifying S3 as Output Destination — Bucket Name Requirement

The bucket name is **mandatory** and must be the first segment after `s3://`:

```
s3://[bucket-name]/[optional-path/]
       ^^^^^^^^^^^^
       Required
```

A valid example: `s3://my-output-bucket/redacted/`
An invalid example (missing bucket): `s3:///redacted/`

### Mixing Input from Local Mount + Output to S3

You can use a mounted directory for input while writing output to S3:

```python Python wrap lines theme={"theme":"poimandres"}
import requests
url = "http://localhost:8080/process/files/uri"
payload = {
    "uri": "/mounted/path/document.pdf",
    "entity_detection": {"return_entity": True},
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
print(response.json())
```

The input is read from the local mount, and the output goes to `PAI_OUTPUT_FILE_DIR` (S3 in this case).

**`Container startup example`**

```shell Docker Command wrap theme={"theme":"poimandres"}
docker run --rm -v <full path to your license.json file>:/app/license/license.json \
-v <full path to files>:<path in container> \
-e PAI_OUTPUT_FILE_DIR=s3://my-output-bucket/redacted/ \
-e PAI_AWS_ACCESS_KEY_ID=<your-access-key-id> \
-e PAI_AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
-e PAI_AWS_DEFAULT_REGION=ap-northeast-1 \
-p 8080:8080 -it crprivateaiprod.azurecr.io/deid:<version>
```

### Mixing Input from S3 + Output to Local Mount

You can also use an `s3://` URI for input while writing output to a locally mounted directory:

```python Python wrap lines theme={"theme":"poimandres"}
import requests
url = "http://localhost:8080/process/files/uri"
payload = {
    "uri": "s3://<your_input_bucket>/path/to/document.pdf",
    "entity_detection": {"return_entity": True},
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
print(response.json())
```

The input is fetched from S3, and the output is written to `PAI_OUTPUT_FILE_DIR` (a local mount in this case).

**`Container startup example`**

```shell Docker Command wrap theme={"theme":"poimandres"}
docker run --rm -v <full path to your license.json file>:/app/license/license.json \
-e PAI_OUTPUT_FILE_DIR=<path to mounted folder in container> \
-e PAI_AWS_ACCESS_KEY_ID=<your-access-key-id> \
-e PAI_AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
-e PAI_AWS_DEFAULT_REGION=ap-northeast-1 \
-p 8080:8080 -it crprivateaiprod.azurecr.io/deid:<version>
```

### How It Works

When an `s3://` URI is provided as input, the container:

1. **Validates** that the S3 bucket exists and is accessible
2. Fetches the file for processing. Nothing is held permanently inside the container.

## Writing to S3

The container processes files synchronously. When writing to different S3 keys at the same time, each request uses its own client and target path, so there are no conflicts.

If multiple requests write to the **same key**, the last one wins — this is standard S3 behavior. To avoid unexpected overwrites, use unique paths per user or session (e.g., include a user ID or timestamp).

## Unsupported Features

### Pre-signed URLs

Pre-signed S3 URLs are not supported.