# Acknowledgements and Licenses Source: https://docs.getlimina.ai/acknowledgement Limina's models are partly trained on data from the C4 and mC4 datasets which are made available under the [ODC Attribution License](https://opendatacommons.org/licenses/by/1-0/). Limina's software utilizes libraries that fall under the BSD license. Please contact [support@getlimina.ai](mailto:support@getlimina.ai) for the complete list. # Detect, Parse, and Validate Entities in Text Source: https://docs.getlimina.ai/configuration-and-operations/advanced-features/analyze-text This guide explains how to use Limina to analyze your text by detecting, parsing, and validating PII in text In order to run the example code in this guide, please sign up for your [free test API key here](https://portal.getlimina.ai/?utm_source=docs\&utm_medium=website\&utm_campaign=guides\&utm_content=guides). In addition to de-identification and redaction, Limina also supports entity detection and validation. The [`analyze/text`](/latest/analyze-text) route described below is an essential tool for exploring and structuring your data as well as creating statistics around your data. In this guide, we demonstrate how to use the [`analyze/text`](/latest/analyze-text) endpoint introduced in `4.1` to return the analysis results of the detected entities, with examples of how these results can be used to meet your own use cases. ## Analyze entities in text (new in 4.1) The [`analyze/text`](/latest/analyze-text) route returns a list of detected entities along with the formatted text for each entity and a description of its subtypes. In this guide, we provide payloads to the Limina's [`analyze/text`](/latest/analyze-text) REST API route and document the associated responses. To better illustrate how this information can be used, we proceed by giving a series of common use cases. ### Validation and custom redaction of credit card numbers Some numerical entities integrate a checksum in their values. This checksum is used to confirm the entity's validity and to minimize the chance of error during transcription. This is the case for credit card numbers, which must satisfy the [Luhn algorithm](https://en.wikipedia.org/wiki/Luhn_algorithm). The [`analyze/text`](/latest/analyze-text) route implements this algorithm on top of the NER model detection. This provides an additional safeguard by ensuring that the detected number is indeed a valid credit card number. Let's look at three specific examples including credit card numbers. ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "Okay, hang on just a second because I got to get it. Okay, it is 6578-7790-4346-2237. Expiration. 1224.", "All right, I'm ready. 800 678-457-7896. Expiration is one. 224.", "CC_type: Diners Club International RuPay Visa JCB Amex CCN: 30569309025904 4242424242424242 4222222222222 6172873484776530 378282246310005 CC_CVC: 480 902 182 765 143 CC_Expiredate: 5/28 6/67 12/67 11/29 9/70" ], "locale": "en-US", "entity_detection": { "accuracy": "high", "entity_types": [ { "type": "ENABLE", "value": ["CREDIT_CARD"] } ] } } ``` ```json Response Body wrap lines theme={"theme":"poimandres"} [ { "entities": [ { "text": "6578-7790-4346-2237", "location": { "stt_idx": 65, "end_idx": 84 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 0.9022786834023215 }, "analysis_result": { "formatted": "6578 7790 4346 2237", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "invalid" } ] } } ], "entities_present": true, "characters_processed": 103, "languages_detected": { "en": 0.9202778935432434 } }, { "entities": [ { "text": "800 678-457-7896", "location": { "stt_idx": 22, "end_idx": 40 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 0.9012922777069939 }, "analysis_result": { "subtypes": [], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 65, "languages_detected": { "en": 0.8164065480232239 } }, { "entities": [ { "text": "30569309025904", "location": { "stt_idx": 60, "end_idx": 74 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 1.0 }, "analysis_result": { "formatted": "3056 9309 025 904", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "valid" } ] } }, { "text": "4242424242424242", "location": { "stt_idx": 75, "end_idx": 91 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 1.0 }, "analysis_result": { "formatted": "4242 4242 4242 4242", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "valid" } ] } }, { "text": "4222222222222", "location": { "stt_idx": 92, "end_idx": 105 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 1.0 }, "analysis_result": { "formatted": "4222 222 222 222", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "valid" } ] } }, { "text": "6172873484776530", "location": { "stt_idx": 106, "end_idx": 122 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 0.9088553956576756 }, "analysis_result": { "formatted": "6172 8734 8477 6530", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "invalid" } ] } }, { "text": "378282246310005", "location": { "stt_idx": 123, "end_idx": 138 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 1.0 }, "analysis_result": { "formatted": "3782 8224 6310 005", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "valid" } ] } } ], "entities_present": true, "characters_processed": 208, "languages_detected": { "en": 0.24319741129875183 } } ] ``` The above request contains two fields, `text` and `entity_detection`, that are shared by the [`analyze/text`](/latest/analyze-text), the [`ner/text`](/latest/ner-text) and the [`process/text`](/latest/process-text) routes. The `text` field contains the text to analyze and the `entity_detection` field contains the NER configurations (e.g., the list of entities to detect). One last field in the request, `locale`, is unique to the [`analyze/text`](/latest/analyze-text) request. The `locale` field is used as a hint to the analyzer to help parse dates and other locale-dependent entities. For example, setting `locale` to `en-US` will force the analyzer to interpret the date `12-10-2020` as December 10, 2020 instead of October 12, 2020. Several example of values that can take these fields are provided below. The full response above is a mouthful, so let's look at the first example's response in more detail. ```json JSON Response with CREDIT_CARD entity wrap lines highlight={13-22} theme={"theme":"poimandres"} { "entities": [ { "text": "6578-7790-4346-2237", "location": { "stt_idx": 65, "end_idx": 84 }, "best_label": "CREDIT_CARD", "labels": { "CREDIT_CARD": 0.9022786834023215 }, "analysis_result": { "formatted": "6578 7790 4346 2237", "subtypes": [], "validation_assertions": [ { "provider": "luhn", "status": "invalid" } ] } } ], "entities_present": true, "characters_processed": 103, "languages_detected": { "en": 0.9202778935432434 } } ``` The response contains three main parts: * the entity information including its **text** and its **location**. Those fields are shared across other routes including the [`ner/text`](/latest/ner-text) and [`process/text`](/latest/process-text) routes and have the same use. * the **formatted** text of the entity. This field is unique to the [`analyze/text`](/latest/analyze-text) route and provides a "standard" format for the entity. This can facilitate the introduction of post-processing logic on detected entities. The formats are described in the following table. | Entity Type | Format | Example | | ---------------------- | --------------------------------------- | ------------------------- | | CREDIT\_CARD | space-separated groups of 3 to 5 digits | 6578 7790 4346 2237 | | DATE | ISO-8601 | 2025-03-20T18:00:00+00:00 | | DOB | ISO-8601 | 2025-03-20 | | AGE | decimal numeral | 12 | | All other entity types | no formatting | - | * a list of **validation assertions** on the entity, which is also unique to the [`analyze/text`](/latest/analyze-text) route. It contains a list of objects that are specific to the entity being detected. In this example, the `provider` is the Luhn algorithm that was run on the credit card number and the result of the algorithm is provided as part of the `status` field. Currently, only credit card numbers contain validation assertions but more assertion providers will be added in the future. The analysis result of this first example can be summed up in the following way. The credit card was successfully parsed and the parsed result is placed in the `formatted` field. However, although the number matches the credit card number format, the Luhn check failed on the number, so it is not a valid credit card number. This could be the result of a transcription error, for example. The information included in the analysis result allows the creation of custom redaction of entities, using the post-processing framework, as shown in [this section](/product-guides/thin-client#custom-redaction-of-credit-card-numbers). ### Date shifting and custom redaction of dates Dates are one type of PII that is encountered in almost every dataset. Redaction is one way to ensure that sensitive dates do not create privacy issues. However, fully redacting dates often reduces the utility of the redacted data. For dates, it is often preferable to use other obfuscation methods that preserve their utility. Two well-known techniques are date shifting and date bucketing. Let's consider three examples containing dates. ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "$MDT $MRK $QRVO $TSS & 5 more stock picks for LONG swings: https://t.co/CbkieXxqoR (July 10 2018) https://t.co/eit53RUY4g", "Short sale volume (not short interest) for $KBE on 2018-07-09 is 42%. https://t.co/7pWbgjJ8Ag $FOXA 38% $TVIX 34% $LITE 54% $HIG 60%", "$WLTW high OI range is 160 to 155 for option expiration 07/20/2018 #options https://t.co/BnVElKBKkJ" ], "locale": "en-US", "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["DATE", "DOB", "DAY", "MONTH", "YEAR"] } ] } } ``` ```json Response Body wrap lines theme={"theme":"poimandres"} [ { "entities": [ { "text": "July 10 2018", "location": { "stt_idx": 89, "end_idx": 101 }, "best_label": "DATE", "labels": { "DATE": 0.9400081038475037, "MONTH": 0.3111259341239929, "DAY": 0.31207050879796344, "YEAR": 0.29245950778325397 }, "analysis_result": { "formatted": "2018-07-10T00:00:00", "subtypes": [ { "text": "10", "formatted": "10", "label": "DAY", "location": { "stt_idx": 94, "end_idx": 96 } }, { "text": "July", "formatted": "7", "label": "MONTH", "location": { "stt_idx": 89, "end_idx": 93 } }, { "text": "2018", "formatted": "2018", "label": "YEAR", "location": { "stt_idx": 97, "end_idx": 101 } } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 126, "languages_detected": { "en": 0.6427053809165955 } }, { "entities": [ { "text": "2018-07-09", "location": { "stt_idx": 51, "end_idx": 61 }, "best_label": "DATE", "labels": { "DATE": 0.9267139077186585, "YEAR": 0.17909334897994994, "MONTH": 0.18299812078475952, "DAY": 0.18503443002700806 }, "analysis_result": { "formatted": "2018-07-09T00:00:00", "subtypes": [ { "text": "09", "formatted": "9", "label": "DAY", "location": { "stt_idx": 59, "end_idx": 61 } }, { "text": "07", "formatted": "7", "label": "MONTH", "location": { "stt_idx": 56, "end_idx": 58 } }, { "text": "2018", "formatted": "2018", "label": "YEAR", "location": { "stt_idx": 51, "end_idx": 55 } } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 132, "languages_detected": { "en": 0.5451536178588867 } }, { "entities": [ { "text": "07/20/2018", "location": { "stt_idx": 56, "end_idx": 66 }, "best_label": "DATE", "labels": { "DATE": 0.9359936833381652, "MONTH": 0.18900736570358276, "DAY": 0.18550281524658202, "YEAR": 0.18460171222686766 }, "analysis_result": { "formatted": "2018-07-20T00:00:00", "subtypes": [ { "text": "20", "formatted": "20", "label": "DAY", "location": { "stt_idx": 59, "end_idx": 61 } }, { "text": "07", "formatted": "7", "label": "MONTH", "location": { "stt_idx": 56, "end_idx": 58 } }, { "text": "2018", "formatted": "2018", "label": "YEAR", "location": { "stt_idx": 62, "end_idx": 66 } } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 99, "languages_detected": { "en": 0.7047932744026184 } } ] ``` Let's look at one specific date entity in the above response. ```json Response body with formatted date entities lines highlight={14-45} theme={"theme":"poimandres"} { "text": "July 10 2018", "location": { "stt_idx": 89, "end_idx": 101 }, "best_label": "DATE", "labels": { "DATE": 0.9400081038475037, "MONTH": 0.3111259341239929, "DAY": 0.31207050879796344, "YEAR": 0.29245950778325397 }, "analysis_result": { "formatted": "2018-07-10T00:00:00", "subtypes": [ { "text": "10", "formatted": "10", "label": "DAY", "location": { "stt_idx": 94, "end_idx": 96 } }, { "text": "July", "formatted": "7", "label": "MONTH", "location": { "stt_idx": 89, "end_idx": 93 } }, { "text": "2018", "formatted": "2018", "label": "YEAR", "location": { "stt_idx": 97, "end_idx": 101 } } ], "validation_assertions": [] } } ``` Many pieces of information are accessible from the `analysis_result` object. First, it is possible to access the formatted date "2018-07-10T00:00:00" from the field `analysis_result.formatted`. If you plan to implement logic on the dates found in the text, it might be easier to access the formatted dates rather than the original, non-standard date formats (e.g., "July 10 2018"). Also, it is possible to directly access the day, month, and year of the date entity via the response fields in `analysis_result.subtypes`. This information can be used to partially redact or to bucket dates. \ An example of redacting the day and month but keeping the year is provided in the [custom redaction of dates guide](/product-guides/thin-client#custom-redaction-of-dates). ### Age bucketing and custom redaction of numbers Similar to dates, it is possible to analyze ages and other numerical entities to create custom redaction. Consider these two examples. ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "A 32-year old Black female German citizen living in Germany wants to travel to the United States for leisure.", "West Point Public School division provides school-based preschool services for children from two through nine years of age who are children at risk and children with identified disabilities or delays." ], "link_batch": false, "locale": "en-US", "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["AGE"] } ] } } ``` ```json Response Body wrap lines theme={"theme":"poimandres"} [ { "entities": [ { "text": "32", "location": { "stt_idx": 2, "end_idx": 4 }, "best_label": "AGE", "labels": { "AGE": 0.9668179750442505 }, "analysis_result": { "formatted": 32, "subtypes": [], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 109, "languages_detected": { "en": 0.9611877202987671 } }, { "entities": [ { "text": "two", "location": { "stt_idx": 93, "end_idx": 96 }, "best_label": "AGE", "labels": { "AGE": 0.9462096095085144 }, "analysis_result": { "formatted": 2, "subtypes": [], "validation_assertions": [] } }, { "text": "nine", "location": { "stt_idx": 105, "end_idx": 109 }, "best_label": "AGE", "labels": { "AGE": 0.9411536455154419 }, "analysis_result": { "formatted": 9, "subtypes": [], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 200, "languages_detected": { "en": 0.9786704778671265 } } ] ``` Using the [Limina python client](https://pypi.org/project/privateai-client/), one can use the above [`analyze/text`](/latest/analyze-text) response to bucketize ages, as shown [here](/product-guides/thin-client#custom-redaction-of-ages). ### Custom redaction of addresses The GDPR and other privacy legislations impose strict requirements regarding the redaction of addresses. In the following scenario, we demonstrate how to partially redact an address by leaving only the less sensitive characters of a zip/postal code and removing all other address information (e.g., civic number, street name, and so on). ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "Please deliver this to 45, Clybaun Heights, Galway City, Ireland H91 AKK3", "3255 M-A-D-D-A-M-S street, huntington, west virginia is his birthplace", "My favorite city is San Francisco, California 94110, United States, 37.7749° N, 122.4194° W" ], "locale": "en-US" } ``` ```json Response Body wrap lines theme={"theme":"poimandres"} [ { "entities": [ { "text": "45, Clybaun Heights, Galway City, Ireland H91 AKK3", "location": { "stt_idx": 23, "end_idx": 73 }, "best_label": "LOCATION_ADDRESS", "labels": { "LOCATION_ADDRESS_STREET": 0.3171827793121338, "LOCATION": 0.9123516889179454, "LOCATION_ADDRESS": 0.9221759648884044, "LOCATION_CITY": 0.16148114204406738, "LOCATION_COUNTRY": 0.05482322678846471, "LOCATION_ZIP": 0.26978740271400004 }, "analysis_result": { "subtypes": [ { "text": "45, Clybaun Heights", "label": "LOCATION_ADDRESS_STREET", "location": { "stt_idx": 23, "end_idx": 42 } }, { "text": "Galway City", "label": "LOCATION_CITY", "location": { "stt_idx": 44, "end_idx": 55 } }, { "text": "Ireland", "label": "LOCATION_COUNTRY", "location": { "stt_idx": 57, "end_idx": 64 } }, { "text": "H91 AKK3", "label": "LOCATION_ZIP", "location": { "stt_idx": 65, "end_idx": 73 } } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 73, "languages_detected": { "en": 0.8342836499214172 } }, { "entities": [ { "text": "3255 M-A-D-D-A-M-S street, huntington, west virginia", "location": { "stt_idx": 0, "end_idx": 52 }, "best_label": "LOCATION_ADDRESS", "labels": { "LOCATION_ADDRESS_STREET": 0.6232224106788635, "LOCATION_ADDRESS": 0.9109632035960322, "LOCATION": 0.8909260371456975, "LOCATION_CITY": 0.07817105106685472, "LOCATION_STATE": 0.12203486328539641 }, "analysis_result": { "subtypes": [ { "text": "3255 M-A-D-D-A-M-S street", "label": "LOCATION_ADDRESS_STREET", "location": { "stt_idx": 0, "end_idx": 25 } }, { "text": "huntington", "label": "LOCATION_CITY", "location": { "stt_idx": 27, "end_idx": 37 } }, { "text": "west virginia", "label": "LOCATION_STATE", "location": { "stt_idx": 39, "end_idx": 52 } } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 70, "languages_detected": { "en": 0.9467829465866089 } }, { "entities": [ { "text": "San Francisco, California 94110, United States, 37.7749\u00b0 N, 122.4194\u00b0 W", "location": { "stt_idx": 20, "end_idx": 91 }, "best_label": "LOCATION", "labels": { "LOCATION_CITY": 0.080466923614343, "LOCATION": 0.8993716637293497, "LOCATION_ADDRESS": 0.200799106930693, "LOCATION_STATE": 0.03926792989174525, "LOCATION_ZIP": 0.12127648045619328, "LOCATION_COUNTRY": 0.07723071426153183, "LOCATION_COORDINATE": 0.4833615819613139 }, "analysis_result": { "subtypes": [ { "text": "San Francisco", "label": "LOCATION_CITY", "location": { "stt_idx": 20, "end_idx": 33 } }, { "text": "California", "label": "LOCATION_STATE", "location": { "stt_idx": 35, "end_idx": 45 } }, { "text": "94110", "label": "LOCATION_ZIP", "location": { "stt_idx": 46, "end_idx": 51 } }, { "text": "United States", "label": "LOCATION_COUNTRY", "location": { "stt_idx": 53, "end_idx": 66 } }, { "text": "37.7749\u00b0 N, 122.4194\u00b0 W", "label": "LOCATION_COORDINATE", "location": { "stt_idx": 68, "end_idx": 91 } } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 91, "languages_detected": { "en": 0.7658711075782776 } } ] ``` The above request contains three examples containing addresses. The corresponding [`analyze/text`](/latest/analyze-text) response contains the result of the analysis. This response, along with the [corresponding Limina client post-processing code](/product-guides/thin-client#custom-redaction-of-locations), can be used to mask street addresses, in order to hide the most sensitive information. ## Relation detection Relation detection refers to the broader natural language processing (NLP) capability of understanding how entities in a text are connected. While entity recognition tells us what the entities are (e.g., a person's name, a company, a location), relation detection tells us how those entities are related. Relation detection covers tasks like [coreference resolution](#coreference-resolution) and [relation extraction](#relation-extraction), both of which are supported, and together provide a deeper understanding of unstructured text. The [`analyze/text`](/latest/analyze-text) route can be used to configure relation detection by using the optional `relation_detection` field in the request. ### Coreference Resolution [Coreference resolution](/configuration-and-operations/entity-detection-and-redaction/customizing-redaction#what-is-coreference-resolution) is the task of identifying different entity mentions in a given text that refer to the same real-world entity. The `relation_detection` field offers a configurable option for coreference resolution: * **coreference\_resolution**: Specifies the method for identifying coreferential entities: * `heuristics`: Uses rule-based methods * `model_prediction`: Uses machine learning models * `combined`: Uses both approaches ```json Request Body lines wrap theme={"theme":"poimandres"} { "text": [ "Nikola Jokić (Serbian Cyrillic: Никола Јокић, pronounced [nǐkola jôkitɕ] ⓘ; born February 19, 1995) is a Serbian professional basketball player who is a center for the Denver Nuggets of the National Basketball Association (NBA). Jokić was born in the city of Sombor in the northern part of Serbia. He grew up in a cramped two-bedroom apartment that housed him and his two brothers." ], "entity_detection": { "accuracy": "high" }, "locale": "en-US", "relation_detection": { "coreference_resolution": "model_prediction" } } ``` ```json Response Body lines theme={"theme":"poimandres"} [ { "entities": [ { "text": "Nikola Jokić", "location": { "stt_idx": 0, "end_idx": 12 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.2300402671098709, "NAME": 0.9172913134098053, "NAME_FAMILY": 0.6867769062519073 }, "coreference_id": "56c15276-33da-4726-bc81-369074049222" }, { "text": "Serbian Cyrillic", "location": { "stt_idx": 14, "end_idx": 30 }, "best_label": "LANGUAGE", "labels": { "LANGUAGE": 0.94222651720047 }, "coreference_id": "0d6296d4-c453-4c73-9415-5abc527a38e5" }, { "text": "Никола Јокић", "location": { "stt_idx": 32, "end_idx": 44 }, "best_label": "NAME_GIVEN", "labels": { "NAME": 0.842899182013103, "NAME_GIVEN": 0.6497380946363721, "NAME_FAMILY": 0.07980045356920787 }, "coreference_id": "56c15276-33da-4726-bc81-369074049222" }, { "text": "nǐkola jôkitɕ", "location": { "stt_idx": 58, "end_idx": 71 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.4644567847251892, "NAME": 0.8982340276241303, "NAME_FAMILY": 0.44664961099624634 }, "coreference_id": "56c15276-33da-4726-bc81-369074049222" }, { "text": "February 19, 1995", "location": { "stt_idx": 81, "end_idx": 98 }, "best_label": "DOB", "labels": { "DOB": 0.9391335248947144 }, "analysis_result": { "formatted": "1995-02-19T00:00:00", "subtypes": [ { "formatted": "19", "label": "DAY" }, { "formatted": "2", "label": "MONTH" }, { "formatted": "1995", "label": "YEAR" } ], "validation_assertions": [] }, "coreference_id": "65e20278-31c8-4cfb-ad73-1e24db5fcd8e" }, { "text": "Serbian", "location": { "stt_idx": 105, "end_idx": 112 }, "best_label": "ORIGIN", "labels": { "ORIGIN": 0.9151841402053833 }, "coreference_id": "43af91fe-7868-4469-a70b-c22cfcd917e2" }, { "text": "professional basketball player", "location": { "stt_idx": 113, "end_idx": 143 }, "best_label": "OCCUPATION", "labels": { "OCCUPATION": 0.8843509753545126 }, "coreference_id": "e8fc3654-89ab-4cec-806a-ec97f13a9673" }, { "text": "center", "location": { "stt_idx": 153, "end_idx": 159 }, "best_label": "OCCUPATION", "labels": { "OCCUPATION": 0.8316260576248169 }, "coreference_id": "8ca28112-9a34-492e-8b1f-4b9fc72c0b1f" }, { "text": "Denver Nuggets", "location": { "stt_idx": 168, "end_idx": 182 }, "best_label": "ORGANIZATION", "labels": { "LOCATION_CITY": 0.48198258876800537, "ORGANIZATION": 0.9154168367385864, "LOCATION": 0.4703272879123688 }, "coreference_id": "de750d67-78eb-4606-8aec-6c2f697e9c50" }, { "text": "National Basketball Association", "location": { "stt_idx": 190, "end_idx": 221 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.9143192768096924 }, "coreference_id": "b907f506-1492-40b2-915a-1c472fc1efe8" }, { "text": "NBA", "location": { "stt_idx": 223, "end_idx": 226 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8653480410575867 }, "coreference_id": "b907f506-1492-40b2-915a-1c472fc1efe8" }, { "text": "Jokić", "location": { "stt_idx": 229, "end_idx": 234 }, "best_label": "NAME_FAMILY", "labels": { "NAME_FAMILY": 0.9177489876747131, "NAME": 0.9098437031110128 }, "coreference_id": "56c15276-33da-4726-bc81-369074049222" }, { "text": "Sombor", "location": { "stt_idx": 259, "end_idx": 265 }, "best_label": "LOCATION_CITY", "labels": { "LOCATION_CITY": 0.925605853398641, "LOCATION": 0.9114498297373453 }, "coreference_id": "4b204f6e-a2ed-4c45-a268-d5a20d477478" }, { "text": "Serbia", "location": { "stt_idx": 290, "end_idx": 296 }, "best_label": "LOCATION_COUNTRY", "labels": { "LOCATION_COUNTRY": 0.9711890816688538, "LOCATION": 0.9073220491409302 }, "coreference_id": "ef54dfa2-5bae-4081-8b9d-2dc0ddf8868c" } ], "entities_present": true, "characters_processed": 381, "languages_detected": { "en": 0.9837551116943359 } } ] ``` The response includes a key element for each entity: * **coreference\_id**: A unique identifier added to each entity that groups coreferential entities under a common label. This behavior matches the `/process/text` endpoint when `processed_text` is set to MARKER and coreference resolution is applied. For example, "Nikola Jokić", "Никола Јокић", "nǐkola jôkitɕ", and "Jokić" all share the same `coreference_id` (`56c15276-33da-4726-bc81-369074049222`), indicating that they refer to the same person. For an example of how to use the coreference information from the API to replace all mentions of a person with their initials, see the [custom redaction of coreferenced names](/product-guides/thin-client#custom-redaction-of-coreferenced-names) example. ### Relation Extraction Relation extraction is the task of identifying meaningful relations between entities in text, such as person-to-person or person-to-location links. It helps unlock document-level understanding by connecting pieces of information and making it easier to de-identify related data. Let's look at an example: ```text Relation Extraction Sample Text wrap theme={"theme":"poimandres"} Nessa Jonsson was born and raised in Sweden. Her father, Erik, emigrated to the United States when she was a baby. He died in 1980. ``` In the text above, there are four entities: * Nessa Jonsson (`NAME`) * Sweden (`LOCATION_COUNTRY`) * Erik (`NAME_GIVEN`) * the United States (`LOCATION_COUNTRY`) * 1980 (`DATE_INTERVAL`) Relation extraction can be used to identify the semantic connections between those entities, such as: * Nessa Jonsson **is born in** Sweden * Nessa Jonsson **is the daughter of** Erik * Erik **is the father of** Nessa Jonsson * Erik **lived in** the United States * Erik **died in** 1980 Relation extraction plays a key role in document understanding by uncovering how entities are connected, which enables systems to move toward structured, contextualized information. In domains like healthcare and finance, this unlocks the potential of unstructured text by identifying relationships like family connections, places of origin, or dates of birth. ### Limina and Relation Extraction (Beta) Limina's de-identification service offers the ability to use relation extraction on its `analyze/text` endpoint. Relation extraction is currently implemented on top of both the named entity recognition (NER) and the coreference resolution models. It is, therefore, limited to predicting relations between clusters of coreferenced entities. Currently, the system supports a single generalized relation type: `RELATED_TO`, which is used to capture all of the supported semantic relations between a person and another entity: * Kinship - a relation between two `NAME`s (or other variants, e.g. `NAME_GIVEN`) indicating family or close personal relationships between individuals. These may include parent-child, siblings, spouses, etc. A kinship relation is always bi-directional. * Place of birth - a relation between `NAME` and `LOCATION` entities, indicating the location where the person was born. This can refer to a city, state, country, or region. * Citizenship - a relation between `NAME` and `LOCATION` or `ORIGIN` entities, indicating nationality or legal citizenship of the person. * Origin - a relation between `NAME` and `ORIGIN` entities, indicating the country a person originally comes from, reflecting ancestry or cultural background rather than legal status or birthplace. * Date of birth - a relation between `NAME` and `DOB` entities, indicating birth date. * Date of death - a relation between `NAME` and `DATE` or `DATE_INTERVAL` entities, indicating the date of death of a person. For the example above, the system will extract the following relations: * Nessa Jonsson → `RELATED_TO` → Sweden * Nessa Jonsson → `RELATED_TO` → Erik * Erik → `RELATED_TO` → Nessa Jonsson The relation extraction feature can be enabled as part of the [`analyze/text`](/latest/analyze-text) endpoint by setting the field `enable_relation_extraction` to `true`. * `enable_relation_extraction`: Controls whether relation extraction is performed during analysis. * `true`: Enables relation extraction * `false` (default): Disables relation extraction **Relation Extraction and Coreference Resolution** Relation extraction relies on coreference resolution to group people mentions in text. Make sure a non-null value is set for `coreference_resolution` before setting `enable_relation_extraction` to true. Here is an example of how to enable relation extraction in your request using the `analyze/text` endpoint. Notice the `enable_relation_extraction` field within the `relation_detection` object. ```json Request Body wrap wrap lines theme={"theme":"poimandres"} { "text": [ "Nessa Jonsson was born on March 17, 1995 in Sweden and currently resides there. Her sister, Erika, has a history of hypertension." ], "entity_detection": { "accuracy": "high" }, "locale": "en-US", "relation_detection": { "coreference_resolution": "model_prediction", "enable_relation_extraction": true } } ``` ```json Response Body wrap lines highlight={19-20, 23-24, 27-28} theme={"theme":"poimandres"} [ { "entities": [ { "text": "Nessa Jonsson", "location": { "stt_idx": 0, "end_idx": 13 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.36283198595046995, "NAME": 0.9023965716361999, "NAME_FAMILY": 0.5529729723930359 }, "coreference_id": "7e688543-9aff-4b9a-a386-5277d2ee8954", "relations": [ { "coreference_id": "60108305-97e7-4019-9d02-0dc0549b27ea", "label": "RELATED_TO" }, { "coreference_id": "c83c10bb-609c-4f5f-b2a1-9ba6ac367614", "label": "RELATED_TO" }, { "coreference_id": "add43460-23ff-4100-b529-b17f8b9a71f4", "label": "RELATED_TO" } ] }, { "text": "March 17, 1995", "location": { "stt_idx": 26, "end_idx": 40 }, "best_label": "DOB", "labels": { "DOB": 0.9576807171106339 }, "analysis_result": { "formatted": "1995-03-17T00:00:00", "subtypes": [ { "formatted": "17", "label": "DAY" }, { "formatted": "3", "label": "MONTH" }, { "formatted": "1995", "label": "YEAR" } ], "validation_assertions": [] }, "coreference_id": "add43460-23ff-4100-b529-b17f8b9a71f4", "relations": [] }, { "text": "Sweden", "location": { "stt_idx": 44, "end_idx": 50 }, "best_label": "LOCATION_COUNTRY", "labels": { "LOCATION_COUNTRY": 0.9481291770935059, "LOCATION": 0.9080604314804077 }, "coreference_id": "60108305-97e7-4019-9d02-0dc0549b27ea", "relations": [] }, { "text": "Erika", "location": { "stt_idx": 92, "end_idx": 97 }, "best_label": "NAME_GIVEN", "labels": { "NAME_GIVEN": 0.9050805866718292, "NAME": 0.8937010765075684 }, "coreference_id": "c83c10bb-609c-4f5f-b2a1-9ba6ac367614", "relations": [ { "coreference_id": "7e688543-9aff-4b9a-a386-5277d2ee8954", "label": "RELATED_TO" } ] }, { "text": "hypertension", "location": { "stt_idx": 116, "end_idx": 128 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9360405206680298 }, "coreference_id": "676cbf92-3eac-46dd-ac39-96d2368c09da", "relations": [] } ], "entities_present": true, "characters_processed": 129, "languages_detected": { "en": 0.9928773641586304 } } ] ``` With relation extraction enabled, each entity in the response now contains an additional field capturing its relations: * `relations`: A list of extracted relations involving the entity. Each relation object includes: * `coreference_id`: The ID of the related entity from the `coreference_id` field of another entity in the response. * `label`: The type of relation detected. Currently, only one relation is supported, the generic `RELATED_TO` relation. **Limitations** The relation extraction model is provided as an experimental feature and is not intended for production use. It currently supports English text and is constrained to inputs of up to **1024 tokens**. Any text beyond this limit will be ignored during processing. Relation predictions may be inaccurate or missed, particularly in complex contexts where related entities occur far apart within the text. ## Synthetic replacements The [`analyze/text`](/latest/analyze-text) endpoint supports generating synthetic replacements for detected entities. This allows you to combine entity analysis with synthetic data generation in a single request. For a general overview of synthetic entity generation and its use cases, see the [Synthetic PII guide](/configuration-and-operations/entity-detection-and-redaction/customizing-redaction#synthetic-pii). To enable synthetic replacements, add the optional `synthetic_replacements` object to your request. This object supports the following fields: * **accuracy**: Selects the synthetic model accuracy. This follows the same options as `synthetic_entity_accuracy` in the [`process/text`](/latest/process-text) route (`standard`, `standard_multilingual`, `standard_automatic`). * **entity\_types**: Following the same pattern as `entity_detection.entity_types`, this field can be used to enable or disable synthetic generation for specific entity types. When synthetic replacements are enabled, each entity in the response will include an additional `synthetic_text` field containing the generated synthetic value for that entity. **Synthetic processing is optional** When the `synthetic_replacements` field is omitted, the response will not include synthetic text for detected entities. Here is an example of how you can enable synthetic replacements in your `analyze/text` request. ```json Request Body wrap lines highlight={9-19} theme={"theme":"poimandres"} { "text": [ "Nessa Jonsson was born on March 17, 1995." ], "entity_detection": { "accuracy": "high" }, "locale": "en-US", "synthetic_replacements": { "accuracy": "standard_automatic", "entity_types": [ { "type": "ENABLE", "value": [ "NAME" ] } ] } } ``` ```json Response Body wrap lines highlight={16-16} theme={"theme":"poimandres"} [ { "entities": [ { "text": "Nessa Jonsson", "location": { "stt_idx": 0, "end_idx": 13 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.36283198595046995, "NAME": 0.9023965716361999, "NAME_FAMILY": 0.5529729723930359 }, "synthetic_text": "Hanna Lindahl" }, { "text": "March 17, 1995", "location": { "stt_idx": 26, "end_idx": 40 }, "best_label": "DOB", "labels": { "DOB": 0.9576807171106339 }, "analysis_result": { "formatted": "1995-03-17T00:00:00", "subtypes": [ { "formatted": "17", "label": "DAY" }, { "formatted": "3", "label": "MONTH" }, { "formatted": "1995", "label": "YEAR" } ], "validation_assertions": [] } } ], "entities_present": true, "characters_processed": 41, "languages_detected": { "en": 0.9928773641586304 } } ] ``` In the request, synthetic replacements are only enabled for `NAME` entities. Therefore, the `synthetic_text` field is omitted in the `DOB` entity within the response. This field contains text which can be used to replace the entity span. For an example on using synthetic replacements from the API to replace names, see this [example](/product-guides/thin-client#combining-synthetic-replacements-with-custom-redaction). # Guide For Integrating Limina DEID Container With Azure OCR Source: https://docs.getlimina.ai/configuration-and-operations/advanced-features/azure-ocr Learn how to seamlessly integrate Limina's DEID container with Azure OCR for efficient redaction of PII in images and documents. This guide provides step-by-step instructions on setting up the Limina container with the Azure OCR for processing images and documents. Both the cloud and containerized versions of Azure OCR are supported. Our general recommendation is Azure Document Intelligence. For Japanese documents, we recommend Azure Computer Vision for more accurate results. ## Prerequisites ### Access to Limina's DEID container See [Grabbing the Image](/installation/grabbing-the-image) for more info on this. ## Integrating Azure OCR with Limina Setup **one** of the following Azure OCR service to use with Limina DEID Container: Please check out our [Guide for OCR Modes Available with Limina DEID Container](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to see the different modes of OCR supported by Limina DEID Container. **Azure Document Intelligence Service** 1. Navigate to the Azure Portal. 2. Create a new Azure Document Intelligence service by following the link: [Create Document Intelligence](https://portal.azure.com/#create/Microsoft.CognitiveServicesFormRecognizer) 3. Note down your Document Intelligence service endpoint and the key. **Or Azure Computer Vision Service** 1. Navigate to the Azure Portal. 2. Create a new Azure Computer Vision service by following the link: [Create Computer Vision](https://portal.azure.com/#create/Microsoft.CognitiveServicesComputerVision) 3. Note down your service endpoint and the key. Open a terminal and run the following command to start the container: ```shell Docker Run Command theme={"theme":"poimandres"} docker run --rm -v "/path/to/license/my-license-file.json":/app/license/license.json \ -e PAI_DISABLE_RAM_CHECK=True \ -e PAI_AZ_DOCUMENT_INTELLIGENCE_URL=https://.cognitiveservices.azure.com/ \ -e PAI_AZ_DOCUMENT_INTELLIGENCE_KEY= \ -p 8080:8080 crprivateaiprod.azurecr.io/deid: ``` The above command starts the container with Document Intelligence, for Computer Vision, please replace the above Document Intelligence environment variables with these. ```shell Azure Computer Vision Environment Variables theme={"theme":"poimandres"} PAI_AZ_COMPUTER_VISION_URL PAI_AZ_COMPUTER_VISION_KEY ``` Please see [Limina deid-examples repository - OCR Examples](https://github.com/privateai/deid-examples/tree/main/docker-compose/OCR%20Examples) for more efficient container handling. Sample docker compose files are available including these topics. * Option 1 - DEID Container with cloud Azure Computer Vision Service * Option 2 - DEID Container with cloud Azure Document Intelligence Service * Option 3 - DEID Container with On-Premise Azure Computer Vision Service Once the containers are running, you can verify their operation by accessing the DEID service's exposed port (e.g., `http://localhost:8080`) and performing a test OCR operation on your documents or images. # LLM Prompt Redaction Source: https://docs.getlimina.ai/configuration-and-operations/advanced-features/llms Learn how to use the headless or API version of Limina to preserve privacy inside applications using LLMs like ChatGPT and GPT4. ## Introduction In this guide, you'll learn how to use Limina to perform prompt redaction and re-hydration. The guide is centred around handling personally identifiable data: you'll deidentify user prompts, send them to OpenAI's ChatGPT, and then re-identify the responses. This ensures confidential information remains safe while interacting with AI models. The process looks like this: Overview of PrivateGPT workflow In addition to this guide, you might find the following notebooks and code examples useful. | Title | LLM Provider | Format | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | ------------- | | [Safely work with Company Confidential Information with Limina](https://github.com/privateai/deid-examples/blob/main/python/LLM%20Examples/Removing%20Confidential%20Financial%20Information%20via%20Redaction%20for%20LLMs.ipynb) | OpenAI | Notebook | | [SEC Filing: Document Summarization and Question Answering](https://github.com/privateai/deid-examples/blob/main/python/LLM%20Examples/PAI-Cohere.ipynb) | Cohere | Notebook | | [SEC Filing: Document Summarization and Question Answering](https://github.com/privateai/deid-examples/blob/main/python/LLM%20Examples/PAI-PaLM.ipynb) | Google | Notebook | | [Secure Prompting with Limina](https://github.com/privateai/deid-examples/blob/main/python/LLM%20Examples/secure_prompt.py) | OpenAI, Google or Cohere | Python Script | | [Secure Prompting with Limina - Files](https://github.com/privateai/deid-examples/blob/main/python/LLM%20Examples/secure_prompt_from_file.py) | OpenAI, Google or Cohere | Python Script | First, we need to pull and run the Limina Docker container, which is responsible for data deidentification and re-identification. Follow the instructions in the [Quickstart Guide](/container-quickstart/quickstart-guide) to set up your Docker container. The next step is setting up the rest of your environment. You will need to install necessary Python libraries, import the required modules in your script, and configure your API keys. Let's install the `privateai_client` and `openai` Python libraries using pip (this has been tested with Limina Client version 1.3.2 and Open AI Client version 0.28.0): ```shell pip install lines wrap theme={"theme":"poimandres"} pip install privateai_client openai ``` Now in your Python script, you'll need to initialize the requisite constants and client libraries: ```python Initialize constants and import library lines wrap theme={"theme":"poimandres"} from privateai_client import PAIClient, request_objects import openai import os # Initialize the openai client openai.api_key = os.getenv('OPENAI_KEY') # initialize the privateai client PRIVATEAI_SCHEME = 'http' PRIVATEAI_HOST = 'localhost' PRIVATEAI_PORT = '8080' client = PAIClient(PRIVATEAI_SCHEME, PRIVATEAI_HOST, PRIVATEAI_PORT) ``` Remember to set a local environment variable named `OPENAI_KEY` with your actual OpenAI [API key](https://platform.openai.com/account/api-keys), and set the `PRIVATEAI_SCHEME`, `PRIVATEAI_HOST`, and `PRIVATEAI_PORT` variables according to where your local Docker container is running. Let's create a function that will redact personal data from user input, and generate entity mappings that will later be used to re-identify the response: ```python Redacting Text Using Limina lines wrap theme={"theme":"poimandres"} def redact_text(text): text_request = request_objects.process_text_obj(text=[text]) return client.process_text(text_request) ``` Next, we'll define a function that does the opposite of the redaction function, enabling the re-identification of redacted entities in the text: ```python Reidentification Using Limina lines wrap theme={"theme":"poimandres"} def reidentify_text(text, entities): entity_mappings = list(map(map_entity, entities)) reid_request = request_objects.reidentify_text_obj(processed_text=[text], entities=entity_mappings) reid_text = client.reidentify_text(reid_request) return reid_text.response.json()[0] ``` We don't need all of the data in the response from the redaction service, so let's create a little mapping function to get just what we need: ```python Mapping Entity lines wrap theme={"theme":"poimandres"} def map_entity(entity): return { 'processed_text': entity['processed_text'], 'text': entity['text'] } ``` We'll also need a function for generating responses from OpenAI's GPT-3.5-turbo chat model. Here's a simple illustration of how to accomplish this: ```python Prompting Chat GPT lines wrap theme={"theme":"poimandres"} def prompt_chat_gpt(text): completion = openai.ChatCompletion.create( model = "gpt-3.5-turbo", messages = [ { "role": "user", "content": text } ] ) return completion.choices[0].message['content'] ``` We're now ready to put all the components together to form our final solution. This is how we redact a prompt, dispatch it to the GPT model, and reidentify the response: ```python Redacting Text And Prompting ChatGPT lines wrap theme={"theme":"poimandres"} def prompt_private_gpt(prompt): print("******* ORIGINAL TEXT *******") print(prompt) redacted_object = redact_text(prompt) print("\n******* REDACTED TEXT **********") print(redacted_object.processed_text) chatgpt_response = prompt_chat_gpt(redacted_object.processed_text[0]) print("\n******* OPEN AI RESPONSE WITH REDACTIONS ********") print(chatgpt_response) reidentified_text = reidentify_text(chatgpt_response, redacted_object.entities[0]) print("\n******* OPEN AI RESPONSE REIDENTIFIED ********") print(reidentified_text) return reidentified_text ``` There you have it - enjoy chatting while preserving your privacy! ```python Private GPT Example lines wrap theme={"theme":"poimandres"} openai_response = prompt_private_gpt("Hey! Im Joe, a developer at Limina!") ``` The entire code is below for you to copy-paste 😉 ```python Complete LLM Prompt Redaction Example lines wrap theme={"theme":"poimandres"} from privateai_client import PAIClient, request_objects import openai import os # Initialize the openai client openai.api_key = os.getenv('OPENAI_KEY') # initialize the privateai client PRIVATEAI_SCHEME = 'http' PRIVATEAI_HOST = 'localhost' PRIVATEAI_PORT = '8080' client = PAIClient(PRIVATEAI_SCHEME, PRIVATEAI_HOST, PRIVATEAI_PORT) def redact_text(text): text_request = request_objects.process_text_obj(text=[text]) return client.process_text(text_request) def reidentify_text(text, entities): entity_mappings = list(map(map_entity, entities)) reid_request = request_objects.reidentify_text_obj(processed_text=[text], entities=entity_mappings) reid_text = client.reidentify_text(reid_request) return reid_text.response.json()[0] def map_entity(entity): return { 'processed_text': entity['processed_text'], 'text': entity['text'] } def prompt_chat_gpt(text): completion = openai.ChatCompletion.create( model = "gpt-3.5-turbo", messages = [ { "role": "user", "content": text } ] ) return completion.choices[0].message['content'] def prompt_private_gpt(prompt): print("******* ORIGINAL TEXT *******") print(prompt) redacted_object = redact_text(prompt) print("\n******* REDACTED TEXT **********") print(redacted_object.processed_text) chatgpt_response = prompt_chat_gpt(redacted_object.processed_text[0]) print("\n******* OPEN AI RESPONSE WITH REDACTIONS ********") print(chatgpt_response) reidentified_text = reidentify_text(chatgpt_response, redacted_object.entities[0]) print("\n******* OPEN AI RESPONSE REIDENTIFIED ********") print(reidentified_text) return reidentified_text openai_response = prompt_private_gpt("Hey! Im Joe, a developer at Limina!") ``` # Named Entity Recognition Source: https://docs.getlimina.ai/configuration-and-operations/advanced-features/ner This guide explains how to detect PII in text and files without redaction [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. In addition to de-identification and redaction, Limina also supports entity detection. This is useful for data discovery and also allows Limina to be used as a general purpose Named Entity Recognition (NER) Engine. In this guide we demonstrate how to use the [`ner/text`](/latest/ner-text) endpoint introduced in `3.9` to return entities in text and describe an approach to do the same in files. ## Detect entities in text (new in 3.9) The [`ner/text`](/latest/ner-text) route introduced in 3.9 returns a list of detected entities. It can be thought of as a cut-down version of [`process/text`](/latest/process-text) that only returns the list of detected entities, [with a key difference described in the next section](#process-vs-detect-entities). In this snippet we use our Python SDK to invoke the [`ner/text`](/latest/ner-text) route on a short sentence and to return a list of detected entities: ```python Text NER wrap lines theme={"theme":"poimandres"} text_request = request_objects.ner_text_obj(text=["My sample name is John Smith"]) resp = client.ner_text(text_request) ``` The list of detected entities is found in the `entities` field: ```python Text NER theme={"theme":"poimandres"} print(json.dumps(resp.entities, indent=4)) ``` Yields: ```json NER Response wrap lines theme={"theme":"poimandres"} [ [ { "text": "John Smith", "location": { "stt_idx": 18, "end_idx": 28 }, "label": "NAME", "likelihood": 0.9105876684188843 }, { "text": "John", "location": { "stt_idx": 18, "end_idx": 22 }, "label": "NAME_GIVEN", "likelihood": 0.9043319821357727 }, { "text": "Smith", "location": { "stt_idx": 23, "end_idx": 28 }, "label": "NAME_FAMILY", "likelihood": 0.9326320886611938 } ] ] ``` ## Process vs Detect Entities There is a key difference between the entities returned in [`process/text`](/latest/process-text) route and [`ner/text`](/latest/ner-text): [`process/text`](/latest/process-text) groups overlapping entity detections into a single entity object, while [`ner/text`](/latest/ner-text) does not. This is evident from the previous example, where *John Smith* detected three different entities: `John Smith`, `John` and `Smith`. The corresponding [`process/text`](/latest/process-text) entity list is: ```json Process Text Json Object theme={"theme":"poimandres"} [ { "processed_text": "NAME_1", "text": "John Smith", "location": { "stt_idx": 18, "end_idx": 28, "stt_idx_processed": 18, "end_idx_processed": 26 }, "best_label": "NAME", "labels": { "NAME": 0.9106, "NAME_GIVEN": 0.4522, "NAME_FAMILY": 0.4663 } } ] ``` The [`ner/text`](/latest/ner-text) provides the raw output of the entity detection engine and is recommended if details about all entities discovered in a text fragment, including overlapping ones are required. With the [`ner/text`](/latest/ner-text) route you will be able to answer questions like *Does this text contain zip codes?* or *Does it contain a complete address?* This extra flexibility implies that you should be ready to implement your own post-processing logic. You should use the [`process/text`](/latest/process-text) if non-overlapping logical entities are required, e.g. to count the number of detected entities. ## Detect entities in files While the [`ner/text`](/latest/ner-text) route only supports text at this time, it is still possible to achieve a similar behaviour for files with the caveat mentioned in the previous section, only grouped entities are accessible for files. In this snippet we use our python sdk to process a file as `base64`. ```python Processing File Via Base64 Route theme={"theme":"poimandres"} # Read from file with open('./sample_pdfs/Letter-of-Intent-pdf.pdf', "rb") as file: b64_file_data = base64.b64encode(file.read()).decode("ascii") # Make the request file_obj = request_objects.file_obj(data=b64_file_data, content_type='application/pdf') request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) ``` Here again, we simply take the `.entities` object from the API response, and add it to a dictionary with the original file path set in the `path` key. In this case we are creating one dictionary to map the file to the entities, but to process an entire directory of files you can build a list where each element is a dictionary as described below, or emit the dictionary to a datastore of your chosing. ```python File NER wrap lines highlight={2-3} theme={"theme":"poimandres"} ner_objects: List[Dict[str, Any]] = [] ner_object = dict(path="./sample_pdfs/Letter-of-Intent-pdf.pdf", entities = resp.entities) ner_objects.append(ner_object) print(json.dumps(ner_objects, indent=4)) ``` Now we have a nice clean dictionary with all the PII detected, and the file location for further inspection if necessary. ```json File NER JSON Response wrap lines theme={"theme":"poimandres"} [ { "path": "./sample_pdfs/Letter-of-Intent-pdf.pdf", "entities": [ { "processed_text": "NAME_1", "text": "Sarah Jackson", "location": { "page": 1, "x0": 0.11588, "x1": 0.23794, "y0": 0.20727, "y1": 0.22227 }, "best_label": "NAME", "labels": { "NAME": 0.9185, "NAME_GIVEN": 0.4492, "NAME_FAMILY": 0.4675 } }, { "processed_text": "ORGANIZATION_2", "text": "Best Capital Corp", "location": { "page": 3, "x0": 0.11706, "x1": 0.27, "y0": 0.87, "y1": 0.88909 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8789 } } ] } ] ``` See how the text *Sarah Jackson* resulted in a single *grouped* entity instead of three different ones for the example above. In this case we have kept the full entities list of dictionaries for simplicity, but during your own implementation you can keep just the components you like. ## Wrap Up Getting a list of entities contained in a text input or in a file is equally simple. The key in this guide is to access the `entities` field in the response. It's that simple 😀. See the API Reference to learn more about the other response fields like `processed_text` and `processed_file`. # Websocket Endpoint Source: https://docs.getlimina.ai/configuration-and-operations/advanced-features/websockets Documentation for using Limina's Websocket endpoint. Beta The Limina container introduced a websocket endpoint in version `3.6.0` to allow users to send requests and receive responses with a single connection. Read more about Websockets [here](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API). It is recommended to use the REST endpoints unless there are specific requirements for websocket. Websockets are available on the Limina container only. ## How to use the websocket endpoint The websocket endpoint can be used directly from the container if it is deployed as a single instance. The websocket endpoint requires additional setup and consideration in your infrastructure to be usable when the container is served behind a production load balancer. The websocket endpoint is available in the container by sending a connection request to the `/ws` endpoint. This can be done with a websocket library such as [websocket-client](https://pypi.org/project/websocket-client/) for Python, or simply by utilizing the [Websockets API](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) directly in Javascript. ## Quickstart example The following is a simple example written in Python to use the websocket endpoint: ```python Websocket Example wrap lines theme={"theme":"poimandres"} import websocket import json ws = websocket.WebSocket() ws.connect("ws://myprivateaicontainer:8000/ws") payload = {"text": "John Smith lives at 123 Fake St."} ws.send(json.dumps(payload)) print(ws.recv()) ws.close() ``` Output: ```json Example Output wrap lines theme={"theme":"poimandres"} { "processed_text": "[NAME_1] lives at [LOCATION_ADDRESS_1].", "entities": [ { "processed_text": "NAME_1", "text": "John Smith", "location": { "stt_idx": 0, "end_idx": 10, "stt_idx_processed": 0, "end_idx_processed": 8 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.4488, "NAME": 0.8994, "NAME_FAMILY": 0.4596 } }, { "processed_text": "LOCATION_ADDRESS_1", "text": "123 Fake St", "location": { "stt_idx": 20, "end_idx": 31, "stt_idx_processed": 18, "end_idx_processed": 38 }, "best_label": "LOCATION_ADDRESS", "labels": { "LOCATION_ADDRESS": 0.9486, "LOCATION": 0.913 } } ], "entities_present": true, "characters_processed": 32, "languages_detected": {"en": 0.8259078860282898} } ``` ## Websocket Payloads The websocket endpoint expects a JSON payload just like the REST API in exactly the same format with all the of the parameters from the [REST API](/latest/process-text/) supported in the same way. The main difference with the websocket endpoint is that the `text` parameter does NOT support a list; we're expecting single request and response exchanges with the container! Below are some examples of payloads and their expected outputs: ```python Websocket Call With JSON payload wrap lines theme={"theme":"poimandres"} # Disabling the NAME entity type and subtypes resulting in no redaction of detected names payload = {"text": "Hello, my name is Greg", "entity_detection": {"entity_types": [{"type":"DISABLE", "value":["NAME", "NAME_GIVEN", "NAME_FAMILY"]}]}} >>>> {"processed_text": "Hello, my name is Greg" ...} ``` ```python Websocket Call with Masked Entities wrap lines theme={"theme":"poimandres"} # Setting the processed text type to MASK and masking entities. payload = {"text": "Hello, my name is John Smith.", "processed_text": {"type":"MASK", "mask_character":"$"}} >>>> {"processed_text": "Hello, my name is $$$$ $$$$$." ...} ``` # Benchmarks Source: https://docs.getlimina.ai/configuration-and-operations/container-management/benchmarks Benchmarks provides some performance numbers on the CPU and GPU containers including recommendations for best throughput. Looking for accuracy benchmarks? Please download our [Whitepaper](https://getlimina.ai/resources/whitepaper?utm_campaign=2024%20Content\&utm_source=docs\&utm_medium=docs\&utm_term=docs) Benchmarked against container version 4.2.0 ## NER Benchmarks The following section provides some NER performance figures for Limina's CPU and GPU containers on various VM instance types, including the hardware in the [system requirements](/installation/prerequisites-and-system-requirements). These numbers have been computed by generating load on the `process/text` route using the default settings (i.e., `HIGH_AUTOMATIC` accuracy mode and `heuristics` coreference). Requests to the `process/text` route were created using an internal dataset of English examples of varied length. The load was scaled to a concurrency level maximizing the throughput of the `process/text` endpoint. Therefore, you could expect a lower latency if you have a lower load. A latency as low as 10ms can be achieved on a 100-words input when using a GPU deployment. ### NER Performance on CPU The table below illustrates the performance of the CPU container on various instance types:
Platform Instance Type Throughput1 (words/sec) Average Latency2 (ms)
Azure
Standard\_E2\_v5 (2 vCPUs, 16GB RAM) 513 1022
Standard\_E8\_v5 (8 vCPUs, 64GB RAM) 1719 304
AWS
m7i.xlarge (4 vCPUs, 16GB RAM) 834 628
m7i.4xlarge (16 vCPUs, 64GB RAM) 1843 285
1 Throughput is given in words per second, where a word denotes a whitespace-separated piece of text. 2 The average example length used for the testing is 131 words. The values in this column are the average latency over all examples. When using the `STANDARD` or `STANDARD_MULTILINGUAL` accuracy mode, you should expect a throughput that is around 4 to 5 times these numbers. Similarly, the `STANDARD_HIGH` and `STANDARD_HIGH_MULTILINGUAL` accuracy mode will deliver a throughput that is around 3 times these numbers. Note that the `coreference_resolution` settings `model_prediction` and `combined` have a big impact on performance. You can expect the throughput to be cut by 10 if you enable these options. Similarly, `SYNTHETIC` entity replacement may reduce throughput by up to 50 times compared to `MARKER` replacements, depending on concurrency and the number of entities replaced. ### NER Performance on GPU The table below contains the benchmarks of the GPU container running on different instance types equipped with a single GPU. Note that the Limina GPU container is designed to run on a single GPU and will not leverage multiple GPUs.
Platform Instance Type Throughput1 Average Latency2
Azure
Standard\_NC4as\_T4\_v3 (4 vCPUs, 28GB RAM) 11900 131
Standard\_NC8as\_T4\_v3 (8 vCPUs, 56GB RAM) 11450 137
AWS
g4dn.2xlarge (8 vCPUs, 32GB RAM) 12100 538
g4dn.4xlarge (16 vCPUs, 64GB RAM) 14000 186
g5.4xlarge (16 vCPUs, 64GB RAM) 28200 453
1 Throughput is given in words per second, where a word denotes a whitespace-separated piece of text. 2 The average example length used for the testing is 131 words. The values in this column are the average latency over all examples. When using the `STANDARD` accuracy mode, you should expect a throughput that is around 4 times these numbers. Similarly, the `STANDARD_HIGH` accuracy mode will deliver a throughput that is 3 times these numbers. It is not recommended to use the `model_prediction` or `combined` coreference resolution modes with the GPU container because of the big impact these modes have on throughput. The `SYNTHETIC` entity replacement option is likewise not recommended with the GPU container, as it largely negates the performance benefits of running on GPU. ## PDF and Image Benchmark Limina recommends that documents are processed on GPU instances. Below are benchmarks of the GPU container with all document features enabled. This includes the default OCR, object detection and NER modes. Note that PDF are processed as images so the processing of a page of PDF is roughly equivalent to the processing of one image. Note also that the processing time of PDF and images may vary depending on the image size and its resolution and the amount of text. | Instance Type | Throughput (pages/sec) | | ------------- | ---------------------- | | g4dn.2xlarge | 1.41 | ## Audio Benchmark The throughput of the Limina audio processing is provided below for the Private GPU and the CPU images on a common AWS instance. | Instance Type | Image Type | Throughput (RTFx1) | | ------------- | ---------- | ----------------------------- | | g4dn.2xlarge | GPU | \~23.0 | | m7i.4xlarge | CPU | \~2.8 | 1 The RTFx (inverse realtime factor) is measuring how many minutes of audio can be processed in 1 minute. A RTFx value of 30 means that an hour of audio recording is processed in 2 minutes. As you can see from the above results, the GPU image is around 10 times faster than the CPU image. ## Additional Guidelines ### Hardware Hardware type matters. `m5zn` instances powered by recent Intel Xeon CPUs with **AVX512 VNNI** support perform over 3X faster than generic instances like `c5`. For this reason, it is recommended to use the hardware specified in the [system requirements](/installation/prerequisites-and-system-requirements). As such, it is best to avoid AWS Fargate, which is typically provisioned with older CPUs like the `c5`. ### Scaling Considerations #### Latency The latency on the `process/text` endpoint scales approximately linearly with the request length. To reduce latency one can call the `process/text` endpoint with smaller requests. This, in general, will not improve throughput. However, feeding the models with very short inputs (i.e., a few words to a couple of sentences) may reduce the models' accuracy because of the lack of context. #### Throughput To maximize throughput, it is recommended to use a large number of concurrent requests. Batching smaller requests together does not improve throughput significantly. For very large deployments, GPU instances are recommended. A single low cost inference instance such as the `g4dn.2xlarge` (\~\$0.752 USD per hour) can process 1GB of unicode text in under an hour. Note that scaling GPU instances requires to balance the GPU and CPU resources. As a rule of thumb, larger models like the `high` accuracy NER model are GPU bound, while smaller models like the `standard` accuracy NER model are CPU bound (on a GPU instance with too few CPU cores). It is best to experiment with a few configurations to find the one that best fit your data. ### Kubernetes Deployments You should expect slightly lower numbers for throughput and latency when running the Limina container on Kubernetes deployments using any of the instance types above. This is due to the overhead of the Kubernetes environment and resources being possibly reserved for the Kubernetes processes. # Concurrency Source: https://docs.getlimina.ai/configuration-and-operations/container-management/concurrency The recommended level of concurrency (the optimal number of simultaneous requests to make to the container) for the CPU and GPU containers. The recommended level of concurrency, i.e. the optimal number of simultaneous requests to make to the container is covered below for the CPU and GPU containers. The recommended concurrency level is driven primarily by the compute requirement of Limina's Neural Network models, such as for PII detection. For an example of how to make concurrent requests, please visit our [examples repository](https://github.com/privateai/deid-examples/blob/main/python/examples/concurrency.py). ## CPU For Neural Network inference workloads, CPUs don't require inputs to be batched together to achieve good hardware utilization. In practice, due to network overhead and pre/post-processing code it is best to use a low level of concurrency such as 1 per container instance. ## GPU Unlike CPUs, GPUs require inputs to be batched together and processed as a single large input to achieve optimal hardware utilization. This means that there is a tradeoff between latency and throughput. A concurrency level of 32 per container instance is a good tradeoff between latency and throughput, however 16 is the recommended number of concurrent connections to optimize for latency. The guidance above is a recommended starting point for most installation types. For both CPU and GPU instance types it is recommend that you tune the volume of simultaneous requests based on your unique traffic patterns and use case specific volumes. Acceptable latency, desired throughput, and tolerance for variability greatly influence how you manage the overall load and performance on your Limina installation. # CORS Configuration Source: https://docs.getlimina.ai/configuration-and-operations/container-management/cors-configuration An introduction to working with CORS and the Limina Container. If you'd like to learn more about CORS and why it is needed on the web, please refer to MDN's [CORS documentation page](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) ## Can I add CORS in the Limina container? To reduce security concerns, we do not support adding CORS in the container. However, there are well known web development patterns that allow us to call APIs from the browser while still respecting CORS headers. ## How to resolve CORS issues from the browser? There are many established patterns for dealing with CORS headers. We will explain two common patterns, however this is not an exhaustive list. The solution you ultimately end up going with will depend on your web technologies and current application architecture. ### 1. Configure a Backend or API Gateway to Allow CORS CORS specifically was made to protect against browser-based vulnerabilities. You can access the Limina container's APIs from an API gateway or backend without any CORS headers required. That API Gateway or backend can then in-turn have CORS configured for the domain of your front-end application. In this diagram, we've set up an API Gateway to allow our website's (website.com) origin to have access to the api-gateway.com resources. Because CORS isn't required for server-to-server interaction, the connection from `api-gateway.com` to `api.getlimina.ai` does not need require CORS headers for it to be successful. ```mermaid actions={false} theme={"theme":"poimandres"} flowchart LR subgraph g1 [Browser] A["`Front End Application _(website.com)_`"] end A --> B["`API Gateway or Backend Server _(api-gateway.com)_`"] B -- "Access-Control-Allow-Origin: website.com" ----> A B --> C["`DEID Container _(api.getlimina.ai)_`"] C --> B ``` ### 2. Use a Proxy Server As the same-origin policy is implemented by internet browsers and not enforced within server-to-server communication, you can use a proxy server to call the external API. A proxy server acts as a middleware between the client and the server. Instead of making a request from the client to the external API directly, you can make a request to the proxy server. The proxy server will make a request to the external API for you and return the response that it receives from the external API. For example, the popular build tool Vite allows you to create a proxy server while you are locally developing your front end application. Details about the proxy server can be found [in the Vite documentation](hhttps://vite.dev/config/server-options). # Environment Variables Source: https://docs.getlimina.ai/configuration-and-operations/container-management/environment-variables Environment Variables provides a comprehensive list of supported environment variables that you can configure with Limina container. The Limina container supports a number of environment variables. The environment variables can be set in the Docker run command as follows: ```shell Docker Command theme={"theme":"poimandres"} docker run --rm -e = -e = -p 8080:8080 -it deid: ``` ## Supported Environment Variables
| Variable Name | Description | | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | | `PAI_ACCURACY_MODES` | Controls which entity detection models are loaded at container start. By default, the container loads all models. Setting this environment variable allows for faster container startup and a lower initial memory footprint. A request specifying an accuracy mode that wasn't loaded will return an error. Allowed values are `high`, `standard_high`, `standard`, `high_multilingual`, `standard_high_multilingual`, `standard_multilingual`. This variable also supports a comma-separated list of values to load multiple models, e.g., `PAI_ACCURACY_MODES=high,standard,standard_high`. The `automatic` accuracy settings are not supported in this variable. | | | `PAI_ALLOW_LIST` | Allows for the allow list to be set globally, instead of passing into each `POST` request. An example could be `PAI_ALLOW_LIST='["John","Grace"]'`. Please see [Processing Text](/latest/process-text/) | | | `PAI_DISABLE_GPU_CHECK` | When defined and set to any value, the startup GPU check is disabled. This variable is only applicable to the GPU container and allows the GPU container to run in fallback CPU mode | | | `PAI_DISABLE_RAM_CHECK` | When defined and set to any value, the sufficient RAM check performed at container startup is disabled. Please note that Limina cannot guarantee container stability if this is switched off | | | `PAI_ENABLED_ENTITIES` | Allows for the enabled classes to be set globally, instead of passing it into each POST request. An example could be `PAI_ENABLED_ENTITIES="NAME"` or `PAI_ENABLED_ENTITIES="NAME,AGE,ORGANIZATION"`. A command sample is located at the bottom. Please see also [Processing Text](/latest/process-text/) | | | `PAI_DISABLED_ENTITIES` | Allows for the disabled classes to be set globally, instead of passing it into each POST request. An example could be `PAI_DISABLED_ENTITIES="ORGANIZATION"` or `PAI_DISABLED_ENTITIES="AGE,LOCATION,ORGANIZATION"`. A command sample is located at the bottom. Please see also [Processing Text](/latest/process-text/) | | | `PAI_ENABLE_AUDIO` | When defined and set to any value, the container loads the functionality to process audio files. This is off by default to save startup time and RAM | | | `PAI_ENABLE_PII_COUNT_METERING` | When defined and set to any value, Aggregated entity detection counts are sent to logstash and the Limina community portal for reporting and visualization inside the dashboard. Note that feature is off by default | | | `PAI_LOG_LEVEL` | Controls the verbosity of the container logging output. Allowed values are `info`, `warning` or `error`. Default is `info` | | | `PAI_MARKER_FORMAT` | Allows for the redaction marker format to be set globally, instead of passing into each `POST` request. Please see [Processing Text](/latest/process-text/) | | | `PAI_OUTPUT_FILE_DIR` | The directory where `/process/files/uri` will write processed files to. Note that this does not need to be specified for `/process/files/base64` | | | `PAI_PROJECT_ID` | Sets a default `project_id` that will be used if a request doesn't contain one. Please see [Processing Text](/latest/process-text/) | | | `PAI_SYNTHETIC_PII_ACCURACY_MODES` | Same as `PAI_ACCURACY_MODES`, except for synthetic entity generation models. This variable also supports a comma-separated list of values to load multiple models, e.g., `PAI_SYNTHETIC_PII_ACCURACY_MODES=standard,standard_multilingual`. Unlike `PAI_ACCURACY_MODES`, this environment variable can be set empty via `PAI_SYNTHETIC_PII_ACCURACY_MODES=` to disable synthetic entity generation | | | `PAI_ENABLE_REPORTING` | Enables reporting to a Logstash server | | | `LOGSTASH_HOST` | The Logstash server's host info | | | `LOGSTASH_PORT` | The port of the Logstash server | | | `LOGSTASH_MONITORING_PORT` | The monitoring API port of the Logstash server | | | `LOGSTASH_TTL` | Sets the time to live value (in seconds) of the data queued for Logstash. Data will be lost if the queued data is not sent successfully before the ttl value. | | | `PAI_REPORT_ENTITY_COUNT` | Enables entity counts (per piece of text deidentified) to be added to reporting | | | `PAI_MAX_IMAGE_PIXELS` | Configures the max allowed pixels in the images processed. Default value is `178956970` | | | `PAI_MAX_FILE_SIZE` | Configures the max allowed file size in bytes. File size check occurs before redaction and produces an error message if it exceeds the value specified. i.e. `2000000` | | | `PAI_WS_LINK_BATCH` | Enables context retention in the websocket endpoint. Default is `true` | | | `PAI_WS_CONTEXT_SIZE` | Sets the context window size (number of previous messages retained in context). Default is `50` | | | `PAI_ENABLE_PDF_TEXT_LAYER` | This environment variable sets the default behaviour for whether a text layer is included in generated PDF files. When set to `true`, the application will include a text layer in the PDFs, allowing for text selection, search, and accessibility features. When set to `false` the text layer will be disabled, resulting in marginally smaller file sizes and faster processing. Note that this option is ignored if the `pdf_options.enable_pdf_text_layer` POST parameter is set in requests to the `/process/files/uri` and `/process/files/base64` endpoints. Default value is `true`. | | | `PAI_OCR_SYSTEM` | Sets the Optical Character Recognition (OCR) system for processing documents and images. Default is `paddleocr` | |
To change the port used by the container, please set the host port as per the command below: ```shell Docker Command theme={"theme":"poimandres"} docker run --rm -v "full path to license.json":/app/license/license.json \ -p :8080 -it crprivateaiprod.azurecr.io/deid: ``` To use `PAI_ENABLED_ENTITIES` to redact only for `NAME` and `ORGANIZATION`, the command will look like: ```shell Docker Command theme={"theme":"poimandres"} docker run --rm -v "full path to license.json":/app/license/license.json \ -e PAI_ENABLED_ENTITIES="NAME,ORGANIZATION" -p :8080 -it crprivateaiprod.azurecr.io/deid: ``` To use `PAI_DISABLED_ENTITIES` to redact all except `ORGANIZATION`, the command will look like: ```shell Docker Command theme={"theme":"poimandres"} docker run --rm -v "full path to license.json":/app/license/license.json \ -e PAI_DISABLED_ENTITIES="ORGANIZATION" -p :8080 -it crprivateaiprod.azurecr.io/deid: ``` Note: If setting entities both in `ENABLE` and `DISABLE`, only `ENABLE` takes effect. If an entity is duplicated both in `ENABLE` and `DISABLE` and `ENABLE` has no other entity set, the request will fail as `400 Bad Request`. # Logstash & Dashboards Source: https://docs.getlimina.ai/configuration-and-operations/container-management/logstash-and-dashboards How to integration Limina with elastic search for monitoring. ## Reporting Integration The Limina container can be configured to send reporting metrics to a logstash server. To enable this feature, the following environment variables can be added to the docker run command:
| Variable Name | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PAI_ENABLE_REPORTING` | Enables Reporting to a Logstash Server | | `LOGSTASH_HOST` | The Logstash server's host info | | `LOGSTASH_PORT` | The port of the Logstash server | | `LOGSTASH_MONITORING_PORT` | The monitoring API port of the Logstash server. | | `LOGSTASH_TTL` | Sets the time to live value (in seconds) of the data queued for logstash. Data will be lost if the queued data is not sent successfully before the ttl value. | | `PAI_REPORT_ENTITY_COUNT` | Enables entity counts (per piece of text deidentified) to be added to reporting |
To run a container with these settings, the following command can be used: ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -p 8080:8080 --mount type=bind,src=$PWD/tests/fixtures/licenses/license.json,dst=/app/license/license.json -e PAI_ENABLE_REPORTING=true -e LOGSTASH_HOST=http://hostname.org -e LOGSTASH_PORT=50000 -e PAI_REPORT_ENTITY_COUNTS=true -it deid:image-name ``` The Logstash pipeline that the data is being sent to must be configured to able to accept JSON objects. A sample pipeline configuration that allows this would be: ```ruby Pipline Config theme={"theme":"poimandres"} input { tcp { port => 50000 codec => json_lines {} } } output { elasticsearch { hosts => "elasticsearch:9200" user => "${LOGSTASH_USER}" password => "${LOGSTASH_PASSWORD}" } } ``` ### What is Being Sent? When `PAI_ENABLE_REPORTING` is enabled in the container metering records will be sent to Logstash in batches, at 5 minute intervals. All records include the following fields:
| Field Name | Description | | --------------------- | ---------------------------------------------------------------------- | | `privateai.SessionId` | A unique id that relates all metering information sent in the interval | | `privateai.Accuracy` | The accuracy used in the requests | | `privateai.ProjectId` | The project id of the request (the default value is main) | | `privateai.Synthetic` | Boolean value indicating if synthetic data was used |
The following meters are currently being used (one record per meter, per project ID):
| Field Name | Description | | --------------------- | -------------------------------------------------- | | `privateai.api_calls` | The amount of api calls sent in the interval | | `privateai.api_chars` | The amount of characters processed in the interval | | `privateai.api_words` | The amount of words processed in the interval |
If `PAI_ENABLE_PII_COUNT_METERING` is enabled in the container, meters for each entity found will also be sent. These meters contain the following fields:
| Field Name | Description | | --------------------- | ----------------------------------------------------------------------------- | | `privateai.pii-count` | The amount of times the entity was found to be the best label in the interval |
If `PAI_REPORT_ENTITY_COUNTS` is enabled in the container, a record of the entities found and their counts for each unit of text processed will be sent. e.g. ```text Metering Records theme={"theme":"poimandres"} privateai.NAME: 2 privateai.OCCUPATION: 1 ``` would indicate that in a unit of text processed, the entity NAME was the best label two times, and OCCUPATION was the best label once. Example dashboard ### Reading Output The following sections describe the typical field lists used when logs are forwarded to the Logstash server within the environment. These examples are provided as a reference for log analysis and field mapping configuration. It contains Limina internal debug information.
| Field Name | privateai Field | Example | Description | | ------------ | ------------------------ | ---------------------------------------------------------- | ---------------------------------------------------------- | | `@version` | | 1 | Logstash internal version # 1 | | `message` | | Sending data to logstash | Log description. Limina set this. | | `type` | | python-logstash | Name of logging handler | | `level` | | INFO | Log level | | `host` | | 471d716d6192 | Host (container) to send logs | | `pid` | | 25 | Uvicorn process pid | | `program` | | /opt/venv/lib/python3.10/site-packages/uvicorn/**main**.py | Uvicorn location | | `@timestamp` | | 2026-03-31T10:44:22.291Z | Log time stamp | | `logsource` | | 471d716d6192 | Log source (equivalent to Host) | | `privateai` | | | Limina specisic section | | | `api-calls` | 3 | Number of Api call | | | `api-chars` | 1234 | Number of characters processed | | | `api-words` | 25 | Number of words processed | | | `logger_name` | logstash | Name of logger | | | `Relext` | false | Relation extraction(beta) used? | | | `line` | 208 | Report function location | | | `logstash_async_version` | 3.0.0 | logstash async handler version | | | `ProjectId` | mytest\_prj1 | Project Id name provided by user | | | `thread_name` | ThreadPoolExecutor-1\_2 | Thread name of processing | | | `RecordType` | `meter` or `entities` | logstash record type | | | `Accuracy` | high\_multilingual | Limina model name used | | | `path` | /app/base\_metering.py | Usage report file name | | | `interpreter_version` | 3.10.20 | Limina container's python version | | | `Synthetic` | false | Synthetic data used? | | | `AppVersion` | 4.3.0-cpu | Limina container version | | | `func_name` | report | Function name of usage reporting | | | `interpreter` | /opt/venv/bin/python | Python path inside container | | | `process_name` | MainProcess | Process name for the pid | | | `SessionId` | 440c5885-f5dd-40ee-abce-5d6a98772f99 | Session Id of sending logs | | | `Coref` | false | Coreference used? | | | `LicenseId` | 412 | Limina license file Id | | | `ApiRoute` | process\_files\_uri | Limina Api name called | | | `TotalEntities` | 2 | Number of redacted entity types | | | `@TimeOfRequest` | 2026-04-02T17:18:29.662365 | Time stamp of TotalEntities logged | | | `[Name of Entity]` | 2 | Count of specific entity | | | `pii-count` | 1 | Count of specific entity (When TotalEntities not selected) |
# Performance Tips Source: https://docs.getlimina.ai/configuration-and-operations/container-management/performance-tips Limina shares tips and recommendations to get the best performance from our PII identification & detection machine learning models. ## Concurrency and Batching The following code examples outline how to increase throughput when using Limina. | Title | Description | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | [AsyncIO](https://github.com/privateai/deid-examples/blob/main/python/examples/async_call.py) | How to use Python's AsyncIO library to make concurrent calls to Limina | | [Threading](https://github.com/privateai/deid-examples/blob/main/python/examples/concurrency.py) | How to use Python's threading and concurrent.futures libraries to make concurrent calls to Limina | | [Batch Requests](https://github.com/privateai/deid-examples/blob/main/python/examples/batching.py) | How to process multiple inputs in a single API call using Python | Please visit the [recommended concurrency levels](/configuration-and-operations/container-management/concurrency) page to set the number of concurrent requests optimally. ## Context Limina relies on Machine Learning to detect PII based on context, instead of pattern matching approaches such as regular expressions. Therefore, for best performance it is advisable to send text through in the largest possible chunks that still meet latency requirements. For example, the following chat log should be sent through in one call with `link_batch` enabled, as opposed to line-by-line: ```text Text Example theme={"theme":"poimandres"} "Hi John, how are you?" "I'm good thanks" "Great, hope Atlanta is treating you well" ``` The [Batch Requests](https://github.com/privateai/deid-examples/blob/main/python/examples/batching.py) code example provided above shows how to implement this. Similarly, text documents should be sent through in a single request, rather than by paragraph or sentence. In addition to improving accuracy, this will minimize the number of API calls made. ## Capitalization The PII detection models are optimised for normal English capitalization, e.g. `"Robert is from Sydney, Australia. Muhab is from Wales"`. If this is not the case for your data, please contact Limina so that we can provide you with the optimal model for your use case. Our solution will still work, but some performance will be lost. This being said, Limina is optimized for processing text with emojis and text containing ASR transcription errors. ## ASR Transcripts When processing audio transcripts, it is recommended to use the following input format: ```text Text Example theme={"theme":"poimandres"} ": , : ," ``` ## Model Tuning Limina's PII detection models generally perform well out of the box. However, model tuning may be beneficial in order to tailor our solution to unique use cases. The tuning process is outlined below. ### **1. Client-supplied Data Sample** Prepare a sample of at least 20 examples illustrating the problem to be addressed. See the table at the bottom of this page for examples. The sample should contain examples that are: * ***anonymized*** such that all PII is manually replaced with synthetic entities * ***long*** enough to provide enough context, ideally including \~30 words before and after the problematic entity * ***diverse*** enough to be representative of the issues encountered ### **2. Secure Data Transfer** Data samples can be shared with Limina via a secure transfer mechanism built on Microsoft Azure. [Contact us](https://getlimina.ai/contact-us/) for an access token. ### **3. Manual Data Anonymization** Limina's data team will again manually anonymize the provided data, to ensure that no personal data is ever stored or used for training. ### **4. Model Tuning & Delivery** Using few-shot learning techniques, Limina improves the PII Detection models by optimizing for your use case. Updated models are released via a new container version. Updates are usually delivered in the next regular release, but can be delivered via a patch release in as little as 72 hours, depending on the SLA and severity of the problem. #### **Example Data Samples** | Potential Deidentification Issue\* | Illustrative Example | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CVV** number not redacted | `Okay. I'm just pulling it up. All right, I can go ahead and take that card number whenever you're ready. Okay, card number 4622-6542-1425-3511. All right, and the expiration date? 0226. And the three digits on the back. Six. 25. Thank you. I'll be charging your card for 243. Let me see what that amount was. 243 16. Would you like your receipt emailed?` | | **DOSE** entities not redacted | `Blood test done, results normal Medications Norvasc (AMLODIPINE 10MG, 1 Tablet(s) PO OD Catapres (CLONIDINE) 75MG, 1 Tablet(s) SL PRN Hydrochlorothiazide 25 MG PO PRN Aspirin 81 MG PO OD Cenolate (ASCORBIC ACID) 500MG, 1 Tablet(s) PO OD Allergies No known allergies Physical Exam Vital signs 200/130 135bpm` | | **DOB** misclassified as **DATE** | `Ms. Richmond, to verify your account can I get your date of birth and your phone number, please? Absolutely, yes. So my number is 907 563 2834 and then February 14, 1990. Okay, February 14, perfect, and it'll just be a moment while I access your file here.` | \*Note that these examples are used only for illustration. Limina correctly picks up the PII in each of these cases. # Security Source: https://docs.getlimina.ai/configuration-and-operations/container-management/security Learn more about the processes and measures the engineering team engages in to build Limina's software in a security conscious manner. ## Team Security Practices Overview Limina builds software in a security conscious manner. This page will give some insight into the processes and measures the engineering team engages in.
Exact policy details on software development practices can be found here: [Limina Trust Center](https://app.vanta.com/private-ai.com/trust/6pwo3ib6rvq44sdba16it). ## Application Security best practices Limina follows security best practices and engages in secure development processes that include regular vulnerability assessments at the code, application and container levels. Vulnerability remediation is conducted according to the SLAs below.
If you have any questions about our processes, feel free to [reach out to the team](https://getlimina.ai/contact-us/)! ## SLA for security patches Limina addresses security issues within the following SLA: Based on the [CVSS v3.0 rating system](https://nvd.nist.gov/vuln-metrics/cvss) * Critical: Within 48 hours of CVE publication and resolution availability * High: Within 2 weeks of CVE publication and resolution availability * Medium and under: Within 1 month of CVE publication and resolution availability ## Container building The Limina container is built on an even *slimmer* slim image to minimize unnecessary dependencies which reduces the container size as well as exposure to dependency vulnerabilities. ## SOC 2, ISO 27001 and other certifications ### Is Limina certified? As a company, Limina is ISO 27001:2022 and SOC2 Type 2 certified and compliant for our internal operations and processes. You can find out more in the [Limina Trust Center](https://app.vanta.com/private-ai.com/trust/6pwo3ib6rvq44sdba16it). ### Does Limina provide proof of X, Y, Z security certification? The Limina product is deployed and managed exclusively within the customer environment, by customer staff, on-prem or within the customer managed cloud. This presents several significant security differences when compared to a SAAS or cloud hosted product: * **ISO 27001 compliance**: Applicable in context of Limina's internal operations * **SOC 2 compliance**: Not applicable, this applies to the operations of the customer environment * **Penetration Testing**: Not applicable, this applies to the security of the customer environment * **Data Processing Agreement (DPA)**: Not applicable, this applies data management within the customer environment * **Business Associate Agreement (BAA)**: Not applicable, this applies data management within the customer environment # Upgrade & Migration Guide Source: https://docs.getlimina.ai/configuration-and-operations/container-management/upgrade-guide This guide will help you migrate from version 3 to version 4 of the Limina product This document covers the breaking changes that must be addressed when migrating between Limina versions. For the complete list of all new features, please see the full release notes in the [Customer Portal](https://portal.getlimina.ai/). ## Upgrading from version 4.3.1 to version 4.4+ For customers with outbound firewall rules. Version 4.4.0+ utilizes new license verification endpoints `verify1.getlimina.ai` and `verify2.getlimina.ai` any existing firewall rules will need to be updated accordingly. The previous endpoints `verify1.private-ai.com` and `verify2.private-ai.com` will continue to function for older containers. ## Upgrading from version 3.X, 4.0.X to version 4.1+ For customers with outbound firewall rules. Version 4.1.0+ utilizes new license verification endpoints `verify1.private-ai.com` and `verify2.private-ai.com` any existing firewall rules will need to be updated accordingly. ## Upgrading from version 3.X to version 4.0 ### API URI change Starting with version 4.0, you will no longer need to specify the container version in your API requests directly to the container. This change will allow customers to run multiple versions of the container simultaneously so that any API consumers can migrate without downtime. We recommend an API Gateway be utilized to insert versioning into the URI path, since it has been removed from the container. For Community and Professional consumers, simply update `v3` to `v4` in your URI. **Localhost Example** ```shell 3.X wrap theme={"theme":"poimandres"} curl --request POST --url http://localhost:8080/v3/process/text \ --json '{"text": ["Hello John"]}' ``` ```shell 4.0 wrap theme={"theme":"poimandres"} curl --request POST --url http://localhost:8080/process/text \ --json '{"text": ["Hello John"]}' ``` **Community Example** ```shell 3.X wrap theme={"theme":"poimandres"} curl --request POST https://api.private-ai.com/community/v3/process/text \ --header 'x-api-key: ' --json '{"text": ["Hello John"]}' ``` ```shell 4.0 wrap theme={"theme":"poimandres"} curl --request POST https://api.getlimina.ai/community/v4/process/text \ --header 'x-api-key: ' --json '{"text": ["Hello John"]}' ``` **Professional Example** ```shell 3.X wrap theme={"theme":"poimandres"} curl --request POST https://api.private-ai.com/professional/v3/process/text \ --header 'x-api-key: ' --json '{"text": ["Hello John"]}' ``` ```shell 4.0 wrap theme={"theme":"poimandres"} curl --request POST https://api.getlimina.ai/professional/v4/process/text \ --header 'x-api-key: ' --json '{"text": ["Hello John"]}' ``` ### GENDER\_SEXUALITY Entity Change Terms denoting gender identity and sexual orientation, previously redacted under a single class `GENDER_SEXUALITY` will now be redacted under two separate classes: `GENDER` and `SEXUALITY` respectively. Please see our [Supported Entity Types](/entities/supported-entity-types) page for descriptions and examples of these entity types. ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": ["My male friend is gay"] } ``` ```json 3.X Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "My [GENDER_SEXUALITY_1] friend is [GENDER_SEXUALITY_2]" } ] ``` ```json 4.0 Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "My [GENDER_1] friend is [SEXUALITY_1]" } ] ``` ### HIPAA and PHI Entity Groupings Change The entity type groupings `HIPAA` and `PHI` have been renamed and the entity types included in each set has also changed. `HIPAA` has been renamed to `HIPAA_SAFE_HARBOR` and, accordingly, only includes entity types covered by the [HIPAA Safe Harbor provision](https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html). The `PHI` grouping has been renamed to `HEALTH_INFORMATION`. For further information on using these selective entity groupings, please see our documentation on [Customizing Detection](/configuration-and-operations/entity-detection-and-redaction/customizing-detection), as well as our [Supported Entity Types](/entities/supported-entity-types) for details on which entity types are included in each set. # Accuracy Modes Source: https://docs.getlimina.ai/configuration-and-operations/entity-detection-and-redaction/accuracy-modes Learn about accuracy modes such as 'standard', 'high', and 'high_multilingual' models compatible with the Limina container for PII detection and redaction. This guide explains the different **Accuracy Modes** available for selecting the model used to identify Personally Identifiable Information (PII) in text. The default mode is [High Automatic](#high-automatic-accuracy), which automatically chooses between [High](#high-accuracy) and [High Multilingual](#high-multilingual-accuracy) models. While the models in the Limina solution are highly optimized (approximately 25x faster than a reference transformer implementation), in high-throughput scenarios, speed can be prioritized over accuracy by choosing Standard models. Standard Accuracy Standard High Accuracy Standard High Multilingual Accuracy Standard High Automatic Accuracy High Accuracy High Multilingual Accuracy High Automatic Accuracy ## Standard Accuracy | Attribute | Rating | | ------------ | ------- | | Accuracy | Lower | | Speed | Highest | | Multilingual | No | The `standard` accuracy mode offers a balanced trade-off between speed and accuracy. It is designed for high-throughput environments where faster processing is required, but slight reduction in accuracy is acceptable. ### When to choose Standard Accuracy? * When speed is the primary concern. * When processing mainly English text. ## Standard High Accuracy | Attribute | Rating | | ------------ | ------ | | Accuracy | Higher | | Speed | Higher | | Multilingual | No | The `standard_high` mode provides higher accuracy than [Standard](#standard-accuracy) but maintains a fast processing speed. It is optimized for English text and doesn’t support multilingual data. ### When to choose Standard High Accuracy? * When increased accuracy is required, but speed is still important. * When processing English-only text. ## Standard High Multilingual Accuracy | Attribute | Rating | | ------------ | ------ | | Accuracy | Higher | | Speed | Higher | | Multilingual | Yes | The `standard_high_multilingual` mode is the multilingual equivalent of the [Standard High](#standard-high-accuracy) model. See our documentation for details on which [languages we support](/languages). ### When to choose Standard High Multilingual Accuracy? * When increased accuracy is required as compared to [Standard](#standard-accuracy) but speed is still important. * When working with multilingual data, including English. ## Standard High Automatic Accuracy | Attribute | Rating | | ------------ | ------ | | Accuracy | Higher | | Speed | Higher | | Multilingual | Yes | The `standard_high_automatic` mode uses a combination of [standard High](#standard-high-accuracy) and [Standard High Multilingual](#standard-high-multilingual-accuracy) models. It dynamically chooses between English-only and multilingual models based on the detected language of the input text, provided the multilingual model is available. While the same model will be used for a single sample, different models can be applied across a batch, making this mode ideal for processing data in mixed languages. See our documentation for details on which [languages we support](/languages). ### When to choose Standard High Automatic Accuracy? * When increased accuracy is required, but speed is still important. * When automatic language detection and model selection are needed. ## High Accuracy | Attribute | Rating | | ------------ | ------- | | Accuracy | Highest | | Speed | Lower | | Multilingual | No | The `high` accuracy mode provides the best accuracy for English text, making it suitable for scenarios where precision is critical. It is slightly slower compared to the `standard` model but delivers the highest detection accuracy for English-only text and documents. GPU support is available for tasks requiring higher accuracy and throughput. ### When to choose High Accuracy? * When the highest level of accuracy for English text is needed. * When slight speed reductions are acceptable for better detection. ## High Multilingual Accuracy | Attribute | Rating | | ------------ | ------- | | Accuracy | Highest | | Speed | Lower | | Multilingual | Yes | The `high_multilingual` mode is the multilingual equivalent of the [High](#high-accuracy) model. It offers high accuracy for multilingual text but is slower compared to the `standard_high_multilingual` version. GPU support is available for tasks requiring higher accuracy and throughput. See our documentation for details on which [languages we support](/languages). ### When to choose High Multilingual Accuracy? * When the highest level of accuracy is needed, and slight speed reductions are acceptable for improved detection. * When working with multilingual data, including English. ## High Automatic Accuracy | Attribute | Rating | | ------------ | ------- | | Accuracy | Highest | | Speed | Lower | | Multilingual | Yes | The `high_automatic` mode uses a combination of [High](#high-accuracy) and [High Multilingual](#high-multilingual-accuracy) models. It dynamically chooses between English-only and multilingual models based on the detected language of the input text, provided the multilingual model is available. While the same model will be used for a single sample, different models can be applied across a batch, making this mode ideal for processing data in mixed languages. GPU support is available for tasks requiring higher accuracy and throughput. See our documentation for details on which [languages we support](/languages). ### When to choose High Automatic Accuracy? * When the highest level of accuracy is needed, and slight speed reductions are acceptable for improved detection. * When automatic language detection and model selection are needed. ## Conclusion Choosing the appropriate accuracy mode depends on your specific needs for speed, precision, and language support. **Standard** modes prioritize speed, making them ideal for high-throughput environments with primarily English text. **High** modes, on the other hand, deliver superior accuracy for scenarios where precision is critical, accepting some reduction in speed. For multilingual data, the **Multilingual** modes ensure comprehensive language coverage. The **Automatic** modes provide the best balance, dynamically selecting the appropriate model based on the language detected. By selecting the right mode, you can optimize performance and ensure accurate PII detection tailored to your data processing needs. For details on our throughput and latency, please see our [benchmarks](/configuration-and-operations/container-management/benchmarks) page. # Customizing Detection Source: https://docs.getlimina.ai/configuration-and-operations/entity-detection-and-redaction/customizing-detection This guide presents advanced techniques to help customize the PII entity detection. [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. Each use case is different and you may sometimes need to adjust the types of entities that Limina detects to address your specific requirements. This guide introduces a few techniques to modify and extend the detection engine to get the most from your data and is organized into two parts: * Part 1: [Enabling and Disabling Entity Types](#enabling-and-disabling-entity-types) covers enabling and disabling the types of entities that are detected. * Part 2: [Filters](#filters) covers allow & block list functionality and including regexes. The techniques described in the following sections apply to most of the Limina APIs. In particular, they can be used to customize the detection in the [NER route](/latest/ner-text), the [Process Text route](//latest/process-text), the [File URI route](/latest/process-files-uri/) and the [File Base64 route](/latest/process-files-base64/). See the specific route documentations for details. The following description will be using example requests and responses from the [Process Text route](//latest/process-text) for simplicity. ## Enabling and Disabling Entity Types Limina detects over 50 unique [entity types](/entities) ranging from personal, credit card and medical information. By default, all non-beta supported types are detected but this can be easily customized via entity selectors. If you need to comply to an existing legislation like GDPR or HIPAA, you may want to de-identify only the entities covered by this regulation. This can easily be done with preset entity groups. Or you may prefer to detect your own set of entities. This can also be done using entity selectors. An example Python script showing how to use entity selectors with Limina's Python client can be found [here](https://github.com/privateai/deid-examples/blob/main/python/examples/enabled_classes.py). ### Configuring Entity Selectors Entity selectors let you *enable* or *disable* entity types as part of your API request. You can, for example, enable `NAME` and `ORGANIZATION` while ignoring all other entity types by using the `ENABLE` selector as shown in this request: ```json Request Body wrap lines highlight={8-9} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["ORGANIZATION", "NAME"] } ] } } ``` As expected, the name and organization mentions are redacted while other entities like dates (i.e. *2019*) and conditions (i.e. *COVID*) are left untouched in the de-identified text: ```text Redacted Text wrap theme={"theme":"poimandres"} "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since 2019 because of COVID. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1]." ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since 2019 because of COVID. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1].", "entities": [ { "processed_text": "ORGANIZATION_1", "text": "Icarus Airways Customer Service", "location": { "stt_idx": 12, "end_idx": 43, "stt_idx_processed": 12, "end_idx_processed": 28 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8451 } }, { "processed_text": "ORGANIZATION_2", "text": "Icarus", "location": { "stt_idx": 134, "end_idx": 140, "stt_idx_processed": 119, "end_idx_processed": 135 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.6382 } }, { "processed_text": "ORGANIZATION_2", "text": "Icarus", "location": { "stt_idx": 262, "end_idx": 268, "stt_idx_processed": 257, "end_idx_processed": 273 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.6274 } }, { "processed_text": "NAME_1", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 366, "end_idx_processed": 374 }, "best_label": "NAME", "labels": { "NAME": 0.8899 } }, { "processed_text": "NAME_1", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 376, "end_idx_processed": 384 }, "best_label": "NAME", "labels": { "NAME": 0.8863 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` It is sometimes simpler to specify the entities to disable. This can be done using a `DISABLE` selector: ```json Request Body wrap lines highlight={8-9} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "entity_detection": { "entity_types": [ { "type": "DISABLE", "value": ["DATE", "DATE_INTERVAL"] } ] } } ``` The above request will redact all entity types except dates and date intervals as shown by the corresponding output: ```text Redacted Text wrap theme={"theme":"poimandres"} "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since 2019 because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1]." ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since 2019 because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1].", "entities": [ { "processed_text": "ORGANIZATION_1", "text": "Icarus Airways Customer Service", "location": { "stt_idx": 12, "end_idx": 43, "stt_idx_processed": 12, "end_idx_processed": 28 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8451 } }, { "processed_text": "ORGANIZATION_2", "text": "Icarus", "location": { "stt_idx": 134, "end_idx": 140, "stt_idx_processed": 119, "end_idx_processed": 135 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.6382 } }, { "processed_text": "ORGANIZATION_2", "text": "Icarus", "location": { "stt_idx": 262, "end_idx": 268, "stt_idx_processed": 257, "end_idx_processed": 273 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.6274 } }, { "processed_text": "CONDITION_1", "text": "COVID", "location": { "stt_idx": 291, "end_idx": 296, "stt_idx_processed": 296, "end_idx_processed": 309 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9187 } }, { "processed_text": "NAME_1", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 374, "end_idx_processed": 382 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3601, "NAME": 0.8899, "NAME_FAMILY": 0.5475 } }, { "processed_text": "NAME_1", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 384, "end_idx_processed": 392 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3714, "NAME": 0.8863, "NAME_FAMILY": 0.53 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` ### Preset Entity Groups If you need to comply with a specific legislation like HIPAA, the de-identification service makes it easy for you. You can simply choose from the list of preset entity groups: `['GDPR', 'GDPR_SENSITIVE', 'HIPAA_SAFE_HARBOR', 'CPRA', 'QUEBEC_PRIVACY_ACT', 'APPI', 'APPI_SENSITIVE', 'PCI', 'HEALTH_INFORMATION']`. For details of what is contained in each group, please consult our [entities page](/entities/supported-entity-types). This is an example on how to enable all entities covered by the GDPR legislation: ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["GDPR"] } ] } } ``` In the response below, the GDPR entities like `NAME` and `CONDITION` are redacted while `ORGANIZATION` mentions are not since there are not part of GDPR: ```text Redacted Text wrap theme={"theme":"poimandres"} "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1]." ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1].", "entities": [ { "processed_text": "CONDITION_1", "text": "COVID", "location": { "stt_idx": 291, "end_idx": 296, "stt_idx_processed": 291, "end_idx_processed": 304 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9187 } }, { "processed_text": "NAME_1", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 369, "end_idx_processed": 377 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3601, "NAME": 0.8899, "NAME_FAMILY": 0.5475 } }, { "processed_text": "NAME_1", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 379, "end_idx_processed": 387 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3714, "NAME": 0.8863, "NAME_FAMILY": 0.53 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` ### Security Considerations There are several reasons that may motivate the use of selectors to limit the set of entities: * You are working with data from a specific domain which may not contain some of the supported entities. For example, medical data are unlikely to contain PCI information like credit card number. Disabling these entities will prevent potential false-positives. * Some entities, while present in your data, may not be regarded as sensitive in your use case. For example, your data may contain generic URLs and filenames that can't be used to identify individuals. It is, however, important to understand that **disabling entities may increase the risk that sensitive information is leaked**. When selecting the list of entities to redact, we encourage you to take extra time and care to think on all the possible implications. A good practice is to have an expert validation to confirm that the redacted contents is free of PII or other sensitive information. Following this validation, the list may be adjusted according to the findings. ### Advanced Topics This section presents advanced techniques to help you get the most of the Limina service. #### **Combining Selectors** It is possible to combine selectors to help create the desired subset of entities. For example, you may be interested in complying with GDPR and HIPAA legislations. This can be done by listing these two groups in an `ENABLE` selector. ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["GDPR", "HIPAA_SAFE_HARBOR"] } ] } } ``` You can also pick and choose the list the entities from groups and individual entity types. ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["GDPR", "ORGANIZATION"] } ] } } ``` The above request will redact all GDPR entities as well as `ORGANIZATION`. It is also possible to combine `ENABLE` and `DISABLE` selectors. ```json Request Body wrap lines highlight={8-9,12-13} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["GDPR"] }, { "type": "DISABLE", "value": ["DATE", "DATE_INTERVAL"] } ] } } ``` This request is enabling all GDPR entities except `DATE` and `DATE_INTERVAL`. **Selector Precedence** When combining several selectors, the list of enable entities is first computed by expanding all entity types and groups in the `ENABLE` selectors. The entity types and groups listed in `DISABLE` selectors are then expended and removed from that list to form the final list of entities. When no `ENABLE` selectors are specified, it is assumed that the all supported entities are enabled. In this case, `DISABLE` selectors will remove from the list of all supported entities. #### **Selective redaction** If you want to redact only one or two entities out of 50+ entity types that we support, you can use the `ENABLE` selector. For example, to redact only `NAME_FAMILY` and `LOCATION_ADDRESS_STREET` in the text below, we use `ENABLE` selector and specify these two entities: ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "Hello there! I am Alice McGee, residing at 325 Sophia St, Port Coquitlam. You can reach me at 235 123-9876 or via email at alicemcgee@gmail.com. I work at SFU." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["NAME_FAMILY", "LOCATION_ADDRESS_STREET"] } ] } } ``` The above request will keep `NAME_GIVEN`, `LOCATION_CITY`, `PHONE_NUMBER`, `EMAIL_ADDRESS` and `ORGANIZATION` visible and redact only `NAME_FAMILY` and `LOCATION_STREET_ADDRESS`. In other cases, you may only want to redact PCI but keep all other entities unredacted, specifically `ACCOUNT_NUMBER` in the following example: ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "His account number at WaveNow Digital is 6787655, and it is connected to his bank account at First National Bank, 987654321. But he also has a credit card on file, ending with 7876, expires on 10/25, and his billing address is 456 41st Avenue, Lower Valley." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": ["PCI"] } ], "enable_non_max_suppression": true } } ``` In this example, `BANK_ACCOUNT`, `CREDIT_CARD`, `CREDIT_CARD_EXPIRATION` and `CVV` will be redacted, while `ACCOUNT_NUMBER` and `LOCATION_ADDRESS` will be visible. #### **Understanding Multi-Label Predictions** At the core of the Limina service lies a model that performs *named entity recognition* (NER). In essence, NER models seek to classify each word in a text into a fixed set of classes: the entity types. NER is often framed as a multi-class multi-label problem since a single word can have several labels. For example, in *Simon Fraser University* the word *Simon* is both part of a name (i.e. Simon Fraser) and an organization (i.e. Simon Fraser University). To create the de-identified or redacted output, the service must select among the predicted labels the one that best represent the entity. To do so, it will often prefer longer entities over shorter ones. For example, the service will redact *Simon Fraser University* as an ORGANIZATION: ```text Example wrap theme={"theme":"poimandres"} I study at Simon Fraser University -> I study at [ORGANIZATION] ``` instead of a combination of a NAME and an ORGANIZATION: ```text Example wrap theme={"theme":"poimandres"} I study at Simon Fraser University -> I study at [NAME] [ORGANIZATION] ``` This leads to a much more natural output for the users. Disabling entities has direct impact on that behaviour. Let's suppose that `ORGANIZATION` has been disabled when redacting the above text. ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "I study at Simon Fraser University" ], "entity_detection": { "entity_types": [ { "type": "DISABLE", "value": ["ORGANIZATION"] } ] } } ``` The resulting response might be surprising at first. ```text Redacted Text wrap theme={"theme":"poimandres"} "I study at [NAME_1] University" ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "I study at [NAME_1] University", "entities": [ { "processed_text": "NAME_1", "text": "Simon Fraser", "location": { "stt_idx": 11, "end_idx": 23, "stt_idx_processed": 11, "end_idx_processed": 19 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.2466, "NAME": 0.4709, "NAME_FAMILY": 0.2037 } } ], "entities_present": true, "characters_processed": 34, "languages_detected": { "en": 0.8937596678733826 } } ] ``` Although `ORGANIZATION` was disabled, a part of the *Simon Fraser University* was redacted. This is explained by the fact that *Simon Fraser* is both part of an organization name but also a person name. Given that `ORGANIZATION` was disabled, the de-identification service picked the *second best* label for these words which is `NAME`. Depending on your use case, you may prefer to keep the full organization name in the output. This can be done with the `enable_non_max_suppression` flag. ```json Request Body wrap lines highlight={12} theme={"theme":"poimandres"} { "text": [ "I study at Simon Fraser University" ], "entity_detection": { "entity_types": [ { "type": "DISABLE", "value": ["ORGANIZATION"] } ], "enable_non_max_suppression": true } } ``` When the `enable_non_max_suppression` flag is set to true, the service will ignore labels with lower likelihoods (i.e. `NAME` in the above example) therefore preventing the redaction of the `ORGANIZATION` as shown below. ```text Redacted Text wrap theme={"theme":"poimandres"} I study at Simon Fraser University ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "I study at Simon Fraser University", "entities": [], "entities_present": false, "characters_processed": 34, "languages_detected": { "en": 0.8937596678733826 } } ] ``` Note that, as with disabled entities, one should use the `enable_non_max_suppression` cautiously. **Setting this flag to true may increase the chance of leaking sensitive information**. #### **Understanding Hierarchical Types** Some of the supported entities in the de-identification service are structured into hierarchies. The labels `NAME` and `LOCATION` are good examples. The `NAME` hierarchy includes `NAME_GIVEN` and `NAME_FAMILY` but also `NAME_MEDICAL_PROFESSIONAL` while the `LOCATION` hierarchy contains `LOCATION_COUNTRY`, `LOCATION_STATE`, `LOCATION_CITY` and so on. However, most labels supported by Limina are not part of a hierarchy. The diagram below shows the different hierarchies. Hierarchies are organized from the most general entity types to the most specific ones. In the diagram, the general types are on the left (level 1). As one moves to the right, the entity types become more specific (level 2 and level 3). The diagram shows also whether the label is a direct identifier, a quasi identifier, or neither (i.e., other). It also shows which of the labels are still in beta. When creating the redacted text, the de-identification service will **prefer to use the most specific label in a hierarchy** instead of the root label. For example, *I live in Canada* will be redacted as `I live in [LOCATION_COUNTRY]` instead of the more generic `I live in [LOCATION]`. This behaviour improves the usability of the data. If you don't want this level of granularity, you can use the information in the graph above to map more specific labels (e.g., `LOCATION_CITY`) to more generic ones (e.g., `LOCATION`). This can be done as a post-processing step on the Limina API output. ## Filters This guide assumes that you have a working knowledge of regular expressions. Limina is using the Python regular expression syntax. You can find more details on the [Python `re` module documentation](https://docs.python.org/3/library/re.html). Sometimes referred to as whitelists and blacklists, filters are specifically designed to allow entities (i.e. leave them in the text) or block entities (i.e. redact them from the text) when the entity text follows an expected format. Filters are built using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression). An example Python script showing how to use filters with Limina's Python client can be found [here](https://github.com/privateai/deid-examples/blob/main/python/examples/block_and_allow.py). ### Allow Filter How can you redact regular phone numbers while keeping companies' toll-free numbers in clear? How would you prevent document ID numbers from being detected as a sensitive numerical number? These are two examples of use cases that can be addressed with Allow filters. Allow filters instruct the detection engine to ignore entities when the entity text match a specific pattern. To create an Allow filter, a regular expression pattern is first created. This pattern is then added to the `filter` list in the `entity_detection` object of your REST request. Let's look at a couple of examples. #### **Allow List** It is possible to feed lists of terms as well as regex patterns to filters. For example, if you want to prevent the detection engine from removing country names you can whitelist them with: ```json Request Body Entity Detection wrap lines theme={"theme":"poimandres"} "entity_detection": { "filter": [ { "type": "ALLOW", "pattern": "Canada|Brazil|Italy" } ] } ``` #### **Allowing toll-free numbers** This is an example of a `process/text` request containing an Allow filter. When run this request will detect and redact phone numbers unless they follow the specific format for toll-free phones: ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "Call me at 438-555-7343 or at work at 1-800-555-1423" ], "entity_detection": { "filter": [ { "type": "ALLOW", "pattern": "(1-)?(800|888|877|866|855|844|833)-\\d{3}-\\d{4}$" } ] } } ``` Gives the following: ```text Redacted Text wrap theme={"theme":"poimandres"} Call me at [PHONE_NUMBER_1] or at work at 1-800-555-1423 ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Call me at [PHONE_NUMBER_1] or at work at 1-800-555-1423", "entities": [ { "processed_text": "PHONE_NUMBER_1", "text": "438-555-7343", "location": { "stt_idx": 11, "end_idx": 23, "stt_idx_processed": 11, "end_idx_processed": 27 }, "best_label": "PHONE_NUMBER", "labels": { "PHONE_NUMBER": 0.9093 } } ], "entities_present": true, "characters_processed": 52, "languages_detected": { "en": 0.695495069026947 } } ] ``` As expected, the first phone number is redacted while the second toll-free number is left in the processed text. **Escaping Regular Expressions** Many regular expressions contain slashes `\` and other special characters. It is important to note that the slash `\` is a reserved character in json. As such, slashes in string must be escaped as `\\` to retain its original meaning. As demonstrated in the example above, the regular expression `r"(1-)?(800|888|877|866|855|844|833)-\d{3}-\d{4}$"` was escaped to `"(1-)?(800|888|877|866|855|844|833)-\\d{3}-\\d{4}$"` in the json request body. #### **Allowing IDs** Let's look at a different example. Suppose that you are de-identifying contracts of the form: ```text Example Text wrap theme={"theme":"poimandres"} CCT-2022-09-12321: Contract between John Doe and Acme Corp. THIS AGREEMENT is made ... ``` It might be difficult for the detection engine to determine if the ID `CCT-2022-09-12321` in the document header is sensitive. The sensitivity may depend for example on other information being publicly available. In this case, the detection engine will flag the IDs. However, if you know that these numbers are not sensitive you may prefer to instruct the detection engine to allow such entities: ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "CCT-2022-09-12321: Contract between John Doe and Acme Corp.\n\nTHIS AGREEMENT is made ..." ], "entity_detection": { "filter": [ { "type": "ALLOW", "pattern": "CCT-\\d{4}-\\d{2}-\\d+" } ] } } ``` This is the `process/text` response to the above request: ```text Redacted Text wrap theme={"theme":"poimandres"} CCT-2022-09-12321: Contract between [NAME_1] and [ORGANIZATION_1].\n\nTHIS AGREEMENT is made ... ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "CCT-2022-09-12321: Contract between [NAME_1] and [ORGANIZATION_1].\n\nTHIS AGREEMENT is made ...", "entities": [ { "processed_text": "NAME_1", "text": "John Doe", "location": { "stt_idx": 36, "end_idx": 44, "stt_idx_processed": 36, "end_idx_processed": 44 }, "best_label": "NAME", "labels": { "NAME": 0.9287, "NAME_GIVEN": 0.3926, "NAME_FAMILY": 0.2851 } }, { "processed_text": "ORGANIZATION_1", "text": "Acme Corp", "location": { "stt_idx": 49, "end_idx": 58, "stt_idx_processed": 49, "end_idx_processed": 65 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.885 } } ], "entities_present": true, "characters_processed": 89, "languages_detected": { "en": 0.9532792568206787 } } ] ``` Without the Allow filter above, the ID `CCT-2022-09-12321` would be redacted as `NUMERICAL_PII` but as expected it was not redacted thanks to the Allow filter. ### Block Filter Let's say that you want to detect some codes or ids sharing a common format in your data. You can rely on the de-identification service to perform the redaction for you, but it may sometimes be preferable to create your own detection logic and provide a specific label for these entities. This is exactly what block filters are for. #### **Block List** Similar to the allow list or whitelist, you can create a block list or blacklist to ensure that some common keywords are always detected and removed like so: ```json Request Body Entity Detection wrap lines theme={"theme":"poimandres"} "entity_detection": { "filter": [ { "type": "BLOCK", "pattern": "Android|iPhone|Pixel", "entity_type": "CELL_TYPE" } ] } ``` #### **Blocking IDs** Let's look at our contract example above. With the help of block filters, you can redact the contract id as `CONTRACT_ID` in the document above: ```json Request Body wrap lines highlight={9-10} theme={"theme":"poimandres"} { "text": [ "CCT-2022-09-12321: Contract between John Doe and Acme Corp.\n\nTHIS AGREEMENT is made ..." ], "entity_detection": { "filter": [ { "type": "BLOCK", "pattern": "CCT-\\d{4}-\\d{2}-\\d+", "entity_type": "CONTRACT_ID" } ] } } ``` This is the `process/text` response to the above request: ```text Redacted Text wrap theme={"theme":"poimandres"} [CONTRACT_ID_1]: Contract between [NAME_1] and [ORGANIZATION_1].\n\nTHIS AGREEMENT is made ... ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "[CONTRACT_ID_1]: Contract between [NAME_1] and [ORGANIZATION_1].\n\nTHIS AGREEMENT is made ...", "entities": [ { "processed_text": "CONTRACT_ID_1", "text": "CCT-2022-09-12321", "location": { "stt_idx": 0, "end_idx": 17, "stt_idx_processed": 0, "end_idx_processed": 15 }, "best_label": "CONTRACT_ID", "labels": { "CONTRACT_ID": 1 } }, { "processed_text": "NAME_1", "text": "John Doe", "location": { "stt_idx": 36, "end_idx": 44, "stt_idx_processed": 34, "end_idx_processed": 42 }, "best_label": "NAME", "labels": { "NAME": 0.9203, "NAME_GIVEN": 0.3381, "NAME_FAMILY": 0.1802 } }, { "processed_text": "ORGANIZATION_1", "text": "Acme Corp", "location": { "stt_idx": 49, "end_idx": 58, "stt_idx_processed": 47, "end_idx_processed": 63 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.7899 } } ], "entities_present": true, "characters_processed": 87, "languages_detected": { "en": 0.9520388245582581 } } ] ``` As expected, the contract id in the text has been redacted with our own custom marker. Here is another example with a more complex pattern to match. #### **Augmenting existing entity type** This is provided as an example and not as a complete solution to redact all ICD numbers. In this example, we are detecting ICD-10 numbers and adding these entities to the existing `CONDITION` entity type: ```json Request Body wrap lines highlight={9-10} theme={"theme":"poimandres"} { "text": [ "ICD-10 References\nJ18.9 | Pneumonia\nE11.52 | Type 2 diabetes mellitus with certain circulatory complications" ], "entity_detection": { "filter": [ { "type": "BLOCK", "pattern": "(?i)([a-t]|[v-z])\\d[a-z0-9](\\.[a-z0-9]{1,4})?", "entity_type": "CONDITION" } ] } } ``` This is the `process/text` response to the above request: ```text Redacted Text wrap theme={"theme":"poimandres"} ICD-10 References\n[CONDITION_1] | [CONDITION_2]\n[CONDITION_3] | [CONDITION_4] with certain circulatory complications ``` ```json Full Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "ICD-10 References\n[CONDITION_1] | [CONDITION_2]\n[CONDITION_3] | [CONDITION_4] with certain circulatory complications", "entities": [ { "processed_text": "CONDITION_1", "text": "J18.9", "location": { "stt_idx": 18, "end_idx": 23, "stt_idx_processed": 18, "end_idx_processed": 31 }, "best_label": "CONDITION", "labels": { "CONDITION": 1 } }, { "processed_text": "CONDITION_2", "text": "Pneumonia", "location": { "stt_idx": 26, "end_idx": 35, "stt_idx_processed": 34, "end_idx_processed": 47 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.8982 } }, { "processed_text": "CONDITION_3", "text": "E11.52", "location": { "stt_idx": 36, "end_idx": 42, "stt_idx_processed": 48, "end_idx_processed": 61 }, "best_label": "CONDITION", "labels": { "CONDITION": 1 } }, { "processed_text": "CONDITION_4", "text": "Type 2 diabetes mellitus", "location": { "stt_idx": 45, "end_idx": 69, "stt_idx_processed": 64, "end_idx_processed": 77 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9196 } } ], "entities_present": true, "characters_processed": 108, "languages_detected": { "en": 0.5311049818992615 } } ] ``` You can see that the results from the block filter results and detection engine have been combined together to create a more comprehensive `CONDITION` entity type. ### Allow Text Filter (new in 3.7) Allow text filters are similar to Allow filters but instead of allowing individual entities, they "mark" sections of your document as safe so that no entities are detected and nothing is redacted or de-identified. Let's consider a simple example. #### **Allowing a section of a document** Suppose that you have a document which contains a *References* section with public information only: ```text Example Text wrap theme={"theme":"poimandres"} Conclusion A section with sensitive information like name (e.g. John Doe) and organization (e.g. Acme Corp). References Berfin Akta¸s, Veronika Solopova, Annalena Kohnert, and Manfred Stede. 2020. Adapting Coreference Resolution to Twitter Conversations. In Findings of EMNLP. Rahul Aralikatte, Heather Lent, Ana Valeria Gonzalez, Daniel Herschcovich, Chen Qiu, Anders Sandholm, Michael Ringaard, and Anders Søgaard. 2019. Rewarding Coreference Resolvers for Being Consistent with World Knowledge. In EMNLP-IJCNLP. ``` By default, this document would be redacted as: ```text Example Processed Text wrap theme={"theme":"poimandres"} Conclusion A section with sensitive information like name (e.g. [NAME_1]) and organization (e.g. [ORGANIZATION_1]). References [NAME_2], [NAME_3], [NAME_4],and [NAME_5]. [DATE_INTERVAL_1]. Adapting Coreference Resolution to Twitter Conversations. In Findings of EMNLP. [NAME_6], [NAME_7], [NAME_8],[NAME_9], [NAME_10], [NAME_11],[NAME_12], and [NAME_13]. [DATE_INTERVAL_2]. Rewarding Coreference Resolvers for Being Consistent with World Knowledge. In EMNLP-IJCNLP. ``` But you may prefer to not de-identify the *References* section since it is not sensitive. This could be done with the Allow Text filter (keeping only the filter in the request for readability): ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": ["..."], "entity_detection": { "filter": [ { "type": "ALLOW_TEXT", "pattern": "References\\s+([\\S\\s]+)", } ] } } ``` Which would result in this processed text: ```text Example Processed Text wrap theme={"theme":"poimandres"} "Conclusion A section with sensitive information like name (e.g. [NAME_1]) and organization (e.g. [ORGANIZATION_1]). References Berfin Akta¸s, Veronika Solopova, Annalena Kohnert, and Manfred Stede. 2020. Adapting Coreference Resolution to Twitter Conversations. In Findings of EMNLP. Rahul Aralikatte, Heather Lent, Ana Valeria Gonzalez, Daniel Herschcovich, Chen Qiu, Anders Sandholm, Michael Ringaard, and Anders Søgaard. 2019. Rewarding Coreference Resolvers for Being Consistent with World Knowledge. In EMNLP-IJCNLP.", ``` where the *References* section was not de-identified. Allow Text filters also support capturing groups in regular expressions. #### **Using capturing groups** [Capturing groups](https://docs.python.org/3/howto/regex.html#grouping) are a very useful feature of regular expressions. By adding capturing groups to your regular expression, you can effectively dissect a matched text into the sections of interest. Consider this document including an audit trail with the editor name and the date of the changes: ```text Example Text wrap theme={"theme":"poimandres"} [Part 1] [John Doe: Fri Mar 10 16:09:20 GMT 2023] [Conclusion] [John Hancock: March 14, 2023] ``` Let's say you want to de-identify the author name but keep the dates of the audit trail in your processed text. One approach is to use Allow filters. However, it might be difficult to create a proper regular expression to allow all possible date formats. Moreover, all date entities would be allowed and not only those in the audit trail. This is where Allow Text filters and capturing groups become useful. The following request contains an Allow Text filter for the audit trail above: ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "text": [ "[Part 1] [John Doe: Fri Mar 10 16:09:20 GMT 2023]\n[Conclusion] [John Hancock: March 14, 2023]" ], "entity_detection": { "filter": [ { "type": "ALLOW_TEXT", "pattern": "\\[[^:]*:([^\\]]*)\\]" } ] } } ``` Notice the capturing group `([^\]]*)` in the second part of the pattern. This group is selecting the date, that is, the section of text from the colon `:` up to the closing square bracket `]`. This informs the Allow Text filter that only this section has to be allowed. This produces this processed text: ```text Example Processed Text wrap theme={"theme":"poimandres"} [Part 1] [[NAME_1]: Fri Mar 10 16:09:20 GMT 2023] [Conclusion] [[NAME_2]: March 14, 2023] ``` where names are masked but dates are shown. **When groups are present, Allow Text filters will only allow the text matching the groups.** This provides the flexibility you need to allow the section of text you want. **A word of caution** The regular expression pattern in filters can be as complex as it needs to be in order to capture the specific text of interest. However, one should be careful to not create filter patterns that are too generic risking to de-identify unnecessary sections of your document or worse to leave sensitive information unredacted. **Quick summary** Limina uses the `re` Python package to evaluate regular expressions. As we have seen so far, the difference between `ALLOW` and `ALLOW_TEXT` can be summarized as follows: `ALLOW` checks whether a regex pattern `match` a detected entity value. If the entity text matches the pattern, the entity is ignored and not redacted. `ALLOW_TEXT` checks whether detected entities are part of any text fragments returned by `re.find` with the provided regex pattern. If the entity text is part of a text fragment that matches the pattern, the entity is ignored and not redacted. # Customizing Redaction Source: https://docs.getlimina.ai/configuration-and-operations/entity-detection-and-redaction/customizing-redaction This guide presents advanced techniques to help customize the creation of redacted output. [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. The Limina APIs offer a lot of flexibility when it comes to create redacted or de-identified content. This guide introduces a few techniques to modify and extend the existing capabilities of Limina APIs. It is organized into four parts: * Part 1: [Configuring a mask](#configuring-a-mask) covers the basics of setting a mask to meet your preferences. * Part 2: [Configuring a marker](#configuring-a-marker) covers the basics of setting the marker format to meet your preferences. * Part 3: [Using synthetic PII](#synthetic-pii) explains how to replace the original PII with synthetic values. * Part 4: [Custom redaction using the NER Text route](#custom-redaction-using-the-ner-text-route) presents an approach to create a fully customized redacted output using the NER route. The techniques described in the first two sections apply to most of the Limina APIs. In particular, they can be used to customize the redaction in the [Process Text route](/latest/process-text), the [File URI route](/latest/process-files-uri) and the [File Base64 route](/latest/process-files-base64). The configuration is done through the shared object `processed_text`, part of the API's request. See the specific route documentations for details. The following description will be using example requests and responses from the [Process Text route](/latest/process-text) for simplicity. When redacting or de-identifying text, a customizable string pattern is used to replace the detected PII in the text. Limina supports these replacement options: * `MASK` containing repeated characters up to the length of the replaced entity. This option provides a redacted text containing no information about the actual entities that were replaced. * `MARKER` containing the type of the entity being replaced. Markers can also be configured to link different mentions of the same entities in the redacted text (*i.e.*, a name that appear twice in the text will have the same unique replacement marker). * `SYNTHETIC` text containing an AI generated replacement for the original entity. This option provides a processed text that is very similar to the original input text except that sensitive PII has been replaced with fake values. ## Configuring a mask Masking is also known as hashing when the `#` character is used. Setting the mask option is as simple as setting the type `MASK` in the `processed_text` object. ```json Request Body wrap lines highlight={6} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MASK" } } ``` This option will replace all names, organizations and other PII mentioned with the default mask character `#`. The redacted text will then look like: ```text Mask Default (redacted text) wrap theme={"theme":"poimandres"} "Hi! This is ###############################. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my ###### Frequent Voyager points. I checked my account today. I had 5,000 points in ####. Now it’s just 3,500. I haven’t flown in ###### since #### because of #####. What happened? May I have your name and account number, ma’am? #############, #############." ``` ```json Mask Default (full response) wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is ###############################. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my ###### Frequent Voyager points. I checked my account today. I had 5,000 points in ####. Now it’s just 3,500. I haven’t flown in ###### since #### because of #####. What happened? May I have your name and account number, ma’am? #############, #############.", "entities": [ { "processed_text": "###############################", "text": "Icarus Airways Customer Service", "location": { "stt_idx": 12, "end_idx": 43, "stt_idx_processed": 12, "end_idx_processed": 43 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8451 } }, { "processed_text": "######", "text": "Icarus", "location": { "stt_idx": 134, "end_idx": 140, "stt_idx_processed": 134, "end_idx_processed": 140 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.7969 } }, { "processed_text": "####", "text": "2019", "location": { "stt_idx": 216, "end_idx": 220, "stt_idx_processed": 216, "end_idx_processed": 220 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9384 } }, { "processed_text": "######", "text": "Icarus", "location": { "stt_idx": 262, "end_idx": 268, "stt_idx_processed": 262, "end_idx_processed": 268 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8285 } }, { "processed_text": "####", "text": "2019", "location": { "stt_idx": 275, "end_idx": 279, "stt_idx_processed": 275, "end_idx_processed": 279 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9393 } }, { "processed_text": "#####", "text": "COVID", "location": { "stt_idx": 291, "end_idx": 296, "stt_idx_processed": 291, "end_idx_processed": 296 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9327 } }, { "processed_text": "#############", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 361, "end_idx_processed": 374 }, "best_label": "NAME", "labels": { "NAME": 0.903, "NAME_GIVEN": 0.3583, "NAME_FAMILY": 0.5411 } }, { "processed_text": "#############", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 376, "end_idx_processed": 389 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3708, "NAME": 0.907, "NAME_FAMILY": 0.5271 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` If you prefer a different masking character in your redacted text, you can specify it in the request. ```json Request Body wrap lines highlight={7} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MASK", "mask_character": "■" } } ``` The above request will redact using the provided mask character: ```text Custom Mask (redacted text) wrap theme={"theme":"poimandres"} "Hi! This is ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my ■■■■■■ Frequent Voyager points. I checked my account today. I had 5,000 points in ■■■■. Now it’s just 3,500. I haven’t flown in ■■■■■■ since ■■■■ because of ■■■■■. What happened? May I have your name and account number, ma’am? ■■■■■■■■■■■■■, ■■■■■■■■■■■■■." ``` ```json Custom Mask (full response) wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my ■■■■■■ Frequent Voyager points. I checked my account today. I had 5,000 points in ■■■■. Now it’s just 3,500. I haven’t flown in ■■■■■■ since ■■■■ because of ■■■■■. What happened? May I have your name and account number, ma’am? ■■■■■■■■■■■■■, ■■■■■■■■■■■■■.", "entities": [ { "processed_text": "■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■", "text": "Icarus Airways Customer Service", "location": { "stt_idx": 12, "end_idx": 43, "stt_idx_processed": 12, "end_idx_processed": 43 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8451 } }, { "processed_text": "■■■■■■", "text": "Icarus", "location": { "stt_idx": 134, "end_idx": 140, "stt_idx_processed": 134, "end_idx_processed": 140 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.7969 } }, { "processed_text": "■■■■", "text": "2019", "location": { "stt_idx": 216, "end_idx": 220, "stt_idx_processed": 216, "end_idx_processed": 220 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9384 } }, { "processed_text": "■■■■■■", "text": "Icarus", "location": { "stt_idx": 262, "end_idx": 268, "stt_idx_processed": 262, "end_idx_processed": 268 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8285 } }, { "processed_text": "■■■■", "text": "2019", "location": { "stt_idx": 275, "end_idx": 279, "stt_idx_processed": 275, "end_idx_processed": 279 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9393 } }, { "processed_text": "■■■■■", "text": "COVID", "location": { "stt_idx": 291, "end_idx": 296, "stt_idx_processed": 291, "end_idx_processed": 296 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9327 } }, { "processed_text": "■■■■■■■■■■■■■", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 361, "end_idx_processed": 374 }, "best_label": "NAME", "labels": { "NAME": 0.903, "NAME_GIVEN": 0.3583, "NAME_FAMILY": 0.5411 } }, { "processed_text": "■■■■■■■■■■■■■", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 376, "end_idx_processed": 389 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3708, "NAME": 0.907, "NAME_FAMILY": 0.5271 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` In both cases, the redacted text does not contain any information about the entities that were replaced beside their length. ## Configuring a marker The marker option allows the redacted text to include the entity type. This may improve the readability of the redacted text. Setting the default marker option is as simple as setting the mask option. ```json Request Body wrap lines highlight={6} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER" } } ``` With the default marker settings, the response will contain markers with the entity type and a unique number. ```text Marker Default (redacted text) wrap theme={"theme":"poimandres"} "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in [DATE_INTERVAL_1]. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since [DATE_INTERVAL_1] because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1]." ``` ```json Marker Default (full response) wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in [DATE_INTERVAL_1]. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since [DATE_INTERVAL_1] because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1].", "entities": [ { "processed_text": "ORGANIZATION_1", "text": "Icarus Airways Customer Service", "location": { "stt_idx": 12, "end_idx": 43, "stt_idx_processed": 12, "end_idx_processed": 28 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8451 } }, { "processed_text": "ORGANIZATION_2", "text": "Icarus", "location": { "stt_idx": 134, "end_idx": 140, "stt_idx_processed": 119, "end_idx_processed": 135 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.7969 } }, { "processed_text": "DATE_INTERVAL_1", "text": "2019", "location": { "stt_idx": 216, "end_idx": 220, "stt_idx_processed": 211, "end_idx_processed": 228 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9384 } }, { "processed_text": "ORGANIZATION_2", "text": "Icarus", "location": { "stt_idx": 262, "end_idx": 268, "stt_idx_processed": 270, "end_idx_processed": 286 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8285 } }, { "processed_text": "DATE_INTERVAL_1", "text": "2019", "location": { "stt_idx": 275, "end_idx": 279, "stt_idx_processed": 293, "end_idx_processed": 310 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9393 } }, { "processed_text": "CONDITION_1", "text": "COVID", "location": { "stt_idx": 291, "end_idx": 296, "stt_idx_processed": 322, "end_idx_processed": 335 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9327 } }, { "processed_text": "NAME_1", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 400, "end_idx_processed": 408 }, "best_label": "NAME", "labels": { "NAME": 0.903, "NAME_GIVEN": 0.3583, "NAME_FAMILY": 0.5411 } }, { "processed_text": "NAME_1", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 410, "end_idx_processed": 418 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3708, "NAME": 0.907, "NAME_FAMILY": 0.5271 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` Notice how, `Nessa Jonsson` and the spelled-out mention `N-E-S-S-A J-O-N-S-S-O-N` have been replaced with the same marker index `NAME_1`. Similarly, two mentions of `Icarus` have been replaced with `ORGANIZATION_2`. When creating the markers with the default settings, the de-identification service will use a unique marker index unless the entity was previously seen in the text. If the entity is repeated more than once in the text, the service will do its best to assign the same unique marker. Read more about keeping the relationship between entities in the [Coreference Resolution](#what-is-coreference-resolution) Section below. You can also create your own marker by providing a format containing one of the *marker keywords* below (*e.g.*, `[ENTITY_TYPE]`): | Marker keywords | Description | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ENTITY_TYPE` | Replace the entity with the type that best describes the entity (*e.g.*, *John* -> `NAME_GIVEN`) | | `ALL_ENTITY_TYPES` | Replace the entity with all the labels that applies (*e.g.*, *John* -> `NAME_GIVEN,NAME`) | | `UNIQUE_ENTITY_TYPE` (default) | Replace the entity with the type that best describes the entity and append a number so that different entities have different markers (e.g. *John* -> `NAME_GIVEN_1`, *Mary* -> `NAME_GIVEN_2`) | | `UNIQUE_HASHED_ENTITY_TYPE` | Similar to `UNIQUE_ENTITY_TYPE` except that to make the markers unique the service appends a hash value instead of an sequential integer. | Limina will replace detected entities with the provided format, using the above keyword as a format specification. Here are some examples of custom markers with the configuration that generated them. Replacing entities with a list of all applicable entity types: ```text ALL_ENTITY_TYPES redacted text wrap theme={"theme":"poimandres"} "Hi! This is . How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Frequent Voyager points. I checked my account today. I had 5,000 points in . Now it’s just 3,500. I haven’t flown in since because of . What happened? May I have your name and account number, ma’am? , ." ``` ```json ALL_ENTITY_TYPES request wrap lines highlight={7} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER", "pattern": "" } } ``` Replacing entities with unique hashed markers: ```text UNIQUE_HASHED_ENTITY_TYPE redacted text wrap theme={"theme":"poimandres"} "Hi! This is -->ORGANIZATION_6SjIF<--. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my -->ORGANIZATION_5AP6v<-- Frequent Voyager points. I checked my account today. I had 5,000 points in -->DATE_INTERVAL_m8wzC<--. Now it’s just 3,500. I haven’t flown in -->ORGANIZATION_5AP6v<-- since -->DATE_INTERVAL_m8wzC<-- because of -->CONDITION_FCj0D<--. What happened? May I have your name and account number, ma’am? -->NAME_HBtjJ<--, -->NAME_HBtjJ<--." ``` ```json UNIQUE_HASHED_ENTITY_TYPE request wrap lines highlight={7} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER", "pattern": "-->UNIQUE_HASHED_ENTITY_TYPE<--" } } ``` These are only a few examples of ways you can customize the markers in your de-identified text. ### What is coreference resolution Coreference resolution is a natural language processing (NLP) task that consists of locating and associating different mentions of real-world entities in unstructured text. Let's look at an example. ```text Text Example theme={"theme":"poimandres"} Hi! I'm Michael but call me Mike please. ``` In the short text above, the same individual is mentioned twice: once as *Michael* and once as *Mike*. Because these two mentions refer to the same person, we say that they are *coreferential*. Coreference resolution does not only address the problem of associating different mentions of an individual, but can also be used to identify mentions of organizations, locations, and so on. Traditionally, the coreference resolution task is often extended to also include pronouns (*e.g.*, "he" and "she") and other nominal forms (*e.g.*, "the PM of Canada"). Coreference resolution has proven to be very useful in many other NLP tasks, including: question answering, sentiment analysis, and document summarization, to name a few. If you are considering using redacted text as input to an existing model (*e.g.*, a LLM) or to train a model for another NLP task, you should consider how coreference resolution might enhance those tasks. ### Limina and coreference resolution (new in 4.0) Limina's de-identification service offers the ability to use coreference resolution on its `process/text` & `analyze/text` endpoint. The current implementation of coreference resolution is done on top of the named entity recognition (NER) model. It is, therefore, limited to returning coreference between entities only. Limina offers three different methods of performing coreference resolution. Whether you need to feed the redacted text to a ML model or simply make it easier to identify the different mentions of entities, you can benefit from Limina's coreference resolution support. | Method Name | Description | Speed | Limitations | | ---------------- | ----------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Heuristics | Uses rule-based methods for linking entities based on string matching. | Fast | Mostly links exact matches and a few minor variations (e.g., difference in casing). It may miss more complex variations and typos. | | Model Prediction | Uses a neural network model to resolve coreferences, allowing for variations. | Slower | Currently only supports `NAME` and `ORGANIZATION` entities in English. This method is much slower than the heuristics one. | | Combined | Combines both heuristics and model prediction for better coverage. | Slowest | Supports all entities with the heuristics method but resolves more complex cases for `NAME` and `ORGANIZATION` with the model prediction method. Slower than heuristics. | For details on using coreference resolution and other supported relations with the `analyze/text` endpoint, refer to the [analyze-text documentation](/configuration-and-operations/advanced-features/analyze-text#relation-detection). Here is an example of how to set coreference resolution in your de-identification request using the `process/text` endpoint. Notice the `coreference_resolution` field part of the `processed_text` object. ```json Request Body wrap lines highlight={8} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER", "pattern": "[UNIQUE_NUMBERED_ENTITY_TYPE]", "coreference_resolution": "model_prediction" } } ``` The `coreference_resolution` field can take one of three values: `heuristics`, `model_prediction` or `combined`. Note that coreference resolution is enabled whenever a unique marker is used (*i.e.*, `UNIQUE_NUMBERED_ENTITY_TYPE` or `UNIQUE_HASHED_ENTITY_TYPE`). By default, the `heuristics` mode will be enabled. The following sections describe each of these options. #### **Heuristics** This method of coreference resolution is based solely on string matching. It is therefore only capable of linking entities that are mentioned in the same way. For example, the entities `Mary` and `mary` will be linked together because the strings match, except for a small difference in casing. However, the entities `John A. Smith` and `Mr Smith` will **not** be linked together in this mode even if they are referring to the same person. As a consequence, the two entities will be assigned different unique markers (e.g., `NAME_1` and `NAME_2`) in the redacted text. Here is the output of the example above using the `heuristics` mode: ```text heuristics response wrap theme={"theme":"poimandres"} "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_2] Frequent Voyager points. I checked my account today. I had 5,000 points in [DATE_INTERVAL_1]. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_2] since [DATE_INTERVAL_1] because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1]." ``` ```json heuristics request wrap lines highlight={8} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER", "pattern": "[UNIQUE_NUMBERED_ENTITY_TYPE]", "coreference_resolution": "heuristics" } } ``` Notice how only two of the three mentions of the Icarus organization were linked together (*i.e.*, using the same `ORGANIZATION_2` marker). The first mention is too different from the two other mentions for it to be resolved using heuristics. However, the two mentions of the person: `Nessa Jonsson` and the spelled-out mention `N-E-S-S-A J-O-N-S-S-O-N` were correctly linked despite the differences between the strings. While the `heuristics` mode has its limitations, it is great when a more **predictable** output is required (*e.g.*, all exact mentions of an entity need to be linked together in a text, no matter how long or difficult the text is). This option is also the **fastest** one and **all entity types** in addition to `NAME` and `ORGANIZATION` are supported. The `heuristics` mode is currently the default one in the `process/text` endpoint. #### **Model prediction** (new in 4.0) The `model_prediction` option was introduced by Limina to work around some of the limitations of the `heuristics` mode of resolution. This option uses a neural network model to resolve coreferences. It is capable of resolving mentions that have different spellings or even mentions containing typos. ```text model_prediction response wrap theme={"theme":"poimandres"} "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_1] Frequent Voyager points. I checked my account today. I had 5,000 points in [DATE_INTERVAL_1]. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_1] since [DATE_INTERVAL_2] because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_2]." ``` ```json model_prediction request wrap lines highlight={8} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER", "pattern": "[UNIQUE_NUMBERED_ENTITY_TYPE]", "coreference_resolution": "model_prediction" } } ``` In the above request, only the value of the `coreference_resolution` field has changed. As you can see, the `model_prediction` mode is capable of linking *Icarus Airways Customer Service* with *Icarus*, leading to a redacted text in which all mentions of the Icarus organization are replaced with the same marker (*i.e.*, `ORGANIZATION_1`). However, the model was unable to link the person's name *Nessa Jonsson* with the spelled-out form. The `model_prediction` option is great if you are dealing with entity mentions that may contain variation or typos. The `model_prediction` option currently only resolves **`NAME` and `ORGANIZATION` entities in English text**. Note also that this option is much **slower than the `heuristics`** one. It is not recommended for text samples that contain more than a few hundred words. #### **Combined** (new in 4.0) As its name suggests, this option combines the two other coreference resolution modes. ```text combined response wrap theme={"theme":"poimandres"} "Hi! This is [ORGANIZATION_1]. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my [ORGANIZATION_1] Frequent Voyager points. I checked my account today. I had 5,000 points in [DATE_INTERVAL_1]. Now it’s just 3,500. I haven’t flown in [ORGANIZATION_1] since [DATE_INTERVAL_1] because of [CONDITION_1]. What happened? May I have your name and account number, ma’am? [NAME_1], [NAME_1]." ``` ```json combined request wrap lines highlight={8} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "MARKER", "pattern": "[UNIQUE_NUMBERED_ENTITY_TYPE]", "coreference_resolution": "combined" } } ``` By leveraging the two approaches for finding coreferential mentions, the `combined` mode is able to resolve all coreferential mentions in the above example. If you plan to process only English text and want the best coverage to identify and resolve coreferential mentions of all types, then this mode is for you! Note that the `combined` mode suffers from the same limitations as the `model_prediction` mode. It is much slower than the `heuristics` mode alone and it is not recommended when processing large volumes of text (*e.g.*, several thousand words). :::note coreference resolution across multiple requests Note that the coreference resolution is only performed **within a single request**. Identical entities across different requests will usually be assigned different markers. If you are processing related text fragments, you may consider passing them as a **batch in a single request** and setting **`link_batch` to True**. This will allow the de-identification service to link entities across these fragments. ::: ## Synthetic PII You may choose to replace the entities in your text with fake or synthetic entities instead of markers and masks. There are a [few reasons](/container-quickstart/api-quickstart#generating-synthetic-entities-beta) to do so. For example, if you train an AI model on your data, synthetic replacements might provide a more realistic input to train your model. Generating synthetic PII is done by setting `processed_text.type` to `SYNTHETIC`. ```json Request Body wrap lines highlight={6} theme={"theme":"poimandres"} { "text": [ "Hi! This is Icarus Airways Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my Icarus Frequent Voyager points. I checked my account today. I had 5,000 points in 2019. Now it’s just 3,500. I haven’t flown in Icarus since 2019 because of COVID. What happened? May I have your name and account number, ma’am? Nessa Jonsson, N-E-S-S-A J-O-N-S-S-O-N." ], "processed_text": { "type": "SYNTHETIC" } } ``` The synthetic text output will be similar to: ```text Synthetic Text wrap theme={"theme":"poimandres"} "Hi! This is United Federal Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my United Federal Customer Service Frequent Voyager points. I checked my account today. I had 5,000 points in 2012. Now it’s just 3,500. I haven’t flown in United Federal Customer Service since 2012 because of COVID. What happened? May I have your name and account number, ma’am? Maria Carlotta, Maria Carlotta." ``` ```json Synthetic Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Hi! This is United Federal Customer Service. How may I be of assistance to you? Hello! I’d like to complain about a discrepancy in my United Federal Customer Service Frequent Voyager points. I checked my account today. I had 5,000 points in 2012. Now it’s just 3,500. I haven’t flown in United Federal Customer Service since 2012 because of COVID. What happened? May I have your name and account number, ma’am? Maria Carlotta, Maria Carlotta.", "entities": [ { "processed_text": "United Federal Customer Service", "text": "Icarus Airways Customer Service", "location": { "stt_idx": 12, "end_idx": 43, "stt_idx_processed": 12, "end_idx_processed": 43 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8451 } }, { "processed_text": "United Federal Customer Service", "text": "Icarus", "location": { "stt_idx": 134, "end_idx": 140, "stt_idx_processed": 134, "end_idx_processed": 165 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.7969 } }, { "processed_text": "2012", "text": "2019", "location": { "stt_idx": 216, "end_idx": 220, "stt_idx_processed": 241, "end_idx_processed": 245 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9384 } }, { "processed_text": "United Federal Customer Service", "text": "Icarus", "location": { "stt_idx": 262, "end_idx": 268, "stt_idx_processed": 287, "end_idx_processed": 318 }, "best_label": "ORGANIZATION", "labels": { "ORGANIZATION": 0.8285 } }, { "processed_text": "2012", "text": "2019", "location": { "stt_idx": 275, "end_idx": 279, "stt_idx_processed": 325, "end_idx_processed": 329 }, "best_label": "DATE_INTERVAL", "labels": { "DATE_INTERVAL": 0.9393 } }, { "processed_text": "COVID", "text": "COVID", "location": { "stt_idx": 291, "end_idx": 296, "stt_idx_processed": 341, "end_idx_processed": 346 }, "best_label": "CONDITION", "labels": { "CONDITION": 0.9327 } }, { "processed_text": "Maria Carlotta", "text": "Nessa Jonsson", "location": { "stt_idx": 361, "end_idx": 374, "stt_idx_processed": 411, "end_idx_processed": 425 }, "best_label": "NAME", "labels": { "NAME": 0.903, "NAME_GIVEN": 0.3583, "NAME_FAMILY": 0.5411 } }, { "processed_text": "Maria Carlotta", "text": "N-E-S-S-A J-O-N-S-S-O-N", "location": { "stt_idx": 376, "end_idx": 399, "stt_idx_processed": 427, "end_idx_processed": 441 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.3708, "NAME": 0.907, "NAME_FAMILY": 0.5271 } } ], "entities_present": true, "characters_processed": 400, "languages_detected": { "en": 0.9167966246604919 } } ] ``` Note how the PII has been replaced with similar looking fake entities. Also you should know that each synthetic data request may have a different response as the synthetic data generation is **non-deterministic**. You can optionally configure the language in which the text is generated using the `synthetic_entity_accuracy` field. For English generation, set this parameter to `standard` for best results. For other languages, set it to `standard_multilingual` and the synthetic model will attempt to predict entities matching the input text language. The default accuracy is `standard_automatic` which will determine the appropriate model (*i.e.*, `standard` or `standard_multilingual`) from the input language. ```json Request Body wrap lines highlight={7} theme={"theme":"poimandres"} { "text": [ "Publié le 03/01/2017 de la baie de Vaitupa, Polynésie française, GPS 17 34.06 S 149 37.1 W\n Nous nous sommes probablement rencontrés chez Yan Labrosse … il ya longtemps. Je suis ton periple avec … beaucoup d’envie!! Yan m’a dit que tu comptais rejoindre l’indonésie." ], "processed_text": { "type": "SYNTHETIC", "synthetic_entity_accuracy": "standard_multilingual" } } ``` The response show how the entities were replaced with French locations and country. ```text Multilingual Synthetic (text only) wrap theme={"theme":"poimandres"} "Publié le 31/08/2014 de la gare de Memphis, Tennessee américain, GPS 20 9sn.10 N ec 15.5 S\n Nous nous sommes probablement rencontrés chez Max Fontaine … il ya longtemps. Je suis ton periple avec … beaucoup d’envie!! Ben m’a dit que tu comptais rejoindre l’Argentine." ``` ```json Multilingual Synthetic (full response) wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Publié le 31/08/2014 de la gare de Memphis, Tennessee américain, GPS 20 9sn.10 N ec 15.5 S\n Nous nous sommes probablement rencontrés chez Max Fontaine … il ya longtemps. Je suis ton periple avec … beaucoup d’envie!! Ben m’a dit que tu comptais rejoindre l’Argentine.", "entities": [ { "processed_text": "31/08/2014", "text": "03/01/2017", "location": { "stt_idx": 10, "end_idx": 20, "stt_idx_processed": 10, "end_idx_processed": 20 }, "best_label": "DATE", "labels": { "DATE": 0.9961 } }, { "processed_text": "gare de Memphis, Tennessee américain", "text": "baie de Vaitupa, Polynésie française", "location": { "stt_idx": 27, "end_idx": 63, "stt_idx_processed": 27, "end_idx_processed": 63 }, "best_label": "LOCATION", "labels": { "LOCATION": 0.8305, "LOCATION_CITY": 0.0939, "LOCATION_STATE": 0.0878, "ORIGIN": 0.0745 } }, { "processed_text": "20 9sn.10 N ec 15.5 S", "text": "17 34.06 S 149 37.1 W", "location": { "stt_idx": 69, "end_idx": 90, "stt_idx_processed": 69, "end_idx_processed": 92 }, "best_label": "LOCATION_COORDINATE", "labels": { "LOCATION_COORDINATE": 0.989, "LOCATION": 0.9506 } }, { "processed_text": "Max Fontaine", "text": "Yan Labrosse", "location": { "stt_idx": 138, "end_idx": 150, "stt_idx_processed": 140, "end_idx_processed": 152 }, "best_label": "NAME", "labels": { "NAME": 0.9953, "NAME_GIVEN": 0.2486, "NAME_FAMILY": 0.7461 } }, { "processed_text": "Ben", "text": "Yan", "location": { "stt_idx": 216, "end_idx": 219, "stt_idx_processed": 218, "end_idx_processed": 221 }, "best_label": "NAME_GIVEN", "labels": { "NAME": 0.9941, "NAME_GIVEN": 0.9915 } }, { "processed_text": "l’Argentine", "text": "l’indonésie", "location": { "stt_idx": 254, "end_idx": 265, "stt_idx_processed": 256, "end_idx_processed": 267 }, "best_label": "LOCATION_COUNTRY", "labels": { "LOCATION": 0.9858, "LOCATION_COUNTRY": 0.9682 } } ], "entities_present": true, "characters_processed": 266, "languages_detected": { "fr": 0.9757143259048462 } } ] ``` See the [Process Text route documentation](/latest/process-text) for additional configuration options for synthetic data generation. ## Custom redaction using the NER Text route As we have seen above, the [Process Text route](/latest/process-text) offers a lot of flexibility in how text and files are redacted. In the event that you have a specific use case that is not completely covered by the API, it is possible to create your own custom redaction function. This section shows how the [NER Text route](/latest/ner-text) that was introduced in `3.9` can be used to create a custom redaction function with more "fine-grained" labels. ### Process Text route redaction Let's say that you want to redact this fragment of text: ```text Text Example wrap theme={"theme":"poimandres"} "ERIC G. BADORREK was born in 1960 and registered to vote on 10 February 2012, giving the address 35933 COLLINS LN, FENWICK WEST, SELBYVILLE, Sussex County, Delaware, U.S.A. BADORREK is registered to vote in the Republican Party. Voter ID number: 100917654" ``` Using the [Process Text route](/latest/process-text), the redacted content will look like: ```text Redacted Text wrap theme={"theme":"poimandres"} "[NAME] was born in [DOB] and registered to vote on [DATE], giving the address [LOCATION_ADDRESS]. [NAME_FAMILY] is registered to vote in the [ORGANIZATION]. Voter ID number: [ACCOUNT_NUMBER]" ``` Notice how all the parts of the name `ERIC G. BADORREK` including the first name, initial and last name were combined into a single `NAME` marker. This grouping of words into a single marker is even more apparent on the address `35933 COLLINS LN, FENWICK WEST, SELBYVILLE, Sussex County, Delaware, U.S.A.` which is redacted as a single `LOCATION_ADDRESS` label. This is certainly making the redacted contents more readable but it is hiding some information that may be useful for your use case. For example, you might want to know if the provided address was containing a zip code or a country which is impossible to determine from the current redacted output. ### Using the NER Text route to create your own redacted content Unlike the Process Text route, the [NER Text route](/latest/ner-text) does not provide a redacted output. However, the entities it returns can be used to create one. Let's see how. Consider this piece of code which is processing the same sample text but with the NER Text route this time. ```python Full Python Code Example wrap lines theme={"theme":"poimandres"} import requests from itertools import groupby text = "ERIC G. BADORREK was born in 1960 and registered to vote on 10 February 2012, giving the address 35933 COLLINS LN, FENWICK WEST, SELBYVILLE, Sussex County, Delaware, U.S.A. BADORREK is registered to vote in the Republican Party. Voter ID number: 100917654" request = { "text": [text] } # TODO - you should be updating this part to point to your local instance of Limina or to one of the Limina cloud API. resp = requests.post("http://localhost:8080/ner/text", json=request).json() # sort the entities so that entities with longest spans are first entities = sorted(resp[0]["entities"], key=lambda e: (e["location"]["stt_idx"], -e["location"]["end_idx"], len(e["label"]))) class NotComparable(str): """Turns a string (_e.g._, a string literal like "e") which would otherwise compare equal to itself non-comparable""" def __init__(self, value: str): self.value = value def __eq__(self, other) -> bool: return False redacted_chunks = [NotComparable(c) for c in text] for entity in entities: start = entity["location"]["stt_idx"] end = entity["location"]["end_idx"] redacted_chunks[start:end] = [f"""[{entity["label"]}]"""] * (end - start) print("".join(key for key, _ in groupby(redacted_chunks))) ``` We first make a request to the NER Text route endpoint, passing the text to analyze. Then we extract and sort the entities from the response. ```python Python Code Example wrap lines theme={"theme":"poimandres"} # TODO - you should be updating this part to point to your local instance of Limina or to one of the Limina cloud API. resp = requests.post("http://localhost:8080/ner/text", json=request).json() # sort the entities so that entities with longest spans are first entities = sorted(resp[0]["entities"], key=lambda e: (e["location"]["stt_idx"], -e["location"]["end_idx"], len(e["label"]))) ``` We are going to use these entities to create a redacted text containing more details about the original text (*e.g.*, whether an address was containing a COUNTRY). Because we are interested in showing "fine-grained" entities (*i.e.*, the one with smaller spans) over "coarser" entities, we are sorting the overlapping entities from the longest to the shortest. The following code will construct the redacted text by iterating over the list of sorted entities. While doing so, it is easier to turn the input text in to a list of characters. This allows us to more easily replace the sensitive contents (*i.e.*, the characters covered by an entity span) with a redaction marker. The following code is converting the input text to a list of characters and then replace each character that is part of an entity with the entity label. A small utility, `NotComparable`, is created to ensure that identical strings are not comparable (*i.e.*, `NotComparable("e") != NotComparable("e")`). This will be useful when outputting the redacted text. ```python Python Code Example wrap lines theme={"theme":"poimandres"} class NotComparable(str): """Turns a string (_e.g._, a string literal like "e") which would otherwise compare equal to itself non-comparable""" def __init__(self, value: str): self.value = value redacted_chunks = [NotComparable(c) for c in text] for entity in entities: start = entity["location"]["stt_idx"] end = entity["location"]["end_idx"] redacted_chunks[start:end] = [f"""[{entity["label"]}]"""] * (end - start) ``` The last step is simply to join all the characters of the original text and the redaction markers into a redacted contents. Since the markers are repeated for each entity characters that were replaced, we use the `groupby` function to only output it once. This is where the `NotComparable` utility plays its role by preventing consecutive identical characters (*e.g.*, the two `R` in `BADORREK`) to be grouped together. ```python Python Code Example wrap theme={"theme":"poimandres"} print("".join(key for key, _ in groupby(redacted_chunks))) ``` The result is a redacted text with all the necessary details. ```text Redacted Text wrap theme={"theme":"poimandres"} "[NAME_GIVEN][NAME][NAME_FAMILY] was born in [DOB] and registered to vote on [DATE], giving the address [LOCATION_ADDRESS_STREET][LOCATION_ADDRESS][LOCATION_CITY][LOCATION_ADDRESS][LOCATION_STATE][LOCATION_ADDRESS][LOCATION_COUNTRY]. [NAME_FAMILY] is registered to vote in the [POLITICAL_AFFILIATION]. Voter ID number: [ACCOUNT_NUMBER]" ``` See how the name `ERIC G. BADORREK` has been replace with `[NAME_GIVEN][NAME][NAME_FAMILY]` instead of a single `NAME` marker and how the address `35933 COLLINS LN, FENWICK WEST, SELBYVILLE, Sussex County, Delaware, U.S.A.` was redacted with much more details `[LOCATION_ADDRESS_STREET][LOCATION_ADDRESS][LOCATION_CITY][LOCATION_ADDRESS][LOCATION_STATE][LOCATION_ADDRESS][LOCATION_COUNTRY]`. From the above redacted text, it becomes clear that the original address contained a city, a state and a country but no zip code. **A parting note about privacy** You may wonder if the redacted results achieved in this section could have been obtained in a simpler way by disabling the `NAME`, `LOCATION` and `LOCATION_ADDRESS` entity types when making the request. While disabling entity types has its use, the technique described above has the advantage of lowering the chances of leaking sensitive data. Consider for example the words `Sussex County` part of the provided address. These words are part of the `LOCATION_ADDRESS` but not part of any other sub-entities. As a result, these words would be left unredacted if both `LOCATION` and `LOCATION_ADDRESS` were disabled. This is applicable to many other entities like *Mount Everest* which is a `LOCATION` but does not match any other location sub-entities. By disabling the `LOCATION` label, we let these entities unredacted. This might not be desirable for some use cases. # Gibberish Detection Source: https://docs.getlimina.ai/configuration-and-operations/entity-detection-and-redaction/gibberish-detection Score how nonsensical the text around each detected entity is, to help surface false positives Beta Gibberish detection scores how nonsensical the text around each detected entity is, which helps you detect false positives. Noisy input, such as text extracted from low-quality scans via OCR, can cause entities to be detected where there is none. When you enable gibberish detection, every detected entity is returned with a `gibberish_score` between `0` and `1`. A higher score suggests that the surrounding text is more characteristic of random character sequences or OCR-generated noise than coherent language. Consequently, entities with higher gibberish scores are more likely to be false positives. ## Why gibberish detection Transformer-based NER models are typically trained on well-formed natural language, where they learn to identify entities using both the words themselves and their surrounding linguistic context. When the input contains gibberish, OCR artifacts, or random character sequences, these contextual cues become unreliable or disappear entirely. As a result, the model is more prone to producing spurious entity predictions or missing true entities, leading to reduced precision and overall performance. Gibberish detection provides an additional quality signal that helps distinguish reliable entity predictions from those made on degraded or nonsensical text. ## Enabling gibberish detection (new in 4.4) Set `enable_gibberish_detection` to `true` at the top level of a request to the [`ner/text`](/latest/ner-text) or [`ner/files`](/latest/ner-files-uri) endpoint: ```json Request Body wrap lines highlight={8} theme={"theme":"poimandres"} { "text": [ "The contract was signed by Laura Bennett on June 3rd.", "Attn: Krell Vontanix, Dept of Aurellian Affiars", "Recipt frm company Nothern Suplly Co, it cnts ref 4471xz asd", "Cotpan lnvol'ce N0 4rZ9 Rrnlk zq## Xy9kp" ], "enable_gibberish_detection": true } ``` ## Reading the score Each entity in the response contains a `gibberish_score` field. The `gibberish_score` field is only present when `enable_gibberish_detection` is set to `true`. When the option is off (the default), the field is omitted entirely. ```json Response wrap lines highlight={13, 24, 35, 46, 66, 77, 88, 99, 119, 130, 150, 161, 172, 183, 194, 205} theme={"theme":"poimandres"} [ { "entities": [ { "text": "Laura", "location": { "stt_idx": 27, "end_idx": 32 }, "label": "NAME_GIVEN", "likelihood": 0.9249095916748047, "validation_results": [], "gibberish_score": 0.0001703500747680664 }, { "text": "Laura Bennett", "location": { "stt_idx": 27, "end_idx": 40 }, "label": "NAME", "likelihood": 0.9158932566642761, "validation_results": [], "gibberish_score": 0.0001703500747680664 }, { "text": "Bennett", "location": { "stt_idx": 33, "end_idx": 40 }, "label": "NAME_FAMILY", "likelihood": 0.9399955868721008, "validation_results": [], "gibberish_score": 0.03785803640799792 }, { "text": "June 3rd", "location": { "stt_idx": 44, "end_idx": 52 }, "label": "DATE", "likelihood": 0.9454799294471741, "validation_results": [], "gibberish_score": 0.1005928173080051 } ], "entities_present": true, "characters_processed": 53, "languages_detected": { "en": 0.9993266463279724 } }, { "entities": [ { "text": "Krell", "location": { "stt_idx": 6, "end_idx": 11 }, "label": "NAME_GIVEN", "likelihood": 0.9069240093231201, "validation_results": [], "gibberish_score": 0.3707108026617384 }, { "text": "Krell Vontanix", "location": { "stt_idx": 6, "end_idx": 20 }, "label": "NAME", "likelihood": 0.906118224064509, "validation_results": [], "gibberish_score": 0.40363886910682156 }, { "text": "Vontanix", "location": { "stt_idx": 12, "end_idx": 20 }, "label": "NAME_FAMILY", "likelihood": 0.9207147657871246, "validation_results": [], "gibberish_score": 0.40363886910682156 }, { "text": "Dept of Aurellian Affiars", "location": { "stt_idx": 22, "end_idx": 47 }, "label": "ORGANIZATION", "likelihood": 0.9109159633517265, "validation_results": [], "gibberish_score": 0.40363886910682156 } ], "entities_present": true, "characters_processed": 47, "languages_detected": { "en": 0.5085869431495667 } }, { "entities": [ { "text": "Nothern Suplly Co", "location": { "stt_idx": 19, "end_idx": 36 }, "label": "ORGANIZATION", "likelihood": 0.8835725784301758, "validation_results": [], "gibberish_score": 0.38497073599686615 }, { "text": "4471xz asd", "location": { "stt_idx": 50, "end_idx": 60 }, "label": "NUMERICAL_PII", "likelihood": 0.8458981911341349, "validation_results": [], "gibberish_score": 0.6274604711816052 } ], "entities_present": true, "characters_processed": 60, "languages_detected": { "en": 0.49856317043304443 } }, { "entities": [ { "text": "N0 4rZ9", "location": { "stt_idx": 16, "end_idx": 23 }, "label": "LOCATION", "likelihood": 0.25443747888008755, "validation_results": [], "gibberish_score": 0.9964572315753586 }, { "text": "N0 4rZ9", "location": { "stt_idx": 16, "end_idx": 23 }, "label": "LOCATION_ZIP", "likelihood": 0.30771060287952423, "validation_results": [], "gibberish_score": 0.9964572315753586 }, { "text": "0 4rZ9", "location": { "stt_idx": 17, "end_idx": 23 }, "label": "VEHICLE_ID", "likelihood": 0.18630204796791078, "validation_results": [], "gibberish_score": 0.9964572315753586 }, { "text": "rZ9", "location": { "stt_idx": 20, "end_idx": 23 }, "label": "PASSWORD", "likelihood": 0.17115056018034616, "validation_results": [], "gibberish_score": 0.9964572315753586 }, { "text": "rnlk zq", "location": { "stt_idx": 25, "end_idx": 32 }, "label": "PASSWORD", "likelihood": 0.3922242820262909, "validation_results": [], "gibberish_score": 0.9964572315753586 }, { "text": "Xy9kp", "location": { "stt_idx": 35, "end_idx": 40 }, "label": "PASSWORD", "likelihood": 0.6181477636098862, "validation_results": [], "gibberish_score": 1 } ], "entities_present": true, "characters_processed": 40, "languages_detected": {} } ] ``` This example demonstates that the higher the score, the stronger the false-positive candidate: 1. Entities in the first, clean sentence have scores close to `0`. 2. The second is a plausible sentence, but includes an invented name and a typo. The entities have `gibberish_score` of around `0.40`. 3. In the third sentence, which has several misspellings, the first entity is surrounded with cleaner text and scores lower (below `0.40`). The second entity has a score higher than `0.60`, because its context is somewhat nonsensical text. 4. The entities in the last sentence, representing a random OCR output, have scores close or equal to `1.0`. The score should be interpreted as a signal rather than a verdict: * A value near `0` means the entity is within a coherent, meaningful text. * A value near `1` means the surrounding text is likely gibberish. * Different thresholds may be selected depending on the data. A higher cut-off value is stricter about what it flags as gibberish. Some entity types are not natural language. Values such as passwords, account and card numbers, API keys, license plates, and other high-entropy identifiers may score high even when they are correctly detected, simply because the value itself looks random. A high score should be treated as a reason to review an entity, instead of a proof that it is a false positive. ## Limitations Text detected as Chinese, Japanese, or Korean is currently not evaluated and returns a `gibberish_score` of `0.0`, as the scoring signals are unreliable for these languages. # Language Detection Source: https://docs.getlimina.ai/configuration-and-operations/entity-detection-and-redaction/language-detection Limina language detection capabilities and implications Limina runs a language detection model on every de-identification request and provides the detected languages as part of the response. It is important to note that the de-identification service model may be selected independently of the language detection results. Refer to the [`accuracy` parameter](/latest/process-text#body-entity-detection-accuracy) description for details on how to force a specific model for de-identification and to learn how to enable automatic model selection. Limina includes language detection capabilities. Here is an example of what you'll see when you use the `process/text` endpoint: ```json Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "... ...", "entities": [... ...], "entities_present": true, "characters_processed": x, "languages_detected": { "en": 0.7678971290588379, "fr": 0.5934884324888 } } ] ``` Limina uses the [fastText language identification model](https://fasttext.cc/docs/en/language-identification.html) to detect and identify languages which is independent of the Limina PII model. The list of supported languages for fastText [can be found here](https://fasttext.cc/docs/en/language-identification.html#list-of-supported-languages). Language detection is **always** run as part of `text` and `file` processing. It is important to note that detection of a language does not mean that the appropriate PII model (English or Multilingual) was used to process the payload or that support for the language is available for PII detection. For example, sending a request such as: ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "Ich würde gerne mit John sprechen. Er wohnt in der Neuhauser Straße" ], "entity_detection": { "accuracy": "high" } } ``` will always use the English-only model to de-identify the input text and, while the response MAY contain entities, the multilingual model will have better performance on languages other than English and should be preferred. Here is the response you receive from this payload: ```json Response wrap lines theme={"theme":"poimandres"} [ { "processed_text": "Ich würde gerne mit [NAME_GIVEN_1] sprechen. Er wohnt in der [LOCATION_ADDRESS_STREET_1]", "entities": [ { "processed_text": "NAME_GIVEN_1", "text": "John", "location": { "stt_idx": 20, "end_idx": 24, "stt_idx_processed": 20, "end_idx_processed": 34 }, "best_label": "NAME_GIVEN", "labels": { "NAME": 0.9956, "NAME_GIVEN": 0.4324 } }, { "processed_text": "LOCATION_ADDRESS_STREET_1", "text": "Neuhauser Straße", "location": { "stt_idx": 51, "end_idx": 67, "stt_idx_processed": 61, "end_idx_processed": 88 }, "best_label": "LOCATION_ADDRESS_STREET", "labels": { "LOCATION": 0.9966, "LOCATION_ADDRESS_STREET": 0.3716 } } ], "entities_present": true, "characters_processed": 67, "languages_detected": { "de": 0.999757707118988 } } ] ``` Although entities are detected and the language was identified as `de` (German), the English PII model `high` was used to process the text. # Configuration & Operations Source: https://docs.getlimina.ai/configuration-and-operations/index Limina's detailed configuration instructions including prerequisites, requirements, deployment, benchmarks, concurrency, and more. The following sections describe usage and operations of the Limina container product. An expanded description of the core features along with guidance on how to utilize those features is provided. Reference material related to container management and overall platform operations is additionally provided. ## Entity Detection & Redaction ## Working with Files ## Advanced Features ## Container Management # Processing Audio Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/audio This guide will get you started with Audio deidentification. Limina supports scanning audio for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. ## How Audio Files Are Processed Audio files are processed as follows: 1. A transcript is produced using an Automatic Speech Recognition (ASR) system 2. The resulting ASR transcript is passed through the text-based PII detection engine 3. If specified by the user, the audio file undergoes pitch distortion 4. Using the ASR timestamps, any sections of the audio file corresponding with PII detections are replaced with a Sine wave or bleep tone 5. The resulting de-identified or redacted audio file and transcript are passed back to the user ## Parameters Please see the `audio_options` object in the [API reference](/latest/process-files-base64#body-audio-options) for a full list of the parameters for audio processing. ## Supported File Types | File Type | Extension | Content Type | Added In | | --------- | ----------- | ------------------------- | -------- | | wave | `.wav` | `audio/wav` | 3.0.0 | | x-wave | `.wav` | `audio/x-wav` | 3.5.0 | | mp3 | `.mp3` | `audio/mpeg`, `audio/mp3` | 3.0.0 | | mp4 | `.mp4` | `audio/mp4` | 3.0.0 | | m4a | `.m4a` | `audio/m4a` | 3.5.0 | | m4a-latm | `.m4a-latm` | `audio/m4a-latm` | 3.5.0 | | webm | `.webm` | `audio/webm` | 3.5.0 | ### VOX Files Note that .vox files are not natively supported by Limina, but can be processed by converting the .vox file to a wav or mp3 using a conversion tool like [SoX](https://sourceforge.net/projects/sox/). Because .vox files are headerless, you will need to know the sample rate and encoding to specify. For example, to take a vox file with a sample rate 8000, mono channel, mu-law encoded: ```shell SoX Command theme={"theme":"poimandres"} sox -t raw -r 8000 -c 1 -e mu-law myfile.vox myfile.wav ``` to generate a wav file. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "audio/wav" }, "entity_detection": { "return_entity": true }, "audio_options": { "bleep_start_padding": 0, "bleep_end_padding": 0 } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.wav)'", "content_type": "audio/wav"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.wav' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.wav" filename_out = "/path/to/output/sample.redacted.wav" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() headers = {"Content-Type": "application/json", "x-api-key": ""} url = "https://api.getlimina.ai/community/v4/process/files/base64" payload = { "file" : { "data": file_content_base64, "content_type": "audio/wav" }, "entity_detection": { "return_entity": True }, "audio_options":{ "bleep_start_padding": 0, "bleep_end_padding": 0 } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.wav" filename_out = "sample.redacted.wav" file_type= "audio/wav" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing CSV Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/csv This guide will get you started with CSV deidentification. Limina supports scanning Comma Separated Value (CSV) files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. ## How CSV Files Are Processed Similar to Excel files, CSV files are processed using the method described for Tabular Data in the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data). The output file retains its original format with labels in place of the detected PII. ## Constraints Please consider writing a handler for your specific application using the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data) to get around any of the constraints listed below. * The file processing routes are synchronous, meaning that large files over 10MB in size may take a long time to process. * The data in the CSV file must be row-oriented (i.e. each row represents a separate record) and the headers must be on the first row. * Files must adhere to the [csv file standards](https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml). ## Additional Options CSV file processing is additionally supported by a specialized pre-processing parameter called `csv_options`. This parameter enables a sampling mode that provides efficiencies in overall file processing performance. The sampling feature looks at a selection of cells within each column of the file, predicts the entity type, and redacts the entirety of the column using those detected entity types. This can have a drastic improvement in performance when working with very large CSV files. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 250 KiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "text/csv" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.csv)'", "content_type": "text/csv"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.csv' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.csv" filename_out = "/path/to/output/sample.redacted.csv" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() headers = {"Content-Type": "application/json", "x-api-key": ""} url = "https://api.getlimina.ai/community/v4/process/files/base64" payload = { "file":{ "data": file_content_base64, "content_type": "text/csv", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.csv" filename_out = "sample.redacted.csv" file_type= "text/csv" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Customizing Object Detection Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/customizing-object-detection This guide presents advanced techniques to help customize object detection. New in 4.0.1 [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. Limina introduces the ability to enable or disable specific object entity types during file processing. This feature provides enhanced control over the types of objects detected and redacted in your files, allowing for more precise customization of redaction processes. The techniques described here apply to both file processing routes of the Limina APIs—the [File URI route](/latest/process-files-uri/) and the [File Base64 route](/latest/process-files-base64/). This customizable object detection feature works seamlessly across all [supported file types](/configuration-and-operations/working-with-files/supported-file-types) where object detection can be performed, including images, PDFs, and Office documents. ## Enabling and Disabling Object Entity Types In file processing, Limina supports the detection of several object types including: license plates, logos, signatures and faces. See [object entity types](/entities/supported-object-entities) for a complete list. By default, all supported types are detected, but this can be easily customized using object entity selectors. ### Configuring Object Entity Selectors Object entity selectors allow you to *enable* or *disable* specific entity types as part of your API request. For example, you can enable detection of `LOGO`, `LICENSE_PLATE` and `SIGNATURE` while ignoring all other entity types by using the `ENABLE` selector as shown in this request: For descriptions of the request fields, refer to the [REST API documentation](/latest/process-files-uri). ```json Request Body wrap lines highlight={6-13} theme={"theme":"poimandres"} { "uri": "path/to/file.png", "entity_detection": { "return_entity": true }, "object_entity_detection": { "object_entity_types": [ { "type" : "ENABLE", "value" : ["FACE", "LICENSE_PLATE", "LOGO"] } ] } } ``` ```json Sample Response wrap lines theme={"theme":"poimandres"} { "result_uri": "path/to/file.redacted.png", "processed_text": "[OCCUPATION_1] Documentation CneQ X(Signature)", "entities": [ { "processed_text": "OCCUPATION_1", "text": "Developer", "location": { "page": 0, "x0": 0.26653, "x1": 0.32143, "y0": 0.3122, "y1": 0.33201 }, "best_label": "OCCUPATION", "labels": { "OCCUPATION": 0.8031 } } ], "objects": [ { "type": "FACE", "location": { "page": 0, "x0": 0.68737, "x1": 0.77033, "y0": 0.1996, "y1": 0.33782 } }, { "type": "LICENSE_PLATE", "location": { "page": 0, "x0": 0.37761, "x1": 0.46474, "y0": 0.74885, "y1": 0.77917 } }, { "type": "LICENSE_PLATE", "location": { "page": 0, "x0": 0.10116, "x1": 0.12824, "y0": 0.66658, "y1": 0.67642 } }, { "type": "LOGO", "location": { "page": 0, "x0": 0.20612, "x1": 0.46193, "y0": 0.20155, "y1": 0.27935 } } ], "entities_present": true, "objects_present": true, "languages_detected": { "en": 0.5266358852386475 }, "audio_duration": null, "page_count": null } ``` As expected, this configuration redacts faces, logos, and license plates while leaving other entities (`SIGNATURE` in this case) untouched in the output file. Input Image Output Image It is sometimes more convenient to specify the object entity types you wish to disable. This can be done using a `DISABLE` selector: ```json Request Body wrap lines highlight={9} theme={"theme":"poimandres"} { "uri": "path/to/file.png", "entity_detection": { "return_entity": true }, "object_entity_detection": { "object_entity_types": [ { "type" : "DISABLE", "value" : ["LICENSE_PLATE"] } ] } } ``` ```json Sample Response wrap lines theme={"theme":"poimandres"} { "result_uri": "path/to/file.redacted.png", "processed_text": "[OCCUPATION_1] Documentation OE66OXC FB-692-NF", "entities": [ { "processed_text": "OCCUPATION_1", "text": "Developer", "location": { "page": 0, "x0": 0.26653, "x1": 0.32143, "y0": 0.313, "y1": 0.33201 }, "best_label": "OCCUPATION", "labels": { "OCCUPATION": 0.864 } } ], "objects": [ { "type": "FACE", "location": { "page": 0, "x0": 0.68737, "x1": 0.77033, "y0": 0.1996, "y1": 0.33782 } }, { "type": "LOGO", "location": { "page": 0, "x0": 0.20612, "x1": 0.46193, "y0": 0.20155, "y1": 0.27935 } }, { "type": "SIGNATURE", "location": { "page": 0, "x0": 0.59876, "x1": 0.90482, "y0": 0.78631, "y1": 0.87875 } } ], "entities_present": true, "objects_present": true, "languages_detected": {}, "audio_duration": null, "page_count": null } ``` The above request will redact all entity types except license plates, as shown below: Output Image ## Security Considerations It is important to note that **disabling object entities may increase the risk of sensitive information being leaked**. When selecting object entities for redaction, take extra time to consider all potential implications. We recommend expert validation to ensure that the redacted content is free of PII or other sensitive information. Following validation, adjust the list of object entities as needed based on the findings. **Object Detection and OCR interactions** In some cases, even when specific object detection types like `LICENSE_PLATE`, `LOGO`, or `SIGNATURE` are disabled, the associated text may still be redacted. This happens because the OCR identifies and processes the text within these objects, such as license numbers, readable text in logos, or legible parts of signatures. For example: * A license plate may be redacted because the license number is recognized as text. * A logo with a company name may be redacted due to the company name being recognized as text. * A handwritten signature may be partially or fully redacted if its characters are interpreted as text. This highlights the interaction between object detection and text redaction, where OCR-based text redaction can occur independently of object detection settings. # Processing DCM/DICOM Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/dcm-dicom This guide will get you started with DICOM deidentification. Beta Limina supports scanning Digital Imaging and Communications in Medicine (DCM or DICOM) files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. ## How DCM/DICOM Files Are Processed DICOM files are processed as follows: 1. The DICOM file's contents which include images, metadata and other elements as described in the [DICOM file standard](https://www.dicomstandard.org/) are extracted. 2. Text components of the DICOM file, including supported properties and metadata are processed by Limina's text module, except for the fields listed below in constraints that are assumed to never contain PII. More information about sensitive fields in [DICOM files can be accessed here](https://dicom.nema.org/dicom/2013/output/chtml/part15/chapter_E.html). 3. Any image components of the DICOM file are processed by Limina's [image](/configuration-and-operations/working-with-files/processing-files/image) module. This process finds and masks any sensitive text, such as scan date and operator name. The scan itself, such as an MRI isn't altered. 4. A new DICOM file is constructed using the de-identified or redacted equivalents created in the steps above. Graphical content where text is present will be OCRed and then redacted. You can configure the OCR System by setting it as an [Environment Variable](/configuration-and-operations/container-management/environment-variables) or sending it in the request object. Check out our [OCR Guide](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to further understand the OCR modes and their usage. ## Constraints * Limina only supports DICOM files with 2D images, DICOM files containing 3D images or videos are not currently supported. Please split 3D images up into a series of 2D images for processing. * The following metadata and parameters are assumed to never contain PII and are always passed through the output DICOM file: ```text Metadata & Parameters theme={"theme":"poimandres"} "AcquisitionNumber", "BitsAllocated", "BitsStored", "Columns", "ConvolutionKernel", "DataCollectionCenter", "DataCollectionCenterPatient", "DataCollectionDiameter", "DetectorBinning", "DistanceSourceToDirector", "DistanceSourceToPatient", "DistanceSourceToDetector", "Exposure", "ExposureTime", "FieldOfViewDimensions", "FieldOfViewShape", "FieldOfViewOrigin", "FilterType", "FrameOfReferenceUID", "FocalSpots", "GantryDetectorTilt", "GeneratorPower", "Grid", "HighBit", "ImageOrientationPatient", "ImagePositionPatient", "ImageType", "ImagerPixelSpacing", "InstanceCreationDate", "InstanceCreationTime", "InstanceNumber", "IrradiationEventUID", "KVP", "LargestImagePixelValue", "Modality", "NumberOfPhaseEncodingSteps", "NumberOfStudyRelatedInstances", "PhotometricInterpretation", "PixelData", "PixelPaddingValue", "PixelRepresentation", "PixelSpacing", "PositionReferenceIndicator", "ReconstructionDiameter", "ReconstructionTargetCenterPatient", "RescaleIntercept", "RescaleSlope", "RotationDirection", "Rows", "SOPClassUID", "SOPInstanceUID", "SamplesPerPixel", "ScanOptions", "SeriesInstanceUID", "SeriesNumber", "SliceLocation", "SliceThickness", "SmallestImagePixelValue", "SoftwareVersions", "SpatialResolution", "SpecificCharacterSet", "StudyInstanceUID", "TableHeight", "WindowCenter", "WindowWidth", "XRayTubeCurrent", "XRayTubeCurrentInuA" ``` ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/dicom" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.dcm)'", "content_type": "application/dicom"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.dcm' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.dcm" filename_out = "/path/to/output/sample.redacted.dcm" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode("ascii") url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/dicom", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.dcm" filename_out = "sample.redacted.dcm" file_type= "application/dicom" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing Excel (XLS/XLSX) Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/excel This guide will get you started with XLSX deidentification. Limina supports scanning Microsoft Excel XLS and XLSX files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. ## How XLSX Files Are Processed Similar to CSV files, cell contents of XLSX files are processed using the method described for Tabular Data in the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data). In addition to cell contents, the following elements are handled: | Property Type | Details | Behaviour | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | Core properties | Author, Category, Comments, Content Status, Identifier, Keywords, Language, Last Modified By, Subject, Title, Version | Redact | | Headers and footers | Any content in headers and footers, such as text and images. Can appear when the document is printed | Passthrough, will change to Redact in a future release | | Images | The [Images](/configuration-and-operations/working-with-files/processing-files/image) page provides a more detailed look at Image processing | Redact, unsupported image types are removed | | Text boxes | Floating text boxes | Passthrough, will change to Remove in a future release | | Embedded links | Hyperlinks to internet pages or documents | Remove | | External elements | Tables and charts embedded from another document or file, such as an Excel chart or table object | Passthrough, please process these separately | | Embedded audio & video | Videos and audio clips | Remove | | Review comments | Comments from document reviews | Passthrough, will change to Remove in a future release | | Shape objects | Shapes containing text | Passthrough, will change to Redact in a future release | Graphical content where text is present will be OCRed and then redacted. You can configure the OCR System by setting it as an [Environment Variable](/configuration-and-operations/container-management/environment-variables) or sending it in the request object. Check out our [OCR Guide](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to further understand the OCR modes and their usage. ## How XLS Files Are Processed XLS files are processed by converting into XLSX files, followed the process described above and then converting back to XLS files. ## Constraints * Cell contents of XLSX files are processed using the method described for Tabular Data in the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data). This requires the data to be column-oriented and the headers to be on the first non-empty row. * Shape objects will not be preserved. * Formulas may not be preserved after redaction. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.xlsx)'", "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.xlsx' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.xlsx" filename_out = "/path/to/output/sample.redacted.xlsx" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.xlsx" filename_out = "sample.redacted.xlsx" file_type= "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing Image Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/image This guide will get you started with Image deidentification. Limina supports scanning images for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. If you'd like to try it yourself, please [sign up for an account](https://portal.getlimina.ai/) to get a free API key. ## How Images Are Processed Image files are processed as follows: 1. Non-text PII such as faces and license plates are detected. 2. Additionally the image is run through an OCR system to detect any text present. 3. The output of the OCR system is then passed to a PII detection module. 4. Any PII found in the previous steps is de-identified via blurring or black box redaction. You can configure the OCR System by setting it as an [Environment Variable](/configuration-and-operations/container-management/environment-variables) or sending it in the request object. Check out our [OCR Guide](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to further understand the OCR modes and their usage. ## Parameters Below are the parameters that control the behaviour of the Image De-identifier. These parameters shall be specified under `image_options`. | Parameter | Explanation | Default | | ---------------- | ------------------------------------------------------------------- | ------- | | `masking_method` | This parameter sets how the entities in an image shall be redacted. | "blur" | ## Image Dimensions * In versions \<4.4.0, images with up to 4000 pixels in a given dimension are supported. * In versions 4.4.0 and up, images with up to 10000 pixels in a given dimension are supported, with a maximum total pixel limit of 16,000,000. ## Constraints * There is limited support for 8-bit PNG images. * Multi-page TIFF images are only supported on versions >=4.2.1. Earlier versions will process and return the first page of a multi-page TIFF. ## Supported File Types | File Type | Extension | Content Type | Added In | | ---------- | --------------- | ----------------------------- | -------- | | JPEG image | `.jpg`, `.jpeg` | `image/jpg`, `image/jpeg` | 3.0.0 | | TIFF image | `.tif`, `.tiff` | `image/tif`, `image/tiff` | 3.0.0 | | PNG image | `.png` | `image/png` | 3.4.0 | | BMP image | `.bmp` | `image/bmp`, `image/x-ms-bmp` | 3.4.0 | | GIF image | `.gif` | `image/gif` | 4.2.1 | ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "image/jpeg" }, "entity_detection": { "return_entity": true } } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.jpeg)'", "content_type": "image/jpeg"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.jpeg' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.jpeg" filename_out = "/path/to/output/sample.redacted.jpeg" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "image/jpeg", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.jpeg" filename_out = "sample.redacted.jpeg" file_type= "image/jpeg" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/index This guide will get you started with file deidentification. [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. Limina supports scanning a multitude of [different file types](/configuration-and-operations/working-with-files/supported-file-types) for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different PII (Personally Identifiable Information) entities, PHI (Protected Health Information) entities, and PCI (Payment Card Industry) entities being detected. Our Supported Languages and Supported Entity Types page provides a more detailed look. ## How Does It Work? Limina support for file processing comes in unified endpoints which works with either base64-encoded files or URIs: `/process/files/base64` and `/process/files/uri`. ### Base64 The `base64` endpoint is the recommended way to process files for most users, as there is no need to mount a volume into the container and ensure that permissions are set correctly. To use the `base64` endpoint, you first need to read the file in memory, encode its content with base64, and send it to the `base64` endpoint. You also need to pass the MIME type of the file as a hint to the file processing pipeline. The [Supported File Types](/configuration-and-operations/working-with-files/supported-file-types) page details extensions and MIME types for both endpoints ### URI Available on the container only, the `uri` endpoint is suitable for larger data volumes and has the following advantages over the base64 endpoint: * No overhead of base64 encoding. * No need to first read the file in memory. * The processed file is saved automatically by the container. API calls are made by pointing to a file on a mounted drive. The redacted contents are automatically saved at a user-specified location with the `.redacted` suffix added to the original name. For example, the `uri` endpoint will access the file `/some/path/my-doc.pdf`, it will redact it and create a file `my-doc.redacted.pdf` with the redacted contents at the location specified by the user. When using the `uri` endpoint, the file extension is used to determine the file type. A mounted drive can be connected to remote object storage or NAS. For instructions on how to connect a mounted drive to S3 please follow the guide here: [Using remote storage](/configuration-and-operations/working-with-files/processing-files/mounted-storage) Passing a file with no extension or with the wrong extension to the `uri` endpoint may lead to unexpected behavior. ## Diving Deeper ### **Processing files with the `base64` endpoint** When using the `/process/files/base64` endpoint, there is no need to mount a folder into the container. ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -v :/app/license/license.json \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid: ``` The file is first read into memory to encode its contents then the encoded contents are passed to the file processing endpoint. On linux, this can be done with the `base64` shell command. Assuming you have the file `sample.pdf` saved in the current folder: ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "'$(base64 -w 0 sample.pdf)'", "content_type": "application/pdf" } } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} echo '{"file": {"data": "'$(base64 -w 0 sample.pdf)'", "content_type": "application/pdf"}}' \ | curl --request POST --url 'http://localhost:8080/process/files/base64' \ -H 'Content-Type: application/json' -d @- | jq -r .processed_file | base64 -d > 'sample.redacted.pdf' ``` ```python Python wrap lines theme={"theme":"poimandres"} import base64 import requests # Specify the input and output file paths filename_in = "sample.pdf" filename_out = "sample.redacted.pdf" # Read the file and do base64 encoding with open(filename_in, "rb") as f: b64_file_content = base64.b64encode(f.read()) b64_file_content = b64_file_content.decode("utf-8") # Make the request and load the results as JSON r = requests.post(url="http://localhost:8080/process/files/base64", json={"file": {"data": b64_file_content, "content_type": "application/pdf"}}) results = r.json() # Decode and write the file to disk with open(filename_out, "wb") as f: f.write(base64.b64decode(results["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 # Specify the input and output file paths filename_in = "sample.pdf" filename_out = "sample.redacted.pdf" file_type= "application/pdf" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key='') # Read from file with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") # Make the request file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) # Write to file with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` This command will redact the file contents and return the redacted document as a base64-encoded string. An example Python script showing how to process files with Limina's Python client using the base64 route can be found [here](https://github.com/privateai/deid-examples/blob/main/python/examples/process_file_base64.py). It is important that the proper MIME type is provided with the base64-encoded string. Failing to pass the proper MIME type may lead to unexpected behavior. Check out the [Supported File Types](/configuration-and-operations/working-with-files/supported-file-types) page for proper MIME types. Check out the API reference for more details on the [base64 endpoint](/latest/process-files-base64). ### **Processing files with the `uri` endpoint** To process files with the `/process/files/uri` endpoint you are required to mount a volume when starting the container. In addition, the service requires access to a folder where the redacted files will be stored. This is done with the `PAI_OUTPUT_FILE_DIR` environment variable. This variable must point to a folder that is mounted into the container as output folder. ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -v :/app/license/license.json \ -v : \ -v : \ -e PAI_OUTPUT_FILE_DIR= \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid: ``` This is an example of a command mounting a `files` folder in the `admin` home folder as input, `output` folder as output location. ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -v /home/admin/license.json:/app/license/license.json \ -v /home/admin/files:/home/admin/files \ -v /home/admin/output:/home/admin/output \ -e PAI_OUTPUT_FILE_DIR=/home/admin/output \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid:3.1.0-cpu ``` **Common Pitfall** Mounting a folder to an existing os or app folder in the container may lead to unexpected behavior. An example Python script showing how to process files with Limina's Python client using the URI route can be found [here](https://github.com/privateai/deid-examples/blob/main/python/examples/process_file_uri.py). Once the container is running with the above command, you can redact files with: ```json Request Body wrap lines theme={"theme":"poimandres"} { "uri": "/home/admin/files/sample.pdf" } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} echo '{"uri": "/home/admin/files/sample.pdf"}' \ | curl --request POST --url 'http://localhost:8080/process/files/uri' \ -H 'Content-Type: application/json' -d @- | jq -r .processed_file | base64 -d > 'sample.redacted.pdf' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests PATH_TO_PDF_FILE = "/home/admin/files/sample.pdf" response = requests.post( "http://localhost:8080/process/files/uri", json={ "uri": PATH_TO_PDF_FILE } ) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key='') filepath = "/home/admin/files/sample.pdf" req_obj = request_objects.file_uri_obj(uri=filepath) resp = client.process_files_uri(req_obj) response.raise_for_status() print(response.json()) ``` Upon successful completion, the above command will save the redacted file under `/home/admin/files/output/sample.redacted.pdf`. **A note on permissions** Files created by the container will have the owner and permissions of the user running the docker service. This is commonly found to be `root` in default installations. However, you can change the user running the container using the docker `--user` option. This command will run the same container with the current user. ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -v /home/admin/license.json:/app/license/license.json \ -e PAI_OUTPUT_FILE_DIR=/home/admin/output \ -v /home/admin/files:/home/admin/files \ -v /home/admin/output:/home/admin/output \ --user $(id -u):$(id -u) \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid:3.1.0-cpu ``` Check out the API reference for more details on the [uri endpoint](/latest/process-files-uri). # Processing JSON Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/json This guide will get you started with json deidentification. Limina supports scanning JSON files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. ## How JSONs Are Processed Similar to XML files, JSON files are processed using the method described in the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data). The output file retains its original format with redaction markers of the format "\[LABEL\_X]" in place of the detected PII. The file passed in should adhere to the [JSON standard](https://datatracker.ietf.org/doc/html/rfc7159) as defined. ## Constraints Please consider writing a handler for your specific application using the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data) to get around any of the constraints listed below. * The file processing routes are synchronous; large files over 5MB in size may take a long time to process. * Only values are redacted. Key names are assumed to never contain PII and are not redacted. * Entity detection numbering is consistent within individual values only, not across the entire file. * The JSON must be entirely human-readable content. Encoded content such as Base64 is not natively supported and may lead to inaccurate PII detection. Please process this content separately. * Due to the [way the data is formatted](/configuration-and-operations/working-with-files/structured-data), deeply hierarchical (>10 levels) JSON structures take proportionally longer to process. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 250 KiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/json" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.json)'", "content_type": "application/json"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.json' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.json" filename_out = "/path/to/output/sample.redacted.json" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/json", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.json" filename_out = "sample.redacted.json" file_type= "application/json" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Using remote storage Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/mounted-storage Using remote storage # Reading files from remote storage By mounting external storage systems to the Private AI container teams can take advantage of a wide array of different technologies to source data for input. This can enable simplification of the overall design for data pipelines and additionally allow teams to take advantage of the feature sets present in those underlying storage providers. ## S3 Object storage When working with s3 buckets it can be useful to mount the bucket and process files directly without the need to encode the file and transfer data over the network. The mini guide below demonstrates how to do this using the Amazon mountpoint application. There are two primary steps required to process files directly from object storage: 1. Configure the container / host machine for S3 access 2. Process files using the `/process/files/uri` endpoint ## Prerequisites and assumptions: The `/process/files/uri` endpoint assumes that all files are available in local storage in the running container. At time of writing there are no connectors or drivers deployed with the Private AI container that enable communication with any additional protocols. As such any storage volume that is mounted to docker is available via the `/process/files/uri` endpoint. For S3 object storage this mini-guide assumes that you have: * Active AWS account * An S3 bucket * A running Private AI instance * AWS CLI installed on the host * Appropriate software installed on the host to enable mounting an S3 bucket as local storage. [https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html) ## Step 1 - Setup S3 mount point Ensure you have connected your AWS account. Follow the documentation here to assist with authentication: [https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials) The format of the command to mount the S3 bucket is fairly straight forward: ```shell theme={"theme":"poimandres"} mount-s3 s3://MY_BUCKETNAME /PATH/TO/FOLDER/ON/HOST --allow-other --uid $(id -u) --gid $(id -g) ``` Take special note of the uid and gid, setting the appropriate user and groups for your configuration. The command above sets the logged in user/group to have permission to access the mount. ## Step 2 - Modify the docker run / container start parameters to mount the local folder The docker run command will now need to be modified to include a new volume linked to the mount point. This is done by utilizing the parameter -v on the command line. Example below: ```shell theme={"theme":"poimandres"} docker run --rm -v /home/some/local/folder/mounted-s3-input:/home/ubuntu/s3-mount-input \ -v /home/some/local/folder/mounted-s3-output:/home/ubuntu/mounted-s3-output \ -e PAI_OUTPUT_FILE_DIR=/home/ubuntu/mounted-s3-output \ -e PAI_FILE_SUPPORT_ENABLED=True -v //home/some/local/folder/license.json:/app/license/license.json \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid:4.2.2-cpu ``` With the steps above complete the container will start and requests can be issued using the following payload format: ```shell curl theme={"theme":"poimandres"} curl -i -X POST \ http://localhostname/process/files/uri \ -H 'Content-Type: application/json' \ -d '{ "uri": "/home/ubuntu/s3-mount-input/example.txt", "entity_detection": { "return_entity": true } }' ``` ## S3 File Support (New in v4.4.0) The DEID container can now process files stored directly in Amazon S3 without requiring any local file mounts or volume bindings. You specify input and output locations using `s3://` URIs, and the container handles downloading and uploading automatically. ### Required Environment Variables To use S3 URIs for input or output, you must set three environment variables: | Variable | Description | | --------------------------- | -------------------------- | | `PAI_AWS_ACCESS_KEY_ID` | Your AWS access key ID | | `PAI_AWS_SECRET_ACCESS_KEY` | Your AWS secret access key | | `PAI_AWS_DEFAULT_REGION` | The target S3 region | These three variables are required whenever an `s3://` URI is used for input or output. Note that `PAI_OUTPUT_FILE_DIR` is always needed for the `/process/files/uri` endpoint, but can be set to either a local path or an `s3://` URI — it is not S3-specific. ### Processing Files from S3 (Input) This is a new way to specify file paths for the `/process/files/uri` endpoint — instead of local mount paths, you can now pass `s3://` URIs directly. For details on the `/process/files/uri` endpoint itself, see [Processing Files with the URI Endpoint](/configuration-and-operations/working-with-files/processing-files/#processing-files-with-the-uri-endpoint). To process files from S3 buckets, you can use either the raw HTTP API or our Python client: ```python Python wrap lines theme={"theme":"poimandres"} import requests url = "http://localhost:8080/process/files/uri" # Input from S3 — output follows PAI_OUTPUT_FILE_DIR payload = { "uri": "s3:///path/to/document.pdf", "entity_detection": {"return_entity": True}, } headers = {"Content-Type": "application/json"} response = requests.post(url, json=payload, headers=headers) results = response.json() print(results) ``` This works for all [supported file types](/configuration-and-operations/working-with-files/supported-file-types). ### Writing Output to S3 (Output) Set `PAI_OUTPUT_FILE_DIR` to an `s3://` URI when starting the container. The redacted file will be written to that path with `.redacted` appended to the original filename (e.g., `s3://my-output-bucket/redacted/document.redacted.pdf`). #### Specifying S3 as Output Destination — Bucket Name Requirement The bucket name is **mandatory** and must be the first segment after `s3://`: ``` s3://[bucket-name]/[optional-path/] ^^^^^^^^^^^^ Required ``` A valid example: `s3://my-output-bucket/redacted/` An invalid example (missing bucket): `s3:///redacted/` ### Mixing Input from Local Mount + Output to S3 You can use a mounted directory for input while writing output to S3: ```python Python wrap lines theme={"theme":"poimandres"} import requests url = "http://localhost:8080/process/files/uri" payload = { "uri": "/mounted/path/document.pdf", "entity_detection": {"return_entity": True}, } headers = {"Content-Type": "application/json"} response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` The input is read from the local mount, and the output goes to `PAI_OUTPUT_FILE_DIR` (S3 in this case). **`Container startup example`** ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -v :/app/license/license.json \ -v : \ -e PAI_OUTPUT_FILE_DIR=s3://my-output-bucket/redacted/ \ -e PAI_AWS_ACCESS_KEY_ID= \ -e PAI_AWS_SECRET_ACCESS_KEY= \ -e PAI_AWS_DEFAULT_REGION=ap-northeast-1 \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid: ``` ### Mixing Input from S3 + Output to Local Mount You can also use an `s3://` URI for input while writing output to a locally mounted directory: ```python Python wrap lines theme={"theme":"poimandres"} import requests url = "http://localhost:8080/process/files/uri" payload = { "uri": "s3:///path/to/document.pdf", "entity_detection": {"return_entity": True}, } headers = {"Content-Type": "application/json"} response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` The input is fetched from S3, and the output is written to `PAI_OUTPUT_FILE_DIR` (a local mount in this case). **`Container startup example`** ```shell Docker Command wrap theme={"theme":"poimandres"} docker run --rm -v :/app/license/license.json \ -e PAI_OUTPUT_FILE_DIR= \ -e PAI_AWS_ACCESS_KEY_ID= \ -e PAI_AWS_SECRET_ACCESS_KEY= \ -e PAI_AWS_DEFAULT_REGION=ap-northeast-1 \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid: ``` ### How It Works When an `s3://` URI is provided as input, the container: 1. **Validates** that the S3 bucket exists and is accessible 2. Fetches the file for processing. Nothing is held permanently inside the container. ## Writing to S3 The container processes files synchronously. When writing to different S3 keys at the same time, each request uses its own client and target path, so there are no conflicts. If multiple requests write to the **same key**, the last one wins — this is standard S3 behavior. To avoid unexpected overwrites, use unique paths per user or session (e.g., include a user ID or timestamp). ## Unsupported Features ### Pre-signed URLs Pre-signed S3 URLs are not supported. # Guide for OCR Modes Available with Limina DEID Container Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/ocr-modes Explore the best OCR modes including Standard OCR, Azure Computer Vision OCR, Azure Document Intelligence, and Hybrid OCR compatible with Limina DEID container for efficient document processing and redaction. If you are using the Limina container, the OCR system is configured through environment variables. For detailed instructions, refer to the sections on [Environment Variables](/configuration-and-operations/container-management/environment-variables) and the [Guide for Integrating Limina DEID container with Azure OCR](/configuration-and-operations/advanced-features/azure-ocr). You can configure more than one OCR systems. Once configured, the OCR system can be selected either via environment variables or directly in the [REST request](/latest/process-files-uri/). This guide outlines the different Optical Character Recognition (OCR) modes available for processing documents and images. Each mode has its unique application, depending on the document's nature and the specific requirements for text extraction. Standard OCR Azure Computer Vision OCR Azure Document Intelligence Hybrid OCR ## Standard OCR | Attribute | Rating | | --------- | ------ | | Accuracy | Medium | | Speed | High | | Cost | Low | Standard OCR is built into the container and is the default OCR engine. It's designed for high-performance text recognition in images and digital documents, focusing on efficiency and accuracy for standard printed text. The control flow for Standard OCR mode is depicted in the following flow chart: ```mermaid theme={"theme":"poimandres"} graph TD subgraph OCR [ OCR ] G[Standard / Azure OCR] end A[Process File] --> B{Document Type?} B -->|PDF| C[Convert to Image]:::test B -->|Office Docs| D[Extract Image] B --> |Image| G C --> G D --> G G --> H[Text DEID] H --> I[Create Redacted Image] I --> J[Create Redacted File] ``` ### When to choose Standard OCR? * Most cost efficient OCR solution is needed. ## Azure Computer Vision OCR | Attribute | Rating | | --------- | ------ | | Accuracy | High | | Speed | High | | Cost | Medium | Azure Computer Vision OCR uses [Microsoft's Computer Vision API](https://azure.microsoft.com/en-us/products/ai-services/ai-vision) to analyze and extract text from a wide range of document types. It's particularly effective for complex documents, including those with handwriting, images, and various fonts, layouts, rotations, and Japanese characters. To use this OCR mode, set `PAI_OCR_SYSTEM=azure_computer_vision` as an environment variable at container startup or specify it directly in the request parameters. Check out our [Guide for Integrating Limina DEID container with Azure OCR](/configuration-and-operations/advanced-features/azure-ocr) for setup information. The control flow is similar to the Standard OCR mode. ### When to choose Azure Computer Vision OCR? * Handwriting support is needed. * Rotated document support is needed. * Enhanced language support is needed. * Wide range of supported languages, such as Japanese. See [Azure OCR Language support](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/language-support) for more details. ## Azure Document Intelligence | Attribute | Rating | | --------- | ------ | | Accuracy | High | | Speed | High | | Cost | High | Azure Document Intelligence uses [Microsoft's Document Intelligence API](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence). On top of what Azure Computer Vision OCR already offers, Document Intelligence provide advanced features for complex layouts like tables and forms. To use this OCR mode, set `PAI_OCR_SYSTEM=azure_doc_intelligence` as an environment variable at container startup or specify it directly in the request parameters. Check out our [Guide for Integrating Limina DEID container with Azure OCR](/configuration-and-operations/advanced-features/azure-ocr) for setup information. The control flow is similar to the Standard OCR mode. ### When to choose Azure Document Intelligence? * Need advanced features for complex layouts like tables and forms. * Handwriting support is needed. * Rotated document support is needed. * Enhanced language support is needed. * Wide range of supported languages. See [Azure OCR Language support](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/language-support/ocr?view=doc-intel-4.0.0\&tabs=read-print%2Clayout-print%2Cgeneral#layout) for more details. ## Hybrid OCR | Attribute | Rating | | --------- | ------ | | Accuracy | Medium | | Speed | High | | Cost | Medium | The Hybrid OCR mode combines Azure Computer Vision OCR and Standard OCR to optimize text extraction and the cost for PDF documents. This mode uses Azure Computer Vision OCR for pages containing graphics and switches to Standard OCR for other pages. The choice between OCR modes is made on a page by page basis. The setup of OCR in hybrid mode is the same as Azure Computer Vision OCR mode. Simply follow our [Guide for Integrating Limina DEID container with Azure OCR](/configuration-and-operations/advanced-features/azure-ocr) & use `PAI_OCR_SYSTEM=hybrid` either as an environment variable or specify it in the request parameters, instead of `PAI_OCR_SYSTEM=azure_computer_vision`. The control flow for Hybrid OCR mode is depicted in the following flow chart: ```mermaid theme={"theme":"poimandres"} graph TD subgraph OCR [ OCR ] F[Standard OCR Flow] G[Azure Computer Vision OCR Flow] end A[Process File] --> B{File is a PDF} B --> |Yes| C[Render Pages] B -->|No| F C --> E{Image present in the page} E -->|Yes| G E -->|No| F G --> H[Text DEID] F --> H H --> I[Create Redacted Image] I --> J[Create Redacted File] ``` ### When to choose Hybrid OCR? * Need a mix of performance and cost effectiveness. * Need to process mainly PDFs. For other document types the Hybrid OCR will work just like the Standard OCR. ## Amazon Textract | Attribute | Rating | | --------- | ------ | | Accuracy | High | | Speed | High | | Cost | High | [Amazon textract](https://aws.amazon.com/textract/) integration is offered for those customers that prefer the AWS ecosystem. To use this OCR mode, set `PAI_OCR_SYSTEM=aws_textract` as an environment variable at container startup or specify it directly in the request parameters. Additionally ensure that following environment variables are set: ```shell theme={"theme":"poimandres"} PAI_AWS_ACCESS_KEY_ID= PAI_AWS_SECRET_ACCESS_KEY= PAI_AWS_DEFAULT_REGION= ``` The control flow is similar to the Standard OCR mode. # Processing PDF Files, Standard versus Enhanced Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/pdf-comparison This guide shows the differences between PDF Processing Types Limina offers two ways of scanning PDF files for PII and creating de-identified or redacted copies. PDF redaction approach could be selected by changing the `approach` parameter under `pdf_options`. ## Standard PDF processing Standard PDF processing involves rendering each PDF page as an image, and processing them as images. Then a new PDF is created using the redacted/de-identified images. Standard PDF processing can include an invisible, de-identified text layer. Standard PDF processing will replace PII with a blurred/black box. More information about Standard PDF processing can be found under [Processing PDF Files (Standard)](/configuration-and-operations/working-with-files/processing-files/pdf-standard). ## Enhanced PDF Processing Enhanced PDF processing allows PDFs to be processed natively. When Enhanced PDF processing is selected, the resulting de-identified PDF will have in-line replacements instead of blurred/black boxes. More information about Enhanced PDF processing can be found under [Processing PDF Files (Enhanced)](/configuration-and-operations/working-with-files/processing-files/pdf-enhanced). The `approach` parameter under `pdf_options` can be set as: * `standard` (Standard, default) * `enhanced` (Enhanced) # Processing PDF Files (Enhanced) Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/pdf-enhanced This guide will get you started with pdf deidentification. Limina supports scanning PDF files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. If you'd like to try it yourself, please [sign up for an account](https://portal.getlimina.ai/) to get a free API key. ## How PDFs Are Processed (Enhanced) PDFs are processed as follows: 1. First, the PDF pages are scanned for text as well as [Object Entities](/object-entities.md) using OCR. 2. The detected objects are replaced with white boxes, and in-place replacements are done for the text. 3. PDF Metadata as well as any included attachments are removed from the PDF. 4. A new PDF is returned with the in-place replacements and without any attachments. ## Parameters Below are the parameters that control the behaviour of the PDF De-identifier. These parameters shall be specified under `pdf_options`. | Parameter | Explanation | Default | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `approach` | This parameter changes which PDF approach is used. | "standard" | | `use_inline_replacement` | This parameter sets whether in-line replacements should be made to the PDF. If set to `False`, the container will use the approach selected in `image_options.masking_method` | True | [PDF Approaches](/configuration-and-operations/working-with-files/processing-files/pdf-comparison) shows the differences between Standard and Enhanced PDF processing. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | No | No | ## Sample Request [Connect with one of our privacy experts](https://private-ai.com/en/company/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines highlight={9-11} theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/pdf" }, "entity_detection": { "return_entity": true }, "pdf_options": { "approach":"enhanced" } } ``` ```shell curl wrap lines highlight={5} theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.pdf)'", "content_type": "application/pdf"}, "entity_detection": {"return_entity": "True"}, "pdf_options": {"approach": "enhanced"}, }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.pdf' ``` ```python python wrap lines highlight={21-23} theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.pdf" filename_out = "/path/to/output/sample.redacted.pdf" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/pdf", }, "entity_detection": { "return_entity": True }, "pdf_options": { "approach": "enhanced" } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.pdf" filename_out = "sample.redacted.pdf" file_type= "application/pdf" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing PDF Files (Standard) Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/pdf-standard This guide will get you started with pdf deidentification. Limina supports scanning PDF files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. If you'd like to try it yourself, please [sign up for an account](https://portal.getlimina.ai/) to get a free API key. ## How PDFs Are Processed PDFs are processed as follows: 1. First, each page in the PDF is rendered as an image. The result is similar to a PDF created by a photocopier scan. This is done to ensure that all PII is properly captured - PDF is a complicated [format](https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf). 2. Each page in the PDF is processed as an [image](/configuration-and-operations/working-with-files/processing-files/image). 3. A new PDF is created using the redacted/de-identified images produced in the previous step. 4. If specified, an invisible, de-identified text layer is created using the OCR system output. This ensures that the resulting PDF is searchable and allows for text to be copy & pasted. You can configure the OCR System by setting it as an [Environment Variable](/configuration-and-operations/container-management/environment-variables) or sending it in the request object. Check out our [OCR Guide](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to further understand the OCR modes and their usage. ## Constraints * Any attachments in a PDF file are removed. * If the PDF document to be de-identified already has an invisible text layer, it will be discarded and replaced with a new text-layer created through the use of OCR. ## Parameters Below are the parameters that control the behaviour of the PDF De-identifier. These parameters shall be specified under `pdf_options`. | Parameter | Explanation | Default | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `approach` | This parameter changes which PDF approach is used. | "standard" | | `density` | PDFs are converted into images using this DPI value. Smaller values result in images with smaller resolutions, which will take up less storage space and process faster, at the cost of output quality & redaction accuracy. | 200 | | `max_resolution` | PDFs are converted into images using the `density` DPI value. Any resulting images with maximum size length larger than this will be resized to this value, while preserving aspect ratio. | 3000 | [PDF Approaches](/configuration-and-operations/working-with-files/processing-files/pdf-comparison) shows the differences between Standard and Enhanced PDF processing. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/pdf" }, "entity_detection": { "return_entity": true }, "pdf_options": { "approach":"standard" } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.pdf)'", "content_type": "application/pdf"}, "entity_detection": {"return_entity": "True"}, "pdf_options": {"approach": "standard"}, }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.pdf' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.pdf" filename_out = "/path/to/output/sample.redacted.pdf" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/pdf", }, "entity_detection": { "return_entity": True }, "pdf_options": { "approach": "standard" } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.pdf" filename_out = "sample.redacted.pdf" file_type= "application/pdf" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing PowerPoint (PPT/PPTX) Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/powerpoint This guide will get you started with PPTX deidentification. Limina supports scanning Microsoft PowerPoint PPT and PPTX files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. If you'd like to try it yourself, please [sign up for an account](https://portal.getlimina.ai/) to get a free API key. ## How PPTX Files Are Processed PPTX files are processed by extracting each element and processing according to the table below. The de-identified or redacted file is created by according to the behaviour specified in the table. | Property Type | Details | Behaviour | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | Core properties | Author, Category, Comments, Content Status, Identifier, Keywords, Language, Last Modified By, Subject, Title, Version | Redact | | Speaker notes | Any content in the speakers notes | Redact | | Tables | Table objects with text and images | Redact | | Images | The [Images](/configuration-and-operations/working-with-files/processing-files/image) page provides a more detailed look at Image processing | Redact, unsupported image types are removed | | Text boxes | Main slide content | Redact | | Embedded links | Hyperlinks to internet pages or documents | Remove | | External elements | Tables and charts embedded from another document or file, such as an Excel chart | Remove external file, redact cached values | | Embedded audio & video | Videos and audio clips | Remove | | Review comments | Comments from document reviews | Redact | | Shape objects | Shapes containing text | Redact | You can configure the OCR System by setting it as an [Environment Variable](/configuration-and-operations/container-management/environment-variables) or sending it in the request object. Check out our [OCR Guide](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to further understand the OCR modes and their usage. ## How PPT Files Are Processed PPT files are processed by converting into PPTX files, followed the process described above and then converting back to PPT files. ## Constraints * If a piece of PII text has more than one style (different fonts, font sizes, underline etc.), the redaction marker will use the first style. * Charts in PPTX files will have all of their numerical data set to 0. * We recommend using Microsoft PowerPoint to open the processed PPT/PPTX files. Other editors may not give ideal results. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/vnd.openxmlformats-officedocument.presentationml.presentation" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.pptx)'", "content_type": "application/vnd.openxmlformats-officedocument.presentationml.presentation"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.pptx' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.pptx" filename_out = "/path/to/output/sample.redacted.pptx" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/vnd.openxmlformats-officedocument.presentationml.presentation", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.pptx" filename_out = "sample.redacted.pptx" file_type= "application/vnd.openxmlformats-officedocument.presentationml.presentation" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing TXT Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/txt This guide will get you started with TXT deidentification. Limina supports scanning TXT files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. If you'd like to try it yourself, please [sign up for an account](https://portal.getlimina.ai/) to get a free API key. ## How TXT Files Are Processed TXT files are processed by simply reading in the contents of the TXT files verbatim and passing it through Limina's text module. The resulting file will contain the labelled and redacted version of contents of the original. ## Constraints * Limina currently only supports `utf-8` encoding for text files. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 250 KiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/en/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "text/plain" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.txt)'", "content_type": "text/plain"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.txt' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.txt" filename_out = "/path/to/output/sample.redacted.txt" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "text/plain", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.txt" filename_out = "sample.redacted.txt" file_type= "text/plain" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing Word (DOC/DOCX) Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/word This guide will get you started with docx deidentification. Limina supports scanning Microsoft Word DOC & DOCX files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. If you'd like to try it yourself, please [sign up for an account](https://portal.getlimina.ai/) to get a free API key. ## How DOCX Files Are Processed Word document support is a new feature. Depending on the complexity of the processed documents, some of their elements might not be properly de-identified. We are working on expanding support; please consider rendering and processing as a [PDF](/configuration-and-operations/working-with-files/processing-files/pdf). This will ensure all content is processed and redacted. DOCX files are processed by extracting each element and processing according to the table below. The de-identified or redacted file is created according to the behaviour specified in the table. | Property Type | Details | Default Behaviour | Options | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------ | | Core properties | Author, Category, Comments, Content Status, Identifier, Keywords, Language, Last Modified By, Subject, Title, Version | Redact | Keep, Redact | | Headers and footers | Any content in headers and footers, such as text and images | Redact | Keep, Redact | | Tables | Table objects with text and images | Redact | Keep, Redact | | Images | The [Images](/configuration-and-operations/working-with-files/processing-files/image) page provides a more detailed look at Image processing | Redact, unsupported image types are removed | Redact | | Text content | Main body content | Redact | Keep, Redact | | Text boxes | Floating text boxes | Redact | Keep, Redact | | Embedded links | Hyperlinks to internet pages or documents | Remove | Keep, Redact | | External elements | Tables and charts embedded from another document or file, such as an Excel chart | Remove external file, redact cached values | Remove external file, redact cached values | | Embedded audio & video | Videos and audio clips | Remove | Remove | | Review comments | Comments from document reviews | Redact | Keep, Redact | | Shape objects | Shapes containing text | Redact | Keep, Redact | | Ink Drawings | Drawings in DOCX documents | Remove | Keep, Remove | See the [API Reference](/latest/process-files-base64) for changing the default behaviour. Graphical content (images) where text is present will be OCRed and then redacted. You can configure the OCR System by setting it as an [Environment Variable](/configuration-and-operations/container-management/environment-variables) or sending it in the request object. Check out our [OCR Guide](/configuration-and-operations/working-with-files/processing-files/ocr-modes) to further understand the OCR modes and their usage. ## How DOC Files Are Processed DOC files are processed by converting into DOCX files, followed the process described above and then converting back to DOC files. ## Constraints * If a piece of PII text has more than one style (different fonts, font sizes, underline etc.), the redaction marker will use the first style. * Charts in DOCX files contain an underlying .XLSX document that is automatically removed during deidentification. The cached chart values are deidentified by default. * Certain MHT files can be natively opened with Microsoft Word by changing the extension from MHT to DOC. Those files are not supported. We recommend that you use Microsoft Word to convert those files to DOCX. * We recommend using Microsoft Word to open the processed DOC/DOCX files. Other editors may not give ideal results. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 10 MiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/en/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.docx)'", "content_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.docx' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.docx" filename_out = "/path/to/output/sample.redacted.docx" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode("ascii") url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.docx" filename_out = "sample.redacted.docx" file_type= "application/vnd.openxmlformats-officedocument.wordprocessingml.document" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing XML Files Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/processing-files/xml This guide will get you started with XML deidentification. Limina supports scanning XML files for PII and creating de-identified or redacted copies. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Languages](/languages) and [Supported Entity Types](/entities) page provides a more detailed look. ## How XML Files Are Processed Similar to JSON files, XML files are processed using the method described in the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data). The output file retains its original format with redaction markers of the format "\[LABEL\_X]" in place of the detected PII. ## Constraints Please consider writing a handler for your specific application using the [Structured Data Guide](/configuration-and-operations/working-with-files/structured-data) to get around any of the constraints listed below. * The file processing routes are synchronous; large files over 5MB in size may take a long time to process. * Only node text contents and attributes are redacted. Node names are assumed to not contain PII and are not redacted. * Entity detections numbering is consistent within individual values only, not across the entire file. * The XML must be entirely human-readable content. Encoded content such as Base64 is not natively supported and may lead to inaccurate PII detection. Please process this content separately. * External sources, such as hyperlinks, images, and references (including custom schemas) in the XML are not loaded, processed, or redacted and will be handled verbatim as text. If you require this type of data to be handled in the XML, please consider writing your own handler. * Because tag and element values are modified, it is possible for the de-identified XML to not match the original file schema. For example, `23` will be redacted as `[MONEY]`. Please adjust the [enabled entity types](/configuration-and-operations/entity-detection-and-redaction/customizing-detection) for your needs. ## Support Matrix | | CPU Container | GPU Container | Community API | Professional API | | --------- | ------------- | ------------- | ------------- | ---------------- | | Supported | Yes | Yes | Up to 250 KiB | No | ## Sample Request [Connect with one of our privacy experts](https://getlimina.ai/en/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ```json Request Body wrap lines theme={"theme":"poimandres"} { "file": { "data": "", "content_type": "application/xml" }, "entity_detection": { "return_entity": true } } ``` ```shell curl wrap lines theme={"theme":"poimandres"} echo '{ "file": {"data": "'$(base64 -w 0 sample.xml)'", "content_type": "application/xml"}, "entity_detection": {"return_entity": "True"} }' \ | curl --request POST --url 'https://api.getlimina.ai/community/v4/process/files/base64' \ -H 'Content-Type: application/json' \ -H 'x-api-key: ' \ -d @- \ | jq -r .processed_file \ | base64 -d > 'sample.redacted.xml' ``` ```python python wrap lines theme={"theme":"poimandres"} import requests import base64 file_url = "https://paidocumentation.blob.core.windows.net/$web/sample.xml" filename_out = "/path/to/output/sample.redacted.xml" file_content = requests.get(file_url).content file_content_base64 = base64.b64encode(file_content).decode() url = "https://api.getlimina.ai/community/v4/process/files/base64" headers = {"Content-Type": "application/json", "x-api-key": ""} payload = { "file":{ "data": file_content_base64, "content_type": "application/xml", }, "entity_detection": { "return_entity": True } } response = requests.post(url, json=payload, headers=headers) with open(filename_out, "wb") as f: f.write(base64.b64decode(response.json()["processed_file"])) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client.objects import request_objects import base64 filename_in = "sample.xml" filename_out = "sample.redacted.xml" file_type= "application/xml" client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") with open(filename_in, "rb") as b64_file: file_data = base64.b64encode(b64_file.read()) file_data = file_data.decode("ascii") file_obj = request_objects.file_obj(data=file_data, content_type=file_type) request_obj = request_objects.file_base64_obj(file=file_obj) resp = client.process_files_base64(request_object=request_obj) with open(filename_out, 'wb') as redacted_file: processed_file = resp.processed_file.encode("ascii") processed_file = base64.b64decode(processed_file, validate=True) redacted_file.write(processed_file) ``` ## Sample Response ```json Response wrap lines theme={"theme":"poimandres"} { "processed_file": "Base64 Encoded File Content of the Redacted File", "processed_text": "string", "entities": "List[Entity]", "entities_present": true, "languages_detected": {"lang_1": 0.67, "lang_2": 0.74} } ``` # Processing Structured Data Sources Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/structured-data This guide illustrates how to use Limina on Structured Data sources such as Databases, JSON, CSV and XML files This guide illustrates how to use Limina on Structured Data sources such as Databases, JSON, CSV and XML files. It is also applicable to Databricks and Snowflake. [Connect with one of our privacy experts](https://getlimina.ai/contact-us/?utm_source=docs\&utm_medium=website) to run this code. ## Tabular Data If it is only required to redact certain Columns such as `VisitNotes` please see our [PII Safe Sentiment Analysis Walkthrough](https://getlimina.ai/blog/sentiment-analysis-anonymization/). In addition to raw text and files, you can use Limina to process tabular data, such as found in a database, CSV or Microsoft Excel file. For example, consider the below table: | ClientNm | DOfB | HlthCareNm | TempFd | VisitNotes | | -------------- | ------ | ---------- | ------ | ----------------------------------------------------------------------------------------------------------------------- | | Stefan Krugler | 150368 | 2342345234 | NYC | Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in | | Hari Seldon | 201030 | 4947384944 | LA | Patient came in for a regular checkup. | One could try to create a rule-based system, but it is difficult: * The column names are all acronyms, `DOfB` in particular is an uncommon way of referring to a date of birth. * Each cell value is written as an integer to save space, removing the hint of `/` or `.` characters in identifying the date. * `TempFd` contains the city the individual lives in, but is stored in a temporary field. * `VisitNotes` is a varchar free text field. This field is particularly useful for ML use cases, for example to determine the main visit reasons or for case resolution. In particular, longer string and blob fields in databases can be difficult. ### Processing As a String A simple approach would be to join each row together and process as a single long string: ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "ClientNm, DOfB, HlthCareNm, TempFd, VisitNotes, Stefan Krugler, 150368, 2342345234, NYC, Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, Hari Seldon, 201030, 4947384944, LA, Patient came in for a regular checkup." ] } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "ClientNm, DOfB, HlthCareNm, TempFd, VisitNotes, Stefan Krugler, 150368, 2342345234, NYC, Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, Hari Seldon, 201030, 4947384944, LA, Patient came in for a regular checkup." ] }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post(url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={"text": ["ClientNm, DOfB, HlthCareNm, TempFd, VisitNotes, Stefan Krugler, 150368, 2342345234, NYC, Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, Hari Seldon, 201030, 4947384944, LA, Patient came in for a regular checkup."]}) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text_request = request_objects.process_text_obj(text=["ClientNm, DOfB, HlthCareNm, TempFd, VisitNotes, Stefan Krugler, 150368, 2342345234, NYC, Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, Hari Seldon, 201030, 4947384944, LA, Patient came in for a regular checkup."]) response = client.process_text(text_request) print(response.processed_text) ``` However the result isn't great: ```shell Response Text wrap theme={"theme":"poimandres"} ['ClientNm, DOfB, HlthCareNm, TempFd, VisitNotes, [NAME_1], [LOCATION_1], Prescribed [NAME_GIVEN_1] [DOSE_1] of [DRUG_1] for his [CONDITION_1]. Will call him on [PHONE_NUMBER_1] when I get the test results in, [NAME_MEDICAL_PROFESSIONAL_1], [LOCATION_2], Patient came in for a regular checkup.'] ``` ### Formatting the Data Correctly We can get better results by formatting the input in a `"key1: value1, key2: value2, ...` pattern, to make sure that each cell entry includes the column name for context. If multiple headers exist, the format should look like: `"key1a: key1b: value1, key2: value2, ...`. Note that it's important to put a colon at the end of each key. ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "ClientNm: Stefan Krugler, DOfB: 150368, HlthCareNm: 2342345234, TempFd: NYC, VisitNotes: Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, ClientNm: Hari Seldon, DOfB: 201030, HlthCareNm: 4947384944, TempFd: LA, VisitNotes: Patient came in for a regular checkup." ] } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "ClientNm: Stefan Krugler, DOfB: 150368, HlthCareNm: 2342345234, TempFd: NYC, VisitNotes: Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, ClientNm: Hari Seldon, DOfB: 201030, HlthCareNm: 4947384944, TempFd: LA, VisitNotes: Patient came in for a regular checkup." ] }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post(url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={"text": ["ClientNm: Stefan Krugler, DOfB: 150368, HlthCareNm: 2342345234, TempFd: NYC, VisitNotes: Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, ClientNm: Hari Seldon, DOfB: 201030, HlthCareNm: 4947384944, TempFd: LA, VisitNotes: Patient came in for a regular checkup."]}) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text_request = request_objects.process_text_obj(text=[ "ClientNm: Stefan Krugler, DOfB: 150368, HlthCareNm: 2342345234, TempFd: NYC, VisitNotes: Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in, ClientNm: Hari Seldon, DOfB: 201030, HlthCareNm: 4947384944, TempFd: LA, VisitNotes: Patient came in for a regular checkup."]) response = client.process_text(text_request) print(response.processed_text) ``` As you can see, the result is much better; it's still susceptible to False-Positive detections on column names. Also as the result is a long string, it is more difficult to relate the predictions back to individual cells: ```shell Response Text wrap theme={"theme":"poimandres"} ['ClientNm: [NAME_1], DOfB: [DOB_1], HlthCareNm: [HEALTHCARE_NUMBER_1], TempFd: [LOCATION_CITY_1], VisitNotes: Prescribed [NAME_GIVEN_1] [DOSE_1] of [DRUG_1] for his [CONDITION_1]. Will call him on [PHONE_NUMBER_1] when I get the test results in, ClientNm: [NAME_2], DOfB: [DOB_2], HlthCareNm: [HEALTHCARE_NUMBER_2], TempFd: [LOCATION_CITY_2], VisitNotes: Patient came in for a regular checkup.'] ``` ### Best Practice: Correct Formatting With Link Batch These examples process the entire table in a single API request to ensure consistent entity numbering. For larger tables it is recommended to process row by row, without numbered markers. We can solve both of the issues with the previous approach by feeding the table in as a list with [link\_batch](/container-quickstart/api-quickstart#processing-related-examples) on. This is functionally equivalent to the example presented above and this allows us to easily get the results of each cell without any post-processing logic. ```json Request Body wrap lines highlight={5} theme={"theme":"poimandres"} { "text": [ "ClientNm:", "Stefan Krugler", ", DOfB:", "150368", ", HlthCareNm:", "2342345234", ", TempFd:", "NYC", ", VisitNotes:", "Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in", "ClientNm:", "Hari Seldon", ", DOfB:", "201030", ", HlthCareNm:", "4947384944", ", TempFd:", "LA", ", VisitNotes:", "Patient came in for a regular checkup." ], "link_batch": true } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "ClientNm:", "Stefan Krugler", ", DOfB:", "150368", ", HlthCareNm:", "2342345234", ", TempFd:", "NYC", ", VisitNotes:", "Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in", "ClientNm:", "Hari Seldon", ", DOfB:", "201030", ", HlthCareNm:", "4947384944", ", TempFd:", "LA", ", VisitNotes:", "Patient came in for a regular checkup." ], "link_batch": true }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post(url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={"text": ["ClientNm:", "Stefan Krugler", ", DOfB:", "150368", ", HlthCareNm:", "2342345234", ", TempFd:", "NYC", ", VisitNotes:", "Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in", "ClientNm:", "Hari Seldon", ", DOfB:", "201030", ", HlthCareNm:", "4947384944", ", TempFd:", "LA", ", VisitNotes:", "Patient came in for a regular checkup."], "link_batch": True}) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text = ["ClientNm:", "Stefan Krugler", ", DOfB:", "150368", ", HlthCareNm:", "2342345234", ", TempFd:", "NYC", ", VisitNotes:", "Prescribed Stefan 10mg of Ibuprofen for his migraines. Will call him on +1 324 4325 5462 when I get the test results in", "ClientNm:", "Hari Seldon", ", DOfB:", "201030", ", HlthCareNm:", "4947384944", ", TempFd:", "LA", ", VisitNotes:", "Patient came in for a regular checkup."] text_request = request_objects.process_text_obj(text=text, link_batch=True) response = client.process_text(text_request) print(response.processed_text) ``` The result is the same, but it is now easy to get the output for each cell by looking at the corresponding response in the list: ```shell Response Text wrap theme={"theme":"poimandres"} ['ClientNm:', '[NAME_1]', ', DOfB:', '[DOB_1]', ', HlthCareNm:', '[HEALTHCARE_NUMBER_1]', ', TempFd:', '[LOCATION_CITY_1]', ', VisitNotes:', 'Prescribed [NAME_GIVEN_1] [DOSE_1] of [DRUG_1] for his [CONDITION_1]. Will call him on [PHONE_NUMBER_1] when I get the test results in', 'ClientNm:', '[NAME_2]', ', DOfB:', '[DOB_2]', ', HlthCareNm:', '[HEALTHCARE_NUMBER_2]', ', TempFd:', '[LOCATION_CITY_2]', ', VisitNotes:', 'Patient came in for a regular checkup.'] ``` In addition to the redacted result, the entity list (not shown) can be used. This approach can be adjusted to column-by-column, but row-by-row usually leads to better detection performance. The examples above process the entire table but they can easily be adapted to process specific rows and columns. ## JSON & XML The approach described in this section is implemented in the `process/files` routes for [JSON](/configuration-and-operations/working-with-files/processing-files/json) and [XML](/configuration-and-operations/working-with-files/processing-files/xml). A similar approach applies for JSON and XML. Consider the following JSON: ```json JSON theme={"theme":"poimandres"} { "VisitDates": ["15/2/2023", "8/5/2023"], "PhysName": "Dr Walter", "PersonAttributes": {"Weight": 78.5, "Height": 189}, "MedLoc": "Mt Western Cancer Center" } ``` It is possible to send the JSON through as text, like one might to an LLM, yet this might corrupt the JSON - a lot of pre- and post-processing logic is necessary for a production application. Instead, we can parse the JSON and arrange it into a `: ...` pattern like above, add colons to the last key before each value and enable [link\_batch](/container-quickstart/api-quickstart#processing-related-examples). The only difference is that a value might have multiple keys in the hierarchy, so we prepend all of those. The request payload then looks like: ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "VisitDates:", "15/2/2023", ", VisitDates:", "8/5/2023", ", PhysName:", "Dr Walter", ", PersonAttributes: Weight:", "78.5", ", PersonAttributes: Height:", "189", ", MedLoc:", "Mt Western Cancer Center" ], "link_batch": true } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "VisitDates:", "15/2/2023", ", VisitDates:", "8/5/2023", ", PhysName:", "Dr Walter", ", PersonAttributes: Weight:", "78.5", ", PersonAttributes: Height:", "189", ", MedLoc:", "Mt Western Cancer Center" ] }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post(url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={"text": ["VisitDates:", "15/2/2023", ", VisitDates:", "8/5/2023", ", PhysName:", "Dr Walter", ", PersonAttributes: Weight:", "78.5", ", PersonAttributes: Height:", "189", ", MedLoc:", "Mt Western Cancer Center"]}) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text_request = request_objects.process_text_obj(text=["VisitDates:", "15/2/2023", ", VisitDates:", "8/5/2023", ", PhysName:", "Dr Walter", ", PersonAttributes: Weight:", "78.5", ", PersonAttributes: Height:", "189", ", MedLoc:", "Mt Western Cancer Center"], link_batch=True) response = client.process_text(text_request) print(response.processed_text) ``` Yields the following: ```shell Response Text wrap theme={"theme":"poimandres"} ['VisitDates:', '[DATE_1]', ', VisitDates:', '[DATE_2]', ', PhysName:', '[NAME_MEDICAL_PROFESSIONAL_1]', ', PersonAttributes: Weight:', '[PHYSICAL_ATTRIBUTE_1]', 'PersonAttributes: Height:', '[PHYSICAL_ATTRIBUTE_2]', ', MedLoc:', '[ORGANIZATION_MEDICAL_FACILITY_1]'] ``` # Supported File Types Source: https://docs.getlimina.ai/configuration-and-operations/working-with-files/supported-file-types Support File Types like PDFs, CSVs, JSON and more. View the full list of supported file types Limina can identify and redact, including images, audio, and documents. Limina can support multiple file types for de-identification. The complete list of supported file types is below. New file types are continually being added, please [contact us](https://getlimina.ai/contact-us/) if you require a file type not in the list below. Limina’s supported entity types function across each file type, with localized variants of different **PII** (Personally Identifiable Information) entities, **PHI** (Protected Health Information) entities, and **PCI** (Payment Card Industry) entities being detected. Our [Supported Entity Types](/entities) page provides a more detailed look at entities. **Supported Languages** Note that while Limina text de-identification service supports more than [50 languages](/languages), the file processing service supports a restricted list of languages. See [supported languages](/languages/#supported-languages) for more details. ## Document File Types | File Type | Extension | Content Type | Added In | Object Detection Support | Beta | | --------------------------------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------- | -------- | :----------------------: | :--: | | [PDF](/configuration-and-operations/working-with-files/processing-files/pdf) | `.pdf` | `application/pdf` | 3.0.0 | ✓ | | | [JSON](/configuration-and-operations/working-with-files/processing-files/json) | `.json` | `application/json` | 3.1.0 | | | | [XML](/configuration-and-operations/working-with-files/processing-files/xml) | `.xml` | `application/xml` | 3.1.0 | | | | [CSV](/configuration-and-operations/working-with-files/processing-files/csv) | `.csv` | `text/csv` | 3.1.0 | | | | [Word](/configuration-and-operations/working-with-files/processing-files/word) | `.doc` | `application/msword` | 3.1.0 | ✓ (partially) | | | [Word Open XML](/configuration-and-operations/working-with-files/processing-files/word) | `.docx` | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | 3.1.0 | ✓ (partially) | | | [Text](/configuration-and-operations/working-with-files/processing-files/txt) | `.txt` | `text/plain` | 3.1.1 | | | | [Excel](/configuration-and-operations/working-with-files/processing-files/excel) | `.xls` | `application/vnd.ms-excel` | 3.2.0 | ✓ (partially) | | | [Excel Open XML](/configuration-and-operations/working-with-files/processing-files/excel) | `.xlsx` | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | 3.2.0 | ✓ (partially) | | | [PowerPoint](/configuration-and-operations/working-with-files/processing-files/powerpoint) | `.ppt` | `application/vnd.ms-powerpoint` | 3.5.0 | ✓ (partially) | | | [PowerPoint Open XML](/configuration-and-operations/working-with-files/processing-files/powerpoint) | `.pptx` | `application/vnd.openxmlformats-officedocument.presentationml.presentation` | 3.5.0 | ✓ (partially) | | | [DICOM](/configuration-and-operations/working-with-files/processing-files/dcm-dicom) | `.dcm` | `application/dicom` | 3.4.0 | | ✓ | **A note on object detection support in Office documents** Object detection in Office files is partially supported. Embedded images in Office documents are only processed for object detection and redaction if they are compatible with our deidentifier. Non-compatible images are replaced with a black placeholder image. This ensures that sensitive data in non-supported formats is always processed, although it is not redacted with the same level of precision as data in supported formats. Additionally: * The bounding box coordinates within the `location` field (`x0`, `x1`, `y0`, `y1`) are relative to the embedded image itself, unlike in PDFs, where they are relative to the document page. * The `page` field value remains `0` for Office files, as page numbering is not currently implemented for this file type. ## Image File Types Processing Image File Types | File Type | Extension | Content Type | Added In | Object Detection Support | | --------- | --------------- | ----------------------------- | -------- | :----------------------: | | JPEG | `.jpg`, `.jpeg` | `image/jpg`, `image/jpeg` | 3.0.0 | ✓ | | TIFF | `.tif`, `.tiff` | `image/tif`, `image/tiff` | 3.0.0 | ✓ | | PNG | `.png` | `image/png` | 3.4.0 | ✓ | | BMP | `.bmp` | `image/bmp`, `image/x-ms-bmp` | 3.4.0 | ✓ | ## Audio File Types Processing Audio File Types | File Type | Extension | Content Type | Added In | | --------- | --------- | ------------------------- | -------- | | wave | `.wav` | `audio/wav` | 3.0.0 | | mp3 | `.mp3` | `audio/mpeg`, `audio/mp3` | 3.0.0 | | mp4 | `.mp4` | `audio/mp4` | 3.0.0 | | m4a | `.m4a` | `audio/m4a` | 3.5.0 | | webm | `.webm` | `audio/webm` | 3.5.0 | **VOX files** .vox files are not natively supported in the Limina container, but can be processed by converting the .vox file to a wav or mp3 using a conversion tool like [SoX](https://sourceforge.net/projects/sox/). Because .vox files are headerless, you will need to know the sample rate and encoding to specify. For example, to take a vox file with a sample rate 8000, mono channel, mu-law encoded: ``` sox -t raw -r 8000 -c 1 -e mu-law myfile.vox myfile.wav ``` to generate a wav file. # Usage Quickstart Source: https://docs.getlimina.ai/container-quickstart/api-quickstart API Quickstart 日本語 This guide walks through the main features of the Limina API. It focuses on pure text applications, but can extend to [processing files](/configuration-and-operations/working-with-files/processing-files/index). This guide uses Limina's cloud API. Get a [free API key](https://portal.getlimina.ai/?from=GettingStarted) to run the examples. If you're using the container instead, follow the [container quickstart](/container-quickstart/quickstart-guide) and replace the endpoint with `http://localhost:8080`. ## Basic Use The `process/text` endpoint accepts a list of text strings and replaces each piece of PII found with a redaction marker. A simple request looks like this: ```json Request Body wrap lines theme={"theme":"poimandres"} { "text": [ "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01." ] } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01." ] }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post( url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={ "text": [ "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01." ] }, ) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text_request = request_objects.process_text_obj(text=["Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01."]) response = client.process_text(text_request) print(response.processed_text) ``` The response contains two main outputs: * `processed_text`, the redacted, masked or synthetic text as defined by `processed_text` in the input * `entities`, a list of each PII found, which is useful for PII detection and NER (Named Entity Recognition) ```json Response wrap lines expandable theme={"theme":"poimandres"} [ { "processed_text": "Thank you for calling the [ORGANIZATION_1]. My name is miss [NAME_GIVEN_1], and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, [SSN_1].", "entities": [ { "processed_text": "ORGANIZATION_1", "text": "Georgia Division of Transportation", "location": { "stt_idx": 26, "end_idx": 60, "stt_idx_processed": 26, "end_idx_processed": 42 }, "best_label": "ORGANIZATION", "labels": { "LOCATION_STATE": 0.2403, "LOCATION": 0.2342, "ORGANIZATION": 0.8967 } }, { "processed_text": "NAME_GIVEN_1", "text": "Johanna", "location": { "stt_idx": 78, "end_idx": 85, "stt_idx_processed": 60, "end_idx_processed": 74 }, "best_label": "NAME_GIVEN", "labels": { "NAME_GIVEN": 0.9127, "NAME": 0.9018 } }, { "processed_text": "SSN_1", "text": "614-5555 01", "location": { "stt_idx": 203, "end_idx": 214, "stt_idx_processed": 192, "end_idx_processed": 199 }, "best_label": "SSN", "labels": { "SSN": 0.913 } } ], "entities_present": true, "characters_processed": 215, "languages_detected": { "en": 0.920992910861969 } } ] ``` ## Processing Related Examples If the elements in the list of strings are related, the `link_batch` parameter can be used to share context throughout the list. ```json Request Body wrap lines highlight={6} theme={"theme":"poimandres"} { "text": [ "My phone number is", "2345435" ], "link_batch": true } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "My phone number is", "2345435" ], "link_batch": true }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post( url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={ "text": [ "My phone number is", "2345435", ], "link_batch": True, }, ) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text_request = request_objects.process_text_obj(text=["My phone number is", "2345435"], link_batch=True) response = client.process_text(text_request) print(response.processed_text) ``` This ensures that the inputs are joined before going to the PII detection system. This way the model sees `My phone number is 2345435` instead of `My phone number is` and `2345435` as two unrelated messages. This allows the phone number to be identified correctly. ```shell Redacted Text wrap theme={"theme":"poimandres"} ["My phone number is", "[PHONE_NUMBER_1]"] ``` ```json Full Response wrap theme={"theme":"poimandres"} [ { "processed_text": "My phone number is", "entities": [], "entities_present": false, "characters_processed": 18, "languages_detected": { "en": 0.8986189365386963 } }, { "processed_text": "[PHONE_NUMBER_1]", "entities": [ { "processed_text": "PHONE_NUMBER_1", "text": "2345435", "location": { "stt_idx": 0, "end_idx": 7, "stt_idx_processed": 0, "end_idx_processed": 16 }, "best_label": "PHONE_NUMBER", "labels": { "PHONE_NUMBER": 0.9166 } } ], "entities_present": true, "characters_processed": 7, "languages_detected": {} } ] ``` ## Customizing Entity Detection With Selective Redaction The above example identifies and removes all non-beta entity types. Granular control over entity detection and redaction can be set using [Entity Selectors](/configuration-and-operations/entity-detection-and-redaction/customizing-detection). For example, to only redact the SSN: ```json Request Body wrap lines highlight={5-14} theme={"theme":"poimandres"} { "text": [ "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": [ "SSN" ] } ] } } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01." ], "entity_detection": { "entity_types": [ { "type": "ENABLE", "value": [ "SSN" ] } ] } }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post( url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={ "text": [ "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01." ], "entity_detection": {"entity_types": [{"type": "ENABLE", "value": ["SSN"]}]}, }, ) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") entity_detection_object = request_objects.entity_detection_obj(entity_types=[request_objects.entity_type_selector_obj(type="ENABLE", value=["SSN"])]) text_request = request_objects.process_text_obj(text=["Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, 614-5555 01."], entity_detection=entity_detection_object) response = client.process_text(text_request) print(response.processed_text) ``` The result of this selective redaction is below: ```text Redacted Text wrap theme={"theme":"poimandres"} Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, [SSN_1]. ``` ```json Full Response wrap theme={"theme":"poimandres"} [ { "processed_text": "Thank you for calling the Georgia Division of Transportation. My name is miss Johanna, and it is a pleasure assisting you today. For security reasons, may I please have your Social Security number? Yes, [SSN_1].", "entities": [ { "processed_text": "SSN_1", "text": "614-5555 01", "location": { "stt_idx": 203, "end_idx": 214, "stt_idx_processed": 203, "end_idx_processed": 210 }, "best_label": "SSN", "labels": { "SSN": 0.913 } } ], "entities_present": true, "characters_processed": 215, "languages_detected": { "en": 0.920992910861969 } } ] ``` ## Adding Allow & Block Lists You can also customize PII detection and redaction using enable/disable [Entity Selectors](/configuration-and-operations/entity-detection-and-redaction/customizing-detection#enabling-and-disabling-entity-types) or regex-based [Filters](/configuration-and-operations/entity-detection-and-redaction/customizing-detection#filters), enabling custom handling for company-specific identifiers such as employee IDs or internal database IDs. The example below shows how to combine [Entity Selectors](/configuration-and-operations/entity-detection-and-redaction/customizing-detection) with [Filters](/configuration-and-operations/entity-detection-and-redaction/customizing-detection#filters) for fine-grained control. In this HR claim scenario, an employee reports a medical injury and requests accommodation. Here, we demonstrate: * Two regex-based [block filters](/configuration-and-operations/entity-detection-and-redaction/customizing-detection#block-filter) defining custom entity types for employee IDs and business units, overriding Limina's defaults. * Disabling the injury entity, which may be required for insurance-related workflows. * Using a list for the `text` payload, as expected in conversational contexts, and enabling `link_batch` to maintain redaction context across the full thread. * Disabling numbering of redaction markers. ```json Request Body wrap lines highlight={22-33} theme={"theme":"poimandres"} { "text": [ "Hello Xavier, can you tell me your employee ID?", "Yep, my Best Corp ID is GID-45434, and my SIN is 690 871 283", "Okay, thanks Xavier, why are you calling today?", "I broke my right leg on the 31st and I'm waiting for my x-ray results. dr. zhang, mercer health centre.", "Oh, so sorry to hear that! How can we help?", "I won't be able to come back to the office in NYC for a while", "No problem Xavier, I will enter a short term work from home for you. You're all set!", "Thanks so much Carole!" ], "link_batch": true, "entity_detection": { "entity_types": [ { "type": "DISABLE", "value": [ "INJURY" ] } ], "filter": [ { "type": "BLOCK", "entity_type": "EMPLOYEE_ID", "pattern": "GID-\\d{5}" }, { "type": "BLOCK", "entity_type": "BUSINESS_UNIT", "pattern": "Best Corp" } ], "return_entity": true }, "processed_text": { "type": "MARKER", "pattern": "[BEST_ENTITY_TYPE]" } } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data ' { "text": [ "Hello Xavier, can you tell me your employee ID?", "Yep, my Best Corp ID is GID-45434, and my SIN is 690 871 283", "Okay, thanks Xavier, why are you calling today?", "I broke my right leg on the 31st and I''m waiting for my x-ray results. dr. zhang, mercer health centre.", "Oh, so sorry to hear that! How can we help?", "I won''t be able to come back to the office in NYC for a while", "No problem Xavier, I will enter a short term work from home for you. You''re all set!", "Thanks so much Carole!" ], "link_batch": true, "entity_detection": { "entity_types": [ { "type": "DISABLE", "value": [ "INJURY" ] } ], "filter": [ { "type": "BLOCK", "entity_type": "EMPLOYEE_ID", "pattern": "GID-\\d{5}" }, { "type": "BLOCK", "entity_type": "BUSINESS_UNIT", "pattern": "Best Corp" } ], "return_entity": true }, "processed_text": { "type": "MARKER", "pattern": "[BEST_ENTITY_TYPE]" } }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post( url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={ "text": [ "Hello Xavier, can you tell me your employee ID?", "Yep, my Best Corp ID is GID-45434, and my SIN is 690 871 283", "Okay, thanks Xavier, why are you calling today?", "I broke my right leg on the 31st and I'm waiting for my x-ray results. dr. zhang, mercer health centre.", "Oh, so sorry to hear that! How can we help?", "I won't be able to come back to the office in NYC for a while", "No problem Xavier, I will enter a short term work from home for you. You're all set!", "Thanks so much Carole!", ], "link_batch": True, "entity_detection": { "entity_types": [{"type": "DISABLE", "value": ["INJURY"]}], "filter": [ { "type": "BLOCK", "entity_type": "EMPLOYEE_ID", "pattern": "GID-\\d{5}", }, { "type": "BLOCK", "entity_type": "BUSINESS_UNIT", "pattern": "Best Corp", }, ], "return_entity": True, }, "processed_text": {"type": "MARKER", "pattern": "[BEST_ENTITY_TYPE]"}, }, ) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") filter_employee = request_objects.filter_selector_obj(type="BLOCK", entity_type="EMPLOYEE_ID", pattern="GID-\\d{5}") filter_bu = request_objects.filter_selector_obj(type="BLOCK", entity_type="BUSINESS_UNIT", pattern="Best Corp") entity_detection_object = request_objects.entity_detection_obj(entity_types=[request_objects.entity_type_selector_obj(type="DISABLE", value=["INJURY"])], filter=[filter_employee, filter_bu], return_entity=True) processed_text_object = request_objects.processed_text_obj(type="MARKER", pattern="[BEST_ENTITY_TYPE]") text_request = request_objects.process_text_obj(text=[ "Hello Xavier, can you tell me your employee ID?", "Yep, my Best Corp ID is GID-45434, and my SIN is 690 871 283", "Okay, thanks Xavier, why are you calling today?", "I broke my right leg on the 31st and I'm waiting for my x-ray results. dr. zhang, mercer health centre.", "Oh, so sorry to hear that! How can we help?", "I won't be able to come back to the office in NYC for a while", "No problem Xavier, I will enter a short term work from home for you. You're all set!", "Thanks so much Carole!" ], link_batch=True, entity_detection=entity_detection_object, processed_text=processed_text_object, ) response = client.process_text(text_request) print(response.processed_text) ``` The above request yields this response: ```python Redacted Text wrap theme={"theme":"poimandres"} ['Hello [NAME_GIVEN], can you tell me your employee ID?', 'Yep, my [BUSINESS_UNIT] ID is [EMPLOYEE_ID], and my SIN is [SSN]', 'Okay, thanks [NAME_GIVEN], why are you calling today?', "I broke my right leg on the [DATE] and I'm waiting for my [MEDICAL_PROCESS] results. [NAME_MEDICAL_PROFESSIONAL], [ORGANIZATION_MEDICAL_FACILITY].", 'Oh, so sorry to hear that! How can we help?', "I won't be able to come back to the office in [LOCATION_CITY] for a while", "No problem [NAME_GIVEN], I will enter a short term work from home for you. You're all set!", 'Thanks so much [NAME_GIVEN]!'] ``` ## Generating Synthetic Entities (Beta) In addition to replacing PII with redaction markers, tokens, or masks, Limina can generate synthetic PII; realistic fake replacements created with an ML model that fit the surrounding context. This offers several advantages: * Synthetic PII preserves most of the original text, reducing the risk of introducing bias compared to generators that create entirely new data, and improving utility for downstream tasks like sentiment analysis. * Even though our [PII detection engine leads the market](https://www.getlimina.ai/en/resources/whitepaper), it isn't perfect. Synthetic PII ensures that any PII detection misses are hidden amongst realistic, fake PII, strengthening protection against re-identification. * Synthetic entities resemble natural language more closely than redaction markers or hashes, minimizing disruption to downstream ML systems. To enable synthetic PII generation, set the `processed_text` object's marker type to `SYNTHETIC` in your API request. ```json Request Body wrap lines highlight={6} theme={"theme":"poimandres"} { "text": [ "Hello, my name is May. I am the aunt of Jessica Parker. We live in Toronto, Canada." ], "processed_text": { "type": "SYNTHETIC" } } ``` ```shell cURL wrap lines theme={"theme":"poimandres"} curl --location 'https://api.getlimina.ai/community/v4/process/text' \ --header 'Content-Type: application/json' \ --header 'x-api-key: ' \ --data '{ "text": [ "Hello, my name is May. I am the aunt of Jessica Parker. We live in Toronto, Canada." ], "processed_text": { "type": "SYNTHETIC" } }' ``` ```python Python wrap lines theme={"theme":"poimandres"} import requests r = requests.post( url="https://api.getlimina.ai/community/v4/process/text", headers={"x-api-key": ""}, json={ "text": [ "Hello, my name is May. I am the aunt of Jessica Parker. We live in Toronto, Canada." ], "processed_text": {"type": "SYNTHETIC"}, }, ) results = r.json() print(results) ``` ```python Python Client wrap lines theme={"theme":"poimandres"} from privateai_client import PAIClient from privateai_client import request_objects client = PAIClient(url="https://api.getlimina.ai/community/v4/", api_key="") text_request = request_objects.process_text_obj(text=["Hello, my name is May. I am the aunt of Jessica Parker. We live in Toronto, Canada."], processed_text=request_objects.processed_text_obj(type="SYNTHETIC")) response = client.process_text(text_request) print(response.processed_text) ``` Yields the following response: ```text Redacted Text wrap theme={"theme":"poimandres"} Hello, my name is Ben. I am the aunt of Michael Morley. We live in Ekshaku, Sweden. ``` ```json Full Response wrap theme={"theme":"poimandres"} [ { "processed_text": "Hello, my name is Ben. I am the aunt of Michael Morley. We live in Ekshaku, Sweden.", "entities": [ { "processed_text": "Ben", "text": "May", "location": { "stt_idx": 18, "end_idx": 21, "stt_idx_processed": 18, "end_idx_processed": 21 }, "best_label": "NAME_GIVEN", "labels": { "NAME_GIVEN": 0.9234, "NAME": 0.8903 } }, { "processed_text": "Michael Morley", "text": "Jessica Parker", "location": { "stt_idx": 40, "end_idx": 54, "stt_idx_processed": 40, "end_idx_processed": 54 }, "best_label": "NAME", "labels": { "NAME_GIVEN": 0.4595, "NAME": 0.9178, "NAME_FAMILY": 0.4567 } }, { "processed_text": "Ekshaku, Sweden", "text": "Toronto, Canada", "location": { "stt_idx": 67, "end_idx": 82, "stt_idx_processed": 67, "end_idx_processed": 82 }, "best_label": "LOCATION", "labels": { "LOCATION_CITY": 0.3177, "LOCATION": 0.9268, "LOCATION_COUNTRY": 0.3185 } } ], "entities_present": true, "characters_processed": 83, "languages_detected": { "en": 0.8507365584373474 } } ] ``` # Deployment planning checklist Source: https://docs.getlimina.ai/container-quickstart/deployment-checklist This introductory guide will help you understand the initial steps required for a successful deployment # The first deployment The following is a snapshot of a typical workflow and checklist when moving from evaluation to validation. The list below represents a set of activities and resources that are commonly used to support engineering, product, dev-ops and legal teams as you prepare for a larger scale deployment. This guide illustrates a typical process flow for customers that have already onboarded with an account manager. [Contact us](https://getlimina.ai/contact-us/) to find out how you can get started! ## First deployment checklist Ensure that all users or service accounts are signed up within the [Limina customer portal](https://portal.getlimina.ai/) and are linked to your organization. Contact support at [support@getlimina.ai](mailto:support@getlimina.ai) if you require assistance. Validate that you can [download the container](/installation/grabbing-the-image) image. *It is always recommended that you "pin" the version of the container image. eg. cpu-4.2.3 even when locally mirroring the container image.* Starting with the basic features please see the [Usage Quickstart](/container-quickstart/api-quickstart). Additionally, check out the [Code Samples](/examples/github-examples) Limina supports a wide variety of file modalities and entity types. It is important to understand the "shape" of your data, the intended outcome, and the anticipated downstream processing steps, as these all impact the features selected and the overall architecture of your pipeline. This includes: An understanding of the [Supported Languages](/languages) An understanding of the [Supported File types](/configuration-and-operations/working-with-files/supported-file-types) An understanding of the [Supported Entities](/entities/supported-entity-types) A review of the [Applicable Regulations](/entities/supported-entity-types#supported-regulations) A review of the detection options in [Customizing detection](/configuration-and-operations/entity-detection-and-redaction/customizing-detection) A review of the redaction options in [Customizing redaction](/configuration-and-operations/entity-detection-and-redaction/customizing-redaction) The design of the data pipeline is highly dependent on the anticipated modalities, latency requirements, throughput requirements and overall traffic patterns. A solid understanding of the [Supported File types](/configuration-and-operations/working-with-files/supported-file-types), and how each [file type is processed](/configuration-and-operations/working-with-files/processing-files/index). As well as the best practices when working with [structured data](/configuration-and-operations/working-with-files/structured-data). Is recommended. Infrastructure questions are most commonly covered by [Minimum Requirements](/installation/prerequisites-and-system-requirements) and [Benchmarks](/configuration-and-operations/container-management/benchmarks). [Kubernetes](/installation/kubernetes-setup-guide) is recommended for production deployments and a customizable [helm chart](https://github.com/privateai/private-ai-helm) is available to support these installations. Lastly, your security and legal teams will want to have a look at [Security](/configuration-and-operations/container-management/security) and our [FAQ](/faq). Paying special attention to the differences in how certification works when the solution is hosted and managed entirely on-prem. # Container Quickstart Source: https://docs.getlimina.ai/container-quickstart/quickstart-guide This quickstart guide will help you get started with setting up the Limina PII de-identification Docker container on your local machine. This quickstart guide covers how to set up and run the Limina container. 日本語 If you'd like to use the Limina-hosted API, which includes the free demo, please create a [portal account](https://portal.getlimina.ai) and use the code examples provided in the portal. ## Getting Started ### Access the Customer Portal When onboarded, you'll receive a customer portal account. Log in to access your license file and container registry credentials. Contact support at [support@getlimina.ai](mailto:support@getlimina.ai) if needed. ### Download License and Pull Container 1. **Download your license file** from the portal (top right of the main page). 2. **Login to the container registry** using the command provided in the portal: ```shell theme={"theme":"poimandres"} docker login -u INSERT_UNIQUE_CLIENT_ID -p INSERT_UNIQUE_CLIENT_PW crprivateaiprod.azurecr.io ``` Container versions: `cpu`, `cpu-text`, `gpu`, `gpu-text` Use text-only versions for text processing only. See [Grabbing the Image](/installation/grabbing-the-image) for details. ## Running the Container ### CPU Version Mount your license file and run: ```shell theme={"theme":"poimandres"} docker run --rm -v "/path/to/license.json":/app/license/license.json \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid:-cpu ``` ### GPU Version Install the [Nvidia Container Toolkit](/installation/prerequisites-and-system-requirements) first, then run: ```shell theme={"theme":"poimandres"} docker run --gpus '"device=0"' --shm-size=4g --rm -v "/path/to/license.json":/app/license/license.json \ -p 8080:8080 -it crprivateaiprod.azurecr.io/deid:-gpu ``` ## Sending a test request Send text for processing: ```shell lines wrap cURL theme={"theme":"poimandres"} curl --request POST --url http://localhost:8080/process/text \ --header 'Content-Type: application/json' \ --data '{"text": ["Hello John"]}' ``` ## Next steps Check out the [Usage quickstart](/container-quickstart/api-quickstart) to learn more about the API features. Review the [Rollout checklist](/container-quickstart/deployment-checklist) for next steps on how to prepare for an initial deployment. # Named Entity Recognition Demo Source: https://docs.getlimina.ai/demo/ner Limina's web demo allows you to easily test out our industry-leading PII identification, redaction, and synthetic PII technology. To try out Limina without deploying the container, please use the Web Demo below or [talk to a solutions expert](https://getlimina.ai/en/contact-us/) to get full access.