Redact Sensitive PII

Search…

In this tutorial, we will create a transform policy to identify and redact or replace PII with fake values. We will then use the CLI to transform a dataset and examine the results.

Sample configuration
Save your configuration to a local file named redact_pii.yaml. Click the link to see all supported info types. The policy below searches for sensitive PII values as defined by Experian (including a custom regex for user IDs), replacing them with fake values when possible, or redacting with a user-defined character.
1
schema_version: "1.0"
2
name: "Redact PII"
3
models:
4
  - transforms:
5
      data_source: "_"
6
      policies:
7
        - name: remove_pii
8
          rules:
9
            - name: fake_or_redact_pii
10
              conditions:
11
                value_label:
12
                  - person_name
13
                  - credit_card_number
14
                  - phone_number
15
                  - us_social_security_number
16
                  - email_address
17
                  - custom/*
18
              transforms:
19
                - type: fake
20
                - type: redact_with_char
21
                  attrs:
22
                    char: X
23
label_predictors:
24
  namespace: custom
25
  regex:
26
    user_id:
27
      patterns:
28
        - score: high
29
          regex: "user_[\\d]{5}"
Copied!
Save the sample dataset below to pii.csv
1
id,name,email,phone,visa,ssn,user_id
2
1,Kimberli Goodman,[email protected],228-229-2479,5108758325678962,108-08-9132,user_93952
3
2,Anna Jackson,[email protected],611-570-4635,5048377302905174,256-28-0041,user_23539
4
3,Sammy Bartkiewicz,[email protected],799-160-2165,5108758273775281,849-46-5175,user_35232
5
4,Matt Parnell,[email protected],985-733-6433,5048376551569087,774-83-5725,user_23529
6
5,Meredith Myers,[email protected],545-861-4923,5108752255128478,180-65-6855,user_92359
Copied!
Create a transformation model
First, create a project to host your transformation models and artifacts.
1
gretel projects create --display-name redact-pii --set-default
Copied!
Next, train your transformation model on your dataset or one with an identical schema.
Currently, only plain text and CSV formats are supported by the Transform API. JSON support is coming soon.
1
gretel models create --config redact_pii.yaml --in-data pii.csv --runner cloud > model-data.json
Copied!
Redact sensitive data
Your model can now be used to redact sensitive data from any dataset with a similar structure or schema.
1
gretel records transform --model-id model-data.json --in-data pii.csv --runner cloud --output .
Copied!
Examine the results
Transform results are downloaded to the local directory in CSV format to the file data.gz. Our policy is set to replace names, addresses, and emails with fake entities, and to redact the user ID regular expression with a character replacement. 
Let's examine the transformed results from the command line.
1
zcat data.gz | column -s, -t
2
​
3
id  name               email                            phone                   visa              ssn          user_id
4
1   Samantha Sandoval  [email protected]               986.089.1149            344661707423210   102-40-4854  XXXX_XXXXX
5
2   Shannon Holmes     [email protected]  (686)646-3171           3519277724227055  554-61-8106  XXXX_XXXXX
6
3   David Chapman      [email protected]   001-946-130-7514x76773  213182470523001   008-06-5773  XXXX_XXXXX
7
4   Crystal Russo      [email protected]               027-327-7306x07952      6011379376191328  628-27-4071  XXXX_XXXXX
8
5   John Allen         [email protected]              (365)502-6954           4047982390743587  740-42-9239  XXXX_XXXXX
Copied!
Next steps
For use cases such as training machine learning models on customer support logs, it is often desirable to replace PII with fake values to maintain semantics in the original data. However, this is not always desirable. Try updating the transformation policy to simply redact all sensitive values with an "*" character.
Video walkthrough
​

Transforms - Previous

CLI Examples

Data De-identification

Last modified 3mo ago

Copy link

Contents

Sample configuration

Create a transformation model

Redact sensitive data

Examine the results

Next steps