Hash and upload the sensitive information source table for exact data match sensitive information types

This article shows you how to hash and upload your sensitive information source table.

Applies to

• Create exact data match sensitive information type new experience
• Create exact data match sensitive information type classic experience

Hash and upload the sensitive information source table

In this phase, you:

1. Set up a custom security group and user account.
2. Set up the Exact Data Match (EDM) Upload Agent tool.
3. Use the EDM Upload Agent tool to hash, with a salt value, the sensitive information source table, and upload it.

You can hash and upload your sensitive data using either the Two-computer method or the Single computer method as described in Hash and upload your data. Best practice is to use two computers to separate the processes of hashing and uploading your sensitive data. Separating the steps across two computers helps ensure that your actual data is never available in clear text form on a computer that might be compromised due to its connection to the internet. This also makes isolating any issues you encounter easier to identify.

Prerequisites

Technology requirements

• A work or school account for Microsoft 365. This account must be added to the EDM_DataUploaders security group.
• A computer with one of the following operating systems. This computer runs the EDM Upload Agent.
  • Windows 11
  • Windows 10
  • Windows Server 2016 with .NET version 4.6.2
  • Windows Server 2019
  • Windows Server 2022
• A directory on the computer you use for uploading your data. This directory contains:
  • The EDM Upload Agent.
  • Your sensitive information data file in .csv, .tsv, or pipe (|) format. By default, the EDM Upload Agent expects your data file to be in .csv format. > [!TIP]

    You can use a file with data that is separated by tabs or pipes (instead of commas), by indicating either the "(Tab)" or "(|)" options with the /ColumnSeparated parameter. For example: EdmUploadAgent.exe /UploadData /DataStoreName PatientRecords /DataFile C:\Edm\Hash\PatientRecords.csv /HashLocation C:\Edm\Hash /Schema edm.xml /AllowedBadLinesPercentage 5

  • The output has and salt files that are created when completing the hash procedure.
  • The datastore name from the edm.xml file. Our example uses PatientRecords.

Security group and user account requirements

1. As a global administrator, go to the admin center using the appropriate link for your subscription and create a security group called EDM_DataUploaders.

2. Add one or more users to the EDM_DataUploaders security group. (These users are the ones who manage the database of sensitive information.)

Exact Data Match Schema

If you used the EDM schema and sensitive information type tool for the new experience or the EDM sensitive information type/rule package for the classic experience, you must download that schema to hash your sensitive information source table. For more information, see Exporting the EDM schema file in XML format.

To download this EDM schema, open a Command Prompt window and run the following command:

EdmUploadAgent.exe /SaveSchema /DataStoreName <schema name> /OutputDir <path to output folder>

Data formatting requirements

Before you hash and upload your sensitive data, run a search for any special characters in the table that might cause problems in parsing the content.

You can validate that the table is in a suitable format by using the EDM Upload Agent with the following syntax:

EdmUploadAgent.exe /ValidateData /DataFile [data file] /Schema [schema file]

Common formatting issues

1. Mismatched number of columns: This issue can be due to the presence of commas or quote characters within values in the table that EDM interprets as column delimiters. Unless they're surrounding a whole value, single and double quotes can cause the tool to misidentify the start and end of individual columns.
2. Single quote characters or commas inside a value: For example, if a person's name includes a single quote such as Tom O'Neil or a city's name starts with an apostrophe such as 's-Gravenhage, you need to modify the data export process used to generate the sensitive information table and surround such columns with double quotes.
3. Double quote characters inside values: Best practice is to use the tab-delimited format for the table. Tab-delimited tables are less susceptible to such issues.

Hash and upload your data

EDM and double-byte character set languages

Exact data match supports double-byte characters, such as those used in Chinese, Japanese, and Korean. However, it doesn't support string matches for corroborative evidence encoded as double byte characters. Neither does it match multi-token CJK text detected in the classified content, unless globalization for EDM is enabled as described later in this document. In all cases, a SIT must be mapped to any multi-token text, both for the primary field and for corroborative evidence fields.

To invoke exact data matching for double-byte characters, take the following steps:

1. Create an EDM Sensitive Information Type (SIT) configured to match on the double-byte character set language, such as Japanese kanji.
2. Ensure that you download and install version 17.01.0495.0 (or later) of the EDM Upload Agent
3. Update the EdmUploadAgent.exe.config file’s globalization parameter to true: <add key=" IsGlobalizationEnabled" value="true">
4. Hash and upload a source table with the data to be matched.

Next steps

• For the new experience: Test an exact data match sensitive information type

• For the classic experience: Create exact data match sensitive information type/rule package