Ingest historical data into your target platform

To ingest exported historical SIEM data into a target platform, first ensure you have chosen a target platform for historical data, selected a data transfer tool for migration ingestion, and stored the data in a staging location.

This article describes how to export data from your legacy SIEM and ingest historical data into Microsoft Sentinel data lake, Azure Data Explorer, or Azure Blob Storage.

Export data from the legacy SIEM

In general, SIEMs can export or dump data to a file in your local file system, so you can use this method to extract the historical data. It’s also important to set up a staging location for your exported files. The tool you use to transfer the data ingestion can copy the files from the staging location to the target platform. For information about the tools you can use for data ingestion, see Select a target Microsoft platform to host the exported historical data.

This diagram shows the high-level export and ingestion process.

To export data from your current SIEM, see one of the following sections:

• Export data from ArcSight
• Export data from Splunk
• Export data from QRadar

Ingest data to Microsoft Sentinel data lake

Microsoft Sentinel data lake is the native data layer of the Microsoft Sentinel platform. It's the simplest path to a unified, queryable history of your security data inside Microsoft Sentinel and the recommended platform for long-term data retention.

To ingest your historical data into Microsoft Sentinel data lake (as shown in the export and ingestion process diagram), use the Custom Log Ingestion tool.

Ingest to Azure Data Explorer (ADX)

Azure Data Explorer (ADX) is a fast, scalable data exploration service. To ingest your historical data into ADX:

1. LightIngest supports Windows only. Install and configure LightIngest on the Windows system where logs are exported, or install LightIngest on another Windows system that has access to the exported logs.
2. If you don't have an existing ADX cluster, create a new cluster and copy the connection string. Learn how to set up ADX.
3. In ADX, create tables and define a schema for the CSV or JSON format (for QRadar). Learn how to create a table and define a schema using sample data or create a table and define a schema without sample data.
4. Run LightIngest with the folder path that includes the exported logs as the path, and the ADX connection string as the output. When you run LightIngest, ensure that you provide the target ADX table name, that the argument pattern is set to *.csv, and the format is set to .csv (or json for QRadar).

Ingest to Azure Blob Storage

To ingest your historical data into Azure Blob Storage:

1. Install and configure AzCopy on the system to which you exported the logs. Alternatively, install AzCopy on another system that has access to the exported logs.
2. Create an Azure Blob Storage account and copy the authorized Microsoft Entra ID credentials or Shared Access Signature token.
3. Run AzCopy with the folder path that includes the exported logs as the source, and the Azure Blob Storage connection string as the output.

Next step