WAF Log Scrubbing XML payloads

Alex Savage 0

Hi, First poster here. I have a SOAP API that is behind an APP GW with WAF and then an APIM.

Some of the payloads are triggering built in WAF rules and causing logs to be recorded. I have configured the log scrubbing to target the named properties inside the XML body but cannot get it to redact them.

I noticed that XML is not supported for skipping built in rules which uses a similar syntax but is it supported in the log scrubbing feature?

Example XML tags

<token>

Is log scrubbing able to find these XML tags and redact the value or is it not supported? Thanks

Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-07T09:42:53.1966667+00:00
Hi @Alex Savage ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

Log scrubbing feature in Azure application gateway WAF does support customization for XML payloads. But it is limited in its ability to handle built-in rules that are triggered by those XML elements. Log scrubbing can efficiently redact specific fields in logs, provided these fields are correctly defined within the rules.

The Web Application Firewall's (WAF's) Log Scrubbing tool helps you to remove sensitive data from your WAF logs. It works by using a rules engine that allows you to build custom rules to identify specific portions of a request that contain sensitive data. Once identified, the tool scrubs that information from your logs and replaces it with ******* For your reference: https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/waf-sensitive-data-protection-configure?tabs=browser#:~:text=The%20Web%20Application,it%20with%20*******

Log scrubbing can handle XML payloads. But the application of log scrubbing for XML data may depend on how the rules are defined. For your reference: https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/waf-sensitive-data-protection#:~:text=Normally%2C%20when%20a,data%20for%20protection

Although log scrubbing is quite effective, there are restrictions when it comes to built-in WAF rules. The built-in rules may not provide the flexibility that is needed to skip certain XML tags directly. In such scenarios, if the log scrubbing feature cannot prevent triggering built-in rules, the sensitive data may still be logged in clear text before the scrubbing takes place. For your reference: https://techcommunity.microsoft.com/t5/azure-network-security-blog/azure-waf-masking-sensitive-data/ba-p/3905356#:~:text=Azure%20WAF%20log%20scrubbing%20tool%20helps%20you%20remove%20sensitive%20data%20from%20your%20WAF%20logs.%20It,custom%20rules.

While Azure WAF log scrubbing supports XML tags, if built-in rules are triggered, those logs might still get recorded before any redaction happens. It is advisable to review your rule setup to ensure that the scrubbing is effectively configured to target the specific properties within your XML payloads. If it is necessary, consider adjusting the rules to avoid triggering built-in ones when possible Fore reference: https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/waf-sensitive-data-protection-configure?tabs=browser https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/waf-sensitive-data-protection-configure?tabs=browser#enable-sensitive-data-protection

Kindly let us know if the above helps or you need further assistance on this issue.

Thanks,

Sai Prasanna.
Alex Savage 0 Reputation points

2024-10-07T09:46:33.8066667+00:00

Thank you so much for the brilliant response.
Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-08T03:51:01.19+00:00

Hi @Alex Savage ,

Thanks for the reply.

Hope you are having a great day!

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

We are happy to assist you.

Regards,

Sai Prasanna.

Alex Savage 0

We were are still unable to get the log scrubbing to work. can you please help with the syntax for the selector to find the xml tag to remove

given the following xml:

<data>
  <item>
    <username>user1</username>
    <password>password1</password>
  </item>
  <item>
    <username>user2</username>
    <password>password2</password>
  </item>
  <item>
    <username>user3</username>
    <password>password3</password>
  </item>
  <item>
    <username>user4</username>
    <password>password4</password>
  </item>
  <item>
    <username>user5</username>
    <password>password5</password>
  </item>
</data>

What would the log scrubbing selector look like to remove the values of the passwords please?

We have tried password and it does not remove the values

Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-09T00:49:55.73+00:00
Hi @Alex savage

Thanks for getting back.

To remove the passwords, you need to specify the XPath expression that targets the <password> elements. The correct selector syntax would be: '//item/password' Please try the above and if it doesn't work out, feel free to reach out to us so that we can help you further. We are pleased to help you. Thanks, Sai Prasanna.
Alex Savage 0 Reputation points

2024-10-09T06:35:30.8766667+00:00
thank you. we did wonder if XPath was available. Is it mentioned in the docs please? Also please can you confirm that it would be a "Post arg name" field.

I cant see it being any of the other ones

//Field names from https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/waf-sensitive-data-protection#:%7E:text=Normally%2C%20when%20a,data%20for%20protection%22https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/waf-sensitive-data-protection#:%7E:text=normally%2c%20when%20a,data%20for%20protection%22

IP address

Request header name

Request cookie name

Request args name

Post arg name

JSON arg name
Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-10T06:45:15.47+00:00
Hi @Alex savage

Greetings!

Please make sure to enter the selector in the "Post arg name" field, as this is typically where you would specify the names of the fields that you want to scrub or modify in your log data.

Also, could you please share which log scrubbing tool you are using so that we can assist you better.

Unfortunately, I couldn't find the exact documentation related to XPath, but I have found some alternate solutions you can try if that doesn't work out.

Using a keyword-based approach:

1 password|pass|passwd|auth_token|api_key|secret

This selector uses a pipe [|] to separate multiple keywords that are commonly used for password fields. You can add or modify these keywords as needed to match your specific use case.

Using a more generic approach with XPath-like syntax:

1 *[contains(lower-case(@name), 'pass') or contains(lower-case(@value), 'pass')]

This selector uses a wildcard to match any element, and the "contains" function checks if the specified string is present in the element's name or value.

Using a regular expression-based approach:

1 (?i)password|pass|passwd|auth_token|api_key|secret

This selector uses a regular expression with the "(?i)" flag to make the match case insensitive. You can modify the regular expression to match your specific use case.

Please note that these selectors might not catch all possible password fields, and you may need to adjust them based on your specific log data and requirements.

Based on the XML structure you provided; I would suggest using the following log scrubbing selector to remove the values of the passwords:

1 //password

This selector uses XPath-like syntax to match the "password" elements in the XML. The // notation is used to search for elements anywhere in the document, regardless of their position.

If this selector does not work, you can try using a more specific selector that takes into account the parent elements:

1 //item/password

This selector matches the password elements that are direct children of the item elements.

If you are still having trouble, you can try using a selector that matches the "password" elements based on their attribute values:

1 //*[local-name()='password']

This selector uses the "local-name() " function to match elements with the local name "password," regardless of their namespace.

Kindly let us know if the above helps or you need further assistance on this issue.
Alex Savage 0 Reputation points

2024-10-10T06:55:11.9066667+00:00

Hi, to confirm this is the built in log scrubbing feature inside the Azure APP GW WAF
Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-14T02:00:01.6666667+00:00

Hi @Alex Savage,

Greetings!

Just following up to see if you had a chance to review the above response and test it out.

If you have any questions, please feel free to ask.

We are pleased to help you.

Thanks,

Sai Prasanna.
Alex Savage 0 Reputation points

2024-10-14T06:44:30.6166667+00:00

I was a little confused by the above code snips being honest. The only interface we have is the Azure APP GW WAF Policy log scrubbing one nothing on ruby or sql. Is there anyway we can ask the team for confirmation on the behaviour please?
Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-14T07:04:34.67+00:00

Hi @Alex Savage,

To troubleshoot further, we will need a specialized 1:1 session, where a support engineer can help you further. If you have a support plan, please file a support ticket for deeper investigation and do share the SR with us, else please do let us know, we will try and help you get a one-time free technical support.

P.S: We will be able to provide you one-time support if and only if your subscription is not managed by a CSP.

Thanks,

Sai Prasanna.
Sai Prasanna Sinde 1,260 Reputation points Microsoft Vendor

2024-10-16T00:46:17.0233333+00:00

Hi @Alex Savage,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

I value your time and look forward to hearing from you on this.

Regards,

Sai Prasanna.

Share via

WAF Log Scrubbing XML payloads

Your answer