The Microsoft Defender for Office 365 Overview dashboard

The Microsoft Defender for Office 365 Overview dashboard in the Microsoft Defender portal shows the effectiveness of Defender for Office 365. You open the dashboard in the Defender portal at Email & collaboration > Overview, or directly at https://security.microsoft.com/emailandcollaborationoverviewreport.

The information on the Overview page is organized into the following areas:

• Defender for Office 365 summary
• Optimize section
• Risky allows section
• Compare solutions section
• Insights section

For the permissions required to view the dashboard and reports, see What permissions are needed to view these reports?.

By default, the data on the page is shown for the last 30 days. But, you can show data for the last 60 days or the last 90 days by selecting the Last 30 days drop down at the top of the page.

Defender for Office 365 summary

The information in the Defender for Office 365 summary at the top of the page is described in the following subsections.

Phish / Malware Efficacy card

The graph on the Phish / Malware Efficacy card visually represents the protection given by Defender for Office 365 against phishing and malware in email messages:

• Pre-delivery: Items detected before they reach the recipient's mailbox.
• Post-delivery: Items removed after the item was delivered to the recipient's mailbox via zero-hour auto purge (ZAP).
• Uncaught: Delivered items that ZAP identified but couldn't remove. For example:
  • Admin deletions or remediations.
  • Admin submissions to Microsoft identifying the message as malware or phishing.
  • User deletions.
  • Non-Microsoft security provider deletions.

The percentage value is the number of messages in each category divided by the total number of malicious malware and phishing email during the review period you selected at the top of the page (30 days (default), 60 days, or 90 days).

Hover over a category in the chart to see the number of messages in each category for the review period. Hover over the percentage to see the total number of messages

Threat detections card

The graph on the Threat detections card shows the number of messages detected by the following technologies during the review period you selected at the top of the page (30 days (default), 60 days, or 90 days):

• Malware: The breakdown of detection technologies is available in the Threat protection status report under View data by Email > Malware and Chart breakdown by Detection Technology.

• Phish: The breakdown of detection technologies is available in the Threat protection status report under View data by Email > Phish and Chart breakdown by Detection Technology.

• Spam: The breakdown of detection technologies is available in the Threat protection status report under View data by Email > Spam and Chart breakdown by Detection Technology.

• Impersonation: Email detected as one of the following threats:

  • Impersonation brand
  • Impersonation user
  • Impersonation domain
  • Mailbox intelligence impersonation

  The breakdown of detections is available in the Threat protection status report user View data by Email > Phish and Chart breakdown by Detection Technology.

• Detonation: Messages detected by opening files and URLs in a virtual environment (detonation) by Safe Links and Safe Attachments in Defender for Office 365. The breakdown of detection technologies is available in the Threat protection status report under:

  • View data by Email > Malware and Chart breakdown by Detection Technology
  • View data by Email > Phish and Chart breakdown by Detection Technology.

• Zap: Harmful email messages detected and removed after delivery by zero-hour auto purge (ZAP). More information is available in the Post-delivery activities report.

• Non-Microsoft Solutions: Detections by non-Microsoft solutions.

Hover over a category in the chart to see the number of messages in each category for the review period.

Optimize section

The information in the Optimize section is described in the following subsections.

Priority accounts card

The graph on the Priority accounts card shows how many message senders you designated as priority accounts to receive tailored protection. For more information about priority accounts, see Manage and monitor priority accounts.

To designate more accounts, select Add employees to go to the Priority accounts page in the Microsoft 365 admin center at https://admin.cloud.microsoft/?#/priorityaccounts.

Hover over a category in the chart to see the number of Onboarded priority accounts and Open slots. The maximum number of priority accounts is 250.

Posture recommendations card

The graph on the Policy recommendations card shows the number of users directly protected by Safe Links and Safe Attachments policies as a percentage of the total number of users (the value 100% means everyone is protected). The numbers are taken from whether the following recommended actions in Microsoft Secure Score have the Status value Completed:

• Safe Links:
  • Ensure Safe Links for Office applications is enabled
  • Create Safe Links policies for email messages
• Safe Attachments:
  • Turn on Safe Attachments in block mode
  • Ensure Safe Attachments policy is enabled

Hover over a category in the chart to see the number of Impacted users (the total number of users in the organization) and Protected users (users protected by Safe Links or Safe Attachment policies as defined by the recommended actions in Microsoft Secure Score).

Notes:

• The Built-in protection preset security policy gives a basic level of Safe Links and Safe Attachments protection to all users by default (and you can exclude recipients).
• Users get a higher level of Safe Links and Safe Attachments protection from either of the following settings:
  • Turn on the Standard or Strict preset security policies, and make sure the users are included in Defender for Office 365 protection.
  • Create custom Safe Links policies or Safe Attachment policies with the users as members.

Risky allows section

The information in the Risky allows section is described in the following subsections.

Messages allowed card

The Messages allowed card shows the count of messages allowed by user or organization allow entries that bypass protection:

• The total number of messages allowed.
• Phish messages allowed.
• Spam messages allowed.
• Good messages allowed.

Tenant allow types card

The Tenant allow types card shows a table with the types of allow entries in the Tenant Allow/Block List that let bad mail get delivered to user mailboxes:

• Allow type column:
  • URL
  • File hash
  • Sender
  • IP allow
• Messages allowed column: The number of messages allowed for the review period you selected at the top of the page (30 days (default), 60 days, or 90 days).

Exchange transport rules card

The Exchange transport rules card shows the mail flow rules (also known as transport rules) that allowed messages that would otherwise be blocked:

• Rule ID
• Messages allowed: The number of messages allowed during the review period you selected at the top of the page (30 days (default), 60 days, or 90 days).

Select Review rules to go to the Rules page in the Exchange admin center (EAC) at https://admin.cloud.microsoft/exchange#/transportrules.

Compare solutions section

The information in the Compare solutions section is described in the following subsections.

Email detections card

The graph on the Email detections shows Microsoft and non-Microsoft detections as part of ICES Vendor Ecosystem integration:

• Defender mail flow detections
• Defender post-delivery detections
• Non-Microsoft post-delivery detections
• Duplicate detections Duplicate post-delivery detections

Hover over a category in the chart to see the number of messages in each category for the review period you selected at the top of the page (30 days (default), 60 days, or 90 days).

Non-Microsoft detections card

The graphs on the Non-Microsoft detections show the following information for non-Microsoft detections as part of ICES Vendor Ecosystem integration:

• Post delivery detections graph:

  • Malware
  • Phish
  • Spam

  Hover over a category in the chart to see the number of messages in each category for the review period you selected at the top of the page (30 days (default), 60 days, or 90 days).

• Efficacy graph: Shows the unique detections by the non-Microsoft service as a percentage of the total detections by Defender for Office 365.

Insights section

The information in the Insights section is described in the following subsections.

Top trending attacks card

The graph on the Top trending attacks card shows the most encountered phishing attack types by volume for the review period you selected at the top of the page (30 days (default), 60 days, or 90 days).

Threat classification in Defender for Office 365 uses advanced technologies such as large language models (LLMs), small language models (SLMs), and machine learning (ML) models to automatically detect and classify email-based threats.

The breakdown of phishing attack types is also available in the Threat protection status report under View data by Email > Phish and Chart breakdown by Threat classification.

Emerging threats card

The Emerging threats card shows any notable campaigns observed by Microsoft security researchers.

Microsoft 365 Secure Email Gateway performance card

The graph on the Microsoft 365 Secure Email Gateway performance card compares the effectiveness of Defender for Office 365 against other secure email gateways. To ensure fairness, the number of missed messages is normalized per 1,000 active users.

Appendix: Advanced hunting efficacy query in Defender for Office 365 Plan 2

Organizations with Defender for Office 365 Plan 2 can use the following query in advanced hunting to generate the same data on the Phish / Malware Efficacy card.

// This query by default will take the last 30 days of data. 
// The query and calculation can be tweaked to meet individual needs, and will update over time to get incrementally more accurate. 
// Ben Harris - Microsoft Defender for Office 365 PM. 
let _startTime = ago(30d); 
let _endTime = now(); 
// Get all mailflow detected as clean at time of delivery 
let EmailEventsClean = materialize( 
    EmailEvents 
    | where Timestamp between (_startTime .. _endTime) and EmailDirection == "Inbound" 
    | where ThreatTypes !contains "Phish" and ThreatTypes !contains "Malware" 
    | project NetworkMessageId,ThreatTypes 
); 
// Get all mailflow detected as phish or malware at time of delivery 
let EmailEventsThreats = materialize( 
    EmailEvents 
    | where Timestamp between (_startTime .. _endTime) and EmailDirection == "Inbound" 
    | where ThreatTypes contains "Phish" or ThreatTypes contains "Malware" 
    | extend MDO_detection = parse_json(DetectionMethods) 
    | extend FirstDetection = iif(isempty(MDO_detection), "Clean", tostring(bag_keys(MDO_detection)[0])) 
    | extend FirstSubcategory = iif(FirstDetection != "Clean" and array_length(MDO_detection[FirstDetection]) > 0, strcat(FirstDetection, ": ", tostring(MDO_detection[FirstDetection][0])), "No Detection (clean)") 
    | project NetworkMessageId,FirstDetection,FirstSubcategory,MDO_detection,ThreatTypes 
); 
// Get all post delivery ZAP / Redelivery events, and arg_max them to ensure we have the latest verdict to work with for each 
let EmailPostDeliveryFiltered = materialize( 
    EmailPostDeliveryEvents 
    | where Timestamp between (_startTime .. datetime_add('day', 7, _endTime)) 
    | where ActionType in ("Malware ZAP","Phish ZAP","Redelivery") 
    | extend Key = strcat(NetworkMessageId , "-", RecipientEmailAddress) 
    | summarize arg_max(Timestamp, *) by Key 
    | project Action,ActionType,ActionResult,ThreatTypes,NetworkMessageId 
); 
// Optional - get all admin submissions for malware or phish, so we can also count these in the miss bucket. 
let CloudAppEventsFiltered = materialize( 
    CloudAppEvents 
    | where Timestamp between (_startTime .. datetime_add('day', 7, _endTime)) 
    | where ActionType == "AdminSubmissionSubmitted" 
    | extend SubmissionType = tostring(parse_json(RawEventData).SubmissionType) 
    | extend NetworkMessageId = tostring(parse_json(RawEventData).ObjectId) 
    | where SubmissionType in ("1", "2") 
    | project SubmissionType,NetworkMessageId 
); 
// get the number of threats caught in mailflow 
let Mal_Phish_Mailflow = toscalar( 
    EmailEventsThreats 
    | summarize count() 
); 
// get the number of threats caught in mailflow which turned out to be false positives (FPs) so we can correct the calculation 
let FP_ZAP = toscalar( 
    EmailPostDeliveryFiltered 
    | where ThreatTypes !contains "Phish" and ThreatTypes !contains "Malware" and ActionType == "Redelivery" 
    | join kind=leftsemi (EmailEventsThreats) on NetworkMessageId 
    | summarize count() 
); 
// get the number of threats successfully cleaned up post delivery, ignoring where administrative policy stopped action 
let FN_ZAP_Successful = toscalar( 
    EmailPostDeliveryFiltered 
    | where ActionType in ("Malware ZAP","Phish ZAP") and ActionResult in ("Success","AdminPolicy") 
    | join kind=leftsemi (EmailEventsClean) on NetworkMessageId 
    | summarize count() 
); 
// get the number of threats unsuccessfully cleaned up post delivery. 
let FN_ZAP_Unsuccessful = toscalar( 
    EmailPostDeliveryFiltered 
    | where ActionType in ("Malware ZAP","Phish ZAP") and ActionResult !in ("Success","AdminPolicy") 
    | join kind=leftsemi (EmailEventsClean) on NetworkMessageId 
    | summarize count() 
); 
// join the administrative submissions to clean mailflow to find the additional miss 
let FN_Admin_Submissions = toscalar( 
    CloudAppEventsFiltered 
    | join kind=rightsemi (EmailEventsClean) on NetworkMessageId 
    | summarize count() 
    ); 
    // print each result, and run the calculation to work out effectiveness at time of delivery and post delivery. 
union withsource=Table 
    (print StatisticName="Mal/Phish Mailflow totals - Minus FPs", Value=toreal(Mal_Phish_Mailflow) - toreal(FP_ZAP)), 
    (print StatisticName="Admin Mal/Phish FNs Submitted", Value=toreal(FN_Admin_Submissions)), 
    (print StatisticName="Mal/Phish FPs Reverse Zapped", Value=toreal(FP_ZAP)), 
    (print StatisticName="Mal / Phish Successfully Zapped", Value=toreal(FN_ZAP_Successful)), 
    (print StatisticName="Mal / Phish UN-Successfully Zapped", Value=toreal(FN_ZAP_Unsuccessful)), 
    (print StatisticName="Effectiveness Post Delivery", Value=abs(round(((toreal(FN_Admin_Submissions)+toreal(FN_ZAP_Unsuccessful))/(toreal(Mal_Phish_Mailflow)+toreal(FN_ZAP_Successful)+toreal(FN_ZAP_Unsuccessful)+toreal(FN_Admin_Submissions)-toreal(FP_ZAP))*100-100),2))), 
    (print StatisticName="Effectiveness Pre-Delivery", Value=abs(round(((toreal(FN_Admin_Submissions)+toreal(FN_ZAP_Unsuccessful)+toreal(FN_ZAP_Successful))/(toreal(Mal_Phish_Mailflow)+toreal(FN_ZAP_Successful)+toreal(FN_ZAP_Unsuccessful)+toreal(FN_Admin_Submissions)-toreal(FP_ZAP))*100-100),2))) 
| project StatisticName, Value