Microsoft-365-Extractor-Suite – A Set Of PowerShell Scripts That Allow For Complete And Reliable Acquisition Of The Microsoft 365 Unified Audit Log

This suite of scripts contains two different scripts that can be used to acquire the Microsoft 365 Unified Audit Log
Read the accompanying blog post on https://invictus-ir.medium.com/introduction-of-the-microsoft-365-extractor-suite-b85e148d4bfe

1. Microsoft365_Extractor, the original script stems from the Office 365 Extractor and provides all features and complete customization. Choose this if you’re not sure what to use.
2. Microsoft365_Extractor_light, lightweight version of the Microsoft365_Extractor that requires minimal configuration and grabs all available logging for the complete period.

Microsoft 365 Extractor

This script makes it possible to extract log data out of a Microsoft 365 environment. The script has four options, which enable the investigator to easily extract logging out of an Microsoft 365 environment.

1. Show available log sources and amount of logging
2. Extract all audit logging
3. Extract group audit logging
4. Extract Specific audit logging (advanced mode)

Show available log sources and amount of logging

Pretty straightforward a search is executed and the total number of logs within the
set timeframe will be displayed and written to a csv file called “Amount_Of_Audit_Logs.csv” the file is prefixed with a random number to prevent duplicates.

Extract all audit logs

Extract all audit logs” this option wil get all available audit logs within the set timeframe and written out to a file called AuditRecords.CSV.

Extract group logging

Extract a group of logs. You can for example extract all Exchange or Azure logging in one go

Extract specific audit logs

Extract specific audit logs” Use this option if you want to extract a subset of the audit logs. To configure what logs will be extracted the tool needs to
be configured with the required Record Types. A full list of recordtypes can be found at the bottom of this page.
The output files will be writen in a directory called ‘Log_Directory” and will be given the name of their recordtype e.g. (ExchangeItem_AuditRecords.csv)

Prerequisites

– PowerShell
– Microsoft 365 account with privileges to access/extract audit logging
– An OS that supports Powershell you should be good. There are some issues with PowerShell on MacOS/Linux related to WinRM so your best option is to use Windows.

Permissions

You have to be assigned the View-Only Audit Logs or Audit Logs role in Exchange Online to search the Microsoft 365 audit log. By default, these roles are assigned to the Compliance Management and Organization Management role groups on the Permissions page in the Exchange admin center. To give a user the ability to search the Office 365 audit log with the minimum level of privileges, you can create a custom role group in Exchange Online, add the View-Only Audit Logs or Audit Logs role, and then add the user as a member of the new role group. For more information, see Manage role groups in Exchange Online.
(https://docs.microsoft.com/en-us/office365/securitycompliance/search-the-audit-log-in-security-and-compliance)

How to use Microsoft365_extractor

1. Download Microsoft365_Extractor.ps1
2. Open PowerShell navigate to the script and run it or right click on the script and press “Run with PowerShell”.
3. Select your prefered option.
4. The logs will be written to ‘Log_Directory’ in the folder where the script is located.

See example video:

How to use Microsoft365_extractor_light

1. Download Microsoft365_Extractor.ps1
2. Open PowerShell navigate to the script and run it or right click on the script and press “Run with PowerShell”.
3. Select StartDate, EndDate and Interval or use the defaults and the script will acquire all logs for the defined period.
4. The logs will be written to ‘Log_Directory’ in the folder where the script is located.

See example video:

Output

Amount_Of_Audit_Logs.csv:
Will show what logs are available and how many for each RecordType.
AuditLog.txt:
The AuditLog stores valuable information for debugging.
AuditRecords.csv:
When all logs are extracted they will be written to this file.
[RecordType]__AuditRecords:
When extracting specific RecordTypes, logs are sorted on RecordType and written to a CSV file.
The name of this file is the RecordType + _AuditRecords.

Available RecordTypes

ExchangeAdmin
ExchangeItem
ExchangeItemGroup
SharePoint
SyntheticProbe
SharePointFileOperation
OneDrive
AzureActiveDirectory
AzureActiveDirectoryAccountLogon
DataCenterSecurityCmdlet
ComplianceDLPSharePoint
Sway
ComplianceDLPExchange
SharePointSharingOperation
AzureActiveDirectoryStsLogon
SkypeForBusinessPSTNUsage
SkypeForBusinessUsersBlocked
SecurityComplianceCenterEOPCmdlet
ExchangeAggregatedOperation
PowerBIAudit
CRM
Yammer
SkypeForBusinessCmdlets
Discovery
MicrosoftTeams
ThreatIntelligence
MailSubmission
MicrosoftFlow
AeD
MicrosoftStream
ComplianceDLPSharePointClassification
ThreatFinder
Project
SharePointListOperation
SharePointCommentOperation
DataGovernance
Kaizala
SecurityComplianceAlerts
ThreatIntelligenceUrl
SecurityComplianceInsights
MIPLabel
WorkplaceAnalytics
PowerAppsApp
PowerAppsPlan
ThreatIntelligenceAtpContent
TeamsHealthcare
ExchangeItemAggregated
HygieneEvent
DataInsightsRestApiAudit
InformationBarrierPolicyApplication
SharePointListItemOperation
SharePointContentTypeOperation
SharePointFieldOperation
MicrosoftTeamsAdmin
HRSignal
MicrosoftTeamsDevice
MicrosoftTeamsAnalytics
InformationWorkerProtection
Campaign
DLPEndpoint
AirInvestigation
Quarantine
MicrosoftForms
LabelContentExplorer
ApplicationAudit
ComplianceSupervisionExchange
CustomerKeyServiceEncryption
OfficeNative
MipAutoLabelSharePointItem
MipAutoLabelSharePointPolicyLocation
MicrosoftTeamsShifts
MipAutoLabelExchangeItem
CortanaBriefing
Search
WDATPAlerts
MDATPAudit
SensitivityLabelPolicyMatch
SensitivityLabelAction
SensitivityLabeledFileAction
AttackSim
AirManualInvestigation
SecurityComplianceRBAC
UserTraining
AirAdminActionInvestigation
MSTIC
PhysicalBadgingSignal
AipDiscover
AipSensitivityLabelAction
AipProtectionAction
AipFileDeleted
AipHeartBeat
MCASAlerts
OnPremisesFileShareScannerDlp
OnPremisesSharePointScannerDlp
ExchangeSearch
SharePointSearch
PrivacyInsights
MyAnalyticsSettings
SecurityComplianceUserChange
ComplianceDLPExchangeClassification
MipExactDataMatch
MS365DCustomDetection
CoreReportingSettings
ComplianceConnector
Source:https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-schema#auditlogrecordtype

Frequently Asked Questions

If I enable mailbox auditing now can I see historical records?
No, additionaly if you enable auditing now it can take up to 24 hours before events will be logged.

I logged into a mailbox with auditing turned on but I don’t see my events?
It can take up to 24 hours before an event is stored in the UAL.

Which date format does the script accepts as input?
The script will tell what the correct date format is. For Start and End data variables it will show between brackets what the format is (yyyy-MM-dd).

Do I need to configure the time period?
No if you don’t specify a time period the script will use the default If you don’t include a timestamp in the value for the StartDate or EndDate parameters, the default timestamp 12:00 AM (midnight) is used.

What about timestamps?
The audit logs are in UTC, and they will be exported as such

What is the retention period?
Office 365 E3 – Audit records are retained for 90 days. That means you can search the audit log for activities that were performed within the last 90 days.

Office 365 E5 – Audit records are retained for 365 days (one year). That means you can search the audit log for activities that were performed within the last year. Retaining audit records for one year is also available for users that are assigned an E3/Exchange Online Plan 1 license and have an Office 365 Advanced Compliance add-on license.

What if I have E5 or other license that has more than 90 days?
Just define a manual startdate instead of the ‘maximum’ because the variable maximum is set to 90 days, which is the default for almost everyone.

Can this script also acquire Message Trace Logs?
At the moment it cannot, but there are several open-source scripts available that can help you with getting the MTL One example can be found here: https://gallery.technet.microsoft.com/scriptcenter/Export-Mail-logs-to-CSV-d5b6c2d6

Known errors

StartDate is later than EndDate
This error occurs sometimes at the final step of the script if you have not defined an endDate. Doublecheck if you have all the logs using Option 1 to validate if you have all logs. Alternative: Define an endDate

Import-PSSession : No command proxies have been created, because all of the requested remote….
This error is caused when the script did not close correctly and an active session will be running in the background. The script tries to import/load all modules again, but this is not necessary since it is already loaded. This error message has no impact on the script and will be gone when the open session gets closed. This can be done by restarting the PowerShell Windows or entering the following command: Get-PSSession | Remove-PSSession

Audit logging is enabled in the Office 365 environment but no logs are getting displayed?
The user must be assigned an Office 365 E5 license. Alternatively, users with an Office 365 E1 or E3 license can be assigned an Advanced eDiscovery standalone license. Administrators and compliance officers who are assigned to cases and use Advanced eDiscovery to analyze data don’t need an E5 license.

Audit log search argument start date should be after
The start date should be earlier then the end date.

New-PSSession: [outlook.office365.com] Connecting to remove server outlook.office365.com failed with the following error message: Access is denied.
The password/username combination are incorrect or the user has not enough privileges to extract the audit logging.

Invalid Argument “Cannot convert value” to type “System.Int32”
Safe to ignore, only observed this on PowerShell on macOS, the script will work fine and continue.