Hi Splunk Community, I’ve set up Azure Firewall logging, selecting all firewall logs and archiving them to a storage account (Event Hub was avoided due to cost concerns). The configuration steps tak...
See more...
Hi Splunk Community, I’ve set up Azure Firewall logging, selecting all firewall logs and archiving them to a storage account (Event Hub was avoided due to cost concerns). The configuration steps taken are as follows: Log Archival: All Azure Firewall logs are set to archive in a storage account Microsoft Cloud Add-On I added the storage account to the Microsoft Cloud Add-On using the secret key with the following permissions: Input/Action API Permissions Role (IAM) Default Sourcetype(s) / Sources Azure Storage Table Azure Storage Blob N/A Access key OR Shared Access Signature: - Allowed services: Blob, Table - Allowed resource types: Service, Container, Object - Allowed permissions: Read, List N/A mscs:storage:blob (Received this) mscs:storage:blob:json mscs:storage:blob:xml mscs:storage:table We are receiving events from the source files in JSON format, but there are two issues: Field Extraction: Critical fields such as protocol, action, source, destination, etc., are not being identified. Incomplete Logs: Logs appear truncated, starting with partial data (e.g., “urceID:…”) and missing “Reso,” which implies dropped or incomplete events (As far as I understand) Few logs were received compared to the traffic on Azure Firewall. Attached is a piece of logs showing errors as mentioned in the question. ________________________________________________________________ Environment Details: • Log Collector: Heavy Forwarder (HF) hosted in Azure.
• Data Flow: Logs are being forwarded to Splunk Cloud Questions: Can it be an issue with using storage accounts and not event-hub? Could the incomplete logs be due to a configuration issue with the Microsoft Cloud Add-On or possibly related to the data transfer between the storage account and Splunk? Has anyone encountered similar issues with field extraction from Azure Firewall JSON logs? Ultimate Goal: Receive Azure Firewall Logs with fields extracted as any other firewall logs received by Syslog (Fortinet for example) Any guidance or troubleshooting suggestions would be much appreciated!