Ingesting Azure Logs (IaaS, PaaS, OS, app, etc.)

[ericl42]

I know there have been a lot of conversations around this topic and technology is constantly changing to make things easier, so I was curious if anyone has had recent experience ingesting all Azure logs to an on-premise Splunk instance.

We're looking at both keeping logs local to Azure and utilize Sentinel but we'd prefer to keep them in Splunk so we have one location for logs no matter where they are (i.e. Azure, AWS, on-premise, etc.) 

Potential solutions:

1. Install a forwarder on every VM and back haul the traffic across the VPN to on-premise indexers.
   • I don't want to pay the VPN costs for this or have that dependency. it's also just not very "cloud-ish" solution.
2. Stand up indexers in Azure and replicate the clusters in Azure & on-premise.
   • These VMs cost a lot of money.
3. Send all logs to Event Hub and use Splunk to pull from there.
   • This seems like a decent solution but I'm not sure of the costs or parsing issues this may entail. 

I would love to hear how others compared Sentinel to Splunk and justified sticking with Splunk in Azure when you had an on-premise Splunk architecture.

Note that we want the infrastructure/platform logs but have a hard requirement to get the OS and app logs (i.e. Windows security, RHEL /var/log/secure, Apache, Squid proxy, etc.)

Thanks!

[Priyankakumari1]

Have you got any answer for this??

[ericl42]

I ended up going down the Event Hub route. You can use the Azure Diagnostic agent to push all Linux/Windows logs to EH and then use Microsoft Azure Add-on for Splunk to then ingest those logs into Splunk.

This allows the traffic to not go over the VPN connection and just pay for back egress traffic outbound to Azure. Since it's SSL, it should do some encryption and save you a few costs.

You can then modify the transforms and props files to help divide the logs up to the appropriate indexes and what not.

[pkolhatk]

HI Cna you share any reference document