How to ensure logs delivery to Splunk

[splunky_diamond]

Hello Splunkers!

Imagine a scenario:

There is a test environment with Splunk being deployed in ubuntu-server 20.04 virtual machine as All-in-One deployment scenario. 

There is a Windows Server 2019, that is sending WindowsEvent Logs from Application and Security using Splunk Universal forwarder along with Splunk add-on for Microsoft Windows. In the normal situation where there is a stable network connection between Windows Server 2019 and ubuntu machine with Splunk, the logs are delivered to Splunk with no problems.

However, imagine there is an adversary who executed a script to disable the network connection on the Windows Server 2019 and performed some malicious actions on that machine and then, went to event viewer application and cleared the security logs so that they never reach Splunk.

My question is, how can we make the Security Logs that were deleted by adversary on Windows Server 2019 through Event Viewer, still reach the Splunk? To clarify, let's say after adversary disabled the network access to Splunk, and then deleted some users in the domain controller, then cleared the Security logs in the event viewer. What can we do, so that we still get these logs of adversary's activity on Windows Server in Splunk for further investigation?

Feel free to ask any additional questions in case this scenario is unclear at some parts. 

Thanks in advance for taking your time reading and replying to this post! 

[gcusello · Accepted solution]

Hi @splunky_diamond ,

for my knowledhe, the only limit of Splunk is that logs must reach splnk and be searcheable!

If you cannot be sure about this there isn't any internal Splunk solution.

If the network connection between UF and Splunk is interrupted, UF locally stores logs for some time and, when the connection, is again available, it sends all logs to Splunk, but an attacker could delete also Splunk temp files, so you cannot do nothing. 

You can be informed that there could be an attack when the data flow is interrupted and when, after the network connection is again available, windows logs are missed.

In other words, Splunk save your data for a while, to avoid data loss during network issue, but these logs could be deleted by an attacker.

The only way is to be notices that there could ne an attack.

Ciao.

Giuseppe

[PickleRick]

Yup. If the threat actor has control over the machine, it could - for example - completely delete the Splunk forwarder from the computer so you cannot be sure of anything after such situation happened (I've seen very sensitive setups where events - not necessarily Windows Event Logs but the general idea is the same - were printed out onto a printer as a non-modifiable medium so that they couldn't be in any way changed after they had been created).

For normal situations where you expect a network downtime from time to time (like sites with unstable network connections, mobile appliances and so on), you can tweak your forwarder's buffer sizes so that it can hold the data back for the needed period of time and then send the queued data when it regains downstream connectivity.

Be aware thought that such setup will create a host of potential problems resulting from the significant lag between the time the event is produced and the time it's being indexed. They can be handled but it needs some preparation and tweaking some limits.

[splunky_diamond]

Thank you very much for your reply, @gcusello !

I have some questions to your post, where can I configure for how long UF stores the logs when the connection is interrupted? Also how can I know the location of where UF stores these logs, is it some file within the add-on? And finally, what's the capacity of that file/those files, where the logs will be stored in this scenario before the connection to Splunk machine is re-established? 

[gcusello]

Hi @splunky_diamond ,

you can use the parameter maxQueueSize in outputs.conf.

about the location,, I'm not sure, but it should be in the $SPLUNK_HOME/var/run/spunk

let me know if I can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Ciao.

Giuseppe

[splunky_diamond]

Hello, @gcusello , thanks for the additional information.

I tested this case in my lab environment and it worked! I just want to clarify some small details. I have added the maxQueueSize in the /SplunkUniversalForwarder/etc/apps/SplunkUniversalForwarder/local outputs.conf, for I have configured that file in that path before in order to send logs to Splunk, but I also found this article 
Howto configure SPLUNK Universal Forwarder (kura2gurun.blogspot.com) , where it says that we should configure outputs.conf file, located at /opt/splunkforwarder/etc/system/local/. 

Is there any impact or difference that I didn't configure outputs in that specific path, but instead did it in the one that I mentioned above?

Cheers,

SplunkyDiamond

[PickleRick]

This is the document you might want to read to understand how Splunk reads the configs.

Also @gcusello 's remark about system/local vs. configured in app is valid.

[gcusello]

Hi @splunky_diamond ,

the only difference is that, if you locate the server.conf in the $SPLUNK_HOME/etc/system/local, you cannot manage it using a Deployment Server, if instead you put this file in one app deployed by DS, you can apply updated and modified configurations using the DS.

There isn't any othe difference.

Ciao.

Giuseppe