Is it recommended to have trendmicro or any antivirus module running on Splunk large scale setup

[ramanapvr]

Dear All,

Can any one please let me know if it recommended to have Trendmicro or any of the Antivirus module running on Splunk large scale setup we are having. We are having a retention of 60 days of data which is ~1.7 TB. Understood that we can run antivirus module on Splunk server by excluding the Splunk partitions and considering other OS partitions. But am scared that it would effect on performance of Splunk.

Can any one give more insight on this perspective pls.

Thanks and Regards,
Peri

[ramanapvr]

Dear Jtacy,

Thanks a lot for the update and your time here. However my question is that we have Splunk cluster architecture with Master, Search Head, Indexer1 and Indexer2 where heavily used from searches perspective, application log monitoring base by alerts created and finally the dashboards created for operations monitoring.

Our servers are having Redhat Linux OS. Now the question is that is it feasible to run antivirus Trendmicro which does real time scanning on servers. Or is it feasible to run Antivirus Trendmicro on OS partitions excluding the Splunk data partitions. Or is it feasible to consider Trendmicro to run on Indexers machines + search head and exclude Master server. Please be noted that our Splunk is not exposed externally to get exploited. So would like to take advice here pls.

Pls do let me know if you need any further inputs. Thanks much

Thanks and Regards,
Peri

[jtacy]

There are a few variables to this question including what OS you're running and what specific AV you're using, but I think it's fair to say that exclusions don't necessarily eliminate the impact of AV. These products require some sort of file system filter that allows them to intercept events so there's always the potential, however small, for problems.

I ran an experiment with an AV product on Linux (CentOS 7) to see what kind of impact it would have on disk performance. In my experience the biggest impact with AV is observed when creating files so I used bonnie++ to measure the latency involved when creating thousands of files. With the AV service stopped, the latency was around 100 microseconds and appeared to be CPU-bound on the testing process. As expected, without any exclusions on the target path the latency was so bad I couldn't wait for the test to finish. The particular product I tested is interesting because it has two different interception methods available: a vendor-provided kernel module and fanotify (which appears to be a generic interface provided by the Linux kernel itself). I tested an exclusion with the vendor-provided module and the performance was outstanding, as though the AV wasn't there at all. On the other hand, with fanotify the latency was about 200 microseconds and was accompanied by a lot of CPU usage on the AV scanner process. I thought it was an interesting test, that's just a 0.1ms difference in file creation time in an extreme test; is that likely to make a difference in any real-world scenario?

On Windows, my understanding is that AV typically uses file system minifilter drivers to intercept events. It looks like it's not unheard of for these drivers to cause unusual problems and I'm sure they could potentially impact performance. However, these products have been developed over many years and are used in production worldwide.

For what it's worth, we use AV with exclusions on Linux indexers and it doesn't worry me at all. The only extra I thing I do is check occasionally with a file creation script to ensure that the exclusions are still active; appending to a file a few thousand times and measuring the time taken is enough to confirm this. That said, comprehensive testing is the only way to be sure your performance isn't being affected. It might be time-consuming to test, but on the other hand if the results indicate that you should seek an exception to the policy, you'll have hard data to take to the appropriate group. Have fun!

[ramanapvr]

Dear Jtacy/All,

Thanks a lot for the update. However would be great if we can get more information on this front. So let me rephrase my question as below with more details.

We have Splunk Cluster Architecture with Master, Search Head, Indexer1 & Indexer2. Pur Splunk is accessed by all our users only in internal network not in externally. Our Splunk servers are flavoured on Redhat OS.

Now the question is that we wanted to run Trendmicro Antivirus agent, and bit scared if that would screw up the performance of Splunk. Have gone through certain articles in Splunk and it was mentioned that we can consider any Antivirus Agent to be in running state, but that should be restricted to do scanning to OS partitions and not to Splunk data partitions where Splunk setup exist.

As we are are running Splunk as critical application Log monitoring solution, by having antivirus running on same server doing real time or offline scanning to OS partitions by not touching to Splunk directory will it create any performance issues on CPU/Memory resulting in to Splunk performance degradation.

Please can you share all possible facts such that we can take advice forward. Please do let me know if any further inputs are do required.

Thanks and Regards,
Peri