What do Splunk Ninjas think are the top three daily Splunk tasks in a large distributed environment?

[brian1_tate]

Hello all,

I am trying to build a workflow for our new Splunk product and want to know what top three regular daily tasks you may do in Splunk Enterprise. This includes anything in regards to ES administration as well and maintenance tasks.

If anyone has suggestions, I would certainly appreciate your feedback. This new environment has 5 indexers in a cluster, three search heads in a cluster and several heavy forwarders with a ton of data sent via forwarders. Comments anyone?

[gjanders]

So, most of what I do is set up alerts to advise me about potential issues, for ES this has included:

1. Sources no longer sending data ( | tstats max(_indextime) where ... or similar is very useful here)
2. Users exceeding there quota (disk quota, search quota), often this is harmless but sometimes I review the limits to see if they are appropriate.
3. Check for badly written/long running searches via the monitoring console (this one is more a manual task for now).

There would be many other potential things on a daily basis, one thing I do try to do is review the error logs...

[brian1_tate]

That's not relevant here. All I'm looking for is general procedural steps for those that use Splunk Enterprise with ES as a SIEM. I have those for ArcSight but after working with a larger Splunk environment, frankly the largest I have ever encountered - one would have them or develop them. So in this case, I am developing them based upon my own experiences and based upon those that others may use in day to day operations.

If you just say, I use Splunk - best wishes in justification of your career...

[a212830]

It's a joke. Lighten up Francis.

[klaxdal]

Don't call me Francis .

https://www.youtube.com/watch?v=IMtvnAmfuf8

[Richfez]

Y'all need to put more smileys in your posts to help us old geezers know you were joking. Geez. 🙂

Plus I am cleaning up a duplicate comment and am slightly rearranging the remaining, since they were responses to one another.

[a212830]

Answer questions on why shouldn't we use ELK/open-source...

[aaraneta_splunk]

@brian1_tate - When you say "ES" are you referring to the app "Splunk Enterprise Security"? Because that is usually what "ES" refers to. Please clarify, thank you!

[brian1_tate]

That's correct but in general, I wanted to get people's feedback on what daily/weekly tasks they perform as I am transitioning from ArcSight to Splunk.