I keep searching all over the Splunk site and I actually think there is TOO much data/information. Maybe I'm looking at the wrong stuff so I have given up and decided to post my question here.
Goal: Install Splunk, forward logs from all nodes (Windows and Linux and at some point maybe AWS)
Where I'm at: I have Splunk installed and I can log in. I got data into it by setting up rsyslog on the Splunk server so Splunk is reading local files. We ended up using snare to pass Windows information to the Splunk server. I looked everywhere and snare was the only thing I could find that was free. I am getting data indexed in Splunk, so I guess technically it's working.
What I want to know is "Is the best/recommended way to set this up"? I keep seeing everything reference the heavy forwarder, but I can't find anything that says you should do xxxxxx. I find a lot of times I click through documentation, click a link and it takes you some where and it seems like it jumped you some where not in the order that you were following along with. Also, I find it a lot of times the documentation says "Oh you are not using the latest version of documentation."
The last thing I wanted to ask was: I know I can search the data, but what should should I search for? What type of dashboards should you have? Is there a best practice that says hey these are the go to/best practices searches or dashboards you should set up first before creating custom ones?
Sorry for the long rant/question, just frustrated and not sure where to go next. Thanks for any help or pointing me in the right direction.
For Forwarding logs from remote systems (Unix and Windows) it is best to set up a Splunk Universal Forwarder that can monitor local files and send them to your Indexer(s) via the splunk receiving port.
What I do is find what I'm looking for via a query. Then you go to the top of the screen, save-as, dashboard panel, then select new or existing dashboard. This saves the query and allows you to fiddle with how it's visualized. You can also save-as an alert if you want to create an alert base on query.
Yes. You can install a forwarder on each node to capture that node's information. I point mine to also monitor a directory where our platform spits out it's logs or where MySQL, Redis, RabbitMQ, whatever spits it's logs out.