Hi everyone, I'm in this situation:
I have a Splunk instance installed on my VM. I would like to send data to another Splunk instance that contains an alert manager and is receiving and triggering data from another VM.
On the first Splunk instance, I have installed the Splunk App for Unix and Linux that is triggering alerts. I would like get these and send them to the other Splunk indexer.
To do this I though, there are 2 different ways. Could someone help me to understand which is the best one?
Does someone of you know a better way to forward this kind of data?
The option 2 is not really an option I believe as light forwarder can't run a search. The option 1 seems feasible.
By the Splunk definition, "an alert is a search with a trigger condition and an action". No forwarder can run an alert, because forwarders cannot search.
If you have installed Splunk indexers on multiple production instances - perhaps you should reconsider your architecture. You can collect the data that is relevant to detecting the alert condition and forward it to an indexer. That sort of seems like what you are asking here.
If you are indexing less than 100 GB per day of data (across all your indexers), then you really only need one indexer. The indexer should reside on its own server (or VM). On the production VMs, whatever they are, collect the data using the universal forwarder and send it to the indexer. You may install the Splunk Technology Add-on (TA) for Unix and Linux on the forwarders. Install the Splunk App for Unix and Linux on the indexer.
Now you have one place to run your searches and alerts (the indexer) but you have data from across the environment.
Thank you for your reply, In the begin the idea was to have 3 or 4 central indexer that received data from the clients. Every client contain a strongbox with syslog that collect logs from windows or linux, but there is no correlation in windows or in linux, so I need to forward all the data. To filter them, I'm afraid I need a further indexer on the client that generate alerts and send them to the main indexer.
But I haven't considered the amount of data, in your opinion which is the threshold to have an instance on the client and one that receive the data from the alert, and which the threshold to have only a forwarder and a central indexer? Is there any way to filter using the forwarder?
I'm already playing with the Add-on for unix, do you know if exist the same for windows?
You can filter on the forwarder, particularly if you use a heavy forwarder. However, unless you are going to filter out more than 50% of the data, you should allow the filtering to happen on the indexer. There is an add-on for Windows.
I think you should look at this page, to understand my explanation. Forwarder deployment topologies
At this point, I strongly advise you not to route or filter anything unless that is necessary to stay within your Splunk license.
I also think that you are over-complicating your setup. Keep it simple by using universal forwarders to send your data to a central Splunk indexer. Run the alerts, reports, etc. on the indexer.
If you are new to Splunk, I recommend reading the following manuals
Coming back to this question: due to network constraints, I'd like to have my heavy forwarder instance sending to my indexer instance only the data related to an alert triggering - is it possible to do this? Thanks!!