topic Can Forwarder Be Configured to Pull Only? in Getting Data In

Can Forwarder Be Configured to Pull Only?

kfalconer — Mon, 07 Nov 2011 12:55:59 GMT

Given our environment requirements, it is not possible to have a forwarder push data to an indexer as needed (because of firewall rules). Is there a way to configure an indexer to make a request to pull data from a forwarder?

Thanks,
Karl

Re: Can Forwarder Be Configured to Pull Only?

Drainy — Mon, 07 Nov 2011 15:42:05 GMT

Not really.
By default the forwarder works through UDP and so it simply forwards all messages out to the indexer, the flow of this can be limited and the local buffer size can be adjusted.
The indexer isn't really designed to poll forwarders for information, mainly to receive the information.

What rules are restricting you? There may be other methods of achieving this that someone else might know

Re: Can Forwarder Be Configured to Pull Only?

kfalconer — Mon, 07 Nov 2011 15:56:26 GMT

I dont know the specific firewall rules, just that the remote machines (of which we want log information) can not initiate requests to a destination outside the firewall.

Re: Can Forwarder Be Configured to Pull Only?

vrossign — Mon, 26 Aug 2013 12:46:47 GMT

I have the same usecase.

Isn't it possible to write the logs into files and have another forwarder close to the indexer retrieve those log files ?

Re: Can Forwarder Be Configured to Pull Only?

rturk — Mon, 26 Aug 2013 13:30:28 GMT

Hi kfalconer,

My first port of call would be to consult your firewall administrators to see whether a rule could be created specifically to allow data out. Splunk can usually address any security concerns that FW admins would have, and as they have a fairly simple comms matrix, (TCP 8089, 9997) it can be locked down. An intermediary forwarder may even be used so that comms only ever originates from one host to simplify matter even more.

I recently had a deployment where the corporate policy did not allow the installation of a Universal Forwarder on some servers, AND data could only be transmitted within a certain window. As a Splunk guy, I can tell you that this was not my idea of fun...

ANYWAY, we managed to get around this by writing custom scripts that would be scheduled to run daily at 2AM, extract Windows event logs for the previous 24 hours, and write them to a shared directory (on a different server) that did have a Universal Forwarder on it to pick up the events and send them to the Indexer.

The end result is that they now have 'Splunk for Active Directory' installed and operational... but only with data for the previous day (they understood this would be the case), and there was a bit of rewrite to get the app working... which I wouldn't recommend if you value your sanity 🙂

TL;DR: If your firewall admins can't help, you might want to look at the following:

Custom scripts to collect data on a scheduled basis
Drop the events on a shared directory that can send data to the indexer
Have a Universal Forwarder on that server send the data to the Indexer as it's received,

PS. If your firewall admins can't help... then they're probably not going to like you taking this approach either 😛

Re: Can Forwarder Be Configured to Pull Only?

Ayn — Mon, 26 Aug 2013 13:39:50 GMT

Absolutely. There are multiple ways of making this happen - the question was just whether a forwarder itself can do some kind of pull instead of push, and it can't. But sure, you can have another forwarder fetch data from the hosts(s) in questions, through a script, fileshare or something else.

Re: Can Forwarder Be Configured to Pull Only?

apfender_splunk — Wed, 04 Feb 2015 12:23:57 GMT

splunk internal communication is using tcp only, no udp