topic Re: data from remote hosts in Deployment Architecture

data from remote hosts

pezcrap — Tue, 22 Apr 2014 01:25:47 GMT

I wish to ingest events from a large number of remote hosts. I cannot install any Splunk infrastructure on these hosts.

I have looked into the various remote interfaces for splunk and none seem appropriate for my needs. It seems likely that I will need to build my own service to collect events from these hosts.

My question is: what is the best way to get data from my service into Splunk? I would like to be able to guarantee that once I have sent an 'ACK' to the remote host, that the data will make its way into splunk. I would also like to be able to scale the infrastructure horizontally.

I could have the server write to a monitored file, but I don't really want to create huge log files just to get data into Splunk.

I could use a FIFO queue, but that would not provide the guarantee I was talking about.

Perhaps I should use a Splunk SDK from within my service?

Can a splunk forwarder help here?

Re: data from remote hosts

yannK — Tue, 22 Apr 2014 01:50:07 GMT

I cannot install any Splunk infrastructure on these hosts.

So you have to bring the logs to splunk somehow, to the indexer or to a forwarder.

for log file, use a shared folder monitored remotely by forwarder
a script to copy the files up to a forwarder
use a syslog server to send the logs ( to another syslog that will write to disk, then monitor with splunk), but avoid UDP of course.

Re: data from remote hosts

pezcrap — Tue, 22 Apr 2014 02:19:25 GMT

This doesn't really answer my question. I already noted I will need to implement my own service to collect events. I was asking how to go about implementing that service.

edit: Thanks a lot for responding though, I sounded a bit ungrateful there 🙂