Deployment Architecture

data from remote hosts

Explorer

I wish to ingest events from a large number of remote hosts. I cannot install any Splunk infrastructure on these hosts.

I have looked into the various remote interfaces for splunk and none seem appropriate for my needs. It seems likely that I will need to build my own service to collect events from these hosts.

My question is: what is the best way to get data from my service into Splunk? I would like to be able to guarantee that once I have sent an 'ACK' to the remote host, that the data will make its way into splunk. I would also like to be able to scale the infrastructure horizontally.

I could have the server write to a monitored file, but I don't really want to create huge log files just to get data into Splunk.

I could use a FIFO queue, but that would not provide the guarantee I was talking about.

Perhaps I should use a Splunk SDK from within my service?

Can a splunk forwarder help here?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

I cannot install any Splunk infrastructure on these hosts.

So you have to bring the logs to splunk somehow, to the indexer or to a forwarder.

  • for log file, use a shared folder monitored remotely by forwarder
  • a script to copy the files up to a forwarder
  • use a syslog server to send the logs ( to another syslog that will write to disk, then monitor with splunk), but avoid UDP of course.
0 Karma

Explorer

This doesn't really answer my question. I already noted I will need to implement my own service to collect events. I was asking how to go about implementing that service.

edit: Thanks a lot for responding though, I sounded a bit ungrateful there 🙂

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!