I wish to ingest events from a large number of remote hosts. I cannot install any Splunk infrastructure on these hosts.
I have looked into the various remote interfaces for splunk and none seem appropriate for my needs. It seems likely that I will need to build my own service to collect events from these hosts.
My question is: what is the best way to get data from my service into Splunk? I would like to be able to guarantee that once I have sent an 'ACK' to the remote host, that the data will make its way into splunk. I would also like to be able to scale the infrastructure horizontally.
I could have the server write to a monitored file, but I don't really want to create huge log files just to get data into Splunk.
I could use a FIFO queue, but that would not provide the guarantee I was talking about.
Perhaps I should use a Splunk SDK from within my service?