Deployment Architecture

data from remote hosts

pezcrap
Explorer

I wish to ingest events from a large number of remote hosts. I cannot install any Splunk infrastructure on these hosts.

I have looked into the various remote interfaces for splunk and none seem appropriate for my needs. It seems likely that I will need to build my own service to collect events from these hosts.

My question is: what is the best way to get data from my service into Splunk? I would like to be able to guarantee that once I have sent an 'ACK' to the remote host, that the data will make its way into splunk. I would also like to be able to scale the infrastructure horizontally.

I could have the server write to a monitored file, but I don't really want to create huge log files just to get data into Splunk.

I could use a FIFO queue, but that would not provide the guarantee I was talking about.

Perhaps I should use a Splunk SDK from within my service?

Can a splunk forwarder help here?

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

I cannot install any Splunk infrastructure on these hosts.

So you have to bring the logs to splunk somehow, to the indexer or to a forwarder.

  • for log file, use a shared folder monitored remotely by forwarder
  • a script to copy the files up to a forwarder
  • use a syslog server to send the logs ( to another syslog that will write to disk, then monitor with splunk), but avoid UDP of course.
0 Karma

pezcrap
Explorer

This doesn't really answer my question. I already noted I will need to implement my own service to collect events. I was asking how to go about implementing that service.

edit: Thanks a lot for responding though, I sounded a bit ungrateful there 🙂

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...