Universal / Heavy Forward Vs Http Event Collector

[nks]

hi,
My requirement is to push a continuous stream of data into Splunk for search and reporting. Data will be provided to me in form of CSV files by a batch job.
these CSV files will be placed in a folder as input and once i am done inserting them into Splunk, these files needs to be deleted from folder for new files to come in.

I am wondering which route should i pick, namely :- A) setup Universal or Heavy Forwarders. They will push the data from CSV to Splunk.. Or B), write my own custom java code to push data from CSV files to HEC endpoints.

[Richfez · Accepted solution]

If you installed a Universal Forwarder and set the files up as a batch input, as long as you have your output configured and have set your timestamping and whatnot right, you are done.

I mean, sure you COULD write custom java code, but why?

Now, if it's REALLY high volume (by that, I mean hundreds of GB/day or more, definitely unlikely for this to be a problem even at dozens of GB/day), HEC may be beneficial.

In the docs I linked, ignore the warnings about doing it only for large uploads of historical data, and also ignore the part about how it works in Splunk Web. They don't matter.

[Richfez]

nks,

If this resolved your issue, could you please mark it Accepted?

If it did not, please post back with more information or what's not working right so we can help finish this up!

Happy Splunking,
Rich