Deployment Architecture

Universal / Heavy Forward Vs Http Event Collector

nks
New Member

hi,
My requirement is to push a continuous stream of data into Splunk for search and reporting. Data will be provided to me in form of CSV files by a batch job.
these CSV files will be placed in a folder as input and once i am done inserting them into Splunk, these files needs to be deleted from folder for new files to come in.

I am wondering which route should i pick, namely :- A) setup Universal or Heavy Forwarders. They will push the data from CSV to Splunk.. Or B), write my own custom java code to push data from CSV files to HEC endpoints.

Tags (1)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

If you installed a Universal Forwarder and set the files up as a batch input, as long as you have your output configured and have set your timestamping and whatnot right, you are done.

I mean, sure you COULD write custom java code, but why?

Now, if it's REALLY high volume (by that, I mean hundreds of GB/day or more, definitely unlikely for this to be a problem even at dozens of GB/day), HEC may be beneficial.

In the docs I linked, ignore the warnings about doing it only for large uploads of historical data, and also ignore the part about how it works in Splunk Web. They don't matter.

View solution in original post

Richfez
SplunkTrust
SplunkTrust

If you installed a Universal Forwarder and set the files up as a batch input, as long as you have your output configured and have set your timestamping and whatnot right, you are done.

I mean, sure you COULD write custom java code, but why?

Now, if it's REALLY high volume (by that, I mean hundreds of GB/day or more, definitely unlikely for this to be a problem even at dozens of GB/day), HEC may be beneficial.

In the docs I linked, ignore the warnings about doing it only for large uploads of historical data, and also ignore the part about how it works in Splunk Web. They don't matter.

Richfez
SplunkTrust
SplunkTrust

nks,

If this resolved your issue, could you please mark it Accepted?

If it did not, please post back with more information or what's not working right so we can help finish this up!

Happy Splunking,
Rich

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...