How will thousands of syslog events sent to Hadoop affect a heavy forwarder?

[a212830]

Hi,

I am currently processing syslog events, using the hfw. This feed is pretty busy - hundreds of files, and I'm being asked to forward all the data to Hadoop. How will this affect the forwarder? These events are critical within Splunk, and I don't want any delay to be added. I'm not sure of the need for real-time here, so my suggestion is going to be Hadoop Connect.

[hsesterhenn_spl]

Hi,

indexing the data using Splunk Enterprise Core in combination with a HFW should not be influenced if you export the data off the indexer using Hadoop Connect.

Remember, Hadoop Connect will run a search and then export the result/data to Hadoop.

Maybe the Hadoop Data Roll feature is a better option if you want to archive buckets instead of exporting files.

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingindexestoHadoop

HTH,

Holger

[a212830]

thousands of events...