We are currently trying to set up a reliable solution for moving data from Splunk to HDFS location. This is not for archiving. We would like to move the data to HDFS location so that we can further process the data in the HDFS cluster using Apache Spark processing framework. We have looked at these options
Forward data from Splunk HF to Apache Nifi Syslog processor to push the data to HDFS
Forward data from Splunk HF to Apache Nifi TcpListener processor to push the data to HDFS
Splunk Hadoop connect (After looking at Splunk documentation, it looks like this plug-in does not work with the latest versions)
Splunk DSP where the data will be moved directly to Kafka and from there move to HDFS
Thanks in advance
Manu Mukundan
... View more