SPLUNK INDEX Discovery

christay — Tue, 14 May 2019 20:08:01 GMT

Hi Guys,

I have configured using index discovery for my forwarder which are forwarding my firewall logs.
I saw from my splunkd.log it seems like the connection to my indexer is successful however, i can't see any logs from my indexer dashboard.

x.x.x.x is my 1st index server
y.y.y.y is my 2nd index server

logs

05-15-2019 03:59:05.238 +0800 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0. using ACK.
05-15-2019 03:59:10.784 +0800 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
05-15-2019 03:59:35.133 +0800 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997
05-15-2019 03:59:35.133 +0800 INFO TcpOutputProc - Connected to idx=y.y.y.y:9997, pset=0, reuse=0. using ACK.
05-15-2019 03:59:40.785 +0800 INFO TailReader - ...continuing.
05-15-2019 03:59:45.785 +0800 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
05-15-2019 04:00:08.725 +0800 INFO TailReader - ...continuing.
05-15-2019 04:01:00.462 +0800 INFO ArchiveProcessor - Handling file=/var/log/fortigate/fortigate.log-20190515.gz
05-15-2019 04:01:00.462 +0800 INFO ArchiveProcessor - new tailer already processed path=/var/log/fortigate/fortigate.log-20190515.gz
05-15-2019 04:01:04.850 +0800 INFO TcpOutputProc - Closing stream for idx=y.y.y.y:9997
05-15-2019 04:01:04.850 +0800 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0. using ACK.
05-15-2019 04:03:04.453 +0800 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997
05-15-2019 04:03:04.453 +0800 INFO TcpOutputProc - Connected to idx=y.y.y.y::9997, pset=0, reuse=0. using ACK.
05-15-2019 04:04:04.270 +0800 INFO TcpOutputProc - Closing stream for idx=y.y.y.y:9997
05-15-2019 04:04:04.270 +0800 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0. using ACK.

The indexes over at the index servers are not updated with any latest event as well.
Any idea how i can troubleshoot on this issue ?

Thanks

Re: SPLUNK INDEX Discovery

codebuilder — Tue, 14 May 2019 22:09:47 GMT

Is your forwarder running as the splunk user? Files and directories under /var/log/ are typically owned by root.
Try changing the user on your forwarder, or change the ownerships of your directory under /var/log/

Re: SPLUNK INDEX Discovery

christay — Wed, 15 May 2019 00:54:21 GMT

I have changed the ownership of /var/log to splunk user, but the result is still the same.
The splunkd.log doesn't share much insight to the issue as well....

topic SPLUNK INDEX Discovery in Getting Data In

SPLUNK INDEX Discovery

logs

Re: SPLUNK INDEX Discovery

Re: SPLUNK INDEX Discovery