Getting Data In

Why is our Splunk forwarder not able to read my snmptrapd file?

New Member

Hi ,

I am trying to read my snmptrap file under /var/log/ path (it has 755 permission as well), but I am not able to see them in Splunk Web.

Here is the log in debug mode. I am using Splunkforwarder 6.3.1 version.

05-26-2016 13:13:17.055 -0700 DEBUG TailReader - Start reading file=/var/log/snmptrapd.log in tailreader0 thread
05-26-2016 13:13:17.055 -0700 DEBUG TailReader -   Have seen this item before (since splunkd was restarted).
05-26-2016 13:13:17.055 -0700 DEBUG TailReader -   Will attempt to read file: /var/log/snmptrapd.log from existing fd.
05-26-2016 13:13:17.055 -0700 DEBUG WatchedFile -   Loading state from fishbucket.
05-26-2016 13:13:17.055 -0700 DEBUG WatchedFile -   Reading for plain initCrc...
05-26-2016 13:13:17.055 -0700 DEBUG WatchedFile - Record found, will advance file by offset=22911097 initcrc=0x3229c20d72db1393.
05-26-2016 13:13:17.055 -0700 DEBUG WatchedFile -   Preserving seekptr and initcrc.
05-26-2016 13:13:17.055 -0700 DEBUG TailReader - About to read data (Reusing existing fd for file='/var/log/snmptrapd.log').
05-26-2016 13:13:17.055 -0700 DEBUG WatchedFile - seeking /var/log/snmptrapd.log to off=22911097
05-26-2016 13:13:17.055 -0700 DEBUG WatchedFile - Reached EOF: fname=/var/log/snmptrapd.log fishstate=key=0x3229c20d72db1393 sptr=22921194 scrc=0x6b79f6d13bb8416f fnamecrc=0x40ddf42b83f38ebd modtime=1464293596
05-26-2016 13:13:17.055 -0700 DEBUG TailReader -   Will doublecheck EOF (in 3000ms)..
05-26-2016 13:13:17.055 -0700 DEBUG TailReader - Finished reading file='/var/log/snmptrapd.log' in tailreader0 thread, disposition=1, deferredBy=3000
05-26-2016 13:13:17.055 -0700 DEBUG TailReader - Defering notification for file=/var/log/snmptrapd.log by 3000ms
05-26-2016 13:13:17.055 -0700 DEBUG TailReader - tailreader0 waiting for jobs
05-26-2016 13:13:20.056 -0700 DEBUG TailingProcessor - Returning disposition: 1
05-26-2016 13:13:20.056 -0700 DEBUG TailingProcessor - ****************************************
05-26-2016 13:13:20.056 -0700 DEBUG TailingProcessor - Deferred notification for path='/var/log/snmptrapd.log'.
05-26-2016 13:13:20.056 -0700 DEBUG TailingProcessor - Returning disposition: 1
05-26-2016 13:13:20.056 -0700 DEBUG TailReader - Enqueued file=/opt/splunkforwarder/var/log/splunk/splunkd.log in tailreader0
05-26-2016 13:13:20.056 -0700 DEBUG TailReader - Enqueued file=/var/log/snmptrapd.log in tailreader0



05-26-2016 13:13:26.061 -0700 DEBUG TailReader - Start reading file=/var/log/snmptrapd.log in tailreader0 thread
05-26-2016 13:13:26.061 -0700 DEBUG TailReader -   Have seen this item before (since splunkd was restarted).
05-26-2016 13:13:26.061 -0700 DEBUG TailReader -   Will attempt to read file: /var/log/snmptrapd.log from existing fd.
05-26-2016 13:13:26.061 -0700 DEBUG WatchedFile -   Loading state from fishbucket.
05-26-2016 13:13:26.061 -0700 DEBUG WatchedFile -   Reading for plain initCrc...
05-26-2016 13:13:26.061 -0700 DEBUG WatchedFile - Record found, will advance file by offset=22946228 initcrc=0x3229c20d72db1393.
05-26-2016 13:13:26.061 -0700 DEBUG WatchedFile -   Preserving seekptr and initcrc.
05-26-2016 13:13:26.061 -0700 DEBUG TailReader - About to read data (Reusing existing fd for file='/var/log/snmptrapd.log').
05-26-2016 13:13:26.061 -0700 DEBUG WatchedFile - seeking /var/log/snmptrapd.log to off=22946228
05-26-2016 13:13:26.061 -0700 DEBUG WatchedFile - Reached EOF: fname=/var/log/snmptrapd.log fishstate=key=0x3229c20d72db1393 sptr=22954419 scrc=0x6b79f6d13bb8416f fnamecrc=0x40ddf42b83f38ebd modtime=1464293604
05-26-2016 13:13:26.061 -0700 DEBUG TailReader -   Will doublecheck EOF (in 3000ms)..
05-26-2016 13:13:26.061 -0700 DEBUG TailReader - Finished reading file='/var/log/snmptrapd.log' in tailreader0 thread, disposition=1, deferredBy=3000
05-26-2016 13:13:26.061 -0700 DEBUG TailReader - Defering notification for file=/var/log/snmptrapd.log by 3000ms
05-26-2016 13:13:26.061 -0700 DEBUG TailReader - tailreader0 waiting for jobs
05-26-2016 13:13:29.062 -0700 DEBUG TailingProcessor - ****************************************
05-26-2016 13:13:29.062 -0700 DEBUG TailingProcessor - Deferred notification for path='/var/log/snmptrapd.log'.
05-26-2016 13:13:29.062 -0700 DEBUG TailingProcessor - Returning disposition: 1
05-26-2016 13:13:29.062 -0700 DEBUG TailingProcessor - Returning disposition: 1
05-26-2016 13:13:29.062 -0700 DEBUG TailReader - Enqueued file=/var/log/snmptrapd.log in tailreader0
05-26-2016 13:13:29.062 -0700 DEBUG TailReader - Enqueued file=/opt/splunkforwarder/var/log/splunk/splunkd.log in tailreader0

Can anyone please guide me what I should do to get the content display in Splunk Web?

Many thanks,

0 Karma

Path Finder

I can't tell you why but I can tell you how I would troubleshoot it:

Monitor another file in a different directory and confirm that you are able to index it.
One by one, change the attributes of new file to mimic the file you can't index. For example you could:

  • Copy the contents from the snmptrap file into the new file
  • Set the same permissions
  • Copy the file over to the directory where snmptrap is stored.

You'll know that the issue is related to the last change you made when the new file stops being indexed.

0 Karma

New Member

I tried the same, it did not work out. Anyways thanks a lot for your suggestion.

0 Karma

Contributor

Hi

Is this particular forwarder managing to transmit any data to the indexer? As romedome suggested "Monitor another file in a different directory and confirm that you are able to index it." Does that work?

If I'm interpreting the DEBUG events correctly, they appear to indicate that the fishbucket pointers are being advanced and data is being read from /var/log/snmptrapd.log. I'd check the metrics.log files on the forwarder to see if there's any reference to data being processed for the source /var/log/snmptrapd.log, or with the appropriate sourcetype. The events you're looking for will contain

group=per_source_thruput, series="/var/log/snmptrapd.log"

or

group=per_sourcetype_thruput, series="sourcetype from inputs.conf"

Can you post the inputs.conf stanza for the /var/log/snmptrapd.log input and how you're searching for the /var/log/snmptrapd.log data in Splunk web.

Dave

0 Karma