UF is not sending data to indexer

[sivaranjiniG]

Some one please help me here..

i am trying to monitor /var/log/audit/audit.log using universal forwarder and sending it to indexer.. but logs are not being sent to indexer..here is the log i m seeing in splunkd of forwarder 

08-06-2020 13:48:17.728 +0530 DEBUG TailingProcessor -   Item '/var/log/audit/audit.log' matches stanza: /var/log/audit/audit.log.
08-06-2020 13:48:17.728 +0530 DEBUG TailingProcessor -   Storing config '/var/log/audit/audit.log' for app ''.
08-06-2020 13:48:17.728 +0530 DEBUG TailingProcessor -   Entry is associated with 1 configuration(s).
08-06-2020 13:48:17.728 +0530 DEBUG TailReader -   Will attempt to read file: /var/log/audit/audit.log.
08-06-2020 13:48:17.730 +0530 DEBUG TailReader -   Got classified_sourcetype='linux_audit' and classified_charset='UTF-8'.
08-06-2020 13:48:17.730 +0530 DEBUG WatchedFile - Storing pending metadata for file=/var/log/audit/audit.log, sourcetype=linux_audit, charset=UTF-8
08-06-2020 13:48:17.730 +0530 DEBUG WatchedFile - setting trailing nulls to false via 'true' or 'false' from conf'
08-06-2020 13:48:17.730 +0530 DEBUG WatchedFile -   Loading state from fishbucket.
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/var/log/audit/audit.log|host::xxx|linux_audit|3 ...
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   Reading for plain initCrc...
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   initcrc has changed to: 0x153ce0cdaa107eee.
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile - Record found, will advance file by offset=12920 initcrc=0x153ce0cdaa107eee.
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   Creating new pipeline input channel with channel id: 4.
08-06-2020 13:48:17.732 +0530 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/var/log/audit/audit.log|host::xxx|linux_audit|4 ...
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - About to read data (Opening file: /var/log/audit/audit.log).
08-06-2020 13:48:17.732 +0530 DEBUG WatchedFile - seeking /var/log/audit/audit.log to off=12920
08-06-2020 13:48:17.732 +0530 DEBUG WatchedFile - Reached EOF: /var/log/audit/audit.log (read 0 bytes)
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Hit EOF immediately.
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Have definitely hit EOF.
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Finished reading file='/var/log/audit/audit.log' in tailreader0 thread, disposition=ACKNOWLEDGE_CHANGE, deferredBy=0.000
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Returning disposition=ACKNOWLEDGE_CHANGE for file=/var/log/audit/audit.log
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Start reading file="/opt/splunkforwarder/var/log/splunk/splunkd.log" in tailreader0 thread

[thambisetty]

is Splunk universal forwarder running with root account?

if not, switch user to account which is used by splunk service and try to read file "/var/log/audit/audit.log" using cat or tail or more.

let me know if you are seeing results or not. if not how did you set permission to read file by splunk user?

[thambisetty]

can you confirm if any other files are monitored on the server being sent to Indexer?

[sivaranjiniG]

yes 

/var/log/messages
/var/log/rhsm
/var/log/secure
/var/log/yum.log
/var/log/cron
/var/log/maillog

these are being sent to indexer without any issue

[aashiqwork]

Go to the Search Head and search with the below search (Make sure you have rights to see internal indexes data):

index=_internal | dedup host | fields host | table host

Look in the list to see if your Forwarder’s hostname is in the list, if it is present that means the Forwarder is connected. 

[sivaranjiniG · Accepted solution]

Thanks For all the Answers...But after deleting fishbucket, data are being pushed to indexer

[gcusello]

Hi @sivaranjiniG,

this means that you're indexing twice logs.

Ciao and next time.

Giuseppe

[sivaranjiniG]

I never have got data in indexer before deleting fishbucket so i thought it didnt read log file..moreover i didnt see any crc related issue in splunkd log..i am not sure how to find out these types of issue in splunkd log

[thambisetty]

check below answer may help you.

https://community.splunk.com/t5/Getting-Data-In/eof-error-when-reading-file/td-p/52833

[sivaranjiniG]

forwarder is not running with root account but Splunk user has access to read auditd log..that should be enough to read logs..i guess