Deployment Architecture

UF is not sending data to indexer

sivaranjiniG
Path Finder

Some one please help me here..

i am trying to monitor /var/log/audit/audit.log using universal forwarder and sending it to indexer.. but logs are not being sent to indexer..here is the log i m seeing in splunkd of forwarder 

 

08-06-2020 13:48:17.728 +0530 DEBUG TailingProcessor -   Item '/var/log/audit/audit.log' matches stanza: /var/log/audit/audit.log.
08-06-2020 13:48:17.728 +0530 DEBUG TailingProcessor -   Storing config '/var/log/audit/audit.log' for app ''.
08-06-2020 13:48:17.728 +0530 DEBUG TailingProcessor -   Entry is associated with 1 configuration(s).
08-06-2020 13:48:17.728 +0530 DEBUG TailReader -   Will attempt to read file: /var/log/audit/audit.log.
08-06-2020 13:48:17.730 +0530 DEBUG TailReader -   Got classified_sourcetype='linux_audit' and classified_charset='UTF-8'.
08-06-2020 13:48:17.730 +0530 DEBUG WatchedFile - Storing pending metadata for file=/var/log/audit/audit.log, sourcetype=linux_audit, charset=UTF-8
08-06-2020 13:48:17.730 +0530 DEBUG WatchedFile - setting trailing nulls to false via 'true' or 'false' from conf'
08-06-2020 13:48:17.730 +0530 DEBUG WatchedFile -   Loading state from fishbucket.
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/var/log/audit/audit.log|host::xxx|linux_audit|3 ...
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   Reading for plain initCrc...
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   initcrc has changed to: 0x153ce0cdaa107eee.
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile - Record found, will advance file by offset=12920 initcrc=0x153ce0cdaa107eee.
08-06-2020 13:48:17.731 +0530 DEBUG WatchedFile -   Creating new pipeline input channel with channel id: 4.
08-06-2020 13:48:17.732 +0530 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/var/log/audit/audit.log|host::xxx|linux_audit|4 ...
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - About to read data (Opening file: /var/log/audit/audit.log).
08-06-2020 13:48:17.732 +0530 DEBUG WatchedFile - seeking /var/log/audit/audit.log to off=12920
08-06-2020 13:48:17.732 +0530 DEBUG WatchedFile - Reached EOF: /var/log/audit/audit.log (read 0 bytes)
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Hit EOF immediately.
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Have definitely hit EOF.
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Finished reading file='/var/log/audit/audit.log' in tailreader0 thread, disposition=ACKNOWLEDGE_CHANGE, deferredBy=0.000
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Returning disposition=ACKNOWLEDGE_CHANGE for file=/var/log/audit/audit.log
08-06-2020 13:48:17.732 +0530 DEBUG TailReader - Start reading file="/opt/splunkforwarder/var/log/splunk/splunkd.log" in tailreader0 thread

 

Labels (2)
0 Karma
1 Solution

sivaranjiniG
Path Finder

Thanks For all the Answers...But after deleting fishbucket, data are being pushed to indexer

View solution in original post

0 Karma

thambisetty
SplunkTrust
SplunkTrust

is Splunk universal forwarder running with root account?

if not, switch user to account which is used by splunk service and try to read file "/var/log/audit/audit.log" using cat or tail or more.

let me know if you are seeing results or not. if not how did you set permission to read file by splunk user?

 

 

 

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

can you confirm if any other files are monitored on the server being sent to Indexer?

————————————
If this helps, give a like below.
0 Karma

sivaranjiniG
Path Finder

yes 

/var/log/messages
/var/log/rhsm
/var/log/secure
/var/log/yum.log
/var/log/cron
/var/log/maillog

these are being sent to indexer without any issue

0 Karma

aashiqwork
Explorer

Go to the Search Head and search with the below search (Make sure you have rights to see internal indexes data):

index=_internal | dedup host | fields host | table host

Look in the list to see if your Forwarder’s hostname is in the list, if it is present that means the Forwarder is connected. 

0 Karma

sivaranjiniG
Path Finder

Thanks For all the Answers...But after deleting fishbucket, data are being pushed to indexer

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sivaranjiniG,

this means that you're indexing twice logs.

Ciao and next time.

Giuseppe

0 Karma

sivaranjiniG
Path Finder

I never have got data in indexer before deleting fishbucket so i thought it didnt read log file..moreover i didnt see any crc related issue in splunkd log..i am not sure how to find out these types of issue in splunkd log

0 Karma

thambisetty
SplunkTrust
SplunkTrust

check below answer may help you.

 

https://community.splunk.com/t5/Getting-Data-In/eof-error-when-reading-file/td-p/52833

————————————
If this helps, give a like below.
0 Karma

sivaranjiniG
Path Finder

forwarder is not running with root account but Splunk user has access to read auditd log..that should be enough to read logs..i guess


0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...