Getting Data In

Linux Event logs not coming over to Splunk cloud

Kwabena13
Engager

I am trying to ingest Linux logs into Splunk. 

1. I have deployed the unix_TA through the deployment server to the Heavy forwarder and to the universal forwarder with the inputs. conf defined in the Local directory. The indexes are defined in the inputs.conf as well.

2. The Universal forwarder has confirmed that the TA is found in the /opt/splunkuniversal forwarder/apps directory with the inputs.conf as deployed.

3. permissions have been granted to the Splunkfwd on the universal forwarder on the Linux server to read var/log .

4. The TA is also installed on the Search Head.

I am able to see the metric logs in the _internal index. However I do not see the event logs. I have run a tcp dump on the heavy forwarder's CLI  and have confirmed that there are logs coming in.

Any ideas on what I am missing?

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Has the UF been restarted?

2. Look for _internal events from that UF regarding monitored files.

3. Did you verify your resulting config with btool?

4. SELinux

0 Karma

Kwabena13
Engager

Thank you. Yes the UF was  restarted. The _internal logs did not have the monitored path in the logs . We also checked the permissions for the var/log and splunkuf had a read access to the file. we tested by logging in as the splunkuf and we were able to see the content of the files.

The Logs are still not showing up in the index. I have checked on the index' configuration and it all checks out with nothing different from the other indexes.

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Based on that output your UF’s cannot read / found those files. Are you absolutely sure that you are using the same account which are used to run splunkd? As @PickleRick said you should check is there any issue with SElinux.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

What 

splunk list inputstatus

shows on UF? It tells what files it has read and how much.

Are you sure that times are correctly picked from files? If there are mismatch between European and USA time format then you must look those events with some other times as now. When you are porting a new source it's useful to use real time search with known hosts / sources for all time. With that way you can catch wrongly recognized timestamps. 

0 Karma

Kwabena13
Engager

Thank you for the answer. When i run the list inputstatus from bin,

I received the  output below;

Kwabena13_0-1720557446183.png

I have verified that Splunkfwd has read access to the var/log

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Just because a user has permissions to read a list of directory contents does not mean it will be able to read individua, files.

2. Again - selinux isssues? Is your selinux in enforcing mode? Have you checked your auditd.log for selinux denied access attempts?

0 Karma
Get Updates on the Splunk Community!

Splunk Certification Support Alert | Pearson VUE Outage

Splunk Certification holders and candidates!  Please be advised of an upcoming system maintenance period for ...

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...