Getting Data In

Linux Event logs not coming over to Splunk cloud

Kwabena13
Engager

I am trying to ingest Linux logs into Splunk. 

1. I have deployed the unix_TA through the deployment server to the Heavy forwarder and to the universal forwarder with the inputs. conf defined in the Local directory. The indexes are defined in the inputs.conf as well.

2. The Universal forwarder has confirmed that the TA is found in the /opt/splunkuniversal forwarder/apps directory with the inputs.conf as deployed.

3. permissions have been granted to the Splunkfwd on the universal forwarder on the Linux server to read var/log .

4. The TA is also installed on the Search Head.

I am able to see the metric logs in the _internal index. However I do not see the event logs. I have run a tcp dump on the heavy forwarder's CLI  and have confirmed that there are logs coming in.

Any ideas on what I am missing?

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Has the UF been restarted?

2. Look for _internal events from that UF regarding monitored files.

3. Did you verify your resulting config with btool?

4. SELinux

0 Karma

Kwabena13
Engager

Thank you. Yes the UF was  restarted. The _internal logs did not have the monitored path in the logs . We also checked the permissions for the var/log and splunkuf had a read access to the file. we tested by logging in as the splunkuf and we were able to see the content of the files.

The Logs are still not showing up in the index. I have checked on the index' configuration and it all checks out with nothing different from the other indexes.

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Based on that output your UF’s cannot read / found those files. Are you absolutely sure that you are using the same account which are used to run splunkd? As @PickleRick said you should check is there any issue with SElinux.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

What 

splunk list inputstatus

shows on UF? It tells what files it has read and how much.

Are you sure that times are correctly picked from files? If there are mismatch between European and USA time format then you must look those events with some other times as now. When you are porting a new source it's useful to use real time search with known hosts / sources for all time. With that way you can catch wrongly recognized timestamps. 

0 Karma

Kwabena13
Engager

Thank you for the answer. When i run the list inputstatus from bin,

I received the  output below;

Kwabena13_0-1720557446183.png

I have verified that Splunkfwd has read access to the var/log

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Just because a user has permissions to read a list of directory contents does not mean it will be able to read individua, files.

2. Again - selinux isssues? Is your selinux in enforcing mode? Have you checked your auditd.log for selinux denied access attempts?

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...