Linux Event logs not coming over to Splunk cloud

[Kwabena13]

I am trying to ingest Linux logs into Splunk. 

1. I have deployed the unix_TA through the deployment server to the Heavy forwarder and to the universal forwarder with the inputs. conf defined in the Local directory. The indexes are defined in the inputs.conf as well.

2. The Universal forwarder has confirmed that the TA is found in the /opt/splunkuniversal forwarder/apps directory with the inputs.conf as deployed.

3. permissions have been granted to the Splunkfwd on the universal forwarder on the Linux server to read var/log .

4. The TA is also installed on the Search Head.

I am able to see the metric logs in the _internal index. However I do not see the event logs. I have run a tcp dump on the heavy forwarder's CLI  and have confirmed that there are logs coming in.

Any ideas on what I am missing?

[PickleRick]

1. Has the UF been restarted?

2. Look for _internal events from that UF regarding monitored files.

3. Did you verify your resulting config with btool?

4. SELinux

[Kwabena13]

Thank you. Yes the UF was  restarted. The _internal logs did not have the monitored path in the logs . We also checked the permissions for the var/log and splunkuf had a read access to the file. we tested by logging in as the splunkuf and we were able to see the content of the files.

The Logs are still not showing up in the index. I have checked on the index' configuration and it all checks out with nothing different from the other indexes.

[isoutamo]

Based on that output your UF’s cannot read / found those files. Are you absolutely sure that you are using the same account which are used to run splunkd? As @PickleRick said you should check is there any issue with SElinux.

[isoutamo]

What 

splunk list inputstatus

shows on UF? It tells what files it has read and how much.

Are you sure that times are correctly picked from files? If there are mismatch between European and USA time format then you must look those events with some other times as now. When you are porting a new source it's useful to use real time search with known hosts / sources for all time. With that way you can catch wrongly recognized timestamps. 

[Kwabena13]

Thank you for the answer. When i run the list inputstatus from bin,

I received the  output below;

I have verified that Splunkfwd has read access to the var/log

[PickleRick]

1. Just because a user has permissions to read a list of directory contents does not mean it will be able to read individua, files.

2. Again - selinux isssues? Is your selinux in enforcing mode? Have you checked your auditd.log for selinux denied access attempts?