Not receiving data from particular source

[Harish2]

Hi 
My sources:
1.  /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC.log

2.  /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-show.log

3.  /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-ignored-sms.log

4.  /app/splunkser/ShiftMinJMC/ShiftMinJMC.log

5.  /app/splunkser/ShiftMinJMC/ShiftMinJMC-show.log

6.  /app/splunkser/ShiftMinJMC/ShiftMinJMC-ignored-sms.log

7.  /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC.log

8.  /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-show.log

9.  /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-ignored-sms.log

I am receive the data from the above sources in SIT  and PROD environment but not receiving  logs from the below sources:

1.  /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC.log

4.  /app/splunkser/ShiftMinJMC/ShiftMinJMC.log

7.  /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC.log

Note: i am getting logs in SIT from all 9 sources but in production the mentioned 1, 4 and 7th sources are not showing up in Production env.

Inputs.conf

[monitor:///app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-show-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-ignored-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftMinJMC/ShiftMinJMC-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftMinJMC/ShiftMinJMC-show-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftMinJMC/ShiftMinJMC-ignored-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-show-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-ignored-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

Props.conf

[app:jmcshift:logs]
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}
SHOULD_LINEMERGE=false
TRUNCATE=99999

Sample logs:
From all 9 sources the events starts with date as shown below:
2023-01-12 23:24:50.245 [error]...........................................

Same inputs.cong and props.conf  in SIT and Production env.
Not sure what could be the issue.

[richgalloway]

Have you checked the permissions on the missing sources to make sure Splunk has read access?

[Harish2]

Hi @richgalloway 
how can i check that, can u please tell me????

[richgalloway]

Sign on to the source server and run 

ls -ls /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC.log /app/splunkser/ShiftMinJMC/ShiftMinJMC.log /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC.log

This will tell you who owns the files and the groups which can access it.  Use the groups command to find out the groups to which the Splunk user belongs.  Contact your Linux admin for specific assistance.

[Harish2]

I checked there is no permission issue, i can see other files with the same permission.

But not able to see data from mentioned sources

[richgalloway]

Here are a few other things to check.

Look in splunkd.log on the forwarders to see if there are messages about reading those sources.

If you use SELinux, have someone verify the settings allow Splunk to read the sources.  If you can sign in as the Splunk user and read the files then Splunk itself should be able to read them.

Verify the sources are going to the right indexes.

Verify the timestamps in the sources are being onboarded correctly.  Incorrect timestamps could make it hard to find data from the source.  Try searching with earliest=0 latest=+1y.

Double-check the SPL used to search for the sources.