topic Re: Not receiving data from particular source in Splunk Search

Not receiving data from particular source

Harish2 — Sat, 28 Jan 2023 20:22:24 GMT

Hi
My sources:
1. /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC.log

2. /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-show.log

3. /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-ignored-sms.log

4. /app/splunkser/ShiftMinJMC/ShiftMinJMC.log

5. /app/splunkser/ShiftMinJMC/ShiftMinJMC-show.log

6. /app/splunkser/ShiftMinJMC/ShiftMinJMC-ignored-sms.log

7. /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC.log

8. /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-show.log

9. /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-ignored-sms.log

I am receive the data from the above sources in SIT and PROD environment but not receiving logs from the below sources:

1. /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC.log

4. /app/splunkser/ShiftMinJMC/ShiftMinJMC.log

7. /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC.log

Note: i am getting logs in SIT from all 9 sources but in production the mentioned 1, 4 and 7th sources are not showing up in Production env.

Inputs.conf

[monitor:///app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-show-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC-ignored-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftMinJMC/ShiftMinJMC-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftMinJMC/ShiftMinJMC-show-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftMinJMC/ShiftMinJMC-ignored-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-show-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

[monitor:///app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC-ignored-*.log]
disabled=0
index=app-jmc-shift-sms
sourcetype=app:jmcshift:logs
blacklist=\.(?:tar|gz)$
crcSalt=<SOURCE>

Props.conf

[app:jmcshift:logs]
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}
SHOULD_LINEMERGE=false
TRUNCATE=99999

Sample logs:
From all 9 sources the events starts with date as shown below:
2023-01-12 23:24:50.245 [error]...........................................

Same inputs.cong and props.conf in SIT and Production env.
Not sure what could be the issue.

Re: Not receiving data from particular source

richgalloway — Sun, 29 Jan 2023 01:00:43 GMT

Have you checked the permissions on the missing sources to make sure Splunk has read access?

Re: Not receiving data from particular source

Harish2 — Sun, 29 Jan 2023 01:52:31 GMT

Hi @richgalloway
how can i check that, can u please tell me????

Re: Not receiving data from particular source

richgalloway — Sun, 29 Jan 2023 03:42:35 GMT

Sign on to the source server and run

ls -ls /app/splunkser/ShiftNonMinJMC/ShiftNonMinJMC.log /app/splunkser/ShiftMinJMC/ShiftMinJMC.log /app/splunkser/ShiftBDRecordJMC/ShiftBDRecordJMC.log

This will tell you who owns the files and the groups which can access it. Use the groups command to find out the groups to which the Splunk user belongs. Contact your Linux admin for specific assistance.

Re: Not receiving data from particular source

Harish2 — Sun, 29 Jan 2023 14:24:35 GMT

I checked there is no permission issue, i can see other files with the same permission.

But not able to see data from mentioned sources

Re: Not receiving data from particular source

richgalloway — Sun, 29 Jan 2023 16:57:36 GMT

Here are a few other things to check.

Look in splunkd.log on the forwarders to see if there are messages about reading those sources.

If you use SELinux, have someone verify the settings allow Splunk to read the sources. If you can sign in as the Splunk user and read the files then Splunk itself should be able to read them.

Verify the sources are going to the right indexes.

Verify the timestamps in the sources are being onboarded correctly. Incorrect timestamps could make it hard to find data from the source. Try searching with earliest=0 latest=+1y.

Double-check the SPL used to search for the sources.