Is there way to find the missing logs using some advanced search?

[acj]

Splunk logs missing for few scheduler jobs

Is there way to find the missing logs using some advanced search

[acj]

There are over 1000 configurations in splunk. I couldnt find anything like inputs.conf. 

Also is there a global search i can run which will search regardless of sourcetype etc

[gcusello]

Hi @acj,

as I said, if you indexed a log, it's searcheable, otherwise you cannot.

you can check the inputs.conf using the btool command by CLI (https://docs.splunk.com/Documentation/Splunk/9.0.1/Troubleshooting/Usebtooltotroubleshootconfigurati...)

I understand that's difficoult, but you shuld know which apps and inputs.conf could address a log.

For this reason it'as a best practice, analyze inputs on paper, to avoid this kind of searches.

Ciao.

Giuseppe

[acj]

The missing logs are from some jobs run daily.  Logs missing few days but present on most days. We are also analysing on whether log really missed part. Is there any checks on splunk to make sure it is not a logging issue. 

what configuration is needed here for analysis

[gcusello]

Hi @acj,

as I said you have to check if the data is logged with a wrong timestamp.

Then you can check if there's another equal log with another sourcetype, this means that you have two inputs that address the same log, but only one is indexed.

if you don't find anything analize your inputs to find the error.

I cannot help you in the first two check, for the third I could help you if you share your inputs.conf and a sample of the missed logs.

Ciao.

Giuseppe

[gcusello]

Hi @acj,

using Splunk you can search all the indexed logs, but if a log is missed (that means not indexed) you cannot search it.

The first question is: are these logs really missed?

if the missed logs have a timestamp in the first eleven days of the month maybe the timestamp recognition failed and you indexed the first of october as the 10th of January (Splunk uses american date format mm/dd/yyy, if you use an european format and you don't declare it maybe it could be read in wrong way).

The second question is: what are the missed logs?

if these logs are already indexed, Splunk doesn't index a log twice, check if there another equal log.

Third question: if a log is really missed, why is it missed?

This is possible only analyzing your data and your input configuration, but it isn't possible with the few information shared.

Ciao.

Giuseppe