Why can't we see indexer internal logs?

[bhsakarchourasi]

Hi All,

we are unable to see the indexers internal logs in _internal index, except mongodb logs. we verified that the input configuration is present in default inputs.conf but while checking in splunkd.log there is no TailReader process logs, the only logs which is related to seekptr (generally seen when there is rollover of the log file). We even tried configuring inputs.conf with different index and sourcetype for splunkd.log but it didn't worked.

Also there are almost half of the UFs stopped reporting to indexers. there are 6 indexers in cluster.

Any idea about the issue will be very helpful.

Thanks,

Bhaskar    

[isoutamo]

Can you check that those logs are present and working on filesystem level? There could be some event on those which told the reason why splunk cannot index those?

[bhsakarchourasi · Accepted solution]

This is solved but I am still not convinced with resolution, after spending hours on troubleshooting we circled back to the changes performed in last one week, then found a small typo in props.conf of one of the app pushed to indexers and search heads. disabling that app in indexers resolved the issue (later corrected the typo in idx and SHs), not sure how a typo in a app halted internal logs of indexers and some of the UFs. We have asked splunk to help investigate the issue.   

[gcusello]

Hi @bhsakarchourasi,

it depends on what's the type you found.

Anyway, good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[isoutamo]

Nice to hear that you solved it. One way to analyze it more, is install all your configurations to standalone instance just like those was on production. Then use btool to check how those are expanded on that node. That way you see why that typo has this kind of side effect.