log parsing failing across multiple sourcetypes

[alemack]

Hi folks,  our field parsing/extraction has broken across all sourcetypes (nginx, log4j, aws:elb's, fix,custom formats as well). The most recent infra event we had was an increase of file storage over a month ago.  If our error were related to a single sourcetype I would assume I have to review my props.conf file for the associated app and sourcetype,but in this case it appears something more systemic is occurring.

As someone with limited knowledge of splunk admin,where can I look to narrow my search to the root cause?  Trying to RTFM, and am familiar with the "general" log structure but not sure exactly what I'm looking for. (an error/exception on restart directly calling out a props.conf file? An index related exception? idk) Would btool help me confirm if my props.conf files are correctly loading? Is there something would indicate a failure of log parsing?

Splunk Enterprise single-instance 9.2.0.1 on an 8 Core 32GB instance

Cheers

//A

[deepakc]

It could any number of things.

If this was working before - what changed is what you want to find out first and work out where the problem is.

I guess if it was working before the props/transforms has either change, overwritten, or removed.

Has someone removed the props/transforms apps that those sourcetypes belong to from /opt/splunk/etc/apps.

You can check by starting here to find out where those sourcetypes live:

/opt/splunk/bin/splunk btool props list --debug

/opt/splunk/bin/splunk btool transforms list --debug

[alemack]

My post can be disregarded,  simple misinformation and not checking what/where people were running their field extractions.  (App vs Global permissions on Field and Transform extractions). Cheers nontheless and thanks for the pointers