Monitoring Splunk

log parsing failing across multiple sourcetypes

alemack
Engager

Hi folks,  our field parsing/extraction has broken across all sourcetypes (nginx, log4j, aws:elb's, fix,custom formats as well). The most recent infra event we had was an increase of file storage over a month ago.  If our error were related to a single sourcetype I would assume I have to review my props.conf file for the associated app and sourcetype,but in this case it appears something more systemic is occurring.

As someone with limited knowledge of splunk admin,where can I look to narrow my search to the root cause?  Trying to RTFM, and am familiar with the "general" log structure but not sure exactly what I'm looking for. (an error/exception on restart directly calling out a props.conf file? An index related exception? idk) Would btool help me confirm if my props.conf files are correctly loading? Is there something would indicate a failure of log parsing?

 

Splunk Enterprise single-instance 9.2.0.1 on an 8 Core 32GB instance

 

Cheers

//A

Labels (3)
0 Karma

deepakc
Builder

It could any number of things.

If this was working before - what changed is what you want to find out first and work out where the problem is.

I guess if it was working before the props/transforms has either change, overwritten, or removed.

Has someone removed the props/transforms apps that those sourcetypes belong to from /opt/splunk/etc/apps.

You can check by starting here to find out where those sourcetypes live:

/opt/splunk/bin/splunk btool props list --debug

/opt/splunk/bin/splunk btool transforms list --debug

alemack
Engager

My post can be disregarded,  simple misinformation and not checking what/where people were running their field extractions.  (App vs Global permissions on Field and Transform extractions). Cheers nontheless and thanks for the pointers

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...