topic Re: troubleshooting props.conf in Splunk Search

troubleshooting props.conf

mitag — Mon, 19 Oct 2020 04:47:57 GMT

If there's an error in a props.conf stanza for a particular sourcetype, where would it show up in the logs? E.g. a key like "SHOULD_LINEMERGE" is misspelled or one of the values is out of bounds or something else where Splunk is having issues with the stanza... Where in the logs would this show up?

My specific case: /opt/splunk/etc/slave-apps/_cluster/local/props.conf on the master (propagated to indexers):

[sweeper:abcnews] SHOULD_LINEMERGE = false MAX_TIMESTAMP_LOOKAHEAD = 30 TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N TIME_PREFIX = ^ TRUNCATE = 100000 MAX_EVENTS = 10000 EXTRACT-sweeper_abcnews = (?s)^\d+-\d+-\d+\s+\d+\:\d+\:\d+\,\d+\s+(?P<module>\S+)\s+\[(?P<processID>.+?)\]\s+(?P<log_level>\S+):\s+(?P<message>.*)$

The primary purposes of the stanza in props.conf is to allow multiline, define event breaks (timestamps, basically) and extract fields.

Splunk however appears to ignore the stanza altogether: multiline events get broken up, no fields are extracted.

The field extraction regex works well elsewhere: tested via "rex" at search time, in "field extractions" at search time, and also in props.conf in a dev splunk instance. It's as if Splunk is ignoring the stanza altogether in the production instance. Why, and how do I troubleshoot this?

Additional context:

/opt/splunk/etc/deployment-apps/_server_app_Linux_Clients/local/inputs.conf in DS, distributed to clients:

[monitor:///var/log/sweeper_abcnews.log] disabled = false index = sweeper sourcetype = sweeper:abcnews

The logs are getting ingested - yet Splunk appears to ignore the relevant stanza in props.conf as if it doesn't exist. Other stanzas in props.conf seem to be working - as multiline events in other sourcetypes do not get broken up.

Appreciate the help!

Re: troubleshooting props.conf

mitag — Mon, 19 Oct 2020 06:11:13 GMT

Just in case, extra context:

Splunk Enterprise 8.04.1, clustered indexers, single search head.

Re: troubleshooting props.conf

gcusello — Mon, 19 Oct 2020 06:55:09 GMT

Hi @mitag,

only one question: what's your architecture? have you any Heavy forwarders?

if yes, the props.conf must be also on HFs.

In addition to debug your parsing, I need a sample of your logs, could you share them?

Anyway, if you want to have multiline events, you should try with SHOULD_LINEMERGE = true.

About the field extraction (at search time) you have to put props.conf in the Search Head,

At least, it isn't clear for me your info: "The logs are getting ingested - yet Splunk appears to ignore the relevant stanza in props.conf as if it doesn't exist. ", what do you mean?

Are the not taken logs the ones in the stanza you shared or a different one?

Check if the logs you want to take in that stanza aren't also in another stanza, because Splunk takes a log once also if you have it in two or more stanzas.

Ciao.

Giuseppe

Re: troubleshooting props.conf

mitag — Mon, 19 Oct 2020 07:28:24 GMT

"what's your architecture? have you any Heavy forwarders?"

Yes to HFs. The logs in question don't pass through them.

Architecture: Master, SH, DS, three clustered indexers.

"In addition to debug your parsing, I need a sample of your logs, could you share them?"

2020-10-16 14:00:49,041 sweeper_abcnews [14839] ERROR: Unhandled exception: Traceback (most recent call last): File "/code/sweeper_abcnews/sweeper_abcnews.py", line 520, in <module> main(args.type, conn) File "/code/sweeper_abcnews/sweeper_abcnews.py", line 349, in main for outline in ascpOutput: File "/code/sweeper_abcnews/sweeper_abcnews.py", line 233, in runascp raise subprocess.CalledProcessError(return_code, fullCmdList) CalledProcessError: Command '['/bin/ascp', '-l', '1G', '-i', '--file-checksum=md5', '--partial-file-suffix=.partial', '--move-after-transfer=/__DONE', '--remove-empty-directories', 'some_user@11.22.33.44:/_UPLOADS/201016_test_file.mp4.ttml', '/Volumes/SomeVolume/File/Path/Here']' returned non-zero exit status 1

"Anyway, if you want to have multiline events, you should try with SHOULD_LINEMERGE = true."

You're probably right yet the same stanza worked in a dev instance for this same log, and works for other similar multiline sourcetypes and keeps the multiline events together. Unsure why. I'll try it, still.

"About the field extraction (at search time) you have to put props.conf in the Search Head"

I use settings -> fields -> field extraction on the SH in Splunk Web and it seems to work.... ("https://sh.splunk.local/en-US/manager/search/data/props/extractions" )

"At least, it isn't clear for me your info: "The logs are getting ingested - yet Splunk appears to ignore the relevant stanza in props.conf as if it doesn't exist. ", what do you mean?"

Single line events get ingested correctly, except for field extraction at index time. Multiline ones - do not: get broken. Field extraction specified in props.conf doesn't work. Beyond that - not sure how to make it clearer.

Re: troubleshooting props.conf

gcusello — Mon, 19 Oct 2020 07:53:52 GMT

Hi @mitag,

if the logs in question don't pass through the HF, this isn't the problem, but remember this issue.

About the pro.conf, I found only two differences with your:

SHOULD_LINEMERGE = True
MAX_TIMESTAMP_LOOKAHEAD = 23

Did you checked if there are other stanzas that address the same logs (files and/or folders)?

Ciao.

Giuseppe