Why does field extraction work in dev environment but not in prod environment?

[cdoebert]

I ingested a CSV into our dev environment, had it create the props stanza with the field extractions I wanted, and copied this over into our prod props.conf.

This works as expected in dev; I can reliably add more logs that it picks up the fields on and does the extracts correctly. When the logs are ingested in prod, however, the source type is picked up but the extracts aren't being done. I open up the Source Types UI in both environments and they're identical.

These are both using 6.4.3 (we're upgrading soon), and Splunk is not barking at me on restart that there are errors in my conf files. The only difference in environments is that dev is a standalone all-in-one, and prod is clustered search heads and non-clustered indexers.

Stanza that was created in dev below:

[obfuscated_stanza]
DATETIME_CONFIG =
FIELD_NAMES = Timestamp,Method,Timing,TransactionID,TrackingNumber,TransactionName,processID,threadID
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true

[jkat54 · Accepted solution]

you could use the EXTRACT or REPORT in props.conf on the search heads if you don't want to use indexed extractions. Tip: most folks don't actually need indexed extractions.

if you share a sample of the data we can help you write a perfect props.conf.

[cdoebert]

Yep, when I deleted the FIELD_NAMES and INDEXED_EXTRACTIONS lines and replaced it with an EXTRACT-Obfuscated regex line, the extracts work!

If you can improve on my regexing for extra credit, please do!

EXTRACT-Obfuscated = ^(?P<timeStamp>[^,]+),(?P<method>[^,]+),(?P<timing>[^,]+),(?P<transactionID>[^,]+),(?P<trackingNumber>[^,]+),(?P<transactionName>[^,]+),(?P<processID>[^,]+),(?P<threadID>[^,]+)

[dmaislin_splunk]

Are your props on the forwarder because if you are using indexed_extractions then the only place it should be is on the forwarder.

[cdoebert]

They are not; only the indexers and search heads.

How would this need to look if that's the case?

[dmaislin_splunk]

Just move the props.conf that are on your indexers for that specific sourcetype to the same folder as the inputs.conf on your forwarder. Those settings for indexed_extractions are only for the forwarders.

[cdoebert]

Is there any way to do these field extracts without putting a props.conf on a forwarder, then?

The only way I can do this the way our environment is structured is to create an entirely new app with just this props.conf for just this stanza and push it out to all of the relevant forwarders and all of our indexers.