JSON Log - Search time field extraction not working on multi machine environment

[rick_harrison]

I have a JSON format log file.

When this is ingested by a single server installation of splunk (4.3.4), fields are correctly extracted at search time (and appear as interesting fields). The following stanaza is used in the props.conf
[alarm_log]
KV_MODE=JSON
TIME_FORMAT=%d/%m/%Y %T.%3N
TIME_PREFIX = LogTimeStamp":"

Moving to QA. I have:
1 Machine with a Heavy Forwader (stanza):
[alarm_log]
TIME_FORMAT=%d/%m/%Y %T.%3N
TIME_PREFIX = LogTimeStamp":"

1 Machine with the index and search head (stanza):
[alarm_log]
KV_MODE=JSON

When the exact same log file is ingested, the fields are not extracted, and I must use spath command in each search to force this extraction.

what am i missing?

[lguinn2]

When you using a heavy forwarder, parsing happens on the forwarder. While I am pretty sure that the KV_MODE should apply only at search time, I wonder if you are missing something because it appears only on the search head / indexer.
I would try adding the KV_MODE=JSON to the heavy forwarder as well as the indexer/search head. It won't hurt to have it in two places. Try it and see if that works. There is really no down side.

You might also consider installing the free Splunk on Splunk app (SoS). This would help you see all the places that might be affecting the configuration of the alarm_log sourcetype. Or try btool