XML field extraction not working on distributed search head

[jplumsdaine22]

I have a 3 node search head cluster that backs on to a single indexer (its a test environment). All servers are 6.3.2. For one particular sourcetype, the search time xml field extractions do not function on the search heads. However, if you run the same search on the Indexer or Deployer UI, the extractions work just fine

KV_MODE = xml is set in props.conf for the relevant sourcetype. Btool output is IDENTICAL on all servers. (except for a single TRANSFORMS on the indexer that renames the source path)

What could be causing this?

Btool output on indexer

/local/mnt/splunk/etc/system/local/props.conf                           [xml_oracleaudit]
/local/mnt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/local/mnt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/local/mnt/splunk/etc/system/local/props.conf                           BREAK_ONLY_BEFORE = <AuditRecord>
/local/mnt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/local/mnt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/local/mnt/splunk/etc/system/local/props.conf                           DATETIME_CONFIG = /etc/datetime.xml
/local/mnt/splunk/etc/system/default/props.conf                         HEADER_MODE =
/local/mnt/splunk/etc/system/local/props.conf                           KV_MODE = xml
/local/mnt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/local/mnt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/local/mnt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/local/mnt/splunk/etc/system/local/props.conf                           MAX_TIMESTAMP_LOOKAHEAD = 200
/local/mnt/splunk/etc/system/local/props.conf                           MUST_BREAK_AFTER = </AuditRecord>
/local/mnt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER =
/local/mnt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE =
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/local/mnt/splunk/etc/system/local/props.conf                           SHOULD_LINEMERGE = true
/local/mnt/splunk/etc/system/local/props.conf                           TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6NZ
/local/mnt/splunk/etc/system/local/props.conf                           TIME_PREFIX = <Extended_Timestamp>
/local/mnt/splunk/etc/system/default/props.conf                         TRANSFORMS =
/local/mnt/splunk/etc/system/local/props.conf                           TRANSFORMS-1_source = abbreviate_oracle_source
/local/mnt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/local/mnt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/local/mnt/splunk/etc/system/default/props.conf                         maxDist = 100
/local/mnt/splunk/etc/system/default/props.conf                         priority =
/local/mnt/splunk/etc/system/default/props.conf                         sourcetype =

Btool output on search_heads

/local/mnt/splunk/etc/apps/search_heads/default/props.conf              [xml_oracleaudit]
/local/mnt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/local/mnt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              BREAK_ONLY_BEFORE = <AuditRecord>
/local/mnt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/local/mnt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              DATETIME_CONFIG = /etc/datetime.xml
/local/mnt/splunk/etc/system/default/props.conf                         HEADER_MODE =
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              KV_MODE = xml
/local/mnt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/local/mnt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/local/mnt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/local/mnt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              MAX_TIMESTAMP_LOOKAHEAD = 200
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              MUST_BREAK_AFTER = </AuditRecord>
/local/mnt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER =
/local/mnt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE =
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/local/mnt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              SHOULD_LINEMERGE = true
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6NZ
/local/mnt/splunk/etc/apps/search_heads/default/props.conf              TIME_PREFIX = <Extended_Timestamp>
/local/mnt/splunk/etc/system/default/props.conf                         TRANSFORMS =
/local/mnt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/local/mnt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/local/mnt/splunk/etc/system/default/props.conf                         maxDist = 100
/local/mnt/splunk/etc/system/default/props.conf                         priority =
/local/mnt/splunk/etc/system/default/props.conf                         sourcetype =

I don't think it\s causing any issues but here is the btool output on that TRANSFORMS stanza I was talking about that is on on the indexer

/local/mnt/splunk/etc/system/local/transforms.conf                           [abbreviate_oracle_source]
/local/mnt/splunk/etc/system/default/transforms.conf                         CAN_OPTIMIZE = True
/local/mnt/splunk/etc/system/default/transforms.conf                         CLEAN_KEYS = True
/local/mnt/splunk/etc/system/default/transforms.conf                         DEFAULT_VALUE =
/local/mnt/splunk/etc/system/local/transforms.conf                           DEST_KEY = MetaData:Source
/local/mnt/splunk/etc/system/local/transforms.conf                           FORMAT = source::/$1/oracle/admin/$2/audit/$3.xml
/local/mnt/splunk/etc/system/default/transforms.conf                         KEEP_EMPTY_VALS = False
/local/mnt/splunk/etc/system/default/transforms.conf                         LOOKAHEAD = 4096
/local/mnt/splunk/etc/system/default/transforms.conf                         MV_ADD = False
/local/mnt/splunk/etc/system/local/transforms.conf                           REGEX = \/(\w+?)\/oracle\/admin?\/(\w+)\/audit\/(\w+?)_
/local/mnt/splunk/etc/system/local/transforms.conf                           SOURCE_KEY = MetaData:Source
/local/mnt/splunk/etc/system/default/transforms.conf                         WRITE_META = False

[jplumsdaine22 · Accepted solution]

So the reason this was failing is that the app was missing the default.meta file. So I created a file called myapp/metadata/default.meta and entered the following

[]
export=system

Redeployed the shcluster bundle and now the field extractions work just fine

[jplumsdaine22]

Thought it was fast mode causing this but the problem still exists.

[krish3]

Try placing your props.conf containing xml extraction on forwarder or datasource which is picking up the file.

I know this sounds weird but I got it worked for JSON and XML by placing the props on fwd or HWF.

Let me know how it goes.