topic Re: Issue: 1 XML file is splitting into 2 event in Getting Data In

Issue: 1 XML file is splitting into 2 event

mariannedave — Mon, 15 Mar 2021 08:27:15 GMT

First Issue: I've been trying to ingest 1 XML file into 1 event only in Splunk. But Splunk always splitting it into 2 events.

Example XML file:
##################
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<IntegrationTransaction>
<TransactionMetaData xmlns="">
<SourceSystemName>SystemNameSource</SourceSystemName>
<TransactionType>ValidTrans</TransactionType>
<UniqueTransactionID>DFGDFGFG</UniqueTransactionID>
<TransactionDateTime>2021-03-12T17:38:02.725+01:00</TransactionDateTime>
</TransactionMetaData>
<Payload xmlns="">
<ValidatedSalesTransactions>
<Transaction>
<RetailID>XZ0051</RetailID>
</Transaction>
</ValidatedSalesTransactions>
</Payload>
</IntegrationTransaction>
##################

transforms.conf
##################
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[accept_xml_files]
REGEX = <?xml version
DEST_KEY = queue
FORMAT = indexQueue
##################

props.conf
##################
[test_XML_sourcetype]
BREAK_ONLY_BEFORE = goblygook
MAX_EVENTS = 200000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
pulldown_type = true
LEARN_MODEL = false
SHOULD_LINEMERGE = true
TRUNCATE = 0
kv_mode = xml
TRANSFORMS-set = setnull, accept_xml_files
##################

inputs.conf
##################
[monitor:///tmp/testXML/*.xml]
index = test_XML_index
sourcetype = test_XML_sourcetype
crcSalt = <SOURCE>
##################

Result in Splunk:
##################
First Event:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<IntegrationTransaction>
<TransactionMetaData xmlns="">
<SourceSystemName>SystemNameSource</SourceSystemName>
<TransactionType>ValidTrans</TransactionType>
<UniqueTransactionID>DFGDFGFG</UniqueTransactionID>
##################
##################
Second Event:
<TransactionDateTime>2021-03-12T17:38:02.725+01:00</TransactionDateTime>
</TransactionMetaData>
<Payload xmlns="">
<ValidatedSalesTransactions>
<Transaction>
<RetailID>XZ0051</RetailID>
</Transaction>
</ValidatedSalesTransactions>
</Payload>
</IntegrationTransaction>
##################

Note: Second event always starts with <TransactionDateTime>

Second Issue: Splunk indexing it not real time. Sometimes it takes 30mins-45mins to be available in Splunk.

Thank you.

Re: Issue: 1 XML file is splitting into 2 event

scelikok — Mon, 15 Mar 2021 09:08:00 GMT

If you do not set timestamp prefix Splunk uses possible timestamp pattern to break events;

Please try below props

[test_XML_sourcetype] BREAK_ONLY_BEFORE=\<\?xml version= CHARSET=UTF-8 MAX_EVENTS=1000 NO_BINARY_CHECK=true SHOULD_LINEMERGE=true TIME_PREFIX=\<TransactionDateTime\> category=Application disabled=false LINE_BREAKER=\<\?xml version= TRANSFORMS-set = setnull, accept_xml_files

Re: Issue: 1 XML file is splitting into 2 event

mariannedave — Mon, 15 Mar 2021 11:32:24 GMT

Thanks for the suggestion. I tried that but, still 1 XML file is being split into 2 event.

Re: Issue: 1 XML file is splitting into 2 event

Vardhan — Mon, 15 Mar 2021 11:43:13 GMT

Hi,

Can you try with below changes.

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\<\?xml version=

Re: Issue: 1 XML file is splitting into 2 event

scelikok — Mon, 15 Mar 2021 11:55:12 GMT

This was a tested config, are you sure you put these settings on indexers?

Re: Issue: 1 XML file is splitting into 2 event

mariannedave — Mon, 15 Mar 2021 11:57:20 GMT

Hi -

Thanks for the suggestion. I've tried that but, still 1 XML file is being split into 2 events.

Re: Issue: 1 XML file is splitting into 2 event

Vardhan — Mon, 15 Mar 2021 12:09:29 GMT

Hi,

Can you take a sample data and ingest it through Add data option. And Go to Advanced settings and Apply the Magic 6 and see where exactly is the issue.

Re: Issue: 1 XML file is splitting into 2 event

mariannedave — Tue, 16 Mar 2021 03:23:13 GMT

Hi - sorry, what is the Magic 6?

Re: Issue: 1 XML file is splitting into 2 event

Vardhan — Tue, 16 Mar 2021 04:37:11 GMT

Hi,

Ingest data through the Add-data option. And apply below six attributes on it.

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\<\?xml version=
TIME_PREFIX=\<TransactionDateTime\>
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q+%z:%z
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=20000

Re: Issue: 1 XML file is splitting into 2 event

mariannedave — Tue, 16 Mar 2021 06:38:57 GMT

I tried the sample data and ingest it through Add data option and config the Advanced settings based on your suggestion. It worked there. But when I deployed it via Deployment Server, 1XML file is still being split into 2 events.

Re: Issue: 1 XML file is splitting into 2 event

Vardhan — Tue, 16 Mar 2021 07:01:48 GMT

Hi,

Can you check the props.conf settings are placed in indexers are not?

Re: Issue: 1 XML file is splitting into 2 event

mariannedave — Tue, 16 Mar 2021 09:22:02 GMT

Thanks. It worked, need to be deployed also in Indexers.