Issue: 1 XML file is splitting into 2 event

[mariannedave]

First Issue: I've been trying to ingest 1 XML file into 1 event only in Splunk. But Splunk always splitting it into 2 events.

Example XML file:
##################
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<IntegrationTransaction>
<TransactionMetaData xmlns="">
<SourceSystemName>SystemNameSource</SourceSystemName>
<TransactionType>ValidTrans</TransactionType>
<UniqueTransactionID>DFGDFGFG</UniqueTransactionID>
<TransactionDateTime>2021-03-12T17:38:02.725+01:00</TransactionDateTime>
</TransactionMetaData>
<Payload xmlns="">
<ValidatedSalesTransactions>
<Transaction>
<RetailID>XZ0051</RetailID>
</Transaction>
</ValidatedSalesTransactions>
</Payload>
</IntegrationTransaction>
##################

transforms.conf
##################
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[accept_xml_files]
REGEX = <?xml version
DEST_KEY = queue
FORMAT = indexQueue
##################

props.conf
##################
[test_XML_sourcetype]
BREAK_ONLY_BEFORE = goblygook
MAX_EVENTS = 200000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
pulldown_type = true
LEARN_MODEL = false
SHOULD_LINEMERGE = true
TRUNCATE = 0
kv_mode = xml
TRANSFORMS-set = setnull, accept_xml_files
##################

inputs.conf
##################
[monitor:///tmp/testXML/*.xml]
index = test_XML_index
sourcetype = test_XML_sourcetype
crcSalt = <SOURCE>
##################

Result in Splunk:
##################
First Event:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<IntegrationTransaction>
<TransactionMetaData xmlns="">
<SourceSystemName>SystemNameSource</SourceSystemName>
<TransactionType>ValidTrans</TransactionType>
<UniqueTransactionID>DFGDFGFG</UniqueTransactionID>
##################
##################
Second Event:
<TransactionDateTime>2021-03-12T17:38:02.725+01:00</TransactionDateTime>
</TransactionMetaData>
<Payload xmlns="">
<ValidatedSalesTransactions>
<Transaction>
<RetailID>XZ0051</RetailID>
</Transaction>
</ValidatedSalesTransactions>
</Payload>
</IntegrationTransaction>
##################

Note: Second event always starts with <TransactionDateTime>

Second Issue: Splunk indexing it not real time. Sometimes it takes 30mins-45mins to be available in Splunk.

Thank you.

[scelikok · Accepted solution]

This was a tested config, are you sure you put these settings on indexers?  

[mariannedave]

Thanks. It worked, need to be deployed also in Indexers.

[mariannedave]

I tried the sample data and ingest it through Add data option and config the Advanced settings based on your suggestion. It worked there. But when I deployed it via Deployment Server, 1XML file is still being split into 2 events.

[Vardhan]

Hi,

Can you check the props.conf settings are placed in indexers are not?

[Vardhan]

Hi,

Can you try with below changes.

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\<\?xml version=

[mariannedave]

Hi - 

Thanks for the suggestion. I've tried that but, still 1 XML file is being split into 2 events.

[Vardhan]

Hi,

Can you take a sample data and ingest it through Add data option. And Go to Advanced settings and Apply the Magic 6 and see where exactly is the issue. 

[mariannedave]

Hi - sorry, what is the Magic 6?

[Vardhan]

Hi,

Ingest data through the Add-data option. And apply below six attributes on it.

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\<\?xml version=
TIME_PREFIX=\<TransactionDateTime\>
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q+%z:%z
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=20000

[scelikok]

If you do not set timestamp prefix Splunk uses possible timestamp pattern to break events;

Please try below props

[test_XML_sourcetype]
BREAK_ONLY_BEFORE=\<\?xml version=
CHARSET=UTF-8
MAX_EVENTS=1000
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIME_PREFIX=\<TransactionDateTime\>
category=Application
disabled=false
LINE_BREAKER=\<\?xml version=
TRANSFORMS-set = setnull, accept_xml_files

[mariannedave]

Thanks for the suggestion. I tried that but, still 1 XML file is being split into 2 event.