Getting Data In

Issue: 1 XML file is splitting into 2 event

mariannedave
Explorer

First Issue: I've been trying to ingest 1 XML file into 1 event only in Splunk. But Splunk always splitting it into 2 events.

Example XML file:
##################
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<IntegrationTransaction>
<TransactionMetaData xmlns="">
<SourceSystemName>SystemNameSource</SourceSystemName>
<TransactionType>ValidTrans</TransactionType>
<UniqueTransactionID>DFGDFGFG</UniqueTransactionID>
<TransactionDateTime>2021-03-12T17:38:02.725+01:00</TransactionDateTime>
</TransactionMetaData>
<Payload xmlns="">
<ValidatedSalesTransactions>
<Transaction>
<RetailID>XZ0051</RetailID>
</Transaction>
</ValidatedSalesTransactions>
</Payload>
</IntegrationTransaction>
##################

transforms.conf
##################
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[accept_xml_files]
REGEX = <?xml version
DEST_KEY = queue
FORMAT = indexQueue
##################

props.conf
##################
[test_XML_sourcetype]
BREAK_ONLY_BEFORE = goblygook
MAX_EVENTS = 200000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
pulldown_type = true
LEARN_MODEL = false
SHOULD_LINEMERGE = true
TRUNCATE = 0
kv_mode = xml
TRANSFORMS-set = setnull, accept_xml_files
##################

inputs.conf
##################
[monitor:///tmp/testXML/*.xml]
index = test_XML_index
sourcetype = test_XML_sourcetype
crcSalt = <SOURCE>
##################

Result in Splunk:
##################
First Event:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<IntegrationTransaction>
<TransactionMetaData xmlns="">
<SourceSystemName>SystemNameSource</SourceSystemName>
<TransactionType>ValidTrans</TransactionType>
<UniqueTransactionID>DFGDFGFG</UniqueTransactionID>
##################
##################
Second Event:
<TransactionDateTime>2021-03-12T17:38:02.725+01:00</TransactionDateTime>
</TransactionMetaData>
<Payload xmlns="">
<ValidatedSalesTransactions>
<Transaction>
<RetailID>XZ0051</RetailID>
</Transaction>
</ValidatedSalesTransactions>
</Payload>
</IntegrationTransaction>
##################

Note: Second event always starts with <TransactionDateTime>

Second Issue: Splunk indexing it not real time. Sometimes it takes 30mins-45mins to be available in Splunk.

Thank you.

 

 

 

Labels (4)
Tags (1)
0 Karma
1 Solution

scelikok
Champion

This was a tested config, are you sure you put these settings on indexers?  

If this reply helps you an upvote is appreciated.

View solution in original post

scelikok
Champion

This was a tested config, are you sure you put these settings on indexers?  

If this reply helps you an upvote is appreciated.

View solution in original post

mariannedave
Explorer

Thanks. It worked, need to be deployed also in Indexers.

0 Karma

mariannedave
Explorer

I tried the sample data and ingest it through Add data option and config the Advanced settings based on your suggestion. It worked there. But when I deployed it via Deployment Server, 1XML file is still being split into 2 events.

0 Karma

Vardhan
Contributor

Hi,

Can you check the props.conf settings are placed in indexers are not?

Vardhan
Contributor

Hi,

Can you try with below changes.

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\<\?xml version=

mariannedave
Explorer

Hi - 

Thanks for the suggestion. I've tried that but, still 1 XML file is being split into 2 events.

0 Karma

Vardhan
Contributor

Hi,

Can you take a sample data and ingest it through Add data option. And Go to Advanced settings and Apply the Magic 6 and see where exactly is the issue. 

mariannedave
Explorer

Hi - sorry, what is the Magic 6?

0 Karma

Vardhan
Contributor

Hi,

Ingest data through the Add-data option. And apply below six attributes on it.

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\<\?xml version=
TIME_PREFIX=\<TransactionDateTime\>
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q+%z:%z
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=20000

scelikok
Champion

If you do not set timestamp prefix Splunk uses possible timestamp pattern to break events;

Please try below props

[test_XML_sourcetype]
BREAK_ONLY_BEFORE=\<\?xml version=
CHARSET=UTF-8
MAX_EVENTS=1000
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIME_PREFIX=\<TransactionDateTime\>
category=Application
disabled=false
LINE_BREAKER=\<\?xml version=
TRANSFORMS-set = setnull, accept_xml_files
If this reply helps you an upvote is appreciated.

mariannedave
Explorer

Thanks for the suggestion. I tried that but, still 1 XML file is being split into 2 event.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!