What is causing issue with data ingestion for xml?

[Strangertinz]

Hi community, 

I have an issue where I am ingesting some xml data but the data coming in is very sporadic. Any idea what could be causing this issue? 

[Strangertinz]

So I have a XML log file that is constantly being written into (about 100 entry per minute) however,  when I search for the data in Splunk I am only seeing sporadic results of the data in Splunk where I see results for 10 minutes then nothing for the next 20 and so on and so forth 

[richgalloway]

Please share the inputs.conf and related props.conf settings for the file.

When data is not received are the events lost or delayed?

What is the query you're using to find the events?

[richgalloway]

How is the data getting from the source to Splunk?

Make sure your data is valid XML as Splunk will not parse invalid XML or events that partially XML.

Please share your inputs.conf and related props.conf setttings.

[Strangertinz]

The data is being parse correctly and the data is being ingested through a UF

[gcusello]

Hi @Strangertinz ,

your information are too poor to try to help you, could you share more datails about your issue?

Ciao.

Giuseppe

[Strangertinz]

@gcusello 

I am having trouble with ingesting my data into Splunk consistently. I have an XML log file that is constantly being written into (about 100 entry per minute) however,  when I search for the data in Splunk I am only seeing sporadic results of the data in Splunk where I see results for 10 minutes then nothing for the next 20 and so on and so forth . 

I have my inputs and props config below. 


inputs config:

[monitor:///var/log/sample_xml_file.xml]
disabled = false
index = sample_xml_index
sourcetype= sample_xml_st

props.conf:

---------------------

[ sample_xml_st ]
CHARSET=UTF-8
KV_MODE=xml
LINE_BREAKER=(<log_entry>)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=FALSE
TIME_FORMAT=%Y%m%d-%H:%M:%S
TIME_PREFIX=<log_time>
TRUNCATE=0
description=describing props config
disabled=false
pulldown_type=1
TZ=-05:00

---------------------



Sample xml log:

<?xml version="1.0" encoding="utf-8" ?>
<log>
  <log_entry>
    <log_time>20230724-05:42:00</log_time>
    <description>some random data 1</description>
  </log_entry>
   <log_entry>
    <log_time>20230724-05:43:00</log_time>
    <description>some random data 2</description>
  </log_entry>
   <log_entry>
    <log_time>20230724-05:43:20</log_time>
    <description>some random data 3</description>
  </log_entry>
</log>

And this xml log file gets constantly written into with the a new log_entry 

[gcusello]

Hi @Strangertinz ,

don't use KV-Mode = XML, but INDEXED_EXTRACTIONS=xml in props.conf:

[ sample_xml_st ]
CHARSET=UTF-8
KV_MODE=none
INDEXED_EXTRACTIONS = XML
LINE_BREAKER=(<log_entry>)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y%m%d-%H:%M:%S
TIME_PREFIX=<log_time>
TRUNCATE=0
description=describing props config
disabled=false
pulldown_type=1
TZ=-05:00

Ciao.

Giuseppe

[Strangertinz]

Hi @gcusello 

Are you suggesting that the KV_MODE=XML setting is causing the data to come in sporadically? 

[gcusello]

Hi @Strangertinz 

I usually use INDEXED_EXTRACTIONS and not KV-Mode.

then if you hav SHOULD_LINEMERGE=false you have an event for each row, maybe this is the issue.

Try my configuration.

Then, as @richgalloway is asking: how did you find that events are sporadically?

then have y9u multiline events or single line events? they should be multiline but with SHOULD_LINEMERGE=false you have single line events.

Ciao.

Giuseppe

[richgalloway]

A still-unanswered question is how you determine the events arrive sporadically.  What SPL are you using to determine that?  Have you verified events are not concatenated (which would explain some of the apparently sporadic behavior)?

[Strangertinz]

I am searching the data with the right index and sourcetype in my SPL and comparing the results with the data that is being written to the log file on the host. I see data for last 15 minutes, and then I don't for another 10 or so minutes when I can still observe the log file being written into per minute..but not at the same rate in Splunk.