topic Re: data ingestion for xml in Getting Data In

What is causing issue with data ingestion for xml?

Strangertinz — Tue, 25 Jul 2023 17:21:52 GMT

Hi community,

I have an issue where I am ingesting some xml data but the data coming in is very sporadic. Any idea what could be causing this issue?

Re: data ingestion for xml

gcusello — Mon, 24 Jul 2023 18:48:43 GMT

Hi @Strangertinz ,

your information are too poor to try to help you, could you share more datails about your issue?

Ciao.

Giuseppe

Re: data ingestion for xml

richgalloway — Mon, 24 Jul 2023 18:54:00 GMT

How is the data getting from the source to Splunk?

Make sure your data is valid XML as Splunk will not parse invalid XML or events that partially XML.

Please share your inputs.conf and related props.conf setttings.

Re: data ingestion for xml

Strangertinz — Mon, 24 Jul 2023 19:02:27 GMT

So I have a XML log file that is constantly being written into (about 100 entry per minute) however, when I search for the data in Splunk I am only seeing sporadic results of the data in Splunk where I see results for 10 minutes then nothing for the next 20 and so on and so forth

Re: data ingestion for xml

Strangertinz — Mon, 24 Jul 2023 19:03:50 GMT

The data is being parse correctly and the data is being ingested through a UF

Re: data ingestion for xml

richgalloway — Mon, 24 Jul 2023 19:27:11 GMT

Please share the inputs.conf and related props.conf settings for the file.

When data is not received are the events lost or delayed?

What is the query you're using to find the events?

Re: data ingestion for xml

Strangertinz — Tue, 25 Jul 2023 17:12:30 GMT

@gcusello

I am having trouble with ingesting my data into Splunk consistently. I have an XML log file that is constantly being written into (about 100 entry per minute) however, when I search for the data in Splunk I am only seeing sporadic results of the data in Splunk where I see results for 10 minutes then nothing for the next 20 and so on and so forth .

I have my inputs and props config below.

inputs config:

[monitor:///var/log/sample_xml_file.xml]
disabled = false
index = sample_xml_index
sourcetype= sample_xml_st

props.conf:

---------------------

[ sample_xml_st ]
CHARSET=UTF-8
KV_MODE=xml
LINE_BREAKER=(<log_entry>)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=FALSE
TIME_FORMAT=%Y%m%d-%H:%M:%S
TIME_PREFIX=<log_time>
TRUNCATE=0
description=describing props config
disabled=false
pulldown_type=1
TZ=-05:00

---------------------

Sample xml log:

<?xml version="1.0" encoding="utf-8" ?>
<log>
  <log_entry>
    <log_time>20230724-05:42:00</log_time>
    <description>some random data 1</description>
  </log_entry>
   <log_entry>
    <log_time>20230724-05:43:00</log_time>
    <description>some random data 2</description>
  </log_entry>
   <log_entry>
    <log_time>20230724-05:43:20</log_time>
    <description>some random data 3</description>
  </log_entry>
</log>

And this xml log file gets constantly written into with the a new log_entry

Re: data ingestion for xml

gcusello — Tue, 25 Jul 2023 18:07:33 GMT

Hi @Strangertinz ,

don't use KV-Mode = XML, but INDEXED_EXTRACTIONS=xml in props.conf:

[ sample_xml_st ] CHARSET=UTF-8 KV_MODE=none INDEXED_EXTRACTIONS = XML LINE_BREAKER=(<log_entry>) NO_BINARY_CHECK=true SHOULD_LINEMERGE=true TIME_FORMAT=%Y%m%d-%H:%M:%S TIME_PREFIX=<log_time> TRUNCATE=0 description=describing props config disabled=false pulldown_type=1 TZ=-05:00

Ciao.

Giuseppe

Re: data ingestion for xml

Strangertinz — Tue, 25 Jul 2023 18:17:35 GMT

Hi @gcusello

Are you suggesting that the KV_MODE=XML setting is causing the data to come in sporadically?

Re: data ingestion for xml

richgalloway — Tue, 25 Jul 2023 18:25:09 GMT

A still-unanswered question is how you determine the events arrive sporadically. What SPL are you using to determine that? Have you verified events are not concatenated (which would explain some of the apparently sporadic behavior)?

Re: data ingestion for xml

Strangertinz — Tue, 25 Jul 2023 19:14:34 GMT

I am searching the data with the right index and sourcetype in my SPL and comparing the results with the data that is being written to the log file on the host. I see data for last 15 minutes, and then I don't for another 10 or so minutes when I can still observe the log file being written into per minute..but not at the same rate in Splunk.

Re: data ingestion for xml

gcusello — Wed, 26 Jul 2023 15:43:48 GMT

Hi @Strangertinz

I usually use INDEXED_EXTRACTIONS and not KV-Mode.

then if you hav SHOULD_LINEMERGE=false you have an event for each row, maybe this is the issue.

Try my configuration.

Then, as @richgalloway is asking: how did you find that events are sporadically?

then have y9u multiline events or single line events? they should be multiline but with SHOULD_LINEMERGE=false you have single line events.

Ciao.

Giuseppe