Getting Data In

What is causing issue with data ingestion for xml?

Strangertinz
Path Finder

Hi community, 

I have an issue where I am ingesting some xml data but the data coming in is very sporadic. Any idea what could be causing this issue? 

 

Labels (2)
0 Karma

Strangertinz
Path Finder

So I have a XML log file that is constantly being written into (about 100 entry per minute) however,  when I search for the data in Splunk I am only seeing sporadic results of the data in Splunk where I see results for 10 minutes then nothing for the next 20 and so on and so forth 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please share the inputs.conf and related props.conf settings for the file.

When data is not received are the events lost or delayed?

What is the query you're using to find the events?

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

How is the data getting from the source to Splunk?

Make sure your data is valid XML as Splunk will not parse invalid XML or events that partially XML.

Please share your inputs.conf and related props.conf setttings.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Strangertinz
Path Finder

The data is being parse correctly and the data is being ingested through a UF

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Strangertinz ,

your information are too poor to try to help you, could you share more datails about your issue?

Ciao.

Giuseppe

0 Karma

Strangertinz
Path Finder

@gcusello 

I am having trouble with ingesting my data into Splunk consistently. I have an XML log file that is constantly being written into (about 100 entry per minute) however,  when I search for the data in Splunk I am only seeing sporadic results of the data in Splunk where I see results for 10 minutes then nothing for the next 20 and so on and so forth . 

I have my inputs and props config below. 


inputs config:


[monitor:///var/log/sample_xml_file.xml]
disabled = false
index = sample_xml_index
sourcetype= sample_xml_st

 

 

 

props.conf:

---------------------

[ sample_xml_st ]
CHARSET=UTF-8
KV_MODE=xml
LINE_BREAKER=(<log_entry>)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=FALSE
TIME_FORMAT=%Y%m%d-%H:%M:%S
TIME_PREFIX=<log_time>
TRUNCATE=0
description=describing props config
disabled=false
pulldown_type=1
TZ=-05:00

---------------------



Sample xml log:

<?xml version="1.0" encoding="utf-8" ?>
<log>
  <log_entry>
    <log_time>20230724-05:42:00</log_time>
    <description>some random data 1</description>
  </log_entry>
   <log_entry>
    <log_time>20230724-05:43:00</log_time>
    <description>some random data 2</description>
  </log_entry>
   <log_entry>
    <log_time>20230724-05:43:20</log_time>
    <description>some random data 3</description>
  </log_entry>
</log>

And this xml log file gets constantly written into with the a new log_entry 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Strangertinz ,

don't use KV-Mode = XML, but INDEXED_EXTRACTIONS=xml in props.conf:

[ sample_xml_st ]
CHARSET=UTF-8
KV_MODE=none
INDEXED_EXTRACTIONS = XML
LINE_BREAKER=(<log_entry>)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y%m%d-%H:%M:%S
TIME_PREFIX=<log_time>
TRUNCATE=0
description=describing props config
disabled=false
pulldown_type=1
TZ=-05:00

Ciao.

Giuseppe

0 Karma

Strangertinz
Path Finder

Hi @gcusello 

 

Are you suggesting that the KV_MODE=XML setting is causing the data to come in sporadically? 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Strangertinz 

I usually use INDEXED_EXTRACTIONS and not KV-Mode.

then if you hav SHOULD_LINEMERGE=false you have an event for each row, maybe this is the issue.

Try my configuration.

Then, as @richgalloway is asking: how did you find that events are sporadically?

then have y9u multiline events or single line events? they should be multiline but with SHOULD_LINEMERGE=false you have single line events.

Ciao.

Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

A still-unanswered question is how you determine the events arrive sporadically.  What SPL are you using to determine that?  Have you verified events are not concatenated (which would explain some of the apparently sporadic behavior)?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Strangertinz
Path Finder

I am searching the data with the right index and sourcetype in my SPL and comparing the results with the data that is being written to the log file on the host. I see data for last 15 minutes, and then I don't for another 10 or so minutes when I can still observe the log file being written into per minute..but not at the same rate in Splunk. 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...