topic Re: Ingest entire XML file in Getting Data In

Ingest entire XML file

nicofantinato — Fri, 13 Nov 2020 14:18:39 GMT

Hello everybody,

we are monitoring via Universal Forwarder several directories with a large XML file in there (around 1000 lines). These files changes every few seconds, and the change also involves the timestamp which is written in the first 256 bytes of the file.

I need to ingest these files entirely at every change but, instead, Splunk ingest me these files only one time every some hours or even days. Do you have any suggestion on how can I fix this?

Here's the props.conf in my heavy forwarders (we have a distributed environment):
[xml_atm]
TRANSFORMS-routing=xmlatm-route
SHOULD_LINEMERGE=true
LINE_BREAKER=(?:restart)([\r\n]+)
CHARSET=ISO-8859-1
CHECK_METHOD = modtime
MAX_EVENTS=4000
TRUNCATE=0
disabled=false
TIME_PREFIX=restart-flag="
REPORT-xmlext=xml-extr

While inputs.conf in UF is this:
[monitor://D:\ABC\Monitor\Monitor\Inputs\*\*.xml]
disabled = 0
host_segment = 5
index = my_index
sourcetype = xml_atm

Thanks in advance.

Re: Ingest entire XML file

richgalloway — Fri, 13 Nov 2020 13:46:48 GMT

Please share the relevant inputs.conf stanza from the UFs.

Re: Ingest entire XML file

nicofantinato — Fri, 13 Nov 2020 13:55:12 GMT

Here is the inputs.conf on the UF
[monitor://D:\ABC\Monitor\Monitor\Inputs\*\*.xml]
disabled = 0
host_segment = 5
index = my_index
sourcetype = xml_atm

Re: Ingest entire XML file

nicofantinato — Fri, 20 Nov 2020 11:32:10 GMT

Hi, do you have any suggestion? I'm still unable to ingest entire XML files every time their modification time changes.

Re: Ingest entire XML file

richgalloway — Fri, 20 Nov 2020 14:30:24 GMT

Splunk tries its best to avoid re-indexing entire files that are ingesting via a monitor stanza. I'm not aware of any setting to override that behavior.

Consider using batch input, instead. Splunk will read the entire file, but will delete it afterward. That means your application must be prepared to re-create the file. It also runs the risk of a race condition between Splunk and your app. Can the application be configured to write a new file instead of overwriting existing files?

Re: Ingest entire XML file

nicofantinato — Wed, 09 Dec 2020 16:54:21 GMT

Hi, turned out we also needed to add directive crcSalt = <SOURCE> in inputs.conf on UFs. Adding this all worked as expected.

inputs.conf became simply:

[monitor://D:\ABC\Monitor\Monitor\Inputs\*\*.xml]
disabled = 0
host_segment = 5
crcSalt = <SOURCE>
index = my_index
sourcetype = xml_atm

Re: Ingest entire XML file

ff9231 — Wed, 26 May 2021 09:18:13 GMT

I have the exact same config as you, the only difference is that I want "source" also, if I define a source value then host names goes back to default servername, in this case host/host_regex/host_segment, nothing works if "source" is defined.

Do you have any suggestion what I can try? I am also configuring on UF.

Sample:

[monitor://D:\ABC\Monitor\Monitor\Inputs\*\*.xml]
disabled = 0
host_segment = 5
crcSalt = <SOURCE>
index = my_index
sourcetype = xml_atm

source = B_wks