topic Re: Why are my Scripted Input XML events truncated to 4 lines? in Getting Data In

Why are my Scripted Input XML events truncated to 4 lines?

jercra — Tue, 29 Sep 2020 13:20:13 GMT

I have scripted input that's calling a simple rest service to get a list of messages. No matter what settings I put in props.conf, I can only see 4 lines of XML on the indexer.

The XML result looks like:

<messages_list>
 <audit>
    <code>30</code>
    <created_at>2017-03-20T13:54:31-06:00</created_at>
    <message>Node ip-172-31-28-26 activated</message>
    <id>4</id>
    <node>02</node>
  </audit>
  <audit>
    <code>31</code>
    <created_at>2017-03-20T13:50:08-06:00</created_at>
    <message>Node ip-172-31-28-26 is deactivated</message>
    <id>3</id>
    <node>02</node>
  </audit>
  <audit>
    <code>30</code>
    <created_at>2017-03-20T13:49:44-06:00</created_at>
    <message>Node ip-172-31-28-26 activated</message>
    <id>2</id>
    <node>02</node>
  </audit>
  <audit>
    <code>32</code>
    <created_at>2017-03-20T13:47:07-06:00</created_at>
    <message>Node ip-172-31-28-26 added to cluster</message>
    <id>1</id>
    <node>02</node>
  </audit>
</messages_list>

The only thing that shows up in Splunk is:

 <messages_list>
 <audit>
    <code>30</code>

I've set TRUNCATE to 1000000 and to 0
I've set a multitude of linebreaking (Before, after, linebreak, linemerge=true|false, etc.)

Nothing changes. It's always only 4 lines.
TCPDump shows the data making it to the indexer.

Current props.conf:
[live_messages]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE=0
MAX_EVENTS = 10000
pulldown_type = 1
KV_MODE=xml
LINE_BREAKER = (<\/messages_list>)
DATETIME_CONFIG = CURRENT

Re: Why are my Scripted Input XML events truncated to 4 lines?

somesoni2 — Thu, 23 Mar 2017 19:48:55 GMT

Can you share your props.conf for this sourcetype? Where is this script runs, UF or full Splunk Enterprise instance?

Re: Why are my Scripted Input XML events truncated to 4 lines?

jercra — Tue, 29 Sep 2020 13:20:25 GMT

I've gone through many, many iterations of props.conf. My current iteration looks like:
[live_messages]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE=0
MAX_EVENTS = 10000
pulldown_type = 1
KV_MODE=xml
LINE_BREAKER = (<\/messages_list>)
DATETIME_CONFIG = CURRENT

The script runs on a forwarder and is calling a webservice on localhost. It's a really simple service that just returns a single XML document.

Re: Why are my Scripted Input XML events truncated to 4 lines?

somesoni2 — Thu, 23 Mar 2017 21:07:10 GMT

Pretty strange. Would you mind trying with this?

[live_messages]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\<messages_list\>)
TRUNCATE=0
MAX_EVENTS = 10000
KV_MODE=xml
DATETIME_CONFIG = CURRENT

I'm assuming live_messages is the sourcetype that you assign to your scripted input in inputs.conf.

Re: Why are my Scripted Input XML events truncated to 4 lines?

jercra — Thu, 23 Mar 2017 21:18:18 GMT

Yeah, the sourcetype is specified in the app's inputs.conf.

Your change seems to have done the trick. Very odd. Thanks. Add it as an answer and I will mark it as such.