Getting Data In

Why are my Scripted Input XML events truncated to 4 lines?

jercra
Explorer

I have scripted input that's calling a simple rest service to get a list of messages. No matter what settings I put in props.conf, I can only see 4 lines of XML on the indexer.

The XML result looks like:

<messages_list>
 <audit>
    <code>30</code>
    <created_at>2017-03-20T13:54:31-06:00</created_at>
    <message>Node ip-172-31-28-26 activated</message>
    <id>4</id>
    <node>02</node>
  </audit>
  <audit>
    <code>31</code>
    <created_at>2017-03-20T13:50:08-06:00</created_at>
    <message>Node ip-172-31-28-26 is deactivated</message>
    <id>3</id>
    <node>02</node>
  </audit>
  <audit>
    <code>30</code>
    <created_at>2017-03-20T13:49:44-06:00</created_at>
    <message>Node ip-172-31-28-26 activated</message>
    <id>2</id>
    <node>02</node>
  </audit>
  <audit>
    <code>32</code>
    <created_at>2017-03-20T13:47:07-06:00</created_at>
    <message>Node ip-172-31-28-26 added to cluster</message>
    <id>1</id>
    <node>02</node>
  </audit>
</messages_list>

The only thing that shows up in Splunk is:

 <messages_list>
 <audit>
    <code>30</code>

I've set TRUNCATE to 1000000 and to 0
I've set a multitude of linebreaking (Before, after, linebreak, linemerge=true|false, etc.)

Nothing changes. It's always only 4 lines.
TCPDump shows the data making it to the indexer.

Current props.conf:
[live_messages]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE=0
MAX_EVENTS = 10000
pulldown_type = 1
KV_MODE=xml
LINE_BREAKER = (<\/messages_list>)
DATETIME_CONFIG = CURRENT

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Pretty strange. Would you mind trying with this?

[live_messages]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\<messages_list\>)
TRUNCATE=0
MAX_EVENTS = 10000
KV_MODE=xml
DATETIME_CONFIG = CURRENT

I'm assuming live_messages is the sourcetype that you assign to your scripted input in inputs.conf.

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Pretty strange. Would you mind trying with this?

[live_messages]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\<messages_list\>)
TRUNCATE=0
MAX_EVENTS = 10000
KV_MODE=xml
DATETIME_CONFIG = CURRENT

I'm assuming live_messages is the sourcetype that you assign to your scripted input in inputs.conf.

0 Karma

jercra
Explorer

Yeah, the sourcetype is specified in the app's inputs.conf.

Your change seems to have done the trick. Very odd. Thanks. Add it as an answer and I will mark it as such.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Can you share your props.conf for this sourcetype? Where is this script runs, UF or full Splunk Enterprise instance?

0 Karma

jercra
Explorer

I've gone through many, many iterations of props.conf. My current iteration looks like:
[live_messages]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE=0
MAX_EVENTS = 10000
pulldown_type = 1
KV_MODE=xml
LINE_BREAKER = (<\/messages_list>)
DATETIME_CONFIG = CURRENT

The script runs on a forwarder and is calling a webservice on localhost. It's a really simple service that just returns a single XML document.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...