Getting Data In

Problem splitting data, lines are lost from scripted input data.

lakromani
Builder

I have a script that works fine.

When I do run it from cli like this, I get correct result:
/opt/splunk/bin/splunk cmd /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh

Flags: X - disabled, I - invalid, D - dynamic
 0  D ;;; upnp 10.10.10.32: Teredo
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=57050
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=57050

 1  D ;;; upnp 10.10.10.84: Skype UDP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=48153

 2  D ;;; upnp 10.10.10.84: Skype TCP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=48153

 3  D ;;; upnp 10.10.10.128: Skype UDP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=43905

 4  D ;;; upnp 10.10.10.128: Skype TCP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=43905

 5  D ;;; upnp 10.10.10.129: Skype UDP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=20139

 6  D ;;; upnp 10.10.10.129: Skype TCP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=20139

 7  D ;;; upnp 10.10.10.125: 3074 UDP
      chain=dstnat action=dst-nat to-addresses=10.10.10.125 to-ports=3074
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=3074

 8  D ;;; upnp 10.10.10.152: WhatsApp (1505943818) ()
      chain=dstnat action=dst-nat to-addresses=10.10.10.152 to-ports=56265
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=56265

 9  D ;;; upnp 10.10.10.152: WhatsApp (1505944513) ()
      chain=dstnat action=dst-nat to-addresses=10.10.10.152 to-ports=61271
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=61271

10  D ;;; upnp 10.10.10.152: WhatsApp (1505945615) ()
      chain=dstnat action=dst-nat to-addresses=10.10.10.152 to-ports=62934
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=62934

11  D ;;; upnp 10.10.10.32: uTorrent (TCP)
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=28816
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=28816

12  D ;;; upnp 10.10.10.32: uTorrent (UDP)
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=28816
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=28816

But in Splunk, I only get 9 events??? It stops at event 7, so 8,9,10,11,12 is missing and result is like this:

25/09/2017
11:21:52.000    
 7  D ;;; upnp 10.10.10.125: 3074 UDP
      chain=dstnat action=dst-nat to-addresses=10.10.10.125 to-ports=3074 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=3074 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 6  D ;;; upnp 10.10.10.129: Skype TCP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139 
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=20139 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 5  D ;;; upnp 10.10.10.129: Skype UDP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=20139 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 4  D ;;; upnp 10.10.10.128: Skype TCP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905 
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=43905 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 3  D ;;; upnp 10.10.10.128: Skype UDP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=43905 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 2  D ;;; upnp 10.10.10.84: Skype TCP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153 
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=48153 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 1  D ;;; upnp 10.10.10.84: Skype UDP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=48153 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 0  D ;;; upnp 10.10.10.32: Teredo
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=57050 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=57050 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
Flags: X - disabled, I - invalid, D - dynamic 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2

I did try to split events by a blank line but did not get it to work),

inputs.conf

[script://$SPLUNK_HOME/etc/apps/MikroTik/bin/mikrotik_upnp.sh]
disabled = false
interval = 300
sourcetype = mikrotik2

props.conf

[mikrotik2]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
LINE_BREAKER =
BREAK_ONLY_BEFORE = \d+\s+D\s
disabled = false

Why does 5 events get lost?
Is it due to my BREAK_ONLY_BEFORE?
Are there a better way to du it (use LINE_BREAKER instead)?

Update.

For some reason I now get all events with one digit ID.
So I get 0 to 9, but not 10,11 or 12.
BREAK_ONLY_BEFORE do contain \d+, so it should take any number.

0 Karma

sbbadri
Motivator

@lakromani

try this,

[ mikrotik2]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\d+\s+\S\s+
CHARSET=UTF-8

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...