Problem splitting data, lines are lost from scripted input data.

lakromani — Tue, 29 Sep 2020 15:51:54 GMT

I have a script that works fine.

When I do run it from cli like this, I get correct result:
/opt/splunk/bin/splunk cmd /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh

Flags: X - disabled, I - invalid, D - dynamic
 0  D ;;; upnp 10.10.10.32: Teredo
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=57050
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=57050

 1  D ;;; upnp 10.10.10.84: Skype UDP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=48153

 2  D ;;; upnp 10.10.10.84: Skype TCP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=48153

 3  D ;;; upnp 10.10.10.128: Skype UDP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=43905

 4  D ;;; upnp 10.10.10.128: Skype TCP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=43905

 5  D ;;; upnp 10.10.10.129: Skype UDP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=20139

 6  D ;;; upnp 10.10.10.129: Skype TCP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=20139

 7  D ;;; upnp 10.10.10.125: 3074 UDP
      chain=dstnat action=dst-nat to-addresses=10.10.10.125 to-ports=3074
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=3074

 8  D ;;; upnp 10.10.10.152: WhatsApp (1505943818) ()
      chain=dstnat action=dst-nat to-addresses=10.10.10.152 to-ports=56265
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=56265

 9  D ;;; upnp 10.10.10.152: WhatsApp (1505944513) ()
      chain=dstnat action=dst-nat to-addresses=10.10.10.152 to-ports=61271
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=61271

10  D ;;; upnp 10.10.10.152: WhatsApp (1505945615) ()
      chain=dstnat action=dst-nat to-addresses=10.10.10.152 to-ports=62934
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=62934

11  D ;;; upnp 10.10.10.32: uTorrent (TCP)
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=28816
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1
      dst-port=28816

12  D ;;; upnp 10.10.10.32: uTorrent (UDP)
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=28816
      protocol=udp dst-address=110.12.197.134 in-interface=ether1
      dst-port=28816

But in Splunk, I only get 9 events??? It stops at event 7, so 8,9,10,11,12 is missing and result is like this:

25/09/2017
11:21:52.000    
 7  D ;;; upnp 10.10.10.125: 3074 UDP
      chain=dstnat action=dst-nat to-addresses=10.10.10.125 to-ports=3074 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=3074 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 6  D ;;; upnp 10.10.10.129: Skype TCP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139 
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=20139 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 5  D ;;; upnp 10.10.10.129: Skype UDP at 10.10.10.129:20139 (3910)
      chain=dstnat action=dst-nat to-addresses=10.10.10.129 to-ports=20139 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=20139 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 4  D ;;; upnp 10.10.10.128: Skype TCP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905 
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=43905 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 3  D ;;; upnp 10.10.10.128: Skype UDP at 10.10.10.128:43905 (3909)
      chain=dstnat action=dst-nat to-addresses=10.10.10.128 to-ports=43905 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=43905 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 2  D ;;; upnp 10.10.10.84: Skype TCP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153 
      protocol=tcp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=48153 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 1  D ;;; upnp 10.10.10.84: Skype UDP at 10.10.10.84:48153 (3904)
      chain=dstnat action=dst-nat to-addresses=10.10.10.84 to-ports=48153 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=48153 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
 0  D ;;; upnp 10.10.10.32: Teredo
      chain=dstnat action=dst-nat to-addresses=10.10.10.32 to-ports=57050 
      protocol=udp dst-address=110.12.197.134 in-interface=ether1 
      dst-port=57050 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2
25/09/2017
11:21:52.000    
Flags: X - disabled, I - invalid, D - dynamic 
host =  Varg source =   /opt/splunk/etc/apps/MikroTik/bin/mikrotik_upnp.sh sourcetype = mikrotik2

I did try to split events by a blank line but did not get it to work),

inputs.conf

[script://$SPLUNK_HOME/etc/apps/MikroTik/bin/mikrotik_upnp.sh]
disabled = false
interval = 300
sourcetype = mikrotik2

props.conf

[mikrotik2]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
LINE_BREAKER =
BREAK_ONLY_BEFORE = \d+\s+D\s
disabled = false

Why does 5 events get lost?
Is it due to my BREAK_ONLY_BEFORE?
Are there a better way to du it (use LINE_BREAKER instead)?

Update.

For some reason I now get all events with one digit ID.
So I get 0 to 9, but not 10,11 or 12.
BREAK_ONLY_BEFORE do contain \d+, so it should take any number.

Re: Problem splitting data, lines are lost from scripted input data.

sbbadri — Tue, 29 Sep 2020 15:56:32 GMT

@lakromani

try this,

[ mikrotik2]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\d+\s+\S\s+
CHARSET=UTF-8

topic Problem splitting data, lines are lost from scripted input data. in Getting Data In

Problem splitting data, lines are lost from scripted input data.

Re: Problem splitting data, lines are lost from scripted input data.