Knowledge Management

Extract multiple lines as one event

fmatera
Explorer

Log snippet:

0416 12:45:59.50:  classify_origination(newcall)
0416 12:45:59.50: colp: 1419371523
0416 12:45:59.50: process_trunk_parm: use_trunk_fields=false
0416 12:45:59.50: ++++ SBCCTL STATS DUMP START (interval = 60) +++
0416 12:45:59.50: STATE: ONLINE
0416 12:45:59.50: uas:   rcvd=13271(2.2e+02 mps) sent=16593(2.8e+02 mps)
0416 12:45:59.50: uac:   rcvd=15253(2.5e+02 mps) sent=11931(2e+02 mps)
0416 12:45:59.50:  mpxy:   rcvd=28322(4.7e+02 mps) sent=13869(2.3e+02 mps)
0416 12:45:59.50:      other:   rcvd=93899(1.6e+03 mps) sent=93899(1.6e+03 mps)
0416 12:45:59.50:   spool:   rcvd=6209(1e+02 mps)    sent=6209(1e+02 mps)
0416 12:45:59.50:   MESSAGES:   rcvd=156954(2615.9 mps) sent=142501(2375 mps)
0416 12:45:59.50: Offered load: total=56846, uac=15253, uas=13271, mpxy=28322
0416 12:45:59.50: OVERFLOWS: uas=0 uac=0 mpxy=0 sbcspool=0 admctl=0
0416 12:45:59.50: QUEUES: mux:0:0:70, uas:0:0:0, uac:0:0:0, mpxy:0:0:69, sbcspool:0:0:0, admctl:0:0:3
0416 12:45:59.50: MSGS UAC/UAS: 6181:5215:129:82:3320:0:1354:1351:5508:5228
0416 12:45:59.50: MSGS MPXY: 4941:0:4949:14453:0:0
0416 12:45:59.50: MSGS SPOOL: 6209:6209
0416 12:45:59.50: CSPS (last 60 seconds): 103.02
0416 12:45:59.50: Call counts: active:2690 (init:43, trying:865, stable:1781, ending:1, teoc:0)
0416 12:45:59.50: Tunnel count: 2531
0416 12:45:59.50: sbcsipuas.1@b12sb01:  CONNECTED       ONLINE  2408/9000
0416 12:45:59.50: sbcsipuas.2@b12sb01:  CONNECTED       ONLINE  282/9000
0416 12:45:59.50: brawt.1@b12sb01:      CLOSED          UNKNOWN 0/9000
0416 12:45:59.50: sbch323uas.1@b12sb01: CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbcsipuac.1@b12sb01:  CONNECTED       ONLINE  2544/9000
0416 12:45:59.50: sbcsipuac.2@b12sb01:  CONNECTED       ONLINE  96/9000
0416 12:45:59.50: sbcsipuac.3@b12sb01:  CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbcsipuac.4@b12sb01:  CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbcsipuac.5@b12sb01:  CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbcsipuac.6@b12sb01:  CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbcsipuac.7@b12sb01:  CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbcsipuac.8@b12sb01:  CONNECTED       ONLINE  0/9000
0416 12:45:59.50: sbch323uac.1@b12sb01: CONNECTED       ONLINE  0/9000
0416 12:45:59.50: mpxy.1@b12sb01:       CONNECTED       ONLINE  332/1300
0416 12:45:59.50: mpxy.2@b12sb01:       CONNECTED       ONLINE  306/1300
0416 12:45:59.50: mpxy.3@b12sb01:       CONNECTED       ONLINE  289/1300
0416 12:45:59.50: mpxy.4@b12sb01:       CONNECTED       ONLINE  313/1300
0416 12:45:59.50: mpxy.5@b12sb01:       CONNECTED       ONLINE  329/1300
0416 12:45:59.50: mpxy.6@b12sb01:       CONNECTED       ONLINE  324/1300
0416 12:45:59.50: mpxy.7@b12sb01:       CONNECTED       ONLINE  327/1300
0416 12:45:59.50: mpxy.8@b12sb01:       CONNECTED       ONLINE  311/1300
0416 12:45:59.50: MPXY Tunnel: timeouts:0, orphans:2, min(ms):0 max(ms):23 count:4939 avg(ms):0.72262 HISTO:4939:0:0:0:0:0:0:0:0:0:0
0416 12:45:59.50: MPXY Packets (Ingress): total:5695213, toss:1988(0.034907%), out_of_seq:104(0.0018261%), lost:941(0.01652%)
0416 12:45:59.50: MPXY Packets (Egress): total:7041461, toss:220(0.0031244%), out_of_seq:74(0.0010509%), lost:3832(0.054391%)
0416 12:45:59.50: Memory Size: 1,205,137,408
0416 12:45:59.50: ++++ SBCCTL STATS DUMP END +++
0416 12:45:59.50:  classify_origination(call)
0416 12:45:59.50: colp: 
0416 12:45:59.50: memory:
0416 12:45:59.50: process_trunk_parm: use_trunk_fields=true

I am trying to extract all of the text between

0416 12:45:59.50: ++++ SBCCTL STATS DUMP START (interval = 60) +++

and

0416 12:45:59.50: ++++ SBCCTL STATS DUMP END +++

and include those lines and all the lines in between into a single event.
This is what I have tried with no luck

inputs:

[monitor:///opt/splunk/testdata/*.log]
sourcetype=test
index = app
disabled = false

props:

[test]
TIME_PREFIX = ^
TIME_FORMAT = %m-%d %H:%M:%S.%2N
MAX_TIMESTAMP_LOOKAHEAD = 32
TRUNCATE = 99999
SHOULD_LINEMERGE = true
TRANSFORMS-set = setnull, setnonull

Transforms:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setnonull]
REGEX = ^.*START.*(?ms).*\+{3}
DEST_KEY = queue
FORMAT = indexQueue

Testing the regex via an online tester selects the lines I am looking to extract. I am not sure of the issue. Any assistance would be greatly appeciated. Tks

Tags (1)
0 Karma

woodcock
Esteemed Legend

Try this:
In props.conf:

[YourSourcetypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?ms)(.*?)\d+\s+\d+:\d+:\d+\.\d+:\s+\+\+\+\+ SBCCTL STATS DUMP START.*?\d+\s+\d+:\d+:\d+\.\d+:\s+\+\+\+\+ SBCCTL STATS DUMP END \+\+\+
TIME_PREFIX = ^
TIME_FORMAT = %m-%d %H:%M:%S.%2N
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS-set = setnull, setnonull

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setnonull]
REGEX = \+\+\+\+ SBCCTL STATS DUMP START
DEST_KEY = queue
FORMAT = indexQueue

fmatera
Explorer

@woodcock Thank you, I was very hopeful in trying this out as I thought it would work, Unfortunately, when implementing these changes, I am not seeing any events via the search head.

Inputs

        [monitor:///opt/splunk/testdata/sbc*.log]
        sourcetype=test
        index = app_sbc
        disabled = false

Props

       [test]
        TIME_PREFIX = ^
        TIME_FORMAT = %m-%d %H:%M:%S.%2N
        MAX_TIMESTAMP_LOOKAHEAD = 32
        SHOULD_LINEMERGE = false
        LINE_BREAKER = (?ms)(.*?)\d+\s+\d+:\d+:\d+\.\d+:\s+\+\+\+\+ SBCCTL STATS DUMP START.*?\d+\s+\d+:\d+:\d+\.\d+:\s+\+\+\+\+ SBCCTL STATS DUMP END \+\+\+
        TRANSFORMS-set = setnull, setnonull

Transforms

        [setnull]
        REGEX = .
        DEST_KEY = queue
        FORMAT = nullQueue

        [setnonull]
        REGEX = \+\+\+\+ SBCCTL STATS DUMP START
        DEST_KEY = queue
        FORMAT = indexQueue
0 Karma

woodcock
Esteemed Legend

It definitely works; I tested it. Did you:

1) Deploy to the first full instance of Splunk that touches the data (HF or Indexer tier)?
2) Restart all splunk instances there?
3) Forward in new data (previous data will stay wrong forever)?
4) Test using `_index_earliest=-5m` to be absolutely sure that it is new data?
0 Karma

somesoni2
Revered Legend

What do you want to do with lines before START and after END??

0 Karma

fmatera
Explorer

Discard them. I thought set all to null and then extract what I need.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...