topic Re: Complex line breaking configuration help needed in Getting Data In

Complex line breaking configuration help needed

jfaldmomacu — Mon, 16 Mar 2020 20:50:37 GMT

Here is a snippet of a log file that I am trying to do line breaking on. I want it to only break when one line has matches "info]*" and the next line has "info]Line"

[2019-12-18 07:00:01.070924-07:00|info]Line 3:     :begin
[2019-12-18 07:00:01.070924-07:00|info]Line 4:     
[2019-12-18 07:00:01.070924-07:00|info]Line 5:     WORKINGDIR "C:\Download\Server1"
[2019-12-18 07:00:01.070924-07:00|info]*Working directory: C:\Download\Server1\
[2019-12-18 07:00:01.070924-07:00|info]Line 6:     
[2019-12-18 07:00:01.070924-07:00|info]Line 7:       FTPLOGON "Server1" /timeout=60
[2019-12-18 07:00:01.070924-07:00|info]*Logging on to <server1> as SFTP (SSH File Transfer Protocol)
[2019-12-18 07:00:01.070924-07:00|info]*Logon in progress...
[2019-12-18 07:00:03.055523-07:00|info]*Logon successful.
[2019-12-18 07:00:03.055523-07:00|info]Line 8:       FTPCD "Extracts"
[2019-12-18 07:00:03.164909-07:00|info]*Current FTP site directory: /Extracts/
[2019-12-18 07:00:03.164909-07:00|info]Line 9:       IFERROR= $ERROR_SUCCESS GOTO Operation1
[2019-12-18 07:00:03.164909-07:00|info]Line 21:    :Operation1
[2019-12-18 07:00:03.164909-07:00|info]Line 22:      FTPGETFILE "*na_alert_subs*" /newest
[2019-12-18 07:00:03.164909-07:00|info]*Hint: FTPGETFILE /newest always returns the newest file
[2019-12-18 07:00:03.430561-07:00|info]Line 22:    *%sitefile has been set to: na_alert_subs_20191217.txt
[2019-12-18 07:00:03.446223-07:00|info]Line 23:      RCVFILE %sitefile /delete
[2019-12-18 07:00:03.446223-07:00|info]*Receiving to "C:\Download\Server1\na_alert_subs_20191217.txt"
[2019-12-18 07:00:12.947244-07:00|info]*Complete, received 1394788 bytes in 9 seconds (1513.44K cps)
[2019-12-18 07:00:13.103506-07:00|info]*File deleted on FTP site.
[2019-12-18 07:00:13.103506-07:00|info]*Download complete, 1 file received.

So in that snippet it would break down into five events.

Re: Complex line breaking configuration help needed

to4kawa — Mon, 16 Mar 2020 21:16:39 GMT

what's line number you want to cut?

Re: Complex line breaking configuration help needed

jfaldmomacu — Mon, 16 Mar 2020 21:33:33 GMT

The five events in my original post should be lines--
1-4,
5-9,
10-11,
12-15,
16-21.

Or like this--

Event 1
 [2019-12-18 07:00:01.070924-07:00|info]Line 3:     :begin
 [2019-12-18 07:00:01.070924-07:00|info]Line 4:     
 [2019-12-18 07:00:01.070924-07:00|info]Line 5:     WORKINGDIR "C:\Download\Server1"
 [2019-12-18 07:00:01.070924-07:00|info]*Working directory: C:\Download\Server1\

Event 2
 [2019-12-18 07:00:01.070924-07:00|info]Line 6:     
 [2019-12-18 07:00:01.070924-07:00|info]Line 7:       FTPLOGON "Server1" /timeout=60
 [2019-12-18 07:00:01.070924-07:00|info]*Logging on to <server1> as SFTP (SSH File Transfer Protocol)
 [2019-12-18 07:00:01.070924-07:00|info]*Logon in progress...
 [2019-12-18 07:00:03.055523-07:00|info]*Logon successful.

Event 3
 [2019-12-18 07:00:03.055523-07:00|info]Line 8:       FTPCD "Extracts"
 [2019-12-18 07:00:03.164909-07:00|info]*Current FTP site directory: /Extracts/

Event 4
 [2019-12-18 07:00:03.164909-07:00|info]Line 9:       IFERROR= $ERROR_SUCCESS GOTO Operation1
 [2019-12-18 07:00:03.164909-07:00|info]Line 21:    :Operation1
 [2019-12-18 07:00:03.164909-07:00|info]Line 22:      FTPGETFILE "*na_alert_subs*" /newest
 [2019-12-18 07:00:03.164909-07:00|info]*Hint: FTPGETFILE /newest always returns the newest file

Event 5
 [2019-12-18 07:00:03.430561-07:00|info]Line 22:    *%sitefile has been set to: na_alert_subs_20191217.txt
 [2019-12-18 07:00:03.446223-07:00|info]Line 23:      RCVFILE %sitefile /delete
 [2019-12-18 07:00:03.446223-07:00|info]*Receiving to "C:\Download\Server1\na_alert_subs_20191217.txt"
 [2019-12-18 07:00:12.947244-07:00|info]*Complete, received 1394788 bytes in 9 seconds (1513.44K cps)
 [2019-12-18 07:00:13.103506-07:00|info]*File deleted on FTP site.
 [2019-12-18 07:00:13.103506-07:00|info]*Download complete, 1 file received.

Re: Complex line breaking configuration help needed

to4kawa — Tue, 17 Mar 2020 11:39:13 GMT

props.conf

SHOULD_LINEMERGE = true
LINE_BREAKER = Line.*[\r\n]\[.*\]\*.*([\r\n])\[.*\]Line

SHOULD_LINEMERGE is with (?msU) Implicitly.

Re: Complex line breaking configuration help needed

jfaldmomacu — Tue, 17 Mar 2020 15:34:18 GMT

That made it so there are no line breaks, or so that everything comes through as one event. The source is a bunch of small files, less than a hundred lines, and each file is now one event.

Re: Complex line breaking configuration help needed

jfaldmomacu — Tue, 17 Mar 2020 16:42:17 GMT

I guess this wasn't as complex as I initially thought. I was getting wrapped up in all the options. In reading the documentation for LINE_BREAKER I was able to get a simple solution. Thank you @to4kawa for prompting me to get to the right answer.

SHOULD_LINEMERGE = false
LINE_BREAKER = .*]\*.*([\r\n]+)\[.*\]Line

I swapped out Line for [^*] as I saw some edge cases where I needed some an event breaking there as well. So I really ended up with this.

SHOULD_LINEMERGE = false
LINE_BREAKER = .*]\*.*([\r\n]+)\[.*\][^\*]

Re: Complex line breaking configuration help needed

woodcock — Tue, 17 Mar 2020 16:44:14 GMT

Like this:

SHOULD_LINEMERGE = false
LINE_BREAKER = [\r\n]+\[\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d+-\d{2}:\d{2}\|info]\*[^\r\n]+([\r\n]+)\[\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d+-\d{2}:\d{2}\|info\]Line

See here:
https://regex101.com/r/P4LwaF/1

Re: Complex line breaking configuration help needed

woodcock — Tue, 17 Mar 2020 16:44:53 GMT

You got the SHOULD_LINEMERGE wrong...

Re: Complex line breaking configuration help needed

to4kawa — Tue, 17 Mar 2020 21:08:39 GMT

I see
When I checked Splunk Add-on Builder, I need SHOULD_LINEMERGE = true
Does your setting cut between 16 and 17?