topic Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ? in Getting Data In

Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ?

Hemnaath — Wed, 30 Sep 2020 01:29:50 GMT

Hi All, Need a help on Line Break Regex and TIME_FORMAT on props.conf, I am ingesting sonarqube logs in to splunk for the below log details with the following source type, but got stuck with the Regex part.

Ce.log details:
2019.07.12 11:05:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:05:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:15:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:15:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:25:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:25:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:35:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:35:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:45:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:45:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs

Source type and stanza details:

[sonarqube:ce]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\d+.\d+.\d+\s+\d+:\d+:\d+\s+\w+
TIME_FORMAT=%Y.%m.%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19

Access.log details:

x.x.x.x- - [11/Jul/2019:17:54:12 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc71kFTVuzYd0BsSB&p=1&ps=500 HTTP/1.1" 200 288 "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACB"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc74DFTVuzYd0BsTO&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACC"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWqYLPh9Yaosnfiy-EtA&p=1&ps=500 HTTP/1.1" 200 3989 "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACD"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWkV46fXeS_Bw5qUD5wC&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACE"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc8AHFTVuzYd0BsWp&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACF"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc8LZFTVuzYd0Bsd5&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACG"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWoCWw4hDVsFMZYP1Pzs&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACH"

[sonarqube:access]
SHOULD_LINEMERGE=false
TIME_FORMAT= %d/%b/%Y:%H:%M:%S %Z
TIME_PREFIX= \d+.\d+.\d+.\d+\s+-\s+-\s+[
MAX_TIMESTAMP_LOOKAHEAD=26
LINE_BREAKER =([\r\n]+)\d+.\d+.\d+.\d

Need help on the above stanza's to parse the sonarqube data in splunk. Kindly guide me on this.

Re: Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ?

gcusello — Wed, 30 Sep 2020 01:26:45 GMT

HI Hemnaath,
at first you don't need LINE_BREAKER if you have SHOULD_LINEMERGE=false because each row is an event.

About the first one, TIME_PREFIX and TIME_FORMAT are correct.

About the second one, I hint to modify TIME_PREFIX adding "^" at the beginning and escaping special chars (e.g. "." and "["), in other words:

TIME_PREFIX=^\d+\.\d+\.\d+\.\d+\s+-\s+-\s+\[

Then TIME_FORMAT is correct.

The best way to check props.conf is to extract some logs in a file and follow the guided procedure for ingestion, in this way you can immediately check your settings.

Bye.
Giuseppe

Re: Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ?

Hemnaath — Wed, 30 Sep 2020 01:29:53 GMT

thanks gcusello, for responding on this. but in our organisation we are following some standard recommendation when you are going to ingest a feed into splunk based on that, it is mandatory to add this stanza's. in this case my LINE_BREAKER regex and Max_time_look-ahead are correct.

[sonarqube:ce]
LINE_BREAKER = ([\r\n]+)\d+.\d+.\d+\s+\d+:\d+:\d+\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=19

[sonarqube:access]
LINE_BREAKER =([\r\n]+)\d+.\d+.\d+.\d
MAX_TIMESTAMP_LOOKAHEAD=26

Please guide me on this.

Re: Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ?

gcusello — Wed, 24 Jul 2019 13:37:32 GMT

HI Hemnaath,
ok, I don't understand but rules are rules!
Anyway, probably the problem are special chars to escape:

[sonarqube:ce]
LINE_BREAKER = ^\d+\.\d+\.\d+\s+\d+:\d+:\d+\s+\w+

[sonarqube:access]
TIME_PREFIX = ^\d+\.\d+\.\d+\.\d+\s+-\s+-\s+\[
LINE_BREAKER = ^\d+\.\d+\.\d+\.\d

Bye.
Giuseppe

Re: Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ?

woodcock — Fri, 26 Jul 2019 23:06:37 GMT

Try this (you had some problems with RegEx):

[sonarqube:ce]
SHOULD_LINEMERGE = false
# LINE_BREAKER = default is fine: do not change
TIME_PREFIX = ^
TIME_FORMAT = %Y.%m.%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

[sonarqube:access]
SHOULD_LINEMERGE = false
# LINE_BREAKER = default is fine: do not change
TIME_PREFIX = ^\d+\.\d+\.\d+\.\d+\s*-\s+-\s+\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 26