Hi All, Need a help on Line Break Regex and TIME_FORMAT on props.conf, I am ingesting sonarqube logs in to splunk for the below log details with the following source type, but got stuck with the Regex part.
Ce.log details:
2019.07.12 11:05:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:05:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:15:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:15:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:25:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:25:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:35:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:35:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:45:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:45:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
Source type and stanza details:
[sonarqube:ce]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\d+.\d+.\d+\s+\d+:\d+:\d+\s+\w+
TIME_FORMAT=%Y.%m.%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19
Access.log details:
x.x.x.x- - [11/Jul/2019:17:54:12 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc71kFTVuzYd0BsSB&p=1&ps=500 HTTP/1.1" 200 288 "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACB"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc74DFTVuzYd0BsTO&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACC"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWqYLPh9Yaosnfiy-EtA&p=1&ps=500 HTTP/1.1" 200 3989 "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACD"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWkV46fXeS_Bw5qUD5wC&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACE"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc8AHFTVuzYd0BsWp&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACF"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc8LZFTVuzYd0Bsd5&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACG"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWoCWw4hDVsFMZYP1Pzs&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACH"
[sonarqube:access]
SHOULD_LINEMERGE=false
TIME_FORMAT= %d/%b/%Y:%H:%M:%S %Z
TIME_PREFIX= \d+.\d+.\d+.\d+\s+-\s+-\s+[
MAX_TIMESTAMP_LOOKAHEAD=26
LINE_BREAKER =([\r\n]+)\d+.\d+.\d+.\d
Need help on the above stanza's to parse the sonarqube data in splunk. Kindly guide me on this.
Try this (you had some problems with RegEx):
[sonarqube:ce]
SHOULD_LINEMERGE = false
# LINE_BREAKER = default is fine: do not change
TIME_PREFIX = ^
TIME_FORMAT = %Y.%m.%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
[sonarqube:access]
SHOULD_LINEMERGE = false
# LINE_BREAKER = default is fine: do not change
TIME_PREFIX = ^\d+\.\d+\.\d+\.\d+\s*-\s+-\s+\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 26
HI Hemnaath,
at first you don't need LINE_BREAKER if you have SHOULD_LINEMERGE=false because each row is an event.
About the first one, TIME_PREFIX and TIME_FORMAT are correct.
About the second one, I hint to modify TIME_PREFIX adding "^" at the beginning and escaping special chars (e.g. "." and "["), in other words:
TIME_PREFIX=^\d+\.\d+\.\d+\.\d+\s+-\s+-\s+\[
Then TIME_FORMAT is correct.
The best way to check props.conf is to extract some logs in a file and follow the guided procedure for ingestion, in this way you can immediately check your settings.
Bye.
Giuseppe
thanks gcusello, for responding on this. but in our organisation we are following some standard recommendation when you are going to ingest a feed into splunk based on that, it is mandatory to add this stanza's. in this case my LINE_BREAKER regex and Max_time_look-ahead are correct.
[sonarqube:ce]
LINE_BREAKER = ([\r\n]+)\d+.\d+.\d+\s+\d+:\d+:\d+\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=19
[sonarqube:access]
LINE_BREAKER =([\r\n]+)\d+.\d+.\d+.\d
MAX_TIMESTAMP_LOOKAHEAD=26
Please guide me on this.
HI Hemnaath,
ok, I don't understand but rules are rules!
Anyway, probably the problem are special chars to escape:
[sonarqube:ce]
LINE_BREAKER = ^\d+\.\d+\.\d+\s+\d+:\d+:\d+\s+\w+
[sonarqube:access]
TIME_PREFIX = ^\d+\.\d+\.\d+\.\d+\s+-\s+-\s+\[
LINE_BREAKER = ^\d+\.\d+\.\d+\.\d
Bye.
Giuseppe