How to write base query to get all the fields from below three multiple logs

[shruti14]

Hi , I have to get the below fields extracted from these three logs to create visulisation: Fields i am interested:

Event_log type,originator_username,object,username,destination,bucket_name,time,type

I have written this regex to create parser but i am not getting all the fields while writing base serach:

^(?:[^ \n]* ){2}(?P<event_log>\w+\s+[a-z_-]+)(?:[^ \n]* ){2}\{"originator_username"\:(?P<originator_username>.[a-z]+")\,"object"\:(?<object>.[a-z]+)[^,\n]*,"extra"\:\{(?P<extra>.[a-z]+)":[^,\n]*(?:[^,\n]*,){6}"time"\:(?P<time>\w+),(?:[^,\n]*,){2}"type"\:(?<type>.[a-z_]+[a-z])"}

2022-01-23 10:19:47,140 WARNING event_log EventLog: {"originator_username":"abc","object":"cluster","extra":{"username":"admin"},"object_type":"cluster","originator_uid":0,"time":164287087,"throttled_event_count":1,"obj_uid":null,"type":"failed_authentication_attempt"}

2022-01-23 07:24:05,479 INFO event_log EventLog: {"originator_username":"abcef","object":"bdb:1","extra":{"destination":{"bucket_name":"dbabucket","type":"s3","subdir":"radar2","filename":""}},"object_type":"bdb","originator_uid":0,"time":164767765,"throttled_event_count":1,"obj_uid":"1","type":"backup_succeeded"}

2022-01-23 07:15:00,294 INFO event_log EventLog: {"originator_username":"adminstrator","object":"bdb:1","object_type":"bdb","originator_uid":0,"time":1642788100,"throttled_event_count":1,"obj_uid":"1","type":"backup_started"}

Can anyone help me what neededd to be fix in regex so i can get all the needed field extracted for base search.

[ITWhisperer]

You could extract the JSON and use spath

| rex "(?<log>\{.*\})"
| spath input=log

[shruti14]

Hi, 

But these logs are not in json format, i.e why i have to parse them manually by writing regex

[ITWhisperer]

The examples you gave have JSON strings at the end which I extracted with the rex before using spath. Are you trying to extract at indexing time or search time?

[shruti14]

I want to extract them at indexing time. So I want to get these field extracted during the index so i get the fields created and then create base search to build dashboards for visualisation. 

The logs are not in regular json format .

[ITWhisperer]

Does this help?

^(?:[^ \n]* ){2}(?P<event_log>\w+\s+[a-z_-]+)(?:[^ \n]* ){2}\{"originator_username"\:(?P<originator_username>"[^"]+")\,"object"\:(?<object>"[^"]+")[^,\n]*,("extra"\:\{(?P<extra>.[a-z]+)":[^,\n])*(?:[^,\n]*,)+"time"\:(?P<time>\w+),(?:[^,\n]*,){2}"type"\:(?<type>.[a-z_]+[a-z])"}

[shruti14]

Yeah it is almost there is there way we can get username and destination also extracted as fields since they also have value i mean this regex gives extra field with value as username and destination but if you can see i have username :admin and destination as bucket name does that further breakage is possible ?

username: admin

"destination":{"bucket_name":"dbabucket","type":"s3","subdir":"radar2","filename":""}}

But yes the regex shared by you is really helpful and i understood what mistake i am doing  thanks ITWhisperer