Help creating search to combine multiple message lines to create timechart

[varadack]

We have Splunk setup in our firm and our application logs writes TLS connections information that span across multiple lines and splunk treats every line as message.

Example of Log:

2022-05-07 20:06:24.712 SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384
2022-05-07 20:06:24.712 Connection protocol=TLSv1.2
2022-05-07 20:06:24.716 Dump of user cache:
2022-05-07 20:06:24.716 LDAP Cache: User 'user1' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-aaaa-prod-rdr'
2022-05-07 20:06:24.717 LDAP Cache: User 'auser2' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-yyyy-prod-wtr'
2022-05-07 20:06:24.717 LDAP Cache: User 'ad_cibgvaprod_rdr' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-yyyy-prod-rdr'
2022-05-07 20:06:24.717 LDAP Cache: User 'ad_vcsmonprod_adm' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-bbbb-prod'
2022-05-07 20:06:24.717 'xxxx-tibems-aaaa-prod-shutdown'
2022-05-07 20:06:24.717 [user1@server1.svr.us.example.net]: Connected, connection id=21879, client id=<none>, type: queue, UTC offset=2

Here line starts with "SSL accepted cipher=" and ends with "ser1@server1.svr.us.example.net]: Connected,"

I would like timecharts cipher (ECDHE-RSA-AES256-GCM-SHA384), user (user1), Server (server1.svr.us.example.net) 

Stats like follows

Date Hour       Cipher   User   Server Count

10-10-20 10:00 ECDHE-RSA-AES256-GCM-SHA384) user1 server1 200

Please let me know if there an elegant solution to this,

Kannan

[varadack]

Thanks just a start as cipher is not able to filter filed just cipher

Example On Cipher it adds extra field as it sees in splunk search

@timestamp: 2022-05-08T15:10:28.291Z
   @version: 1
   Broker: broker
   a_time: 2022-05-08 15:10:28.275
   app_id: appid
   host: server
   message:  SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384
   path: /apps/broker/port/logs/server.log
   port: port
   type: type
}

Show as raw text

• host = server host = server message = SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384

ECDHE-RSA-AES256-GCM-SHA384","host":"xxxx","path":"/apps/broker/port/logs/server.domain.log","broker":"broker-NE1","type":"type","@version":"1","a_time":"2022-05-08

similarly

user and server also  comes as comes with other texts

like

user

/DC=net/DC=company/DC=vvvv/CN=zzz]","host":"server","path":"/apps/broker/port/logs/server.log","Broker":"broker-NE1","type":"type","

server also clobbered.

Thanks,

Kannan

[richgalloway]

The regular expressions in my first reply are correct for the example events provided in the OP.  They may need to be adapted for live data.  If you need help with that, please provide actual events with sensitive data masked, but preserve the original format.

[varadack]

The message is too long is there a way attach as a file here for your reference?

Kannan

[richgalloway]

You can use the transaction command to fit the events together, then extract the desired fields with rex.  Finally, the stats command will give the counts.

| transaction startswith="SSL accepted cipher" endswith="]: Connected"
| rex "cipher=(?<cipher>\S+)"
| rex "\[(?<user>[^@]+)@(?<server>[^\]]+)"
| stats count by _time,cipher,user,server

[varadack]

Hi richgalloway,

Your starting query was very promising and used to restrict the regex modifying as 

rex "cipher=(?<cipher>\S+)\",\"(type|Broker)" |

But it includes type and Broker also in the cipher filed and counts as below but I wanted only cipher

cipher user server count

ECDHE-RSA-AES256-GCM-SHA384","type":"TYPE	user1	server1	21
ECDHE-RSA-AES256-GCM-SHA384","Broker":"Broker"	user2	server1	25

[richgalloway · Accepted solution]

Now that we have more sample data, we can refine the regular expressions.  Try this:

| rex "cipher=(?<cipher>[^\\\"]+)"
| rex "type\\\":\\\"(?<type>[^\\\"]+)"
| rex "Broker\\\":\\\"(?<Broker>[^\\\"]+)"

[varadack]

Thanks a lot and it completely worked as expected and responded quick to solve this