topic Re: Help creating search to combine multiple message lines to create timechart in Splunk Search

Help creating search to combine multiple message lines to create timechart

varadack — Sun, 08 May 2022 05:18:32 GMT

We have Splunk setup in our firm and our application logs writes TLS connections information that span across multiple lines and splunk treats every line as message.

Example of Log:

2022-05-07 20:06:24.712 SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384
2022-05-07 20:06:24.712 Connection protocol=TLSv1.2
2022-05-07 20:06:24.716 Dump of user cache:
2022-05-07 20:06:24.716 LDAP Cache: User 'user1' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-aaaa-prod-rdr'
2022-05-07 20:06:24.717 LDAP Cache: User 'auser2' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-yyyy-prod-wtr'
2022-05-07 20:06:24.717 LDAP Cache: User 'ad_cibgvaprod_rdr' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-yyyy-prod-rdr'
2022-05-07 20:06:24.717 LDAP Cache: User 'ad_vcsmonprod_adm' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-bbbb-prod'
2022-05-07 20:06:24.717 'xxxx-tibems-aaaa-prod-shutdown'
2022-05-07 20:06:24.717 [user1@server1.svr.us.example.net]: Connected, connection id=21879, client id=<none>, type: queue, UTC offset=2

Here line starts with "SSL accepted cipher=" and ends with "ser1@server1.svr.us.example.net]: Connected,"

I would like timecharts cipher (ECDHE-RSA-AES256-GCM-SHA384), user (user1), Server (server1.svr.us.example.net)

Stats like follows

Date Hour Cipher User Server Count

10-10-20 10:00 ECDHE-RSA-AES256-GCM-SHA384) user1 server1 200

Please let me know if there an elegant solution to this,

Kannan

Re: Splunk Search to combine multiple message lines to create timechart

richgalloway — Sun, 08 May 2022 00:17:43 GMT

You can use the transaction command to fit the events together, then extract the desired fields with rex. Finally, the stats command will give the counts.

| transaction startswith="SSL accepted cipher" endswith="]: Connected" | rex "cipher=(?<cipher>\S+)" | rex "\[(?<user>[^@]+)@(?<server>[^\]]+)" | stats count by _time,cipher,user,server

Re: Help creating search to combine multiple message lines to create timechart

varadack — Sun, 08 May 2022 15:26:18 GMT

Thanks just a start as cipher is not able to filter filed just cipher

Example On Cipher it adds extra field as it sees in splunk search

@timestamp: 2022-05-08T15:10:28.291Z
   @version: 1
   Broker: broker
   a_time: 2022-05-08 15:10:28.275
   app_id: appid
   host: server
   message: SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384
   path: /apps/broker/port/logs/server.log
   port: port
   type: type
}

Show as raw text

host = server host = server message = SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384

ECDHE-RSA-AES256-GCM-SHA384","host":"xxxx","path":"/apps/broker/port/logs/server.domain.log","broker":"broker-NE1","type":"type","@version":"1","a_time":"2022-05-08

similarly

user and server also comes as comes with other texts

user

/DC=net/DC=company/DC=vvvv/CN=zzz]","host":"server","path":"/apps/broker/port/logs/server.log","Broker":"broker-NE1","type":"type","

server also clobbered.

Thanks,

Kannan

Re: Help creating search to combine multiple message lines to create timechart

richgalloway — Mon, 09 May 2022 12:25:33 GMT

The regular expressions in my first reply are correct for the example events provided in the OP. They may need to be adapted for live data. If you need help with that, please provide actual events with sensitive data masked, but preserve the original format.

Re: Help creating search to combine multiple message lines to create timechart

varadack — Mon, 09 May 2022 23:16:08 GMT

The message is too long is there a way attach as a file here for your reference?

Kannan

Re: Splunk Search to combine multiple message lines to create timechart

varadack — Tue, 10 May 2022 00:48:16 GMT

Hi richgalloway,

Your starting query was very promising and used to restrict the regex modifying as

rex "cipher=(?<cipher>\S+)\",\"(type|Broker)" |

But it includes type and Broker also in the cipher filed and counts as below but I wanted only cipher

cipher user server count

ECDHE-RSA-AES256-GCM-SHA384","type":"TYPE	user1	server1	21
ECDHE-RSA-AES256-GCM-SHA384","Broker":"Broker"	user2	server1	25

Re: Splunk Search to combine multiple message lines to create timechart

richgalloway — Tue, 10 May 2022 13:32:11 GMT

Now that we have more sample data, we can refine the regular expressions. Try this:

| rex "cipher=(?<cipher>[^\\\"]+)" | rex "type\\\":\\\"(?<type>[^\\\"]+)" | rex "Broker\\\":\\\"(?<Broker>[^\\\"]+)"

Re: Splunk Search to combine multiple message lines to create timechart

varadack — Wed, 11 May 2022 10:01:36 GMT

Thanks a lot and it completely worked as expected and responded quick to solve this