Splunk Search

Help creating search to combine multiple message lines to create timechart

varadack
Engager

We have Splunk setup in our firm and our application logs writes TLS connections information that span across multiple lines and splunk treats every line as message.

Example of Log:

2022-05-07 20:06:24.712 SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384
2022-05-07 20:06:24.712 Connection protocol=TLSv1.2
2022-05-07 20:06:24.716 Dump of user cache:
2022-05-07 20:06:24.716 LDAP Cache: User 'user1' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-aaaa-prod-rdr'
2022-05-07 20:06:24.717 LDAP Cache: User 'auser2' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-yyyy-prod-wtr'
2022-05-07 20:06:24.717 LDAP Cache: User 'ad_cibgvaprod_rdr' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-yyyy-prod-rdr'
2022-05-07 20:06:24.717 LDAP Cache: User 'ad_vcsmonprod_adm' is a member of group(s):
2022-05-07 20:06:24.717 'xxxx-tibems-bbbb-prod'
2022-05-07 20:06:24.717 'xxxx-tibems-aaaa-prod-shutdown'
2022-05-07 20:06:24.717 [user1@server1.svr.us.example.net]: Connected, connection id=21879, client id=<none>, type: queue, UTC offset=2

 

Here line starts with "SSL accepted cipher=" and ends with "ser1@server1.svr.us.example.net]: Connected,"

 

I would like timecharts cipher (ECDHE-RSA-AES256-GCM-SHA384), user (user1), Server (server1.svr.us.example.net) 

Stats like follows

Date Hour       Cipher   User   Server Count

10-10-20 10:00 ECDHE-RSA-AES256-GCM-SHA384) user1 server1 200

 

Please let me know if there an elegant solution to this,

Kannan

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Now that we have more sample data, we can refine the regular expressions.  Try this:

| rex "cipher=(?<cipher>[^\\\"]+)"
| rex "type\\\":\\\"(?<type>[^\\\"]+)"
| rex "Broker\\\":\\\"(?<Broker>[^\\\"]+)"
---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

varadack
Engager

Thanks just a start as cipher is not able to filter filed just cipher

Example On Cipher it adds extra field as it sees in splunk search

@timestamp2022-05-08T15:10:28.291Z
   @version1
   Brokerbroker
   a_time2022-05-08 15:10:28.275
   app_idappid
   hostserver
   message SSL accepted cipher=ECDHE-RSA-AES256-GCM-SHA384
   path/apps/broker/port/logs/server.log
   portport
   typetype

}
 
Show as raw text

ECDHE-RSA-AES256-GCM-SHA384","host":"xxxx","path":"/apps/broker/port/logs/server.domain.log","broker":"broker-NE1","type":"type","@version":"1","a_time":"2022-05-08

 

similarly

user and server also  comes as comes with other texts

like

user

/DC=net/DC=company/DC=vvvv/CN=zzz]","host":"server","path":"/apps/broker/port/logs/server.log","Broker":"broker-NE1","type":"type","

 

server also clobbered.

 

Thanks,

Kannan

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The regular expressions in my first reply are correct for the example events provided in the OP.  They may need to be adapted for live data.  If you need help with that, please provide actual events with sensitive data masked, but preserve the original format.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

varadack
Engager

The message is too long is there a way attach as a file here for your reference?

Kannan

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You can use the transaction command to fit the events together, then extract the desired fields with rex.  Finally, the stats command will give the counts.

| transaction startswith="SSL accepted cipher" endswith="]: Connected"
| rex "cipher=(?<cipher>\S+)"
| rex "\[(?<user>[^@]+)@(?<server>[^\]]+)"
| stats count by _time,cipher,user,server

 

---
If this reply helps you, an upvote would be appreciated.
0 Karma

varadack
Engager

Hi richgalloway,

Your starting query was very promising and used to restrict the regex modifying as 

rex "cipher=(?<cipher>\S+)\",\"(type|Broker)" |

But it includes type and Broker also in the cipher filed and counts as below but I wanted only cipher

 

cipher user server count

ECDHE-RSA-AES256-GCM-SHA384","type":"TYPEuser1server121
ECDHE-RSA-AES256-GCM-SHA384","Broker":"Broker"user2server125
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Now that we have more sample data, we can refine the regular expressions.  Try this:

| rex "cipher=(?<cipher>[^\\\"]+)"
| rex "type\\\":\\\"(?<type>[^\\\"]+)"
| rex "Broker\\\":\\\"(?<Broker>[^\\\"]+)"
---
If this reply helps you, an upvote would be appreciated.
0 Karma

varadack
Engager

Thanks a lot and it completely worked as expected and responded quick to solve this

0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

<P style=" text-align: center; "><span class="lia-inline-image-display-wrapper lia-image-align-center" ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

<FONT size="5"><FONT size="5" color="#FF00FF">Get the latest news and updates from the Splunk Community ...