Splunk Search

How to extract the numeric value and IP address from a string in my sample data?

chandukreddi
Path Finder

hello,

My log contains below entries.

2017-10-06T04:19:25.658+0000 I NETWORK [initandlisten] connection accepted from 12.34.56.789:12345 #192 (10 connections now open)

I am looking for 2 things.

  1. I want to create a timechart for "Totalconnections". This information will come from the string "(10 connections now open)" and I want to timechart the number 10
  2. I want to count the IPaddress to know how many connections there are per IP.
1 Solution

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi Chandukreddi,

can you please try below search??

YOUR SEARCH | rex "(?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+)" | chart sum(ConnectionCount) as ConnectionCount over _time by  IPAddress

View solution in original post

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi Chandukreddi,

can you please try below search??

YOUR SEARCH | rex "(?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+)" | chart sum(ConnectionCount) as ConnectionCount over _time by  IPAddress

chandukreddi
Path Finder

Thanks Kamlesh! it worked!!

could you please help me on above session count timechart? Somesoni was trying to help me but still I am not getting expected output.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi ChandukReddi,
Sure ..
Can you please let me know what you expect as session count from below sample event?

2017-10-06T04:05:53.268+0000 I NETWORK [initandlisten] connection accepted from IP:PORT #187 (12 connections now open)
0 Karma

chandukreddi
Path Finder

Hi Kamlesh,

I wan to see a number of open connections in timechart graph from above sample log.

2017-10-06T04:05:53.268+0000 I NETWORK [initandlisten] connection accepted from IP:PORT #187 (12 connections now open)

At time "2017-10-06T04:05:53" there were total "12 connections now open", I want to see this session count in graph.

and also in this ring we have multiple hosts and each host will have same kind of log, so want to get the count by host.

Mostly I will see open sessions count graph in last 1 hr per minute.

Please let me know if it is not clear.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi Chandu,

Can you please execute below search for last 1 hour??

 | rex "(?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+)" | bucket _time span=1m | chart sum(ConnectionCount) as ConnectionCount over _time by  IPAddress

Here I have considered "IP" as a host.
This search will provide you host wise sum of session count by every minute.

Kindly correct me if I'm wrong in IP considerations.

0 Karma

chandukreddi
Path Finder

Hi Kamlesh,

No we should not count by IP addres, Here is IP address is client ip address.

I am just looking for total connection in cluster (we have 3 nodes in cluster) and in our logs it shows how many connections were open at that particular time period.

I just want to filter "12 connections now open" this string from bellow sample log and grep for number 12 and show them in the graph.

2017-10-06T04:05:53.268+0000 I NETWORK [initandlisten] connection accepted from IP:PORT #187 (12 connections now open)

Please let me know if I am not clear.

Thanks for your help!

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi Chandu,
I'm just trying to understand log files and how they forwarded to the indexer.

These logs are coming from clusters ( means from all the hosts of the cluster). Am I right ?? So we can use host field as cluster host.

Please check below search. This search will show you a timeline of host wise connection.

YOUR SEARCH | rex "(?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+)" | chart values(ConnectionCount) as ConnectionCount over _time by  host

If you found multiple values in ConnectionCount then we have to take latest value from them. So In this case use below search.

YOUR SEARCH | rex "(?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+)" | chart latest(ConnectionCount) as ConnectionCount over _time by  host
0 Karma

somesoni2
SplunkTrust
SplunkTrust

You need to first capture those IP and connection number into field, like this

your base search | rex "connection accepted from (?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+) connections"

Now you can do total connection timechart like this

above search | timechart sum(ConnectionCount) as TotalConnections

For count of connections per IP address

above search | timechart sum(ConnectionCount) as TotalConnections by IPAddress

chandukreddi
Path Finder

Below query is giving me output 500 to 1100 connections but as per my logs connections are in between 10 to 30.
ring=xxxx source=xxxx "NETWORK" earliest=-4h | rex "connection accepted from (?\d+.\d+.\d+.\d+):[^(]+((?\d+) connections"| timechart sum(ConnectionCount) as TotalConnections

lets not worry about Connection per IP for now, I just need connection count i.e 10
from this string (10 connections now open) because these are real connections.

Here is the log sample:

2017-10-06T04:01:24.889+0000 I NETWORK [conn183] end connection xxx (9 connections now open)

0 Karma

somesoni2
SplunkTrust
SplunkTrust

In stats, use max or latest instead of sum.

your base search | rex "connection accepted from (?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+) connections" | timechart max(ConnectionCount) as TotalConnections

OR

your base search | rex "connection accepted from (?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+) connections" | timechart latest(ConnectionCount) as TotalConnections

chandukreddi
Path Finder

Thanks Somesoni!

Can't I get exact connected sessions graph instead of Max/Latest/avg?

Example:
From log entries at 04:05:53.268 I have 12 open connections (I just want to see in my graph 12 at that timestamp) and at 4:19:25.658 I have 10 connections open, so when I do plot a graph I want to see exact count so that I will get idea how many sessions were active at particular time.

2017-10-06T04:05:53.268+0000 I NETWORK [initandlisten] connection accepted from IP:PORT #187 (12 connections now open)

2017-10-06T04:19:25.658+0000 I NETWORK [initandlisten] connection accepted from IP:Port (10 connections now open)

2017-10-06T04:23:55.733+0000 I NETWORK [initandlisten] connection accepted from #193 (10 connections now open)

Sorry I am very new to splunk, we just started using this.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can actually do this

your base search | rex "connection accepted from (?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+) connections" | table _time ConnectionCount

This will display all the points with corresponding connection count. Please note that there is a limit on how many points can be plotted in the chart so it may not show all points based on how much data you select. See this for more details:
https://docs.splunk.com/Documentation/Splunk/7.0.0/Viz/ChartDisplayissues#Time_charting

0 Karma

chandukreddi
Path Finder

Excellent that worked Somesoni!

But I have multiple hosts on that ring, how do I get per host level?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this (will create a new field with same name as value of field host, and that new field will contains corresponding connection count value)

your base search | rex "connection accepted from (?<IPAddress>\d+\.\d+\.\d+\.\d+):[^\(]+\((?<ConnectionCount>\d+) connections" | table _time host ConnectionCount | eval {host}=ConnectionCount | fields - host ConnectionCount
0 Karma

chandukreddi
Path Finder

Somesoni, My graph is not showing as timestamp based, it's giving random time results.

Graph showing like this.
example:
First it showing 13:19:31 sessions count, second 13:48:01 sessions count and then 13:39:03 timestamp sessions count, it just shows random order.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...