Splunk Search

Lookup Table and Regex

aksampat81
New Member

Hi,
I need some help with lookup table combined with regular expressions.

I have the an apache log file which looks like:

*25/Jul/2018:10:17:30.999;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/balanceviewer/ HTTP/1.0;200;111;12421;15yjLFielsdx424u6S7Y136sHTvfpFWP;-;+dF15erUTMPMyqlLYEUugh7j7eJjtKtub5pRai/Z3ckGx0AkbJnDI3PPwHrwzRdk+bVwDNNU0ducGu0IZ8cKVA==;host/195 CFNetwork/651.5.4 Darwin/11.7.1;1493;"-";
25/Jul/2018:10:17:30.999;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/balanceviewer/ HTTP/1.0;200;111;12421;18y3LF87lsdx4643AS7Y13ZsHT4bpFWP;-;+dF15erUTMPMyqlLYEUugh7j7eJjtKtub5pRai/Z3ckGx0AkbJnDI3PPwHrwzRdk+bVwDNNU0ducGu0IZ8cKVA==;host/195 CFNetwork/651.5.4 Darwin/11.7.1;1493;"-";
25/Jul/2018:10:17:30.999;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/balanceviewer/ HTTP/1.0;200;111;12421;17yh6F9tlsd34A43AS7Y19ZseevbpFWP;-;+dF15erUTMPMyqlLYEUugh7j7eJjtKtub5pRai/Z3ckGx0AkbJnDI3PPwHrwzRdk+bVwDNNU0ducGu0IZ8cKVA==;host/195 CFNetwork/651.5.4 Darwin/11.7.1;1493;"-";
25/Jul/2018:10:17:30.997;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/getyoursettings/N1vAO8zgb38oYh6MogfhwQ HTTP/1.0;200;111;1111;155eUDXOjDT/G2QzEf2WvZulbirv4Fw2;https://host.temp.nl/finance/us/pay/transactions.html?account=A1vAO8zgbR8oghPMogfhwQ;n0hJVF5icTKeT+zO/5Xhs+Ul/QHwyqsrngxzEC6hPmU1RNpc5zF/51fSh07kfpHzp96YMt0bZiwZj76ze+N9GA==;Mozilla/5.0 (Linux; Android 7.0; HUAWEI NXT-L29 Build/HUAWEINXT-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Crosswalk/19.49.514.5 Mobile Safari/537.36;2362;"-";
25/Jul/2018:10:17:30.997;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/getyoursettings/Q2vAO8zg5R8oY4PMogfhwQ HTTP/1.0;200;111;1111;155gUDX4jDT/G2rzEf2WvZulbirv4Fwt;https://host.temp.nl/finance/us/pay/transactions.html?account=s1vAO8zgbs8oYhPMogfhwQ;n0hJVF5icTKeT+zO/5Xhs+Ul/QHwyqsrngxzEC6hPmU1RNpc5zF/51fSh07kfpHzp96YMt0bZiwZj76ze+N9GA==;Mozilla/5.0 (Linux; Android 7.0; HUAWEI NXT-L29 Build/HUAWEINXT-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Crosswalk/19.49.514.5 Mobile Safari/537.36;2362;"-";
25/Jul/2018:10:17:30.997;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/getyoursettings/A3vAO8zg5R8o4hPMogfhwQ HTTP/1.0;200;111;1111;155gUDXOfDT/G2QzEf2WvZulbirv4Ftv;https://host.temp.nl/finance/us/pay/transactions.html?account=f1vAO8zdbR8oYhPMogfhwQ;n0hJVF5icTKeT+zO/5Xhs+Ul/QHwyqsrngxzEC6hPmU1RNpc5zF/51fSh07kfpHzp96YMt0bZiwZj76ze+N9GA==;Mozilla/5.0 (Linux; Android 7.0; HUAWEI NXT-L29 Build/HUAWEINXT-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Crosswalk/19.49.514.5 Mobile Safari/537.36;2362;"-";*

I want to use the lookup and regex command to get an output based on the URL hit:

URL,TransactionName, avg_Responsetime,Hits
 /api/balances/balanceviewer/,BALANCEVIEWER,1493,3
/api/balances/getyoursettings,GETYOURSETTINGS,2362,3

Does anyone know how I can make this kind of a table/report with splunk?

Thanks in advance.

Amit

0 Karma

sudosplunk
Motivator

As stated by @DalJeanis, if the assumptions are correct, then your search would look something like this:

Your_basic_search
| rex field = _raw ".+finance\S+\;GET\s(?<URL>\/\w+\/\w+\/\w+\/)"
| rex field = URL "\/\w+\/\w+\/(?<TransactionName>\w+)
| rex field = _raw ".+\;(?<avg_Responsetime>\d+)\;\"\-\""
|  eval TransactionName=upper(TransactionName)
| stats count by TransactionName as Hits
| table URL TransactionName avg_Responsetime Hits
| outputlookup foo.csv

Note: If your search heads are clustered, then use outputlookup command, otherwise use outputcsv command.

0 Karma

DalJeanis
Legend

We've made some assumptions here that you will need to review with your real data. See the second second for your code and a discussion.

The first section is just a test data generator:

| makeresults | eval myRaw="25/Jul/2018:10:17:30.999;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/balanceviewer/ HTTP/1.0;200;111;12421;15yjLFielsdx424u6S7Y136sHTvfpFWP;-;+dF15erUTMPMyqlLYEUugh7j7eJjtKtub5pRai/Z3ckGx0AkbJnDI3PPwHrwzRdk+bVwDNNU0ducGu0IZ8cKVA==;host/195 CFNetwork/651.5.4 Darwin/11.7.1;1493;\"-\";!!!!25/Jul/2018:10:17:30.999;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/balanceviewer/ HTTP/1.0;200;111;12421;18y3LF87lsdx4643AS7Y13ZsHT4bpFWP;-;+dF15erUTMPMyqlLYEUugh7j7eJjtKtub5pRai/Z3ckGx0AkbJnDI3PPwHrwzRdk+bVwDNNU0ducGu0IZ8cKVA==;host/195 CFNetwork/651.5.4 Darwin/11.7.1;1493;\"-\";!!!!25/Jul/2018:10:17:30.999;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/balanceviewer/ HTTP/1.0;200;111;12421;17yh6F9tlsd34A43AS7Y19ZseevbpFWP;-;+dF15erUTMPMyqlLYEUugh7j7eJjtKtub5pRai/Z3ckGx0AkbJnDI3PPwHrwzRdk+bVwDNNU0ducGu0IZ8cKVA==;host/195 CFNetwork/651.5.4 Darwin/11.7.1;1493;\"-\";!!!!25/Jul/2018:10:17:30.997;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/getyoursettings/N1vAO8zgb38oYh6MogfhwQ HTTP/1.0;200;111;1111;155eUDXOjDT/G2QzEf2WvZulbirv4Fw2;https://host.temp.nl/finance/us/pay/transactions.html?account=A1vAO8zgbR8oghPMogfhwQ;n0hJVF5icTKeT+zO/5Xhs+Ul/QHwyqsrngxzEC6hPmU1RNpc5zF/51fSh07kfpHzp96YMt0bZiwZj76ze+N9GA==;Mozilla/5.0 (Linux; Android 7.0; HUAWEI NXT-L29 Build/HUAWEINXT-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Crosswalk/19.49.514.5 Mobile Safari/537.36;2362;\"-\";!!!!25/Jul/2018:10:17:30.997;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/getyoursettings/Q2vAO8zg5R8oY4PMogfhwQ HTTP/1.0;200;111;1111;155gUDX4jDT/G2rzEf2WvZulbirv4Fwt;https://host.temp.nl/finance/us/pay/transactions.html?account=s1vAO8zgbs8oYhPMogfhwQ;n0hJVF5icTKeT+zO/5Xhs+Ul/QHwyqsrngxzEC6hPmU1RNpc5zF/51fSh07kfpHzp96YMt0bZiwZj76ze+N9GA==;Mozilla/5.0 (Linux; Android 7.0; HUAWEI NXT-L29 Build/HUAWEINXT-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Crosswalk/19.49.514.5 Mobile Safari/537.36;2362;\"-\";!!!!25/Jul/2018:10:17:30.997;2.2.2.255;temp-prod.nl;https://finance.host.nl;GET /api/balances/getyoursettings/A3vAO8zg5R8o4hPMogfhwQ HTTP/1.0;200;111;1111;155gUDXOfDT/G2QzEf2WvZulbirv4Ftv;https://host.temp.nl/finance/us/pay/transactions.html?account=f1vAO8zdbR8oYhPMogfhwQ;n0hJVF5icTKeT+zO/5Xhs+Ul/QHwyqsrngxzEC6hPmU1RNpc5zF/51fSh07kfpHzp96YMt0bZiwZj76ze+N9GA==;Mozilla/5.0 (Linux; Android 7.0; HUAWEI NXT-L29 Build/HUAWEINXT-L29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Crosswalk/19.49.514.5 Mobile Safari/537.36;2362;\"-\";"
| makemv delim="!!!!" myRaw
| mvexpand myRaw
| rename myRaw as _raw
| rename COMMENT as "The above just enters your data"

Now, here's the actual code and assumptions made:

| rename COMMENT as "Here we assume the URL must start with /api and is all lower case plus slashes."
| rex "(?<URL>/api[a-z0-9//]*)" 

| rename COMMENT as "Here we assume the transaction name is whatever is between the last two slashes, and must be converted to upper case."
| rex field=URL "[//](?<TransactionName>[a-z0-9]*)[//]$"
| eval TransactionName=upper(TransactionName)

| rename COMMENT as "Here we assume the response time is a numeric entry between the two semicolons that are third and second from the end of each record."
| rex ";(?<Resp>\d+);[^;]*;$"

| rename COMMENT as "Now we calculate the count and average for each combination."
| stats avg(Resp) as AvgResp count as Hits by URL TransactionName
| eval AvgResp=round(AvgResp)

If you need help chatting about regular expressions, please get connected up to our Slack channel. There is a #regex subchannel where you can get quick iterations on minor changes to assumptions like the above.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...

Introduction to Splunk AI

How are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. Lucky for ...