topic Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords? in Splunk Search

How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid — Tue, 29 Sep 2020 16:45:06 GMT

I have a set of keywords which I need to search and generate the output similar to
https://answers.splunk.com/answers/588828/how-do-you-display-the-of-requests-per-24-hours-of.html#comment-589033

keywords:

DCBClaimSearch
CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count=
etc

log format:

1x.xx.xxx.xxx - - 1xxxx4 [04/Oct/2017:00:03:44 -0400] - /xxx/Gxxxt/ibxx_xxxxxxxxxxxxxxxxxxxxxxx[[/!TABTHREAD1 HTTP/1.1 xxxxxxxxxxx.net TIME:0/49626 "POST /pxxxb/Gxxxxt/ixxxU_xxxxxxxxxxxxxxxxxxxxxxxxx/!TABTHREAD1?pyActivity=xxxxxxxxxxxxxxxxxxxxxxxx%20&Request_Type=&xxxxxxxxxxxx_CD=Nxxx_xxxxEL&Exxxxxxxxx_CD=&Lxxxxxxxxxe=&CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count= HTTP/1.1" 200 737

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

kamlesh_vaghela — Fri, 10 Nov 2017 12:21:13 GMT

Hi @saifullakhalid,

Can you please try this search?

YOUR_SEARCH
| rex field=_raw Count=(.*)\s.*\s(?<COUNT>.*) 
| stats count AS "Count" sum(COUNT) as "Sum of COUNT" min(_time) AS start_time max(_time) as end_time by date_month,date_mday,date_hour
| eval "Start Time"=strftime(start_time,"%d/%m/%Y %I:%M:%S:%p") 
| eval "End Time"=strftime(end_time,"%d/%m/%Y %I:%M:%S:%p") 
| table "Start Time" "End Time" "Count" "Sum of COUNT"

Thanks

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid — Tue, 29 Sep 2020 16:45:31 GMT

This is displaying me start time , end time Count and total count, How should I display the keys words below along with these. I tried adding COUNT, but it was empty. I tried adding like below but it is displaying the full request, I just wanted keywords to display.

index="prod_4oct_508" pyActivity="" | rex field=_raw Count=(.)\s.\s(?.)
| stats count AS "Count" sum(COUNT) as "Sum of COUNT" min(_time) AS start_time max(_time) as end_time by date_month,date_mday,date_hour,pyActivity
| eval "Start Time"=strftime(start_time,"%d/%m/%Y %I:%M:%S:%p")
| eval "End Time"=strftime(end_time,"%d/%m/%Y %I:%M:%S:%p") | dedup pyActivity | sort –count
| table "Start Time" "End Time" "pyActivity" "Count" "Sum of COUNT"

keywords:

DCBClaimSearch
CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count=

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

kamlesh_vaghela — Tue, 29 Sep 2020 16:45:37 GMT

Hi @saifullakhalid,

Do you want to extract these keywords?

DCBClaimSearch
CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count=

Below search will extract "CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count="

index="prod_4oct_508" pyActivity="" 
| rex field=_raw "&CountryCode=(?<FIELD1>.*)\s(.*)\s(.*)\s(?<COUNT>.*)"
| table _time FIELD1 COUNT

Can you please let me know the pattern of keyword "DCBClaimSearch" ?

Thanks

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid — Tue, 29 Sep 2020 16:45:39 GMT

?pyActivity=...............................................PreActivity=DCBClaimSearch&HeaderButtonSectionName.................HTTP/1.1" 200 4502
?pyActivity=...........................LanguageCode=&CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count=..........HTTP/1.1" 200 3402

Every keyword has different pattern and it lies in middle of requests but all will start with ?pyActivity=

For example:
?pyActivity=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx%20&Request_Type=&xxxxxxxxxxxxxx_xxxxxxxxxxxxxxx&ELEMENT_CD=&LanguageCode=&CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count= HTTP/1.1" 200 5092

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

kamlesh_vaghela — Fri, 10 Nov 2017 16:32:30 GMT

Hi @saifullakhalid,

Can you please try this?

 index="prod_4oct_508" pyActivity="*" 
| rex field=_raw "&CountryCode=(?<FIELD1>.*)\s(.*)\s(.*)\s(?<COUNT>.*)"
| rex field=_raw "pyActivity=(.*)PreActivity=(?<PreActivity>.*)&" 
| table _time PreActivity _time FIELD1 COUNT

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid — Tue, 29 Sep 2020 16:45:44 GMT

I have written a java program which reads the list of keywords (61 keywords )from csv and parse the logs and generate 24 hours reports like below for each keyword. Can we do this in splunk if so then please suggest.

First report format:

User Time Protocol Url Elapsed Time (Seconds) call Status Size logname
1432029 17/Oct/04 05:19:05 HTTP/1.1 xxxxxxxxxxx.net 0.150266 POST /xxxx/xxxxt/xxxxxxxxxxxxxx*/!TABTHREAD4?pyActivity=Reloadxxxx&pzxxxxxxxx=xxxxxxxxx&pzFromFrame=xxxxxxxxx&pzxxxxe=pyxxxe&pzxxxxx=false&StxxxxName=xxxxxe&xxxx&inStandardsMode=false&AJAXTrackID=3&pzHarnessID=xxx36749 HTTP/1.1 200 2207 \507\access_log_10_04_2017

second report format:

Transaction keyword Start Time End Time Total # of executions Avg # of Executions per Hour Min Resp Time Max Resp Time Avg Resp Time 90th percentile Resp Time Std Dev Of Resp Time Min Size of Response Max Size of Response Avg Size of Response 90th percentile Size of Response Std Dev Of Size of Response

TC01 DCBClaimSearch 17/Oct/04 00:11:46 17/Oct/04 23:24:05 2398 104 0.029761 27.35804 0.108725955 0.1099671 0.594170951 82 10342 4302.940784 4543 424.216474

3rd report:
Transaction keyword Start Time End Time Total # of executions Avg # of Executions per Hour Min Resp Time Max Resp Time Avg Resp Time 90th percentile Resp Time Std Dev Of Resp Time Min Size of Response Max Size of Response Avg Size of Response 90th percentile Size of Response Std Dev Of Size of Response

TC01 DCBClaimSearch 17/Oct/04 00:00:00 17/Oct/04 00:59:59 4 4 0.056125 0.070999 0.0613225 0.070999 0.00671778 3617 4533 3886.75 4533 437.5083809
TC01 DCBClaimSearch 17/Oct/04 01:00:00 17/Oct/04 01:59:59 3 3 0.058215 0.080105 0.066264 0.080105 0.012039662 3780 4548 4036 4548 443.4050067

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid — Fri, 10 Nov 2017 16:47:26 GMT

looks like it is extracting values which has the pattern "&CountryCode" and c
But is it possible to read these keywords values which needs to be extracted using csv. because as I said the pattern is every line in the code starts with ?pyActivity= , but the keywords falls in middle of this request. like in above comment "CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count=" is part of the line ?pyActivity=

Re: How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid — Fri, 10 Nov 2017 17:25:03 GMT

I have created a different question for reports mentioned below https://answers.splunk.com/answers/590577/how-do-you-create-the-report-for-the-log-access-be.html?minQuestionBodyLength=80