How to get certain fields from the logs

[xvxt006]

Hi, I am brand new to splunk, sorry if i am asking very basic questions. i have data in the below format (I have put 3 sample requests)

i would like to know how many times each command is being called from the logs in a tabular format. For example from the first request below, i need to extract "search"
and display the count.

For the first request the pattern is - the command will always be preceded by /Company/directory and ends with .shtml.

For the 2nd request the pattern is - always preceded by /typeahead (as i need to capture TypeaheadQueryResponder).

3rd request is SEO url - after /Company i would like to capture till ? (URI).

34.234.42.184 - - [26/Sep/2012:12:01:21 -0500] "GET
/Company/directory/search.shtml?searchQuery=desk+lights&op=search&btr=desk+lights&N=0&GlobalSearch=true HTTP/1.1" 200

237.189.83.254 - - [26/Sep/2012:12:01:21 -0500] "POST /typeahead/TypeaheadQueryResponder HTTP/1.1"
200

55.242.45.133 - - [26/Sep/2012:12:01:21 -0500] "GET
/Company/hand-protection/safety/ironclad/category/werwerre/No-48/WORK+GLOVES?Ner=textsearchesinbase%2Btrue HTTP/1.1"

Can someone help me with this. Thank you for your help in advance.

[lguinn2]

You have several choices:

• Create fields by editing props.conf. This will create a "permanent" search-time field that everyone can use
• Use the Interactive Field Extractor to create the same fields as editing props.conf
• Use the rex command to create fields "on-the-fly"

Extract and add new fields describes all of these options.

For all of them, it will be helpful to know regular expressions. Also, is this log indexed as sourcetype access_combined or access_combined_wcookie? If so, you have some existing fields that may help.

Following are some regular expressions that may work. I have shown them with the rex command.

1 - Extract the command field

 yoursearchhere | rex "/Company/directory/(?<command>.*?).shtml"

2 - Extract TypeaheadQueryResponder

yoursearchhere | rex "/typeahead/(?<TypeaheadQueryResponder>.*?)\s"

3 - Extract file

yoursearchhere | rex "/Company/(?<file>.*?)\?" 

[xvxt006]

Hi,

To consolidate all the results, i have used the 2 rex commands in the same search (consolidated the first and 2nd ).

my search | rex "/typeahead/(?.*?)[\s|/]" | rex

"/Company/directory/(?.*?).shtml" | top limit=5000 command

If i do that, i am getting results but when i do the search individually (having one rex command only), the search results are different. Can someone help me how to get the results consistently whether i do the search separately or having the all rex commands in the same search.

[lguinn2]

You could do it all in one search like this:

yoursearchhere
| rex "/Company/directory/(?<command>.*?).shtml"
| rex "/typeahead/(?<TypeaheadQueryResponder>.*?)\s"
| rex "/Company/(?<file>.*?)\?"
| yourstatisticshere

You might be able to put it all into one giant regular expression. But when I thought about that, it made my head hurt.

[xvxt006]

Thank you. They are giving results individually. Do you guys know how to combine all of these results? i have tried to have multiple rex (in the same search) and it is complaining. Any clue?

[xvxt006]

Thanks to both of you. Response to my question is blazing fast. I will try one of these solutions and let you guys know how it goes. Thx again.