Splunk Search

How to get certain fields from the logs

Contributor

Hi, I am brand new to splunk, sorry if i am asking very basic questions. i have data in the below format (I have put 3 sample requests)

i would like to know how many times each command is being called from the logs in a tabular format. For example from the first request below, i need to extract "search"
and display the count.

For the first request the pattern is - the command will always be preceded by /Company/directory and ends with .shtml.

For the 2nd request the pattern is - always preceded by /typeahead (as i need to capture TypeaheadQueryResponder).

3rd request is SEO url - after /Company i would like to capture till ? (URI).

34.234.42.184 - - [26/Sep/2012:12:01:21 -0500] "GET
/Company/directory/search.shtml?searchQuery=desk+lights&op=search&btr=desk+lights&N=0&GlobalSearch=true HTTP/1.1" 200

237.189.83.254 - - [26/Sep/2012:12:01:21 -0500] "POST /typeahead/TypeaheadQueryResponder HTTP/1.1"
200

55.242.45.133 - - [26/Sep/2012:12:01:21 -0500] "GET
/Company/hand-protection/safety/ironclad/category/werwerre/No-48/WORK+GLOVES?Ner=textsearchesinbase%2Btrue HTTP/1.1"

Can someone help me with this. Thank you for your help in advance.

Tags (2)
0 Karma

Legend

You have several choices:

  • Create fields by editing props.conf. This will create a "permanent" search-time field that everyone can use
  • Use the Interactive Field Extractor to create the same fields as editing props.conf
  • Use the rex command to create fields "on-the-fly"

Extract and add new fields describes all of these options.

For all of them, it will be helpful to know regular expressions. Also, is this log indexed as sourcetype access_combined or access_combined_wcookie? If so, you have some existing fields that may help.

Following are some regular expressions that may work. I have shown them with the rex command.

1 - Extract the command field

 yoursearchhere | rex "/Company/directory/(?<command>.*?).shtml"

2 - Extract TypeaheadQueryResponder

yoursearchhere | rex "/typeahead/(?<TypeaheadQueryResponder>.*?)\s"

3 - Extract file

yoursearchhere | rex "/Company/(?<file>.*?)\?" 

Contributor

Hi,

To consolidate all the results, i have used the 2 rex commands in the same search (consolidated the first and 2nd ).

my search | rex "/typeahead/(?.*?)[\s|/]" | rex

"/Company/directory/(?.*?).shtml" | top limit=5000 command

If i do that, i am getting results but when i do the search individually (having one rex command only), the search results are different. Can someone help me how to get the results consistently whether i do the search separately or having the all rex commands in the same search.

0 Karma

Legend

You could do it all in one search like this:

yoursearchhere
| rex "/Company/directory/(?<command>.*?).shtml"
| rex "/typeahead/(?<TypeaheadQueryResponder>.*?)\s"
| rex "/Company/(?<file>.*?)\?"
| yourstatisticshere

You might be able to put it all into one giant regular expression. But when I thought about that, it made my head hurt.

0 Karma

Contributor

Thank you. They are giving results individually. Do you guys know how to combine all of these results? i have tried to have multiple rex (in the same search) and it is complaining. Any clue?

0 Karma

Contributor

Thanks to both of you. Response to my question is blazing fast. I will try one of these solutions and let you guys know how it goes. Thx again.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!