topic Re: Field Extraction in All Apps and Add-ons

Field Extraction

jamesmcgee — Tue, 08 Nov 2011 16:09:27 GMT

Am trying to create a field extraction to report on at search time, based on data like the below

W3SVC1 IP.IP.IP.40 GET /service/79/ClientA/Default.aspx
W3SVC1 IP.IP.IP.17 GET /service/77/ClientB/Default.aspx
W3SVC3 IP.IP.IP.16 GET /service/77/ClientB/Default.aspx
W3SVC1 IP.IP.IP.40 GET /service/77/ClientC/Default.aspx
W3SVC7 IP.IP.IP.40 GET /service/79/ClientA/Default.aspx
W3SVC3 IP.IP.IP.16 GET /service/77/ClientB/Default.aspx

So, that I can report on "Get requests", to standard logon page, by client.

Am really struggling with either the IFX app, or the default "Field Extractor", and think there should be an easy way...

Any help appreciated.

Thanks

Re: Field Extraction

kristian_kolb — Tue, 08 Nov 2011 16:53:33 GMT

Hi,

If you are using a standard sourcetype for your log format, the method (GET, POST, OPTIONS, TRACE etc) should probably already be extracted.

Extract the client by

\d+\/(?<client>.*)\/Default\.aspx$

hope this helps,

Kristian

Re: Field Extraction

jamesmcgee — Wed, 09 Nov 2011 01:43:56 GMT

Top man, that did the job... but....How do I now count all "hits" by client... (sorry!)

So, rather than just "extracting" pages/hits with default.aspx, I can also count anything by client

i.e

/service/12/CLIENTA/logo.gif
/service/12/CLIENTA/manual.pdf
/service/12/CLIENTB/logo.gif
/service/12/CLIENTC/logo.gif
/service/12/CLIENTD/logo.gif

Now, want to count the "hits" by client, but am somehow pulling in anything beyond the CLIENTx/ mark no matter how I try to get the regex to work.

Thanks

Re: Field Extraction

kristian_kolb — Wed, 09 Nov 2011 14:33:07 GMT

Do you need help with the search query, or with the field extractions or both. I don't really understand your last sentence.

Re: Field Extraction

jamesmcgee — Wed, 09 Nov 2011 14:42:14 GMT

So, with what you gave me earlier, I can report on how many times default.aspx was hit, by client, using the extraction. What I am trying to do know, is show "all" hits, by client.

So, current search is soemthing like...

Source=iis | rex "\d+\/(?.*)\/Default.aspx" | timechart count by client

What this gives me, is fine, to show me "landing page" (default.aspx) numbers, by client, but what I'd like to show now, is how many hits in total, by client, so the easiest thing for me to do (I think), would be to

run search/chart on

/service/12/CLIENTA/.

/service/23/CLIENTB/.

/service/12/CLIENTC/.

/service/28/CLIENTD/.

So, that no matter what page they request, it get's counted in the logs as a hit, and I can extract it by client.

But, no matter what tweaking I do, do the regex you gave me earlier, I seem to pull in too much data. I'm not interest in specific pages, gifs/logos etc, but really how many "hits" there were overall, by client.

(or, do you have a link to regex for dummies)?

Thanks

Re: Field Extraction

kmattern — Wed, 09 Nov 2011 16:22:06 GMT

It sounds to me like you want to break out the path into multiple fields. The handy way to do that is to use multivalue field extraction. Assuming that this field,

"/service/12/CLIENTA/logo.gif"

is named Path, try something like this

| makemv delim="/" Path | eval cli=mvindex(Path,2) | stats count(cli) as Client_Count

Remember that the index starts with 0 and not 1. You can also say something like

| eval WebPage=mvindex(Path,3)

This will extract just the page from the longer string so you can count the web pages. You can even say something like

| stats count(cli) as ClientCount by WebPage

to get the count of each web page by client. I use this method all the time to show the top web pages hit on my portal.

Ken

Re: Field Extraction

kristian_kolb — Wed, 09 Nov 2011 20:33:10 GMT

Well, yes.. hmm, just what do the numbers (12,23,28) mean?

How do you mean that the regex pulls in too much data? Do you not get a field called 'client', containing values like CLIENTA, CLIENTB etc? Or are you getting everything from CLIENTx to the end of the line?

As I understand your post above, you want a count of all log lines (i.e. all requests regardless of the resource being requested), grouped on the .. 'instance' for lack of a better word. Is this 'instance' just the CLIENTx value or the CLIENTx value in combination with the preceeding number?