topic Re: Creating usage reports from logfiles in Splunk Search

Creating usage reports from logfiles

huaraz — Sat, 27 Aug 2011 13:18:13 GMT

Hi,

I am new to splunk and heard it can do nearly every type of reporting. I have an ADSL router creating logs in the following format:

Aug 25 23:00:22 Vigor: Local User: 192.168.1.8:50829 -> 212.58.244.67:80 (TCP)Web

How can I create a pie chart showing for each source (e.g. 192.168.1.8) to which destinations (e.g. 212.58.244.67) they went.

I can import the logs and select the appropriate lines, but I don't know I can define variables (are these fields in splunk ?) for src and dst and plot them.

Thank you
Markus

I read a bit about custom fields. I see without any search regex the follwoing events
Aug 25 23:00:22 Vigor: Local User: 192.168.1.8:50829 -> 212.58.244.67:80 (TCP)Web host=ip-10-17-23-243 Options| sourcetype=router-kiwi Options| source=/home/markus/data/router-kiwi-2011-08-25.txt Options

I created the following field extractions for host ip-10-17-23-243 (as it forces me to use either host, source or sourcetype)

"User:\s*(?<mysrc>:.*)
and
->\s*(?<mydst>:.*)"

But when I create a search mysrc="192.168.1.8" I don't get anything. What do I do wrong ?
How can I check the fields are correct ? When I do a search with | rex field=_raw "User:\s*(?<mysrc>:.*) ->\s*(?<mydst>:.*)" I don't get an error, but I also don't know what is mysrc nor mysdst.

Markus

Re: Creating usage reports from logfiles

Ayn — Sun, 28 Aug 2011 16:24:46 GMT

How are you creating the fields? As part of your search using the rex operator, through the field extractor or directly through a configuration file such as props.conf?

Re: Creating usage reports from logfiles

chris — Mon, 29 Aug 2011 12:40:31 GMT

Hi Huaraz

To extract the ips you can try to add this to your search:
| rex field=_raw "User:\s+(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*\s+->\s+(?<dst>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"

Splunk doesn't tell you that the regex you apply to a search is not working the way you want, because it can only detect syntax errors.

There are different ways to get the desired result:

Check if the fields you just tried to create appear in the List if you click on "Pick fields" to the left of your search results (check the values that were found)
Splunk can help you generate the regex if you select "Extract Fields" from the context menu of an event that contains the values you want to extract into fields/variables. You can then test and save field extractions
You could also use 3rd party tool to help you with your regexes (http://regex.larsolavtorvik.com/)

Once you have your fields you can append a reporting command to your search (then click on show report to format the report):

Popular destinations(pie chart:

| chart count(src) by dst

Active Sources (pie chart):

| chart count(dst) by src

Show when a source is active (line chart):

| timechart count(src) by src

Or you could also just create a table of your sources and destination tuples:

| rex field=_raw "User:\s+(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*\s+->\s+(?<dst>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" | where isnotnull(src) | table src dst | sort src

Re: Creating usage reports from logfiles

huaraz — Wed, 31 Aug 2011 21:59:37 GMT

I used Manager » Fields » Field extractions to add a new field

Markus

Re: Creating usage reports from logfiles

huaraz — Wed, 31 Aug 2011 22:46:01 GMT

That worked
Thank you
Markus