topic Setting fields from logs with different row values in Splunk Search

Setting fields from logs with different row values

juancarlos_pola — Fri, 10 Oct 2014 07:45:24 GMT

Hello everybody,

I have a question that might have been responded before but I have a log file from a server that looks like this

10:01pm load_disk:  200
10:01pm sessions: 2
10:01pm maxpage: 201
10:01pm datadisk: 30
10:01pm memory: 10
10:02pm load_disk:201
10:02pm sessions: 3
10:02pm maxpage: 202
10:02pm datadisk: 31
10:02pm memory: 11
....

As you can see, it is a log file that shows many different values on each row but the values repeat (in this case) every 5 rows. I would like to extract a field for each line that defines a log:
field1 -> load_disk
field2 -> sessions
...

When I try to extract using the Extract Field option in the search, it does not show me all the lines of the log ( this log has a huge number of lines that are different and repeat ) maybe because of window size but I cannot see all the lines of the log (before repeating).

Best regards,

Juan

Re: Setting fields from logs with different row values

tom_frotscher — Fri, 10 Oct 2014 09:43:51 GMT

I do not completely understand your question. Do you want to put all fields, in this case load_disk, sessions, maxpage, datadisk and memory on one row?

So you want to gor from your example to something like this:

10:01pm load_disk: 200, sessions: 2, maxpage: 201, datadisk: 30, memory: 10
10:02pm load_disk: 201, sessions: 3, maxpage: 202, datadisk: 31, memory: 11

Re: Setting fields from logs with different row values

juancarlos_pola — Tue, 14 Oct 2014 04:49:54 GMT

Hello Tom,

Thank you for your reply and I am sorry for the long silence.

What I actually wanted from the previous log was to set a field for each log variable as follows:

field1 = load_disk
field2 = sessions
field3 = maxpage
field4 = datadisk
field5 = memory

so I can draw a line chart with all these fields inside one chart:

source=/admin/server/* | timechart first(field1) first(field2) avg(field3) first(field4) avg(field5)

My issue now is that the REAL log has more than 80 different variables and they repeat every 10 minutes, so I would like to set a field for each log variable. I saw some usage of the regex editor but I am quite new in Splunk so I would like some advice or if you have had previous experience with such kind of log files

Thank you in advance.

Re: Setting fields from logs with different row values

lguinn2 — Tue, 14 Oct 2014 05:09:36 GMT

You could do this:

in props.conf

[yoursourcetypehere]
REPORT-eaf=extract-all-fields

in transforms.conf

[extract-all-fields]
FORMAT = $1::$2
REGEX  = (\S+?)\s*:\s*(\S+)

This should give you all the field extractions.