Custom field extraction and table output

[senthamilselvan]

Hi Team,

I have the below sample log file. I want to filter all the lines starting with "NET," and also want to create a table output with the field value paired data. The field name should be "NET,Network I/O ga016d02,en11-read-KB/s,en13-read-KB/s etc" and their respective values.
Here in our scenario, every line will be separate event in Splunk and the header will also be one of the events. So we have to pick that event as header and manipulate other events to get the values. Also, it will be great if this is possible using search query.

NET,T0005,108376.3,3.9,3199.0,0.1,908.2,51672.7,859.3,12676.2,0.1,908.2
NETPACKET,T0005,91328.5,33.8,4038.7,1.5,107.1,91061.3,28.0,2716.1,1.2,107.1
NET,Network I/O ga016d02,en11-read-KB/s,en13-read-KB/s,en12-read-KB/s,en15-read-KB/s,lo0-read-KB/s,en11-write-KB/s,en13-write-KB/s,en12-write-KB/s,en15-write-KB/s,lo0-write-KB/s
PROC,T0006,14.21,0.01,235490,709352,8010,4281,67,52,28095,9005,0,0,0
FILE,T0006,0,9679,0,313480806,5976928,0,0,0
NET,T0006,119694.1,7.6,1517.8,0.1,304.5,43834.8,1712.4,23185.6,0.1,304.5
FILE,T0010,0,9950,0,116724025,6390813,0,0,0
NET,T0010,130704.1,11.4,951.4,0.2,5.2,46862.3,2573.1,26190.6,0.1,5.2
NETPACKET,T0010,92532.1,95.1,3382.2,1.6,15.6,92412.0,81.1,1968.8,1.2,15.6

Thanks Selvan J

[alemarzu]

Hi there @senthamilselvanj

Something like this might work, just rename field names by your header fieldnames.

... | rex "^\NET,(?<field1>[^\,]+),(?<field2>[^\,]+),(?<field3>[^\,]+),(?<field4>[^\,]+),(?<field5>[^\,]+),(?<field6>[^\,]+),(?<field7>[^\,]+),(?<field8>[^\,]+),(?<field9>[^\,]+),(?<field10>[^\,]+),(?<field11>[^\s]+)" | table field1 field2 field3 ...

Hope it helps

[praphulla1]

field1,2,3 are part of another event

NET,Network I/O ga016d02,en11-read-KB/s,en13-read-KB/s,en12-read-KB/s,en15-read-KB/s,lo0-read-KB/s,en11-write-KB/s,en13-write-KB/s,en12-write-KB/s,en15-write-KB/s,lo0-write-KB/s

so its not possible to use the rex provided.

[alemarzu]

You are not being clear enough.

If field1~3 has to be discarded then this regex should work.

... | rex "^\NET,[^\,]+,[^\,]+,[^\,]+,(?<field4>[^\,]+),(?<field5>[^\,]+),(?<field6>[^\,]+),(?<field7>[^\,]+),(?<field8>[^\,]+),(?<field9>[^\,]+),(?<field10>[^\,]+),(?<field11>[^\s]+)"

[praphulla1]

Let me reframe my sentence. I meant to say that the we are not sure how many fields would be available so the regex cannot be prepared. The search query has to dynamically treat the below line as header and from a table based on the rest of the events.

CSV Header event
NET,Network I/O ga016d02,en11-read-KB/s,en13-read-KB/s,en12-read-KB/s,en15-read-KB/s,lo0-read-KB/s,en11-write-KB/s,en13-write-KB/s,en12-write-KB/s,en15-write-KB/s,lo0-write-KB/s

[praphulla1]

field1,2,3 are part of another event
NET,Network I/O ga016d02,en11-read-KB/s,en13-read-KB/s,en12-read-KB/s,en15-read-KB/s,lo0-read-KB/s,en11-write-KB/s,en13-write-KB/s,en12-write-KB/s,en15-write-KB/s,lo0-write-KB/s

how can be pass those values from that event to rex ?

[peterchenadded]

That's nmon data, you should look into http://nmon-for-splunk.readthedocs.io/en/latest/Userguide.html and see how they do it.