topic Re: Should we perform a field extraction or field transformation on this sample log? in Splunk Search

Should we perform a field extraction or field transformation on this sample log?

splunker9999 — Fri, 13 Jan 2017 18:44:54 GMT

Hi,

We are looking transform fields from log events, can some one please help.

we need to translate to below codes:
I = Upload
j = Errored
k = Aborted
o = Successful
p =Errored
q = Aborted

In the below example "i" in bold is where we need to translate.

sat dec 13 10:01:17 2017 0 10.01.42.1 42288 /home/cat/wat/g1/T24_94291_20170113_093008_20161213110014510.txt b s **i** r user 0 *

Thanks

Re: Should we perform a field extraction or field transformation on this sample log?

AnthonyTibaldi — Fri, 13 Jan 2017 18:57:15 GMT

I would recommend doing a field extraction using regex to capture the i,j,k,o,p and q into a field named code.

Then in your search use the case command to transform the letter code to the word.

| eval Status_Code=case(code==i,"Upload",code==j,"Errored",code==k,"Aborted" etc..... )

Re: Should we perform a field extraction or field transformation on this sample log?

somesoni2 — Fri, 13 Jan 2017 19:29:36 GMT

Is the "i" (or any other possible value) already being extracted as field? Do you want to modify raw data (non recommended due to overhead on indexing) or just need a field with translated status.

Re: Should we perform a field extraction or field transformation on this sample log?

niketn — Tue, 29 Sep 2020 12:22:56 GMT

You can create a lookup table for above status abbreviation and status description values table as status_description_csv lookup definition via csv file.

status_abbr,status_description
i,Upload
j,Errored
k,Aborted
o,Successful
p,Errored
q,Aborted

You can then use interactive Field Extractor to Extract new fields from the Splunk Search screen (you can either use Regular Expressions or use space as a delimiter)

For example following is the regular expression for status_abbr field: ^(?:[^.\n]*.){4}\w+\s+\w+\s+\w+\s+(?P\w+).

 <Your Base search> 
| lookup status_description_csv status_abbr output status_description
| table _time status_abbr status_description _raw
| stats count by status_description

Re: Should we perform a field extraction or field transformation on this sample log?

splunker9999 — Sun, 15 Jan 2017 03:49:28 GMT

NO , we didn't extracted anything,

Yes we would prefer translated status on field would be great.

Thanks

Re: Should we perform a field extraction or field transformation on this sample log?

splunker9999 — Mon, 16 Jan 2017 01:09:48 GMT

Hi , above regular expression will only work if we 4 ".", but for some of the events we have more "."
Below is one of the example, cna you please provide me the regex:

  sat dec 13 10:01:17 2017 0 10.01.42.1 42288 /home/cat/wat/g1/T24_94291_20170113_093008_20161213110014510.txt.gdp b s **i** r user 0 *

  sat dec 13 10:01:17 2017 0 10.01.42.1 42288 /home/cat/wat.1/g1/T24_94291_20170113_093008_20161213110014510.txt. b s **u** r user 0 *

  sat dec 13 10:01:17 2017 0 10.01.42.1 42288 /home/cat/wat/g1.abc.xls/T24_94291_20170113_093008_20161213110014510.cat b s **o** r user 0 *

Re: Should we perform a field extraction or field transformation on this sample log?

somesoni2 — Mon, 16 Jan 2017 01:41:15 GMT

Try with this regex

^(\S+\s+){11}(?<status_abbr>\w)

Re: Should we perform a field extraction or field transformation on this sample log?

splunker9999 — Mon, 16 Jan 2017 15:52:01 GMT

Hi Somesh,

Just found another isssue, this is picking when we have only 11 spaces , but for few other events it is different.

here below event has more than 11 spaces, is there a way we can include if it 11 or more spaces some thing using rex.

Mon Feb 14 09:35:26 2017 236 100.81.24.1 8189 /cat2/main/y2/jila/itfs/589 y2 jila FS.zip b s o r jila ssh 0 *

Re: Should we perform a field extraction or field transformation on this sample log?

somesoni2 — Mon, 16 Jan 2017 16:02:23 GMT

Since the nearby words are also single aphabets, it makes it difficult. Also, for other events, the status was 12th from start and 5th from last, but in your new samples it doesn't follow that as well. Assuming the status field can have values from range (I , j, k, o, p, q) and nearby characters don't include any of status values, give this a try

^(\S+\s+){11,12}(?<status_abbr>(I|j|k|o|p|q))\s