topic Re: How do I move a string cat operation from the search and store it as an extracted field in events? in Splunk Search

How do I move a string cat operation from the search and store it as an extracted field in events?

HattrickNZ — Mon, 19 Dec 2016 02:37:35 GMT

How do I move a string cat operation from the search and store it in an extracted field option that Splunk offers under events? This way the string cat won't appear in the search.

I can use string cat to create my fields the way I want with this format subrackNo "-" SlotNo "-" boardType

... | strcat subrackNo "-" SlotNo "-" boardType fields | timechart max(kpi) by fields

And this give me this format
0-1-SPUb

0-11-SPUb
0-16-GOUa

THis is one of the rows from my data set:

2016-12-18 23:59:59,DeviceName=Device1,subrackNo=2,boardType=GOUc,SlotNo=23,SubsystemNo=1,CPUoccupancy=3,Throughputoccupancy=0

Now what I want to be able to do is do this in the extracted field option that splunk offers, so I don't have to do it in the search as I am doing with the string cat above. So this way my search could be like this with the string cat removed.

... | timechart max(kpi) by fields

EDIT1

this might the path to take using regex - https://regex101.com/r/nPatfn/1

Re: How do I move a string cat operation from the search and store it as an extracted field in events?

acharlieh — Mon, 19 Dec 2016 03:24:31 GMT

Why Splunk has a separate strcat command, is beyond me... using eval you can use the concatenation operator and do the same like so:

... | eval fields = subrackNo . "-" .  SlotNo . "-" . boardType | timechart max(kpi) by fields

Now with an eval expression you can instead define it as a calculated field.

Re: How do I move a string cat operation from the search and store it as an extracted field in events?

HattrickNZ — Mon, 19 Dec 2016 04:10:55 GMT

tks, but I don't want the command to be in the search, if that makes sense. I want it to be a defined field. I think it can be done under event fields - extracted field. Just not sure. I have added a regex to try and demonstrate this in my question.

Re: How do I move a string cat operation from the search and store it as an extracted field in events?

niketn — Mon, 19 Dec 2016 05:00:09 GMT

Splunk Extracted Fields work on existing data they do not offer expression evaluations. If you want to move your above string concatenation to Knowledge Objects so that you don't have to write the extraction every time during search you should consider moving this to Settings --> Knowledge --> Calculated fields, where you can write any valid evaluation expression.

You would need to provide other details like your Destination Splunk app name, Sourcetype to apply eval expression, name of eval field and the eval expression. In your case

Name= fields
Eval expression= subrackNo."-".SlotNo."-".boardType

Once you do the above fields field should be available under Interesting Fields, when the search is run in Smart or Verbose mode.

PS: You should also make sure that you push only those eval expressions to Calculated Fields which are universal to your sourcetype, like your data contains bytes field and all your report/dashboards show kilobyte, then you would want to move the eval expression to Calculated fields i.e. eval kilobyte=round(bytes/1024,0).

In your case, if not all your dashboards use above string concatenation there is no point evaluating the same upfront through Calculated fields, when they are not going to be used in all of your searches. In other words, through Calculated fields, you pipe an additional eval expression to your existing base search. If left unused this will be an overhead in the SPL. So please think it over before creating a Calculated fields knowledge object.

Also another thing to consider is the precedence of Calculated Fields

1) search-time field extraction
2) field aliasing
3) Calculated Fields
4) derivation of lookup fields.

So, what it implies is that, Calculated fields can not be used on your lookup table fields. Refer to Splunk documentation on Calculated Fields: http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/definecalcfields

Re: How do I move a string cat operation from the search and store it as an extracted field in events?

acharlieh — Mon, 19 Dec 2016 05:03:11 GMT

That's what I'm saying, you want to define a calculated field, and step one of getting there is determining the eval expression to use. With this you define the calculated field, and then drop the eval expression from your searches, letting Splunk apply it for your sourcetype.

For search time field extractions, you are not able to concatenate together disparate segments to make a single field ( transforms.conf.spec says as much under the FORMAT attribute ).

If you do not want to use a calculated field, as they are all right next to eachother in your original data, you could extract the entire segment as a single field, but subrackNo=2,boardType=GOUc,SlotNo=23is a lot more verbose than your desired format of 2-23-GOUc. A second option would be to create an index-time extraction for your field, where you could extract all segments and piece them together, but this could cause its own headaches as well.

Re: How do I move a string cat operation from the search and store it as an extracted field in events?

HattrickNZ — Mon, 19 Dec 2016 21:21:03 GMT

tks for me it was settings - fields - calculated fileds. then like you say
Name= fields
Eval expression= subrackNo."-".SlotNo."-".boardType

and had to seletect which source type to apply it to.

This was slightly different than the way I thought I had to do it which was:
Under the events tabs - select an event - event actions - extract fields. But this way way there is a regular expression or delimiter option, and that is probably not needed here as the fields are already there, I just want to cat them together.