How to make a dynamic string substitution to insert specific parameters into specific place in string

[mmacalik]

Dear Splunk community

I need help with a presumably easy task, but it had already cost me quite a while.

I'm trying to make a dynamic string substitution to insert specific parameters into specific place in string.

in example:

| makeresults 
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]" 
| eval param="param1:param2:param3"

Where %1 is the respective position in param string (colon separated)

The resulting string would be (note that it is not in param index order):

"blablabla [param2] blablabla [param1] blablabla [param3]"

The number of parameters and indexes in message varies (usually from 1 to 4, but can also be none).

I've tried to split it into mv fields and make some multi value indexed substitution,  and then use a foreach statement or mvjoin but frankly i failed.

I've also considered some hard regex work but i'm not even sure if its possible to work

Please note that I am limited to Splunk Enterprise 6.5.10

Regards 

MM

[mmacalik]

Additional details:

The data I'm analysing consists (among other) production process parameter's and human readable message template to populate with those parameters.

Basically its thousands of message - param pairs which needs to be combined into human readable format.

More insight example:

| makeresults 
| eval message="Product [%2] is on hold in process [%1] hold description: [%3]" 
| eval param="PID1000:ABCD123:visual defect"

Which would result in human readable format:

Product [ABCD123] is on hold in process [PID1000] hold description: [visual defect]

another example:

| makeresults 
| eval message="Material [%1] already assigned to [%2]" 
| eval param="MAT1234:ABCD123"

Should result in :
Material [MAT1234] already assigned to [ABCD123]

Although making  spearate  eval would work for specific example, the number of parameters can differ as well their order in message (thus numerated indexes ).

For example if i would do it in any typical language i would split the param string into a list (or array), and then inserted indexed values into proper positions in message string.

Is something like that possible?

[ITWhisperer · Accepted solution]

| makeresults
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]"
| eval param="paramA:paramB:paramC"
| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
| stats values(message) as message values(param) as param by arg
| eval arg="arg".arg
| xyseries message arg param
| foreach arg*
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    | eval message=replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>)]

[mmacalik]

I modified it a little so it can work with multiple rows.

In production environment i have much more fields that can distinguish a row but for the sake of example i added counter and param as |stats by argument.

|makeresults count=4
|streamstats count
|eval message=case(count=1, "blablabla [%2] blablabla [%1] blablabla [%3]", count=2, "blablabla [%2] blablabla [%1] blablabla", count=3, "blablabla [%2] blablabla [%1] blablabla [%3]", count=4, "blablabla [%1]")
|eval param=case(count=1, "paramA:paramB:paramC", count=2, "paramD:paramE", count=3, "paramF:paramG:paramH", count=4, "paramI")
|rename count as counter
|fields message, param, counter

| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
| stats values(message) as message values(param) as params by counter, param arg 
| eval arg="arg".arg
| eval message=counter.":".message
| xyseries message arg param
| foreach arg*
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    | eval message=if(isnull(<<FIELD>>), message, replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>))]
| eval message=mvindex(split(message,":"),1)

[ITWhisperer]

Yes, streamstats count is the way I would have done it too 😀

[mmacalik]

Thanks,  @ITWhisperer   that's exactly what i was looking for.

Now to analyze what is actually going on... 😉

@Software-Simian 

The lookup aproach seems also good, but not feasable when there are more than 4000 message templates in 4 languages (different syntax thus different param order).

[ITWhisperer]

``` Comments don't work this way until later versions of splunk, but using this format for convenience ```
| makeresults
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]"
| eval param="paramA:paramB:paramC"
``` Extract all the argument placeholder numbers ```
| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
``` Separate them into different events - you might need to add some extra commands here to maintain which arguments came from which event. Let me know if you are unsure how to do this ```
| stats values(message) as message values(param) as param by arg
``` Prefix the placeholder numbers with a known string ```
| eval arg="arg".arg
``` Convert the placeholder rows to columns ```
| xyseries message arg param
``` For each placeholder column (field) - this is why the known prefix is important ```
| foreach arg*
    ``` Use MATCHSEG1 to get the placeholder number from the field name ```
    ``` Use the placeholder number to get the corresponding parameter value ```
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    ``` Replace the placeholder in the message with the parameter value ```
    | eval message=replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>)]
``` Robert is your father's brother! ```

[Software-Simian]

For example if i would do it in any typical language i would split the param string into a list (or array), and then inserted indexed values into proper positions in message string.

So you have a basic set of messages with placeholders...okay where is the problem?

You define a lookup with those message and assign an ID...depending on the ID a specific message template is loaded. Now the only thing missing are the values...well those are being retrieved with the same ID and then inserted into the template as demonstrated.

Depending on the amount this can be done with CASE or a separate lookup table

| makeresults

| eval ID = 2
| eval var1 = Case(ID=="1", "11", 1==1, "21")
| eval var2 = Case(ID=="1", "12", 1==1, "22")
| eval var3 = Case(ID=="1", "13", 1==1, "23")
| eval message = Case(ID=="1", "Product [" . $var1$ . "] is on hold in process [" . $var2$ . "] hold description: [" . $var3$ . "]",
1==1, "Material [" . $var1$ . "] already assigned to [" . $var2$ . "]")

| table message

Please change ID value and append Case options OR create Lookups

[isoutamo]

If/when @Software-Simian 's solutions didn't help you, you could try to splunk's printf conversion function. But then you have issues about orders of parameters and also how to create arguments as a list of values.

So probably the easiest and most robust solution is to create your own splunk external command which take those message and params as an arguments and then return a string which contains what you are needing. I suppose that there are already some examples on splunkbase, which you can use as a starting point.

r. Ismo

@ITWhisperer seems to solve this already 😉 Thanks @ITWhisperer I learn something new!

[isoutamo]

This seems to be a solution to some issue, but can you describe what was your original challenge which you are trying to solve? Maybe there are some other way to do it?

[Software-Simian]

Hi,

i am not entirely sure i understood your issue...

is that what you need?

| makeresults
| eval var1 = "11"
| eval var2 = "12"
| eval var3 = "13"
| eval message="blablabla [" . $var1$ . "] blablabla [" . $var2$ . "] blablabla [" . $var3$ . "]"
| table message