Splunk Search

How to make a dynamic string substitution to insert specific parameters into specific place in string

mmacalik
Explorer

Dear Splunk community

I need help with a presumably easy task, but it had already cost me quite a while.

I'm trying to make a dynamic string substitution to insert specific parameters into specific place in string.

in example:

 

 

| makeresults 
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]" 
| eval param="param1:param2:param3"

 

 

Where %1 is the respective position in param string (colon separated)

The resulting string would be (note that it is not in param index order):

"blablabla [param2] blablabla [param1] blablabla [param3]"

The number of parameters and indexes in message varies (usually from 1 to 4, but can also be none).

 

I've tried to split it into mv fields and make some multi value indexed substitution,  and then use a foreach statement or mvjoin but frankly i failed.

I've also considered some hard regex work but i'm not even sure if its possible to work

Please note that I am limited to Splunk Enterprise 6.5.10

 

Regards 

MM

 

Labels (2)
Tags (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| makeresults
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]"
| eval param="paramA:paramB:paramC"
| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
| stats values(message) as message values(param) as param by arg
| eval arg="arg".arg
| xyseries message arg param
| foreach arg*
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    | eval message=replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>)]

View solution in original post

mmacalik
Explorer

Additional details:

The data I'm analysing consists (among other) production process parameter's and human readable message template to populate with those parameters.

Basically its thousands of message - param pairs which needs to be combined into human readable format.

More insight example:

 

| makeresults 
| eval message="Product [%2] is on hold in process [%1] hold description: [%3]" 
| eval param="PID1000:ABCD123:visual defect"

 

Which would result in human readable format:

Product [ABCD123] is on hold in process [PID1000] hold description: [visual defect]

another example:

 

| makeresults 
| eval message="Material [%1] already assigned to [%2]" 
| eval param="MAT1234:ABCD123"

 

Should result in :
Material [MAT1234] already assigned to [ABCD123]

Although making  spearate  eval would work for specific example, the number of parameters can differ as well their order in message (thus numerated indexes ).

For example if i would do it in any typical language i would split the param string into a list (or array), and then inserted indexed values into proper positions in message string.

Is something like that possible?

 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| makeresults
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]"
| eval param="paramA:paramB:paramC"
| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
| stats values(message) as message values(param) as param by arg
| eval arg="arg".arg
| xyseries message arg param
| foreach arg*
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    | eval message=replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>)]

mmacalik
Explorer

I modified it a little so it can work with multiple rows.

In production environment i have much more fields that can distinguish a row but for the sake of example i added counter and param as |stats by argument.

 

 

|makeresults count=4
|streamstats count
|eval message=case(count=1, "blablabla [%2] blablabla [%1] blablabla [%3]", count=2, "blablabla [%2] blablabla [%1] blablabla", count=3, "blablabla [%2] blablabla [%1] blablabla [%3]", count=4, "blablabla [%1]")
|eval param=case(count=1, "paramA:paramB:paramC", count=2, "paramD:paramE", count=3, "paramF:paramG:paramH", count=4, "paramI")
|rename count as counter
|fields message, param, counter

| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
| stats values(message) as message values(param) as params by counter, param arg 
| eval arg="arg".arg
| eval message=counter.":".message
| xyseries message arg param
| foreach arg*
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    | eval message=if(isnull(<<FIELD>>), message, replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>))]
| eval message=mvindex(split(message,":"),1)

 

 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Yes, streamstats count is the way I would have done it too 😀

0 Karma

mmacalik
Explorer

Thanks,  @ITWhisperer   that's exactly what i was looking for.

Now to analyze what is actually going on... 😉

@Software-Simian 

The lookup aproach seems also good, but not feasable when there are more than 4000 message templates in 4 languages (different syntax thus different param order).

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
``` Comments don't work this way until later versions of splunk, but using this format for convenience ```
| makeresults
| eval message="blablabla [%2] blablabla [%1] blablabla [%3]"
| eval param="paramA:paramB:paramC"
``` Extract all the argument placeholder numbers ```
| rex field=message max_match=0 "\[\%(?<arg>\d+)\]"
``` Separate them into different events - you might need to add some extra commands here to maintain which arguments came from which event. Let me know if you are unsure how to do this ```
| stats values(message) as message values(param) as param by arg
``` Prefix the placeholder numbers with a known string ```
| eval arg="arg".arg
``` Convert the placeholder rows to columns ```
| xyseries message arg param
``` For each placeholder column (field) - this is why the known prefix is important ```
| foreach arg*
    ``` Use MATCHSEG1 to get the placeholder number from the field name ```
    ``` Use the placeholder number to get the corresponding parameter value ```
    [| eval <<FIELD>>=mvindex(split(<<FIELD>>,":"),<<MATCHSEG1>>-1)
    ``` Replace the placeholder in the message with the parameter value ```
    | eval message=replace(message,"\[\%<<MATCHSEG1>>\]",<<FIELD>>)]
``` Robert is your father's brother! ```
0 Karma

Software-Simian
Path Finder

 

For example if i would do it in any typical language i would split the param string into a list (or array), and then inserted indexed values into proper positions in message string.

So you have a basic set of messages with placeholders...okay where is the problem?

You define a lookup with those message and assign an ID...depending on the ID a specific message template is loaded. Now the only thing missing are the values...well those are being retrieved with the same ID and then inserted into the template as demonstrated.

Depending on the amount this can be done with CASE or a separate lookup table

| makeresults

| eval ID = 2
| eval var1 = Case(ID=="1", "11", 1==1, "21")
| eval var2 = Case(ID=="1", "12", 1==1, "22")
| eval var3 = Case(ID=="1", "13", 1==1, "23")
| eval message = Case(ID=="1", "Product [" . $var1$ . "] is on hold in process [" . $var2$ . "] hold description: [" . $var3$ . "]",
1==1, "Material [" . $var1$ . "] already assigned to [" . $var2$ . "]")

| table message

Please change ID value and append Case options OR create Lookups

isoutamo
SplunkTrust
SplunkTrust

If/when @Software-Simian 's solutions didn't help you, you could try to splunk's printf conversion function. But then you have issues about orders of parameters and also how to create arguments as a list of values.

So probably the easiest and most robust solution is to create your own splunk external command which take those message and params as an arguments and then return a string which contains what you are needing. I suppose that there are already some examples on splunkbase, which you can use as a starting point.

r. Ismo

@ITWhisperer seems to solve this already 😉 Thanks @ITWhisperer I learn something new!

0 Karma

isoutamo
SplunkTrust
SplunkTrust
This seems to be a solution to some issue, but can you describe what was your original challenge which you are trying to solve? Maybe there are some other way to do it?
0 Karma

Software-Simian
Path Finder

Hi,

 

i am not entirely sure i understood your issue...

is that what you need?

| makeresults
| eval var1 = "11"
| eval var2 = "12"
| eval var3 = "13"
| eval message="blablabla [" . $var1$ . "] blablabla [" . $var2$ . "] blablabla [" . $var3$ . "]"
| table message

 

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...