Splunk Search

Assistance with SPL for Default Argument Values and mvexpand Usage

oussama1
Loves-to-Learn Lots

I am working with event data in Splunk where each event contains a command with multiple arguments. I'm extracting these arguments and their associated values using regex, resulting in multi-value fields within Splunk. However, I'm encountering a challenge where some arguments do not have an associated value, and for these cases, I would like to set their values to `true`.

Here's the SPL I'm using for extraction:
| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"

What I need is to refine this SPL so that after extraction, any argument without a value is automatically assigned a value of `true`. After setting the default values, I would then like to use `mvexpand` to separate each argument-value pair into its own event.

Could you provide guidance on how to adjust my regex or SPL command to accomplish this within Splunk?

Labels (1)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

Assuming regex is working, and you have successfully extracted all flag's.  fillnull and coalesce does not help because they only take effect when value is completely void.

In olden days, people use mvzip-split string maneuvers to handle these conditions.  But since 8.2, Splunk added a set of JSON functions so you can represent data structure more expressively.  The following is a possibility:

 

| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"
| eval flag=trim(flag, "-")
| eval flagidx = mvrange(0, mvcount(flag))
| eval compact = mvmap(flagidx, json_object("flag", mvindex(flag, flagidx), "value", mvindex(value, flagidx)))
| fields - flagidx
| mvexpand compact
| eval value = json_extract(compact, "value"), flag = json_extract(compact, "flag")
| eval value = if(isnull(flag), null(), coalesce(value, "true"))

 

Using the examples you gave, here is an emulation you can play with and compare with real data.

 

| makeresults format=csv data="Aptlauncher_cmd
launch test -config basic_config.cfg -system test_system1 -retry 3
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4
launch update -email user@example.com -domain test.domain.com -port 8080
launch deploy -verbose -dry_run -force
launch schedule -task \"Deploy task\" -at \"2023-07-21 10:00:00\" -notify \"admin@example.com\"
launch clean -@cleanup -remove_all -v2.5
launch start -config@version2 --custom-env "DEV-TEST" --update-rate@5min
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent
launch execute -file script.sh -next-gen --flag -another-flag value
launch execute process_without_any_flags
launch special -@@ -##value special_value --$$$ 100
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2"
``` data emulation above ```

 

Emulative results are

Aptlauncher_cmdflagvalue
launch test -config basic_config.cfg -system test_system1 -retry 3configbasic_config.cfg
launch test -config basic_config.cfg -system test_system1 -retry 3systemtest_system1
launch test -config basic_config.cfg -system test_system1 -retry 3retry3
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4con-figadvanced-config_v2.cfg
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4sys_temtest_system_2
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4re-try4
launch update -email user@example.com -domain test.domain.com -port 8080emailuser@example.com
launch update -email user@example.com -domain test.domain.com -port 8080domaintest.domain.com
launch update -email user@example.com -domain test.domain.com -port 8080port8080
launch deploy -verbose -dry_run -forceverbosetrue
launch deploy -verbose -dry_run -forcedry_runtrue
launch deploy -verbose -dry_run -forceforcetrue
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"task"Deploy
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"at"2023-07-21
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"notify"admin@example.com"
launch clean -@cleanup -remove_all -v2.5@cleanuptrue
launch clean -@cleanup -remove_all -v2.5remove_alltrue
launch clean -@cleanup -remove_all -v2.5v2.5true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5minconfig@version2DEV-TEST
launch start -config@version2 --custom-env DEV-TEST --update-rate@5mincustom-envtrue
launch start -config@version2 --custom-env DEV-TEST --update-rate@5minupdate-rate@5mintrue
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentenvDEV
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentbuild-version1.0.0
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent@retry-limit5
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentlog-leveldebug
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentsilenttrue
launch execute -file script.sh -next-gen --flag -another-flag valuefilescript.sh
launch execute -file script.sh -next-gen --flag -another-flag valuenext-genvalue
launch execute -file script.sh -next-gen --flag -another-flag valueflagtrue
launch execute -file script.sh -next-gen --flag -another-flag valueanother-flagtrue
launch execute process_without_any_flags  
launch special -@@ -##value special_value --$$$ 100@@special_value
launch special -@@ -##value special_value --$$$ 100##value100
launch special -@@ -##value special_value --$$$ 100$$$true
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2add5
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2subtract3
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2multiply@2.5true
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2divide@2true

Hope this helps.  (As you can see, the regex has some difficulty with quoted values.)

Tags (1)
0 Karma

oussama1
Loves-to-Learn Lots

 I still have on last small issue:

so what the spl does, if values are missing it adds true to the last ones not the correct ones :

Example :  -config Conf-console -ntpsync no --check_core no
shoul give :
Flag : config Value: Conf
Flag : console Value: true
Flag : ntpsync Value: no
Flag : check_core Value: no

but it extract all the values and then add true to the last values so it gives this which is incorrect:
Flag : config Value: Conf
Flag : console Value: no
Flag : ntpsync Value: no
Flag : check_core Value: true

so I think if you can show me a way to move those without values to the last so they will be matched with the true values, it would be really helpful

0 Karma

yuanliu
SplunkTrust
SplunkTrust

Ignore the deleted answer.  When there is a missed space between "Conf" and "-console", I couldn't see the problem you were trying to fix. (It's really important to illustrate data accurately when asking data analytics questions.)

So, the regex doesn't handle inputs like "-config Conf -console -ntpsync no --check_core no".  In fact, the regex is already too heavy handed.  Instead of adding more regex tax, the following method semantically expresses command line syntax, and does not require composition and decomposition. (Command line allows multiple spaces and such.  But nothing another trim couldn't fix.)

 

| eval flag = mvindex(split(Aptlauncher_cmd, " -"), 1, -1)
| eval flag = trim(flag, "-")
| mvexpand flag
| eval flag = split(flag, " ")
| eval value = mvindex(flag, 1), flag = mvindex(flag, 0)
| eval value = if(isnull(flag), null(), coalesce(value, "true"))

 

The samples (including "-config Conf -console -ntpsync no --check_core no") will give

Aptlauncher_cmdflagvalue
launch test -config Conf -console -ntpsync no --check_core noconfigConf
launch test -config Conf -console -ntpsync no --check_core noconsoletrue
launch test -config Conf -console -ntpsync no --check_core nontpsyncno
launch test -config Conf -console -ntpsync no --check_core nocheck_coreno
launch test -config basic_config.cfg -system test_system1 -retry 3configbasic_config.cfg
launch test -config basic_config.cfg -system test_system1 -retry 3systemtest_system1
launch test -config basic_config.cfg -system test_system1 -retry 3retry3
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4con-figadvanced-config_v2.cfg
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4sys_temtest_system_2
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4re-try4
launch update -email user@example.com -domain test.domain.com -port 8080emailuser@example.com
launch update -email user@example.com -domain test.domain.com -port 8080domaintest.domain.com
launch update -email user@example.com -domain test.domain.com -port 8080port8080
launch deploy -verbose -dry_run -forceverbosetrue
launch deploy -verbose -dry_run -forcedry_runtrue
launch deploy -verbose -dry_run -forceforcetrue
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"task"Deploy
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"at"2023-07-21
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"notify"admin@example.com"
launch clean -@cleanup -remove_all -v2.5@cleanuptrue
launch clean -@cleanup -remove_all -v2.5remove_alltrue
launch clean -@cleanup -remove_all -v2.5v2.5true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5minconfig@version2true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5mincustom-envDEV-TEST
launch start -config@version2 --custom-env DEV-TEST --update-rate@5minupdate-rate@5mintrue
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentenvDEV
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentbuild-version1.0.0
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent@retry-limit5
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentlog-leveldebug
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silentsilenttrue
launch execute -file script.sh -next-gen --flag -another-flag valuefilescript.sh
launch execute -file script.sh -next-gen --flag -another-flag valuenext-gentrue
launch execute -file script.sh -next-gen --flag -another-flag valueflagtrue
launch execute -file script.sh -next-gen --flag -another-flag valueanother-flagvalue
launch execute process_without_any_flags  
launch special -@@ -##value special_value --$$$ 100@@true
launch special -@@ -##value special_value --$$$ 100##valuespecial_value
launch special -@@ -##value special_value --$$$ 100$$$100
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2add5
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2subtract3
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2multiply@2.5true
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2divide@2true
Tags (2)
0 Karma

oussama1
Loves-to-Learn Lots

Thank you, that was really helpful but I still have on last small issue:

so what the spl does, if values are missing it adds true to the last ones not the correct ones :

Example :  -config Conf-console -ntpsync no --check_core no
shoul give :
Flag : config Value: Conf
Flag : console Value: true
Flag : ntpsync Value: no
Flag : check_core Value: no

but it extract all the values and then add true to the last values so it gives this which is incorrect:
Flag : config Value: Conf
Flag : console Value: no
Flag : ntpsync Value: no
Flag : check_core Value: true

so I think if you can show me a way to move those without values to the last so they will be matched with the true values, it would be really helpful

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @oussama1 ,

I think your problem is because of regex output. If some of your flags have no associated values from regex output, it is not possible to match flag and values. Maybe you should change your regex to create an output that can be parsed by SPL. If you have an anonymous sample events , we can try to help.

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

oussama1
Loves-to-Learn Lots


Flags are specifically those strings that start with - or -- and are preceded by a space (Ex : -config basic_config.cfg or --config basic_config.cfg). We also need to handle cases where a flag is followed directly by its value, and cases where the flag stands alone indicating a boolean value of true (Ex : aptlaunch clean -@cleanup -remove_all -v2.5). And values are the strings separated from the Flags with a space (Ex : -config basic_config.cfg) or separated with the flags by an = (Ex : -config=basic_config.cfg):

So basically my regex should pass this test cases :

1. Basic flags with alphanumeric values:
cmd: launch test -config basic_config.cfg -system test_system1 -retry 3

2. Flags with dashes and underscores in names and values:
cmd: launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4

3. Flags with special characters (@, ., :):
cmd: launch update -email user@example.com -domain test.domain.com -port 8080

4. Standalone flags (boolean flags):
cmd: launch deploy -verbose -dry_run -force

5. Flags with values including spaces (should be wrapped in quotes):
cmd: launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"

6. Mixed alphanumeric and special character flags without values:
cmd: launch clean -@cleanup -remove_all -v2.5

7. Complex flags with mixed special characters:
cmd: launch start -config@version2 --custom-env "DEV-TEST" --update-rate@5min

8. Multiple flags with and without values, mixed special characters:
cmd: launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent

9. Flags that could be misinterpreted as values:
cmd: launch execute -file script.sh -next-gen --flag -another-flag value

10. Command with no flags at all (to ensure it's skipped properly):
cmd: launch execute process_without_any_flags

11. Command with flags having only special characters:
cmd: launch special -@@ -##value special_value --$$$ 100

12. Flags with numeric values and mixed special characters:
cmd: launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2

 

0 Karma

oussama1
Loves-to-Learn Lots

its still not working so basically this is the spl i am using :

| eval Aptlauncher_cmd = replace(Aptlauncher_cmd,"=", " =")
| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"
| fillnull value="true" flag value
| eval flag=trim(flag, "-")
| eval value=coalesce(value, "true")
| where isnotnull(flag) AND flag!=""
| table flag, value

but I am still having the same issue true is not added since its a multi value field I guess and mvexpand doesnt separate them as pairs when I use it 

problem.png

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @oussama1,

You can add fillnull command  after your rex command;

| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"
| fillnull value="true" flag value

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...