Assistance with SPL for Default Argument Values and mvexpand Usage

[oussama1]

I am working with event data in Splunk where each event contains a command with multiple arguments. I'm extracting these arguments and their associated values using regex, resulting in multi-value fields within Splunk. However, I'm encountering a challenge where some arguments do not have an associated value, and for these cases, I would like to set their values to `true`.

Here's the SPL I'm using for extraction:
| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"

What I need is to refine this SPL so that after extraction, any argument without a value is automatically assigned a value of `true`. After setting the default values, I would then like to use `mvexpand` to separate each argument-value pair into its own event.

Could you provide guidance on how to adjust my regex or SPL command to accomplish this within Splunk?

[yuanliu]

Assuming regex is working, and you have successfully extracted all flag's.  fillnull and coalesce does not help because they only take effect when value is completely void.

In olden days, people use mvzip-split string maneuvers to handle these conditions.  But since 8.2, Splunk added a set of JSON functions so you can represent data structure more expressively.  The following is a possibility:

| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"
| eval flag=trim(flag, "-")
| eval flagidx = mvrange(0, mvcount(flag))
| eval compact = mvmap(flagidx, json_object("flag", mvindex(flag, flagidx), "value", mvindex(value, flagidx)))
| fields - flagidx
| mvexpand compact
| eval value = json_extract(compact, "value"), flag = json_extract(compact, "flag")
| eval value = if(isnull(flag), null(), coalesce(value, "true"))

Using the examples you gave, here is an emulation you can play with and compare with real data.

| makeresults format=csv data="Aptlauncher_cmd
launch test -config basic_config.cfg -system test_system1 -retry 3
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4
launch update -email user@example.com -domain test.domain.com -port 8080
launch deploy -verbose -dry_run -force
launch schedule -task \"Deploy task\" -at \"2023-07-21 10:00:00\" -notify \"admin@example.com\"
launch clean -@cleanup -remove_all -v2.5
launch start -config@version2 --custom-env "DEV-TEST" --update-rate@5min
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent
launch execute -file script.sh -next-gen --flag -another-flag value
launch execute process_without_any_flags
launch special -@@ -##value special_value --$$$ 100
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2"
``` data emulation above ```

Emulative results are

Aptlauncher_cmd	flag	value
launch test -config basic_config.cfg -system test_system1 -retry 3	config	basic_config.cfg
launch test -config basic_config.cfg -system test_system1 -retry 3	system	test_system1
launch test -config basic_config.cfg -system test_system1 -retry 3	retry	3
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4	con-fig	advanced-config_v2.cfg
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4	sys_tem	test_system_2
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4	re-try	4
launch update -email user@example.com -domain test.domain.com -port 8080	email	user@example.com
launch update -email user@example.com -domain test.domain.com -port 8080	domain	test.domain.com
launch update -email user@example.com -domain test.domain.com -port 8080	port	8080
launch deploy -verbose -dry_run -force	verbose	true
launch deploy -verbose -dry_run -force	dry_run	true
launch deploy -verbose -dry_run -force	force	true
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"	task	"Deploy
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"	at	"2023-07-21
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"	notify	"admin@example.com"
launch clean -@cleanup -remove_all -v2.5	@cleanup	true
launch clean -@cleanup -remove_all -v2.5	remove_all	true
launch clean -@cleanup -remove_all -v2.5	v2.5	true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5min	config@version2	DEV-TEST
launch start -config@version2 --custom-env DEV-TEST --update-rate@5min	custom-env	true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5min	update-rate@5min	true
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	env	DEV
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	build-version	1.0.0
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	@retry-limit	5
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	log-level	debug
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	silent	true
launch execute -file script.sh -next-gen --flag -another-flag value	file	script.sh
launch execute -file script.sh -next-gen --flag -another-flag value	next-gen	value
launch execute -file script.sh -next-gen --flag -another-flag value	flag	true
launch execute -file script.sh -next-gen --flag -another-flag value	another-flag	true
launch execute process_without_any_flags
launch special -@@ -##value special_value --$$$ 100	@@	special_value
launch special -@@ -##value special_value --$$$ 100	##value	100
launch special -@@ -##value special_value --$$$ 100	$$$	true
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	add	5
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	subtract	3
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	multiply@2.5	true
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	divide@2	true

Hope this helps.  (As you can see, the regex has some difficulty with quoted values.)

[oussama1]

 I still have on last small issue:

so what the spl does, if values are missing it adds true to the last ones not the correct ones :

Example :  -config Conf-console -ntpsync no --check_core no
shoul give :
Flag : config Value: Conf
Flag : console Value: true
Flag : ntpsync Value: no
Flag : check_core Value: no

but it extract all the values and then add true to the last values so it gives this which is incorrect:
Flag : config Value: Conf
Flag : console Value: no
Flag : ntpsync Value: no
Flag : check_core Value: true

so I think if you can show me a way to move those without values to the last so they will be matched with the true values, it would be really helpful

[yuanliu]

Ignore the deleted answer.  When there is a missed space between "Conf" and "-console", I couldn't see the problem you were trying to fix. (It's really important to illustrate data accurately when asking data analytics questions.)

So, the regex doesn't handle inputs like "-config Conf -console -ntpsync no --check_core no".  In fact, the regex is already too heavy handed.  Instead of adding more regex tax, the following method semantically expresses command line syntax, and does not require composition and decomposition. (Command line allows multiple spaces and such.  But nothing another trim couldn't fix.)

| eval flag = mvindex(split(Aptlauncher_cmd, " -"), 1, -1)
| eval flag = trim(flag, "-")
| mvexpand flag
| eval flag = split(flag, " ")
| eval value = mvindex(flag, 1), flag = mvindex(flag, 0)
| eval value = if(isnull(flag), null(), coalesce(value, "true"))

The samples (including "-config Conf -console -ntpsync no --check_core no") will give

Aptlauncher_cmd	flag	value
launch test -config Conf -console -ntpsync no --check_core no	config	Conf
launch test -config Conf -console -ntpsync no --check_core no	console	true
launch test -config Conf -console -ntpsync no --check_core no	ntpsync	no
launch test -config Conf -console -ntpsync no --check_core no	check_core	no
launch test -config basic_config.cfg -system test_system1 -retry 3	config	basic_config.cfg
launch test -config basic_config.cfg -system test_system1 -retry 3	system	test_system1
launch test -config basic_config.cfg -system test_system1 -retry 3	retry	3
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4	con-fig	advanced-config_v2.cfg
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4	sys_tem	test_system_2
launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4	re-try	4
launch update -email user@example.com -domain test.domain.com -port 8080	email	user@example.com
launch update -email user@example.com -domain test.domain.com -port 8080	domain	test.domain.com
launch update -email user@example.com -domain test.domain.com -port 8080	port	8080
launch deploy -verbose -dry_run -force	verbose	true
launch deploy -verbose -dry_run -force	dry_run	true
launch deploy -verbose -dry_run -force	force	true
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"	task	"Deploy
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"	at	"2023-07-21
launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"	notify	"admin@example.com"
launch clean -@cleanup -remove_all -v2.5	@cleanup	true
launch clean -@cleanup -remove_all -v2.5	remove_all	true
launch clean -@cleanup -remove_all -v2.5	v2.5	true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5min	config@version2	true
launch start -config@version2 --custom-env DEV-TEST --update-rate@5min	custom-env	DEV-TEST
launch start -config@version2 --custom-env DEV-TEST --update-rate@5min	update-rate@5min	true
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	env	DEV
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	build-version	1.0.0
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	@retry-limit	5
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	log-level	debug
launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent	silent	true
launch execute -file script.sh -next-gen --flag -another-flag value	file	script.sh
launch execute -file script.sh -next-gen --flag -another-flag value	next-gen	true
launch execute -file script.sh -next-gen --flag -another-flag value	flag	true
launch execute -file script.sh -next-gen --flag -another-flag value	another-flag	value
launch execute process_without_any_flags
launch special -@@ -##value special_value --$$$ 100	@@	true
launch special -@@ -##value special_value --$$$ 100	##value	special_value
launch special -@@ -##value special_value --$$$ 100	$$$	100
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	add	5
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	subtract	3
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	multiply@2.5	true
launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2	divide@2	true

[oussama1]

Thank you, that was really helpful but I still have on last small issue:

so what the spl does, if values are missing it adds true to the last ones not the correct ones :

Example :  -config Conf-console -ntpsync no --check_core no
shoul give :
Flag : config Value: Conf
Flag : console Value: true
Flag : ntpsync Value: no
Flag : check_core Value: no

but it extract all the values and then add true to the last values so it gives this which is incorrect:
Flag : config Value: Conf
Flag : console Value: no
Flag : ntpsync Value: no
Flag : check_core Value: true

so I think if you can show me a way to move those without values to the last so they will be matched with the true values, it would be really helpful

[scelikok]

Hi @oussama1 ,

I think your problem is because of regex output. If some of your flags have no associated values from regex output, it is not possible to match flag and values. Maybe you should change your regex to create an output that can be parsed by SPL. If you have an anonymous sample events , we can try to help.

[oussama1]

Flags are specifically those strings that start with - or -- and are preceded by a space (Ex : -config basic_config.cfg or --config basic_config.cfg). We also need to handle cases where a flag is followed directly by its value, and cases where the flag stands alone indicating a boolean value of true (Ex : aptlaunch clean -@cleanup -remove_all -v2.5). And values are the strings separated from the Flags with a space (Ex : -config basic_config.cfg) or separated with the flags by an = (Ex : -config=basic_config.cfg):

So basically my regex should pass this test cases :

1. Basic flags with alphanumeric values:
cmd: launch test -config basic_config.cfg -system test_system1 -retry 3

2. Flags with dashes and underscores in names and values:
cmd: launch test -con-fig advanced-config_v2.cfg -sys_tem test_system_2 -re-try 4

3. Flags with special characters (@, ., :):
cmd: launch update -email user@example.com -domain test.domain.com -port 8080

4. Standalone flags (boolean flags):
cmd: launch deploy -verbose -dry_run -force

5. Flags with values including spaces (should be wrapped in quotes):
cmd: launch schedule -task "Deploy task" -at "2023-07-21 10:00:00" -notify "admin@example.com"

6. Mixed alphanumeric and special character flags without values:
cmd: launch clean -@cleanup -remove_all -v2.5

7. Complex flags with mixed special characters:
cmd: launch start -config@version2 --custom-env "DEV-TEST" --update-rate@5min

8. Multiple flags with and without values, mixed special characters:
cmd: launch run -env DEV --build-version 1.0.0 -@retry-limit 5 --log-level debug -silent

9. Flags that could be misinterpreted as values:
cmd: launch execute -file script.sh -next-gen --flag -another-flag value

10. Command with no flags at all (to ensure it's skipped properly):
cmd: launch execute process_without_any_flags

11. Command with flags having only special characters:
cmd: launch special -@@ -##value special_value --$$$ 100

12. Flags with numeric values and mixed special characters:
cmd: launch calculate -add 5 -subtract 3 --multiply@2.5 --divide@2

[oussama1]

its still not working so basically this is the spl i am using :

| eval Aptlauncher_cmd = replace(Aptlauncher_cmd,"=", " =")
| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"
| fillnull value="true" flag value
| eval flag=trim(flag, "-")
| eval value=coalesce(value, "true")
| where isnotnull(flag) AND flag!=""
| table flag, value

but I am still having the same issue true is not added since its a multi value field I guess and mvexpand doesnt separate them as pairs when I use it 

[scelikok]

Hi @oussama1,

You can add fillnull command  after your rex command;

| rex max_match=0 field=Aptlauncher_cmd "\s(?<flag>--?[\w\-.@|$|#]+)(?:(?=\s--?)|(?=\s[\w\-.\/|$|#|\"|=])\s(?<value>[^\s]+))?"
| fillnull value="true" flag value