Dynamically generated field names for table or other operation

[fatsug]

I've been trying to solve this every which way and another and I always come up just short of the target.

When searching linux audit log, the type=EXECVE has the most detailed information regarding commands executed. However, if you are interrested in anything other than the command/binary (a0), there will be field unspecific wildcard searches.

Depending on the command and number of options there is a dynamic number of "aX"s where the total number equals another field value (argc) minus 1.

For an event, argc=3 means that there are fields a0, a1, and a2 (three "arguments")

What I wanted was a way to run a base search which returns a large number of events with varying number of "a" fields (a0 ... a(argc-1)) and preferably place these in a table with the correct observed maximum number of argument (aX) fields as columns dynamically. In other words NOT THIS:

index="linux" source="/var/log/audit/audit.log" type="EXECVE"
| table a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 ......

This works, though I really don't like having to hardcode exessive values like this. What I would prefer is a way to, based on the fields observed in the base search, generate matching number (maximum) of columns (aX values) in a table.

a0	a1	a2	a3	a4	a5	a6
value1	value2	value3	value4	value5	value6	value7

Maybe even merging the values of fields a0 to a(argc-1) into a single new field with the entire command.

Command

value1 value2 value3 value4 ... value7

I got started like this:

index="linux" source="/var/log/audit/audit.log" type="EXECVE"
| eval argc_numeric = tonumber(argc)
| eval args = mvrange(0, argc_numeric)
| mvexpand args 
| dedup args
| eval arg = "a" + tostring(args)

Which actually produces a field with the correct number of aX values (field names), though it feels like I'm taking a long way to produce something which is already present (field names) in the base search. And I have no idea how to use the "arg" values as input for table (or stats or whatever). I was thinking of something along the lines of

for i in arg:
  print(arg)  

as input for table. Though this may be a horrible way to approach this problem.

So, hopefully these chaotic notes is enough to kind of explain what I am trying to achieve and have not idea at all how to approach in a good and effective way.

All suggestions are welcome

[ITWhisperer]

If I understand correctly, your events have a field called argc and a number of fields (a0 to aX-1) where X is the maximum number of arguments in your events, and what you are trying to do is reconstruct the full command line "a0 a1 a2 ...." for each event?

index="linux" source="/var/log/audit/audit.log" type="EXECVE"
``` Tag each of the events with an id so it can be reconstructed later ```
| streamstats count as _event
``` Rename argc so it does not start with "a" ```
| rename argc as Xargc
``` Multiplicate each event depending on number of arguments present in the event ```
| eval Xargs = mvrange(0, argc)
| mvexpand Xargs
``` Copy relevant field to command field ```
| foreach a*
    [| eval command=if("<<MATCHSTR>>"==Xarg,<<FIELD>>,command)]
``` Gather command components together for event ```
| stats list(command) as command by _event
| nomv command

[fatsug]

Yes, pretty much like argc is the count of arguments and aX are a zero based list with argc number of aXes

"Field 'Xargs' does not exist in the data."

But I'll play around with it a bit, thank you

[ITWhisperer · Accepted solution]

I missed the X in the mvrange line - try something like this

index="linux" source="/var/log/audit/audit.log" type="EXECVE"
``` Tag each of the events with an id so it can be reconstructed later ```
| streamstats count as _event
``` Rename argc so it does not start with "a" ```
| rename argc as Xargc
``` Multiplicate each event depending on number of arguments present in the event ```
| eval Xargs = mvrange(0, Xargc)
| mvexpand Xargs
``` Copy relevant field to command field ```
| foreach a*
    [| eval command=if("<<MATCHSTR>>"==Xarg,<<FIELD>>,command)]
``` Gather command components together for event ```
| stats list(command) as command by _event
| nomv command

[fatsug]

One small typo with a missing "s"

``` Copy relevant field to command field ```
| foreach a*
    [| eval command=if("<<MATCHSTR>>"==Xargs,<<FIELD>>,command)]

And then I did get exactly what I asked for, so I think its appropriate that I mark this as the solution.

Now, if I can re-connect the command with time and host I have what I actually wanted but did not ask for 🙂

_time

host

command

Thank you for helping me solve this!

[ITWhisperer]

index="linux" source="/var/log/audit/audit.log" type="EXECVE"
``` Tag each of the events with an id so it can be reconstructed later ```
| streamstats count as _event
``` Rename argc so it does not start with "a" ```
| rename argc as Xargc
``` Multiplicate each event depending on number of arguments present in the event ```
| eval Xargs = mvrange(0, Xargc)
| mvexpand Xargs
``` Copy relevant field to command field ```
| foreach a*
    [| eval command=if("<<MATCHSTR>>"==Xargs,<<FIELD>>,command)]
``` Gather command components together for event ```
| stats list(command) as command by _time host _event
| nomv command

[fatsug]

Yup, I tried sticking it after 'nomv' first, which did not work great 🙂 Though adding to the 'stats' part did the trick!

Thank you again 

[richgalloway]

The table command supports wildcards, which makes it easy to handle use cases like this.

index="linux" source="/var/log/audit/audit.log" type="EXECVE"
| table a*

[fatsug]

Well *face* *palm*...

One problem, the sorting messes up the order of columns. Now it goes a0, a10, a11, a12, a2 etc.

So I figure I'll have to sort this in one way or another if this is going to work