Searching for Multiple Commands In Succession

[cxr5971]

Hello all,

I am looking at endpoint data and I want to see if I can make a search query to look at certain commands that are run in fairly quick succession, say a minute or two. Essentially, someone running cp, then chown, then using kextcache or something of that nature.

The idea is to gather results of people performing these commands right after another or within a minute or so to see someone trying to install something malicious. I saw another post talking about streamstats but it doesn't seem to work the way I want it. To be honest, I don't have much of a query yet so I would appreciate any input that could describe how to do this.

Thanks

[woodcock · Accepted solution]

You should avoid transaction except for limited ad-hoc cases; it should NEVER be used in a scheduled search. Try something like this (assumes your data has a field named command😞

index=YouShouldAlwaysSpecifyANIndex AND sourcetype=AndSourcetypeToo AND [|inputlookup YourLookupFileWithOneCommandOnEachLineWithTheFIeldNamedTheSameAsYourDataUses.csv]
| streamstats values(command) AS commands dc(command) AS commandCount BY host user _time span=2m
| where commandCount >= 2

[jnudell_2]

Hi @cxr5971 ,

You could play around with the transaction command to see if it gives you what you're looking for:

index=[your command history index] sourcetype=[your command sourcetype] command IN ("cp", "chown", ... etc)

| transaction user maxspan=2m

This will group all events for the field user together as long as they are in a 2 minute span of time. You could also use startswith to tell transaction only to group events that start with a certain event, command=cp as an example. There is also endswith to put an end to the transaction chain. But you would only use those if you have a specific starting and ending point for the list of commands.
You can reference the transaction docs for more information (https://docs.splunk.com/Documentation/Splunk/7.3.0/SearchReference/Transaction)

This way, if you have a group of commands that get found by transaction in a 2 minute period of time, if the count is greater than 1, you could create an alert.

[cxr5971]

Transactions seem to be very useful and almost exactly what I need except of course they are very resource intensive and draining on our infrastructure as a whole. Does anyone have some tips on making transaction searches more efficient or an SPL heavy statement that could do something very similar?

[jnudell_2]

Yes, it is resource intensive. And as @woodcock mentions, you should try to avoid using it if you can. It does have its uses and can be considered in limited situations. You can try his suggestion below, using streamstats. Just be aware that streamstats is memory restricted (default is 200MB per search), so if you have a very large set of data to search through, streamstats will stop calculating at some point.

[skalliger]

Instead of doing a heavy SPL I would suggest you try to take a look at Workflow Actions.
That is what they're here for - doing things in sequence.

Skalli

[cxr5971]

I don't think that this will work for me as I am looking to use this search in different alerting mechanisms so I don't believe that works very well with workflow actions.

[skalliger]

Alright, I tried. I'll convert to a comment. I don't have other ideas right now. 🙂

[cxr5971]

I appreciate the post though! Got me looking into workflow actions more 🙂

[adonio]

how does your data looks like?
can you share some samples?
is it the good old .bash history? do you have auditd?

[cxr5971]

Hello @adonio I can't really share what the data looks like in picture form but I can't describe it the best I can. Each event has one command that a user performed called "CommandLine". Each event also has a bunch of other data, but the important thing here is a UID and other things to identify each user.

I'm looking for something that will look at all the data, and say hey he typed this command, let's say we look for a user typing "cp -R abc.kext /" and then not even a minute later there's another event via the same UID saying "chown -R root:wheel". Then another event with him linking certain things. I want to show this data somehow. I want to clump those events together and say "here's the string of commands that he did".

Also it may be beneficial to search for these commands in the order they occurred.

Is this possible?

[cxr5971]

I've looked into transactions as well. Is this a viable solution? Any sort of pseudo-spl would be great 🙂