topic Re: Searching for Multiple Commands In Succession in Splunk Search

Searching for Multiple Commands In Succession

cxr5971 — Wed, 26 Jun 2019 20:10:39 GMT

Hello all,

I am looking at endpoint data and I want to see if I can make a search query to look at certain commands that are run in fairly quick succession, say a minute or two. Essentially, someone running cp, then chown, then using kextcache or something of that nature.

The idea is to gather results of people performing these commands right after another or within a minute or so to see someone trying to install something malicious. I saw another post talking about streamstats but it doesn't seem to work the way I want it. To be honest, I don't have much of a query yet so I would appreciate any input that could describe how to do this.

Thanks

Re: Searching for Multiple Commands In Succession

adonio — Thu, 27 Jun 2019 02:40:38 GMT

how does your data looks like?
can you share some samples?
is it the good old .bash history? do you have auditd?

Re: Searching for Multiple Commands In Succession

cxr5971 — Thu, 27 Jun 2019 12:31:33 GMT

Hello @adonio I can't really share what the data looks like in picture form but I can't describe it the best I can. Each event has one command that a user performed called "CommandLine". Each event also has a bunch of other data, but the important thing here is a UID and other things to identify each user.

I'm looking for something that will look at all the data, and say hey he typed this command, let's say we look for a user typing "cp -R abc.kext /" and then not even a minute later there's another event via the same UID saying "chown -R root:wheel". Then another event with him linking certain things. I want to show this data somehow. I want to clump those events together and say "here's the string of commands that he did".

Also it may be beneficial to search for these commands in the order they occurred.

Is this possible?

Re: Searching for Multiple Commands In Succession

skalliger — Thu, 27 Jun 2019 16:36:38 GMT

Instead of doing a heavy SPL I would suggest you try to take a look at Workflow Actions.
That is what they're here for - doing things in sequence.

Skalli

Re: Searching for Multiple Commands In Succession

cxr5971 — Thu, 27 Jun 2019 18:52:00 GMT

I don't think that this will work for me as I am looking to use this search in different alerting mechanisms so I don't believe that works very well with workflow actions.

Re: Searching for Multiple Commands In Succession

cxr5971 — Thu, 27 Jun 2019 19:42:49 GMT

I've looked into transactions as well. Is this a viable solution? Any sort of pseudo-spl would be great 🙂

Re: Searching for Multiple Commands In Succession

skalliger — Thu, 27 Jun 2019 21:23:31 GMT

Alright, I tried. I'll convert to a comment. I don't have other ideas right now. 🙂

Re: Searching for Multiple Commands In Succession

jnudell_2 — Thu, 27 Jun 2019 22:08:25 GMT

Hi @cxr5971 ,

You could play around with the transaction command to see if it gives you what you're looking for:

index=[your command history index] sourcetype=[your command sourcetype] command IN ("cp", "chown", ... etc)

| transaction user maxspan=2m

This will group all events for the field user together as long as they are in a 2 minute span of time. You could also use startswith to tell transaction only to group events that start with a certain event, command=cp as an example. There is also endswith to put an end to the transaction chain. But you would only use those if you have a specific starting and ending point for the list of commands.
You can reference the transaction docs for more information (https://docs.splunk.com/Documentation/Splunk/7.3.0/SearchReference/Transaction)

This way, if you have a group of commands that get found by transaction in a 2 minute period of time, if the count is greater than 1, you could create an alert.

Re: Searching for Multiple Commands In Succession

cxr5971 — Fri, 28 Jun 2019 12:26:11 GMT

I appreciate the post though! Got me looking into workflow actions more 🙂

Re: Searching for Multiple Commands In Succession

cxr5971 — Fri, 28 Jun 2019 14:00:00 GMT

Transactions seem to be very useful and almost exactly what I need except of course they are very resource intensive and draining on our infrastructure as a whole. Does anyone have some tips on making transaction searches more efficient or an SPL heavy statement that could do something very similar?

Re: Searching for Multiple Commands In Succession

woodcock — Sat, 29 Jun 2019 15:02:29 GMT

You should avoid transaction except for limited ad-hoc cases; it should NEVER be used in a scheduled search. Try something like this (assumes your data has a field named command😞

index=YouShouldAlwaysSpecifyANIndex AND sourcetype=AndSourcetypeToo AND [|inputlookup YourLookupFileWithOneCommandOnEachLineWithTheFIeldNamedTheSameAsYourDataUses.csv]
| streamstats values(command) AS commands dc(command) AS commandCount BY host user _time span=2m
| where commandCount >= 2

Re: Searching for Multiple Commands In Succession

jnudell_2 — Sat, 29 Jun 2019 15:46:58 GMT

Yes, it is resource intensive. And as @woodcock mentions, you should try to avoid using it if you can. It does have its uses and can be considered in limited situations. You can try his suggestion below, using streamstats. Just be aware that streamstats is memory restricted (default is 200MB per search), so if you have a very large set of data to search through, streamstats will stop calculating at some point.