topic Re: How would I exclude events that normally occur after a restart of a server? in Splunk Search

How would I exclude events that normally occur after a restart of a server?

jmaple — Wed, 27 Jul 2016 14:55:41 GMT

On our Linux servers, we see that audit policies are re-applied to the audit service whenever the service is restarted. I'm trying to find a way to only generate events if a restart of the server has not occurred within a given period of time (likely 10-15) minutes. How would I configure my search to search for audit change events that are not within 15 minutes of a server restart?

Search for finding a change in audit settings:

index=os* key="MAC-policy" type=CONFIG_CHANGE

Search for finding server restarts:

index=os* type=SYSTEM_SHUTDOWN

Re: How would I exclude events that normally occur after a restart of a server?

sundareshr — Tue, 29 Sep 2020 10:22:51 GMT

See if this works. What this should do, is group all events that start with Shutdown and have a CONFIG_CHANGE within 15min timespan. All other events will not be grouped. closed_txn=f, returns ungrouped events. You may have to experiment with keepevicted vs keeporphans

index=os | transaction maxspan=16m startswith="SYSTEM_SHUTDOWN" endswith="CONFIG_CHANGE" keepevicted=t | where closed_txn=0

http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Transaction

Re: How would I exclude events that normally occur after a restart of a server?

jmaple — Tue, 29 Sep 2020 10:22:56 GMT

Wouldn't marking all transactions as open mean that the "CONFIG_CHANGE" wasn't taking place? 100% of the time it will take place after a reboot so I need to be able to see this event if the "SYSTEM_SHUTDOWN" doesn't occur.

Re: How would I exclude events that normally occur after a restart of a server?

sundareshr — Tue, 29 Sep 2020 10:22:59 GMT

This doesn't mark all transaction as open. It will mark transactions as open if it DOES NOT meet any of these criteris

1) Find SYSTEM_SHUTDOWN
2) Find CONFIG_CHANGE within 15mins of SYSTEM_SHUTDOWN

Once it groups the transactions, (SYSTEM_SHUTDOWN & CONFIG_CHANGE within 15min span) remaining events will be marked as "open" transactions. The where clause will keep only the open transactions.

To make this cleaner, you could add ("SYSTEM_SHUTDOWN" OR "CONFIG_CHANGE") to your base search to minimize the events returned.

Were you able to try the query? What results did you get?

Re: How would I exclude events that normally occur after a restart of a server?

jmaple — Wed, 27 Jul 2016 17:19:14 GMT

The results showed transactions but the ones I saw didn't start with "SYSTEM_SHUTDOWN" so I flipped the keepevicted value to false and the events that were generated are not showing now but I need to do an effective test by rebooting my test machine then adding an audit rule.

Update: Looks like that did not produce an event when I added an audit rule. Here is what happened:

First I kept the "keepevicted=t". This produced events but the events produced are within the span of time. If the idea was to produce events outside the span of time, this did not work.

Next I changed it to "keepevicted=f". This produced no events so I changed to span down to 1s to force a result and still no result so this did not work.

Next I changed it from "keepevicted=f" to "keeporphans=t". This produced no results so it did not work.

Next I changed it to "keeporphans=f". This produced no results so it did not work.

Re: How would I exclude events that normally occur after a restart of a server?

sundareshr — Tue, 29 Sep 2020 10:23:01 GMT

My understanding is you DO NOT want CONFIG_CHANGE" right after a "SYSTEM_SHUTDOWN". If that's true, you should set keepevicted=t. You want all events that are NOT grouped in a transaction. To verify the findings, do a table _time _raw closed_txn and compare the events where closed_txn=0 and closed_txn=1. The one's with 1 should have "SYSTEM_SHUTDOWN" AND/OR "CONFIG_CHANGE" where as the the other should only have "CONFIG_CHANGE"

Re: How would I exclude events that normally occur after a restart of a server?

jmaple — Tue, 29 Sep 2020 10:23:07 GMT

I want to see the "CONFIG_CHANGE" event ONLY if the "SYSTEM_SHUTDOWN" event hasn't occurred in the last 15m. That would be the only time the "CONFIG_CHANGE" event would be relevant to our security team.

Re: How would I exclude events that normally occur after a restart of a server?

sundareshr — Wed, 27 Jul 2016 19:14:47 GMT

Let's try a different approach

index=os ("SYSTEM_SHUTDOWN" OR "CONFIG_CHANGE") | rex "(?<action>SYSTEM_SHUTDOWN|CONFIG_CHANGE)" |  autogress action as next_action | where action=next_action

Re: How would I exclude events that normally occur after a restart of a server?

jmaple — Tue, 29 Sep 2020 10:23:18 GMT

That does the trick.

So I understand it, are you skipping the first CONFIG_CHANGE event that occurs after a SYSTEM_SHUTDOWN event?

Re: How would I exclude events that normally occur after a restart of a server?

sundareshr — Wed, 27 Jul 2016 19:31:05 GMT

That's Correct

Re: How would I exclude events that normally occur after a restart of a server?

jmaple — Wed, 27 Jul 2016 19:32:47 GMT

That works for me. Thanks.