Splunk Search

Expire Rows in a Lookup After 30 Days

dmbr
Explorer

Hi Splunkheads, 

Need some advice here. I have built a simple lookup table and simple search for known bad ip addresses. My search runs across the lookup table, and returns a table for any matches across the environment. 

Here is my search:

 

| tstats summariesonly=t fillnull_value="MISSING" count from datamodel=Network_Traffic.All_Traffic by All_Traffic.src, All_Traffic.dest, All_Traffic.dest_port, _time, All_Traffic.action, All_Traffic.bytes, index, sourcetype
| lookup ioc_ip.csv ioc_ip as All_Traffic.src OUTPUT ioc_ip as src_found
| lookup ioc_ip.csv ioc_ip as All_Traffic.dest OUTPUT ioc_ip as dest_found
| where !isnull(src_found) OR !isnull(dest_found)
| fields - src_found, dest_found
| sort -_time

 

I have been asked to auto-expire rows in the lookup after 30 days. The logic would be something like:

If date ioc_email_date older than 30 days:
Delete row
else:
Run search
 
I have added dates to my lookup table. Here is a dummy example of my lookup table:
Splunk Screenshot 2021-05-21 134637.png
My questions:
1. Best format for the ioc_ip_date column? Would it be best to use Epoch time? Currently using this "2021-18-05" as per above. Happy to convert to any format.
2. Any suggestions on how to add the logic to the above search sample?
 
 
Labels (2)
Tags (4)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Since you are going to be doing some sort of calculation and comparison on the date, epoch time would be best, although you could store both just to make it easier to check.

| inputlookup ...
| where now - date < 30*60*60*24
| outputlookup ...

The lookup file will only get smaller with this process, so you need to add additional events to this process or have an additional process to keep the file current.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...