Splunk Search

Rescan Splunk raw data in one search

OldManEd
Builder

Looking for some ideas. I have a search that runs fine. I was given the task of modifying it so, under certain conditions, the raw data is scanned a second time before any calculations and the final table is created. I looked at transactions, stats, macros and eval but nothing looks like it will do the job. Here is a rough example of the code:

index=my_test_index 
NOT (   src=SourceServer01
     OR src=`SourceServer02
     OR src=SourceServer03
    )
NOT (   dest=1.1.1.0/23 
     OR dest=2.2.2.0/23 AND (   fieldA=AAAA 
                             OR FieldA=BBBB
                             OR FieldA=CCCC
                            ) 
    )
|table _time, dest, FieldA, src 

The record I have to also dismiss is if, after the second "NOT", I have a record that has a "FieldA" value, (not one of those that was filtered), and passed the "src" and "dest" filters, go back and scan the raw data again to see if another record with the same new "FieldA" value has already been dismissed with a filtered "src" value and happened within a certain amount of time like 10 seconds, (5 before and 5 after). If a record was found then run the record in question through another "dest" filter.

I hope my explanation makes a little sense. At this point I'm just trying to figure out how to approach this problem.

Thanks for any ideas that may be sent this way.

0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

Okay, here's a general rule about Splunk queries... if the problem is phrased "first I do THIS, then I go back and do THAT" then it's phrased wrong. That is going to be a really really really bad idea.

So, try it this way -

Pull ALL the records.

Flag the ones that are going to be dismissed.

THEN use eventstats or streamstats to ROLL that flag over to the additional records that need to be dismissed.

In this case, I would tend to use streamstats because of the fuzzy time logic. subtract 5 seconds from the _time of the records that are going to be dismissed, then resort all the records | sort 0 _time, and streamstats with time_window=10s.

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

Okay, here's a general rule about Splunk queries... if the problem is phrased "first I do THIS, then I go back and do THAT" then it's phrased wrong. That is going to be a really really really bad idea.

So, try it this way -

Pull ALL the records.

Flag the ones that are going to be dismissed.

THEN use eventstats or streamstats to ROLL that flag over to the additional records that need to be dismissed.

In this case, I would tend to use streamstats because of the fuzzy time logic. subtract 5 seconds from the _time of the records that are going to be dismissed, then resort all the records | sort 0 _time, and streamstats with time_window=10s.

DalJeanis
SplunkTrust
SplunkTrust

@oldmaned -

If I read your specs right, then your pseudocode should look something like this.

 index=my_test_index
| eval killme=case(src=SourceServer01 OR src=SourceServer02 OR src=SourceServer03, "killme",
    dest="1.1.1.0/23","killme", 
    dest="2.2.2.0/23 AND (fieldA="AAAA" OR FieldA="BBBB" OR FieldA="CCCC"), "killme"      
    )
| eventstats max(killme) as killme by cs1 cs6
| table _time, dest, FieldA, src 

Obviously, the comparison tests for dest will need to be set up correctly ( like(src,"1.1.1.%") or cidrmatch("1.1.1.0/23",src) or whatever.

0 Karma

jplumsdaine22
Influencer

Its going to depend on what the raw data looks like. Can you possibly post a series of events with the same field A to show us the flow you're talking about?

0 Karma

OldManEd
Builder

They are FireEye NX logs. If you need more I'll get some.

0 Karma

jplumsdaine22
Influencer

Just post a couple of rows and show us the output you need.

0 Karma

OldManEd
Builder

These are FireEye logs. Nothing special. The top one is filtered out but the second one passed through. The issue is, all this came from a load balancer configuration. Both were initiated from the same event. I need to figure out how to filter the second log entry if and only if a ~similar~ event has already been filtered within a few seconds based on the same signature. I can match on the signature (cs1) and command entry (cs6) but that's about it.

Jul 22 02:54:18 <Splunk server> fenotify-1423236.alert: CEF:0|FireEye|MPS|7.9.5.676262|MC|malware-callback|7|rt=Jul 22 2018 08:54:11 UTC src=<srcIP A> cn3Label=cncPort cn3=<port> cn2Label=sid cn2=00000001 shost=<Host server name A> proto=tcp spt=24310 dst=<Dest IP A> cs5Label=cncHost cs5=<IP> dvchost=<Host Name> dvc=<DVC IP> smac=<MAC Address> cn1Label=vlan cn1=24 dpt=<Port> externalId=1423236 cs4Label=link cs4=https://10.165.252.21/event_stream/events_for_bot?ev_id\=1423236&lms_iden\=0CC47A30CBF8 act=notified cs6Label=channel cs6=Gh0st dmac=<MAC Address>  cs1Label=sname cs1=<Signature Name> devicePayloadId=b5b255db-87dc-4ce3-b07c-f90965e80047 

Jul 22 02:54:19 <Splunk server> fenotify-1423235.alert: CEF:0|FireEye|MPS|7.9.5.676262|MC|malware-callback|7|rt=Jul 22 2018 08:54:11 UTC src=<srcIP B> cn3Label=cncPort cn3=<port> cn2Label=sid cn2=00000001 shost=<HHost server name B> proto=tcp spt=41988 dst=<Dest IP B> cs5Label=cncHost cs5=<IP> dvchost=Host Name> dvc=<DVC IP> smac=0<MAC Address> cn1Label=vlan cn1=23 dpt=<Port> externalId=1423235 cs4Label=link cs4=https://10.165.252.21/event_stream/events_for_bot?ev_id\=1423235&lms_iden\=0CC47A30CBF8 act=notified cs6Label=channel cs6=Gh0st dmac=<MAC Address> cs1Label=sname cs1=<Signaure Name>  devicePayloadId=b5d14648-8237-43df-a341-484f03fdace0  
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...