Rescan Splunk raw data in one search

[OldManEd]

Looking for some ideas. I have a search that runs fine. I was given the task of modifying it so, under certain conditions, the raw data is scanned a second time before any calculations and the final table is created. I looked at transactions, stats, macros and eval but nothing looks like it will do the job. Here is a rough example of the code:

index=my_test_index 
NOT (   src=SourceServer01
     OR src=`SourceServer02
     OR src=SourceServer03
    )
NOT (   dest=1.1.1.0/23 
     OR dest=2.2.2.0/23 AND (   fieldA=AAAA 
                             OR FieldA=BBBB
                             OR FieldA=CCCC
                            ) 
    )
|table _time, dest, FieldA, src 

The record I have to also dismiss is if, after the second "NOT", I have a record that has a "FieldA" value, (not one of those that was filtered), and passed the "src" and "dest" filters, go back and scan the raw data again to see if another record with the same new "FieldA" value has already been dismissed with a filtered "src" value and happened within a certain amount of time like 10 seconds, (5 before and 5 after). If a record was found then run the record in question through another "dest" filter.

I hope my explanation makes a little sense. At this point I'm just trying to figure out how to approach this problem.

Thanks for any ideas that may be sent this way.

[DalJeanis · Accepted solution]

Okay, here's a general rule about Splunk queries... if the problem is phrased "first I do THIS, then I go back and do THAT" then it's phrased wrong. That is going to be a really really really bad idea.

So, try it this way -

Pull ALL the records.

Flag the ones that are going to be dismissed.

THEN use eventstats or streamstats to ROLL that flag over to the additional records that need to be dismissed.

In this case, I would tend to use streamstats because of the fuzzy time logic. subtract 5 seconds from the _time of the records that are going to be dismissed, then resort all the records | sort 0 _time, and streamstats with time_window=10s.

[DalJeanis]

@oldmaned -

If I read your specs right, then your pseudocode should look something like this.

 index=my_test_index
| eval killme=case(src=SourceServer01 OR src=SourceServer02 OR src=SourceServer03, "killme",
    dest="1.1.1.0/23","killme", 
    dest="2.2.2.0/23 AND (fieldA="AAAA" OR FieldA="BBBB" OR FieldA="CCCC"), "killme"      
    )
| eventstats max(killme) as killme by cs1 cs6
| table _time, dest, FieldA, src 

Obviously, the comparison tests for dest will need to be set up correctly ( like(src,"1.1.1.%") or cidrmatch("1.1.1.0/23",src) or whatever.

[jplumsdaine22]

Its going to depend on what the raw data looks like. Can you possibly post a series of events with the same field A to show us the flow you're talking about?

[OldManEd]

They are FireEye NX logs. If you need more I'll get some.

[jplumsdaine22]

Just post a couple of rows and show us the output you need.

[OldManEd]

These are FireEye logs. Nothing special. The top one is filtered out but the second one passed through. The issue is, all this came from a load balancer configuration. Both were initiated from the same event. I need to figure out how to filter the second log entry if and only if a ~similar~ event has already been filtered within a few seconds based on the same signature. I can match on the signature (cs1) and command entry (cs6) but that's about it.

Jul 22 02:54:18 <Splunk server> fenotify-1423236.alert: CEF:0|FireEye|MPS|7.9.5.676262|MC|malware-callback|7|rt=Jul 22 2018 08:54:11 UTC src=<srcIP A> cn3Label=cncPort cn3=<port> cn2Label=sid cn2=00000001 shost=<Host server name A> proto=tcp spt=24310 dst=<Dest IP A> cs5Label=cncHost cs5=<IP> dvchost=<Host Name> dvc=<DVC IP> smac=<MAC Address> cn1Label=vlan cn1=24 dpt=<Port> externalId=1423236 cs4Label=link cs4=https://10.165.252.21/event_stream/events_for_bot?ev_id\=1423236&lms_iden\=0CC47A30CBF8 act=notified cs6Label=channel cs6=Gh0st dmac=<MAC Address>  cs1Label=sname cs1=<Signature Name> devicePayloadId=b5b255db-87dc-4ce3-b07c-f90965e80047 

Jul 22 02:54:19 <Splunk server> fenotify-1423235.alert: CEF:0|FireEye|MPS|7.9.5.676262|MC|malware-callback|7|rt=Jul 22 2018 08:54:11 UTC src=<srcIP B> cn3Label=cncPort cn3=<port> cn2Label=sid cn2=00000001 shost=<HHost server name B> proto=tcp spt=41988 dst=<Dest IP B> cs5Label=cncHost cs5=<IP> dvchost=Host Name> dvc=<DVC IP> smac=0<MAC Address> cn1Label=vlan cn1=23 dpt=<Port> externalId=1423235 cs4Label=link cs4=https://10.165.252.21/event_stream/events_for_bot?ev_id\=1423235&lms_iden\=0CC47A30CBF8 act=notified cs6Label=channel cs6=Gh0st dmac=<MAC Address> cs1Label=sname cs1=<Signaure Name>  devicePayloadId=b5d14648-8237-43df-a341-484f03fdace0