Getting Data In

Sub Search - how do I compare a search event to an index, only displaying the table of events not found in index?

manta0101
Engager

 

I have created a search that contains a field that is unique. I am using this search to populate the index. however for some reason when I try and check to see if the record is in the index it doesn't work for me. The closest I have come is this:

| localop | rest ....

```first search key field```
|eval soar_uuid= id+"_RecordedFuture"

|append [search index=rf-alerts soar_uuid|rename soar_uuid as ExistingKey]
| table soar_uuid,triggered,rule.name,title,classification,url,ExistingKey


The above returns  a list of new records with a blank ExistingKey field, and matching keys for soar_uuid  of existing records with a blank soar_uuid field. 
If I could just populate either with the other field, then I could remove all the duplicates.
I want to remove the new records that match the existing records before writing the events to the index. appendsearch instead of append doesn't seem to return the existing records. 

Labels (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Assuming there is only one event for each soar_uuid in either of the two searches, i.e. it is unique in its search but possibly duplicated in the other search, you could do comething like this

| localop | rest ....

```first search key field```
|eval soar_uuid= id+"_RecordedFuture"

|append [search index=rf-alerts soar_uuid]
| eventstats count by soar_uuid
| table soar_uuid,triggered,rule.name,title,classification,url,count

count would then be 2 if it is duplicated in the appended search

View solution in original post

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Assuming there is only one event for each soar_uuid in either of the two searches, i.e. it is unique in its search but possibly duplicated in the other search, you could do comething like this

| localop | rest ....

```first search key field```
|eval soar_uuid= id+"_RecordedFuture"

|append [search index=rf-alerts soar_uuid]
| eventstats count by soar_uuid
| table soar_uuid,triggered,rule.name,title,classification,url,count

count would then be 2 if it is duplicated in the appended search

0 Karma

manta0101
Engager

That worked, here is the updated SPL using your concept.

 

|eval soar_uuid= id+"_RecordedFuture"

|append [search index=rf-alerts soar_uuid]
| eventstats count by soar_uuid |where count<2
| table soar_uuid,triggered,rule.name,title,classification,url,count
0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...