Splunk ES Not Generating Notable Events from BOTSv1 (nessus, suricata) Data

[woo]

I am trying to test Splunk Enterprise Security (ES) by feeding it sample data, but I am unable to get any Notable Events generated.

My Goal: To ingest sample logs (BOTSv1 nessus-scan and suricata) using Eventgen and have Splunk ES generate corresponding Notable Events in the notable index.

Current Situation:

1. Platform: Splunk ES is successfully installed and running on my Splunk Enterprise server.

2. Data Source: I am using nessus-scan.json and suricata.json ("Complete" data) from the official splunk/botsv1 GitHub repository.

3. Ingestion: I am using splunk-eventgen to replay this data into Splunk. I have confirmed that the data is being ingested correctly and is visible in the main index (or its target index).

4. Time Range: I am searching using the "All time" time range, as the BOTS data has historical timestamps.

The Problem: Expected Behavior: Splunk ES Correlation Searches should detect this activity and create alerts in the notable index. Actual Behavior: The notable index is empty. Running index=notable | stats count always returns 0.

Troubleshooting Steps Already Taken:

1. Enabled Correlation Searches: I went to Configure > Content Management. I enabled Vulnerability Scanner Detected (by events) and Vulnerability Scanner Detected (by targets) for the Nessus data. I also searched for and enabled relevant IDS and intrusion related Correlation Searches (like Threat - IDS Attack Detected - Rule) for the Suricata data.

2. Accelerated Data Models: I went to Settings > Data Models. I confirmed that both the Vulnerabilities and Intrusion Detection data models are Enabled and Accelerated (they show the green checkmark).

3. Data Model Population Check: I ran | tstats count from datamodel=Vulnerabilities where sourcetype="nessus:scan" and | tstats count from datamodel=Intrusion_Detection where sourcetype="suricata". Both queries return counts greater than 0, which confirms that the data is correctly populating the accelerated data models.

My Question: Even though the data is being indexed, the data models are accelerated and populated, and the relevant Correlation Searches are enabled, no notable events are being created.

What am I missing? Are there other configurations (like Threat Intelligence setup) that are mandatory for these specific rules to fire? How can I further debug why these enabled searches are not triggering?

Alternatively, is there another method I can use to reliably generate any sample notable events just to confirm the system is working? It doesn't necessarily have to be from the BOTSv1 sample data. My main goal right now is just to see something in the notable index.


I downloaded only these two data for testing

(https://github.com/splunk/botsv1)

1. nessus:scan -> https://s3.amazonaws.com/botsdataset/botsv1/json-by-sourcetype/botsv1.nessus-scan.json.gz
2. suricata -> https://s3.amazonaws.com/botsdataset/botsv1/json-by-sourcetype/botsv1.suricata.json.gz

Thank you for your help.

[PickleRick]

OK. In order to produce a notable several things must happen.

1. The search must produce results. Have you verified if the search - when run interactively - yields results?

2. The search must be properly dispatched by the scheduler, run to its completion and produce results. Have you verified if the search was dispatched and if/how many results it returned? (you can do this in the normal reports/alerts part of GUI)

3. If the search had been run and produced results which matched the alerting criteria it should have created the entry in the notable index. If I remember correctly this should also be processed by ES internal mechanics and produce entry in kvstore but I don't recall the details here.

So there are still quite a few things to debug.