Splunk Enterprise Security

Configuring the Splunk App for Enterprise Security on a new Search Head (attached to larger Indexer list), why can't I get the Threat Activity to load?

klawman
Explorer

I'm working to migrate ES to a new search head that has network visibility to indexers in multiple Business Units and more indexers. I am seeing my network traffic counts increase as I am now picking up the new architecture but I can't get my 'new' Threat Activity Dashboard to report anything.

I can see that the Threat Intelligence Downloads are operational and (as far as I've been told) both platforms should be equal, other than the additional feeds available to the new system.

I'm just not sure where to start when the only response is "no results found".

0 Karma

richard_griffit
Engager

I had the same problem with the same dashboard. I found that the index=threat_activity wasn't being populated either.

Splunk support had me do the following:

Please remove the following from etc/apps/TA-paloalto/local/macros.conf

[tstats]

definition = tstats summariesonly=t
# definition = tstats prestats=true local=`tstats_local`

Several of the searches were not completing due to scheduler limits. I would look for status=skipped in the scheduler.log file.

07-09-2015 12:54:56.128 -0700 INFO  SavedSplunker - savedsearch_id="nobody;DA-ESS-ThreatIntelligence;_ACCELERATE_DM_DA-ESS-ThreatIntelligence_Threat_Intelligence_ACCELERATE_", user="nobody", app="DA-ESS-ThreatIntelligence", savedsearch_name="_ACCELERATE_DM_DA-ESS-ThreatIntelligence_Threat_Intelligence_ACCELERATE_", status=skipped, reason="maxsearches limit reached", scheduled_time=1436471400

Modified /opt/splunk/etc/system/local/limits.conf and increased the two settings for the scheduler section.

[scheduler]
max_searches_perc = 70
auto_summary_perc = 75
0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

Does the files in /opt/splunk/etc/apps/SA-ThreatIntelligence/lookups are populated ?
What is the result of the following command in search (from ES): | inputlookup threatintel_by_cidr
Is it the same ES version ? Which one ? Same OS ?

0 Karma

klawman
Explorer

Yes it is accelerated, 100% complete and about .2MB on disk. (same on both systems)

0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

Of course, other dashboards (like the traffic one) are populated , correct ?

0 Karma

klawman
Explorer

Yes, Traffic Center IS populating.

There IS a difference here in that both systems are picking up "pan-traffic" from the client's Palo Alto firewalls and on the new system I am working to get their Cisco ASA traffic tagged appropriately using the Splunk Add-on for Cisco ASA. (again, different BU's working with different technology)

I am not yet properly seeing the ASA traffic but I was/am assuming I should still be able to get the matches from the Palo Altos.

0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

Did you tried to add in the local threat list an IP that is used in one of your log ? I just want to be sure that some traffic IP are matching the ones from the threat lists ?

0 Karma

klawman
Explorer

I think I am getting closer to the issue. I followed your advice and as I attempt to find the IP addresses in local_threatlist, I'm seeing that "ess_lookup_lists" does not populate on the new server.

Essentially, the new deployment isn't reporting ANY lists and lookups. Is there a configuration piece that got missed?

0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

Hummmm... how did you migrate ES to the new server ? Did you copy the files or reinstall a fresh copy of ES ? And when (just a few hours ago, or several days ago) ?
Do you have the both ES running in parallel ?

0 Karma

klawman
Explorer

I installed ES as a fresh install on the new server approximately one week ago (technically a week ago last Friday). The platform used was an existing search that was re-purposed for ES. It is (now) a dedicated, distributed Search Head with no other apps installed that are not CIM compliant.

Yes, both ES systems are running in parallel.

0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

In the _audit index (index=_audit), do you see searchs labbeled "Threat - Threat Intelligence By System - Lookup Gen" or similar ? How often ? Did one run after you entered the new entries in the local threat list ?
What is your Splunk installation path ?

0 Karma

klawman
Explorer

No, I do not see any search with "threat" listed in the _audit index at all. (Honestly, I don't see any audit events with "threat" on either ES server)

The ES deployment path for the newer linux install is "/opt/splunk/etc/apps"

0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

If you have no such events, even for a long period, you should contact support. This is not normal. Is it related to your problem, this might be, but to be confirmed.

Regards

0 Karma

mdessus_splunk
Splunk Employee
Splunk Employee

What is the status of the Threat_Intelligence datamodel (in the Data Model audit) ?
Is it accelerated ? Is it complete ? And the disk size ?

0 Karma

klawman
Explorer

To answer your questions in order:

1) Yes, opt/splunk/etc/apps/SA-ThreatIntelligence/lookups are poplulating on both servers, old an new. Latest updates are from 6/28/15

2) | inputlookup thretintel_by_cidr gives a list of ip_intel addresses on both systems

3) Yes, it's the same ES version, 3.3.0. The "new" server is on RHEL (no feed) the "old" is Win2012 Server (working).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...