I'm working to migrate ES to a new search head that has network visibility to indexers in multiple Business Units and more indexers. I am seeing my network traffic counts increase as I am now picking up the new architecture but I can't get my 'new' Threat Activity Dashboard to report anything.
I can see that the Threat Intelligence Downloads are operational and (as far as I've been told) both platforms should be equal, other than the additional feeds available to the new system.
I'm just not sure where to start when the only response is "no results found".
I had the same problem with the same dashboard. I found that the index=threat_activity wasn't being populated either.
Splunk support had me do the following:
Please remove the following from etc/apps/TA-paloalto/local/macros.conf
[tstats] definition = tstats summariesonly=t # definition = tstats prestats=true local=`tstats_local`
Several of the searches were not completing due to scheduler limits. I would look for status=skipped in the scheduler.log file.
07-09-2015 12:54:56.128 -0700 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-ThreatIntelligence;_ACCELERATE_DM_DA-ESS-ThreatIntelligence_Threat_Intelligence_ACCELERATE_", user="nobody", app="DA-ESS-ThreatIntelligence", savedsearch_name="_ACCELERATE_DM_DA-ESS-ThreatIntelligence_Threat_Intelligence_ACCELERATE_", status=skipped, reason="maxsearches limit reached", scheduled_time=1436471400
/opt/splunk/etc/system/local/limits.conf and increased the two settings for the scheduler section.
[scheduler] max_searches_perc = 70 auto_summary_perc = 75
Does the files in /opt/splunk/etc/apps/SA-ThreatIntelligence/lookups are populated ?
What is the result of the following command in search (from ES): | inputlookup threatintel_by_cidr
Is it the same ES version ? Which one ? Same OS ?
Yes, Traffic Center IS populating.
There IS a difference here in that both systems are picking up "pan-traffic" from the client's Palo Alto firewalls and on the new system I am working to get their Cisco ASA traffic tagged appropriately using the Splunk Add-on for Cisco ASA. (again, different BU's working with different technology)
I am not yet properly seeing the ASA traffic but I was/am assuming I should still be able to get the matches from the Palo Altos.
Did you tried to add in the local threat list an IP that is used in one of your log ? I just want to be sure that some traffic IP are matching the ones from the threat lists ?
I think I am getting closer to the issue. I followed your advice and as I attempt to find the IP addresses in local_threatlist, I'm seeing that "ess_lookup_lists" does not populate on the new server.
Essentially, the new deployment isn't reporting ANY lists and lookups. Is there a configuration piece that got missed?
Hummmm... how did you migrate ES to the new server ? Did you copy the files or reinstall a fresh copy of ES ? And when (just a few hours ago, or several days ago) ?
Do you have the both ES running in parallel ?
I installed ES as a fresh install on the new server approximately one week ago (technically a week ago last Friday). The platform used was an existing search that was re-purposed for ES. It is (now) a dedicated, distributed Search Head with no other apps installed that are not CIM compliant.
Yes, both ES systems are running in parallel.
In the _audit index (index=_audit), do you see searchs labbeled "Threat - Threat Intelligence By System - Lookup Gen" or similar ? How often ? Did one run after you entered the new entries in the local threat list ?
What is your Splunk installation path ?
No, I do not see any search with "threat" listed in the _audit index at all. (Honestly, I don't see any audit events with "threat" on either ES server)
The ES deployment path for the newer linux install is "/opt/splunk/etc/apps"
To answer your questions in order:
1) Yes, opt/splunk/etc/apps/SA-ThreatIntelligence/lookups are poplulating on both servers, old an new. Latest updates are from 6/28/15
2) | inputlookup thretintel_by_cidr gives a list of ip_intel addresses on both systems
3) Yes, it's the same ES version, 3.3.0. The "new" server is on RHEL (no feed) the "old" is Win2012 Server (working).