You will definitely see any search heads configured as cluster clients in your Cluster Manager View. Did you go to settings > distributed search > search peers and add the new search heads there as search peers and remove the old ones? ... View more

Are you forwarding all of your splunk internal logs from your new search heads to your indexers? Maybe look to see if there are any errors or warnings in the splunkd log in the DistributedPeerManager component on your DMC search head. ... View more

Are you sure you have network communication functioning on tcp-8089 (splunk default mgmt) between the DMC and the new peers you are adding? ... View more

This is typically a symptom of automated field extractions not being applied to sourcetypes correctly before they get mapped into the datamodels that Enterprise Security uses. This is part of the tuning process when you install/configure enterprise security. Make sure that all your authentication related events are sourcetyped correctly so that the field extractions and tagging occurs properly. To troubleshoot this, you will have to take a look at the raw event logs, see what sourcetype they are assigned to and find the Technology Addon that formats/parses that data. Make sure that the appropriate Technology Add on has field extractions configured in props.conf and transforms.conf, if needed, you can customize these under the /local directory of the Technology Addon. (or create your own Technology Addon if it is a custom datasource, there is a decent elearning module on splunk.com that explains what goes into creating a Technology Addon) ... View more

Have you installed all of the required libraries on your server? Do you have network communication between the Splunk server and the checkpoint management server? My experience is that when these things are in place, using the web interface is definitely the easiest way to get it configured. On a 64-bit Redhat/CentOS 6.x server I've had to do the following: yum install glibc.i686 yum install pam.i686 ln -s /opt/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/libcpc++libc6.1-2.so.3 /lib/libcpc++libc6.1-2.so.3 It seemed the scripts in the TA required these libraries in the couple cases I've worked on. ... View more

did you try setting KV_MODE=json? ... View more

When you have an app with a lookup in it properly working on your test search head, you would have to verify app/file permissions on all the files in the app, need to be owned by the account running splunk on all the search heads, if not running as root. Do you have the lookup table assigned to a sourcetype? One other thing to check would be the default.meta file that your lookup app contains. Make sure that you export everything in the app to system, so that it can be used on any search head by any splunk app. I've been bitten by permission issues like that plenty of times. ... View more

you could run a search like this: index=_internal host=* source=*metrics.log group=queue blocked=true | rename host as Indexer | chart count(blocked) as "Queue Blocks" by Indexer, name to create a chart of the count of blocks by indexer and queue. ... View more

instead of using one or more space characters in your regex after the { character, try using a negative character class instead like so: \,\s+\{[^\"]+\"id\"\: ... View more

Did you try escaping some of the literal characters in your regex: BREAK_ONLY_BEFORE=,s*{s*"id": this should be: BREAK_ONLY_BEFORE = \,\s*\{\s*\"id\"\: actually, I would also use \s+ instead of \s*, if you know that there will be at least one space character. The BREAK_ONLY_BEFORE directive specifies a regex pattern, what you had in your configuration example wasn't regex, it was just a string of literal characters. \s is a space character in regex, not s. This answer is assuming what is in your example is exactly what's in your props.conf config and splunk answers didn't strip out the escape characters out for you when you posted it. ... View more

does the DOMAIN/username account exist in your MSSQL instance and does it have access rights to the ePO database? ... View more

Since you are referring to an operating system level account and not one within the Splunk application, you will need to use a samba-client component on your Linux server such as winbind. You will install those components and configure it to participate in Active Directory. This will allow you to create an Active Directory synced account on your operating system. Then you can use that account and follow the Splunk installation procedure for installing with a no privileged account. ... View more

After you upgraded your java version, did you verify in dbconnect that you configured the app with the new, correct java path? Also, I don't remember dbx2 coming with the jar file for the driver. Take a look at this answers post which clears a couple things up. http://answers.splunk.com/answers/233188/db-connect-and-java-versions.html ... View more

yes you will. You will be able to use a combination of two splunk configuration files, authentication.conf and authorization.conf to configure ldap authentication for Splunk and create/map splunk roles to security groups in Active Directory. Here are two good references covering the ways Splunk can integrate with Active Directory/LDAP, all at the application level, so it won't matter what operating system you are running it on. http://blogs.splunk.com/2009/08/13/ldap-auth-configuration-tips/ http://docs.splunk.com/Documentation/ActiveDirectory/1.2.2/DeployAD/ConfiguretheSA-ldapsearchsupportingaddon ... View more

Did you reduce the default time range of the data model? I believe out of the box, it is set for a year. Also, what is the data volume of your pan logs? ... View more

ES relies a lot more on tags then it does sourcetypes. If you are creating your own sourcetypes, take a look through the common information model documentation and the ES dashboard matrix. You'll need your custom sourcetypes to map into the datamodels that ship with ES and have the proper field extractions. If you can map your data to sourcetypes from the technology add-ons that ship with ES, you will save yourself a ton of work. By the way, an entire book could be written on this set of questions. ... View more

I would imagine that you have a mismatch of the pass4SymmKey between the indexer and the license master. This can be set in server.conf : [general] pass4SymmKey = mylicense (this string has to be the same between all of the license slaves and the license master) ... View more

Typically, to search a data model using acceleration would look something like this: | tstats count from datamodel=Network_Traffic where something= . . . (http://docs.splunk.com/Documentation/Splunk/6.2.3/Knowledge/Acceleratedatamodels) ... View more

Are there any updates for this issue? I'm currently seeing this on a very busy Enterprise Security Search head. It seems to happen about 20 times in a month period. Would this get resolved by increasing the ulimit settings on the linux server, or is there some limits.conf setting that should be looked at on the search head? This is Splunk Enterprise 6.2, so it is still happening on newer versions. One other caveat, it resides in amazon cloud., ... View more

If this scripted input isn't working, then the data in question is not in the index = checkpoint_lea, so it is not indexed yet. Is the certificate from the checkpoint management station in the path ./certs ? and named SplunkLEA.p12? Can you test network communication on port 18185 between the splunk server and the management station? You should be able to look on the checkpoint management station and verify that you see successful logons from Splunk. you need to verify that you have the correct opsec_entity_sic_name and opsec_sic_name. I remember their being some library dependencies that the script required as well. You can manually run the script from the operating system of the splunk server to verify the it operates correctly. You should also verify that the /etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh file exists or not, because that is what this error is complaining about. ... View more

1st, you need to test the triggering condition, then manually run your search and verify that you are capturing results with the searches that you are using for your correlations. ... View more

I would take a serious look at time synchronization across your environment. Correlation searches depend heavily on all the monitored devices having the same time. They should all be using ntp and syncing from the same times source. ... View more

You can always restrict the time frame of the datamodel accelerations that you don't want to be expensive that your instance of ES won't be using. That way, even if they have a lot of data in them, and you aren't relying on ES to use them, they won't hit you as hard. ... View more