When you are loading the data into the KVStore, you want to rename the field you're using as _key, therefore it will be used as the primary key record instead of what Splunk autogenerated for you. I'm using a Python script to generate the documents I put into KVStore as part of another process, but an example of how I build the document is: doc = dict(_key=str(item['IP']), User=str(item['User']), Domain=str(item['Domain']), Last_Update=int(float(item['Last_Update'])), Last_PA_Send=int(0)) consider the item[xxxx] as my fields from my search results, since I wanted my IP field to be the primary key I renamed it to _key prior to sending it to KVStore. ... View more

Thanks! This is all fantastic information! ... View more

Would you be able to add some comments regarding behavior when performing an OS/System level reboot? ... View more

Thanks for pointing all of this out! That's some good digging into the problem, and it lines up with some of the things I'm seeing. I'll see if I can figure out what's going on further, since most of the TA scripts call that retry function in the event of a failure, either its a defect with the SDK or potentially with the usage of the boto library by the TA itself. ... View more

We have the same license amount and currently run an AWS solution as well, so here are our specs as an example for you, please note that we use Splunk Enterprise Security so I've substituted M4 instances for C4 instances where they would be more appropriate without ES (I did it so it wouldn't moan about RAM). Indexers: 4 x M4.4xlarge (100 GB GP2 Volume "/opt/splunk", 3 TB GP2 Volume "/data/splunk_hot", 8 TB ST1 Volume "/data/splunk_cold", 1 TB ST1 Volume "/data/splunk_frozen" ~ 9 months live retention based on our current loads). Search heads: 1 x C4.4xlarge (100 GB GP2 Volume "/opt/splunk") - General Use 1 x M4.4xlarge (100 GB GP2 Volume "/opt/splunk") - Enterprise Security Cluster Master Server: 1 x C4.xlarge (100 GB GP2 Volume "/opt/splunk") Settings: Multisite - true 2 indexers and 1 sh per site, sites are defined by AWS Availability Zones, Master sits on site1 All indexes are replicated, rep factor is origin:1, total:2 for rep and search factor. Summary replication - true UseACK for forwarders - true I've not listed other devices as they're not relevant to our setup, things like deployment servers and heavy forwarders are defined more by process requirements/geolocation for my Org (we have both cloud and on prem forwarders). The above core system handles search load for ~20 non security users, ~12 security users, and the system search load. ... View more

Since the addon probably is expecting the native log file lines and you're probably getting a syslog header attached to your lines, you'll likely to do something like this: - Create a dummy sourcetype like "ibm-websphere-syslog" and monitor your rsyslog files with that. - In the dummy sourcetype, setup a props and transforms so that it seds out the syslog header, then transforms the line to the "proper" ibm websphere sourcetype from the addon. - Verify that your data is being properly parsed as the real sourcetype and the fields expected from the addon are appearing. You could also redo the props.conf and transforms.conf statements to account for your syslog header and throw them in the local dir of the addon. ... View more

Since this app does the scripted input/cron job to get data, you'll need to install it on a HF or a Search head for your data collection, plus any indexers you plan on using to index the data and the search heads you plan on searching the data from (including Splunk ES). Once you've done that, you'll need to go to the node you've chosen as the data collector (either a SH or HF) and generate the inputs either using the web config for the TA or by hand editing the conf files provided in the TA (web gui is easier, but you'll want to change your app context to this TA before you go to settings and set everything up so the new conf files get stored within the app). Once your Rapid7 detections are run and the input picks up the new data you'll then be able to search and so on. Not this app appears to not meet the specs for index cluster deployment as the bundles fail validation with this app (did not include a spec file for a custom .conf they have, also thinks one of the default values is invalid likely due to the lack of spec file). ... View more

Kiran, based on my knowledge of Splunk Cloud, you'll have to pass them through on on prem Heavy Forwarder or Universal forwarder where you can send data over UDP and/or SCP. Since the forwarders that connect to Splunk Cloud use SSL certs, it's safe to say you'll be unable to do a "direct" input to Splunk Cloud. ... View more

Randino, You'll need to copy the logs files over from your ESA device using SCP to a Splunk UF and then monitor those files and assign the proper sourcetype from there. ... View more

You should modify the search so it does not match events which fall under the maint window using a combination of the date_wday and date_hour and if it's on specific days every month then date_mday. Ex, a maint window every sunday from 00:00:00 to 02:00:00, you could use "date_wday!=sunday AND (date_hour!=0 OR date_hour!=1)" Another idea would be to put your maintenance schedule into a lookup CSV, either as their epoch times or a "normal" timestamp you'll have to convert with strptime. Then just filter out events with times which match those ranges in the search. ... View more

I got around to testing this and it has speed up the responsiveness of the input box pretty dramatically (using an inputlookup for the population), it sounds like for this scenario this is a pretty good workaround. ... View more

For your DEV/QA environment, you should try using data cloning at the forwarder level over to those indexers instead of snapshot copying. Then just set the data retention in the DEV/QA environment to a much smaller level. You should be able to have the same apps on the search head and indexer as normal but they would be separate from your prod environment (much more cost effective if you're not using a cluster / multisite cluster as DEV/QA as well). You can deploy apps on the search head using the deployment server (I use the DS to deploy tech addons and additional configurations to the search heads, however be careful of deploying apps to a prod environment where you allow users to edit dashboards, reports, and saved searches with global permissions, as the deployment server likes to keep the apps' synced). ... View more

It should exactly the same as if you were going to do it on a RHEL install so I'd say go for it. However, I've not heard of much usage in monitoring the red hat hypervisor (in fact I only know of Splunk built monitoring for VMware systems and HyperV via some of the MS tech), so that sounds like a pretty interesting use case. ... View more

Yea they're all bad, but it's a side effect of the odd design choice to have a "deployment" master server which then can start having radically different versions from the captain and the other SHC members. Good idea for a feature request, seems like something they need to address in one way or the other. ... View more

The ways I can think of would be to either: 1. Stop all SHC members, remove the local folder for the app, start them up again. 2. Move the app off the SHC deployer, let the SHC delete the app from themselves, redeploy. 3. I've never tried this, but maybe it could work: instead of using the SHC deployer, deploy everything to the SHC captain from a regular deployment server or manually and see if it'll replicate everything across the rest of the members anyways. ... View more

I've been playing with both, it seems the security app is more focused as a "alert action" or ES action item for notable events. The Splunk app and addon for ServiceNow seem to be focused on monitoring your servicenow environment using Splunk similar to other apps (such as the infrastructure focused apps), and working as an alternative to SNOWs reporting and performance analytics items. It also has the added benefits of creating incidents and events, though I don't think it is as refined as the "Security" app (but it is only for incidents). I'm definitely interested to hear about this from an expert though. In my experience so far, if you're very good with the SNOW app for Splunk then you don't need to use the Security app, however the Security app is much easier to setup and use. In my situation, I'm planning on using the SNOW app on our "regular" search head for all of those items, but using the "Security" SNOW app on the ES search head to save time and resources on the devices. ... View more

Have you checked the permissions of the /opt/splunk folder on your install? If those look good, check the permissions of the objects in /opt/splunk/etc/auth and especially make sure splunk_secret is set to read only. From those error logs you posted it looks like the server cannot find the pages internally, sounds most likely due to a permissions issue or security mismatch of some kind, though it's believable that the upgrade install may have broken the paths. What OS are you running Splunk on? ... View more

Have you done any network diagnostics from that box to the Splunk cloud endpoint? Make sure the ports Splunk cloud is asking you to use for data transmission are working correctly. After that verify everything in the SSL config is as they say it should be, and if there is a password for the cert file, put it in plaintext back in the config and reboot the box so it can be resalted. I've experienced similar issues before and it was because the SSL config was not perfect (however I'm on an on-prem deployment), you should start with network diagnostics then move onto triple checking the SSL config. ... View more

This is compared with our index cluster which is three boxes of 16 vCPU and 16 GB RAM. ... View more

Oh sorry I didn't answer that part of the question, our HF is only at 4 vCPU and 8GB Ram, and even then it doesn't ever sustain high load. Since it's only doing the parsing and it's not dealing with a ton of data the lower specs are fine. You should be fine with a pretty low spec box, but it doesn't hurt to watch the utilization and adjust as needed, great use case for VMs if that's your plan. ... View more

Thanks for the clarification on this! Yes I was looking at the spec file primarily and that's where I started getting a bit confused by the description of the attribute. ... View more