ditto. just to be clear there are disclaimers written all over the shuttle repo: Shuttl development has stalled. There's no known developer working on this project. Shuttl seems to work for Splunk 6.x when it's built off the develop branch, but it's experimental. ... WARNING: The examples provided in this guide are for testing only and should not be used in production environments without consulting splunk documentation. ... View more

actually to clarify, i was also seeing messages as found here in this question. this answer helped me (was actually a loadbalancing issue). https://answers.splunk.com/answers/205904/search-head-cluster-error-usermanagerpro-failed-to.html ... View more

[SPLUNK] ... i'm not 100% sure but figured i should clarify that the stanza name above "SPLUNK" needs to match whatever stanza name. so in my case i put "ldap-auth" as the stanza name in both files. this is honestly the best answer i've seen presented so far between the docs and one other answer. the method proposed in the documentation of just simply copying the file over into master apps is not a smart one... it's better to have some automation drop the file in etc/system/local on each system instead. hopefully splunk can fix this type of settings in the future to be more streamlined like with indexer clustering. setup and maintenance of SHC is way too complicated and not documented strongly enough IMHO. ... View more

@mgonter_splunk - any updates on this? i am using LDAP auth in a Search Head cluster. ... View more

i think this response is much more complete and accurate: https://answers.splunk.com/answers/103900/is-there-a-walk-through-on-how-to-deploy-authentication-conf.html ... View more

awesome insight @Colin Humphreys! thanks man! ... View more

cheers @williamwar! 2 follow-ups for you: any caution about renaming existing apps? like if i have one called "foo_indexes", any issues if i rename it to "acme_NYC_foo_indexes" to match my new naming convention? just confirming - but following Configuration File Precedence - if i split these out from already-existing system-local files, i'll need to 1) also blow away system/local copies of those entries on each system and 2) restart splunk. ... View more

@jcoates i think there are actually multiple things going on here. we found the following issue migrating from a homegrown aggregator with a single queue, to a queue which serves up notifications from tons of s3buckets. i can confirm this was an issue in both 1.1.1 and 2.0 versions of the TA. records would download fine in most cases, but then we'd see a stacktrace before any events were ingested. we think it was possibly treating the directory / folder itself as a cloudtrail log, and just giving up. 2015-10-28 01:59:04,521 INFO pid=10768 tid=MainThread file=aws_cloudtrail.py:process_S3_notifications:453 | fetched 31 records, wrote 31, discarded 0, redirected 0 from s3:foo-cloudtrails/nnnnnnnn/folder/yyyyyyyyyy/CloudTrail/region/2015/10/28/aaaaaaaaaaaa-foo.json.gz 2015-10-28 01:59:35,704 CRITICAL pid=10798 tid=MainThread file=aws_cloudtrail.py:stream_events:331 | Outer catchall - Traceback: Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py", line 269, in stream_events s3_completed, s3_keys_to_delete, s3_failed=self.process_S3_notifications(s3_conn, s3_notifications) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py", line 405, in process_S3_notifications message['s3Bucket'], key, type(e).__name__, e)) KeyError: 's3Bucket' digging further with a cleaned up debug message... (which is super tricky in splunk TBH) seems like it was a formatting error actually printing the error itself and killing the script 😕 : 2015-10-28 05:16:14,005 ERROR pid=26577 tid=MainThread file=aws_cloudtrail.py:process_S3_notifications:407 | problems reading json from s3:foo-cloudtrails/nnnnnnnn/folder/yyyyyyyyyy/: ValueError No JSON object could be decoded so effectively, we DoS'd ourselves until we patched the script. the main line for us was below, but other exceptions around there would also fail similarly: except ValueError as e: message_failed=True logger.log(logging.ERROR, "problems reading json from s3:{}/{}: {} {}".format( message['s3']['bucket']['name'], key, type(e).__name__, e)) #logger.log(logging.ERROR, "problems reading json from s3:{}/{}: {} {}".format( # message['s3Bucket'], key, type(e).__name__, e)) is there a github or bitbucket repo which we can use to suggest changes? for now, i will toss them in bitbucket: https://bitbucket.org/awurster/splunk-ta-aws/commits/311f3828e422bf1583148039771a7c051e3cc6c0 ... View more

more digging and it seems that splunk 6.3 introduced an extra field instance_type to the keys. i'll see if i can't set this to something and work around it. ○ → diff splunk-6.*/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py 35,37c35,40 < if cherrypy.request.cookie.get('uid') is None: < cherrypy.response.cookie['uid'] = splunk.util.uuid4().upper() # for consistency as splunkd returns uppercase guid < cherrypy.response.cookie['uid']['expires'] = 5 * 365 * 24 * 3600 --- > req_cookie = cherrypy.request.cookie > resp_cookie = cherrypy.response.cookie > if req_cookie.get('splunkweb_uid') is None: > resp_cookie['splunkweb_uid'] = req_cookie.get('uid').value if req_cookie.get('uid') else splunk.util.uuid4().upper() > resp_cookie['splunkweb_uid']['path'] = '/' > resp_cookie['splunkweb_uid']['expires'] = 5 * 365 * 24 * 3600 39,40c42,43 < cherrypy.response.cookie['uid']['secure'] = 1 < --- > cherrypy.response.cookie['splunkweb_uid']['secure'] = 1 > 343a347 > server_info['entry'][0]['content']['instance_type'] = cherrypy.config['instance_type'] 630c634 < for k in ['build_number', 'cpu_arch', 'version_label', 'is_free_license', 'is_trial_license', 'license_state', 'os_name', 'guid', 'master_guid', 'license_desc', 'install_type', 'django', 'addOns', 'activeLicenseGroup', 'product_type']: --- > for k in ['build_number', 'cpu_arch', 'version_label', 'is_free_license', 'is_trial_license', 'license_state', 'os_name', 'guid', 'master_guid', 'license_desc', 'install_type', 'django', 'addOns', 'activeLicenseGroup', 'product_type', 'instance_type']: ... View more

does it have to be called splunk_httpinput ?? IIRC deployment server / splunk .conf guides recommend following an app naming convention, for which that would be bucking the trend 😕 ... View more

so after running Duo's uninstall.sh it looks like Duo either restored an older version of the file or something else i can't figure out. this left splunk in an unusable state... and so here's how i restored the busted account.py file in a jiffy: curl -L -o splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64.tgz "http://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.3.0&product=splunk&filename=splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64.tgz&wget=true" tar -xzvf splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64.tgz mv /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py.duo_restored.old cp splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/ ~/bin/splunk restart ... View more

thanks but i'm mostly just looking for a pythonic way to access the local system's timezone. i'm writing logs to ~/var/log/splunk/foo.log which lands in the server's _internal index. it might be redundant to log a TZ, but i still am curious what the answer is. i should also clarify that i'd like to do it natively as much as possible, without requiring a 3rd party lib like pytz to be bundled with my add-on. ... View more

none of these answers seem to work in 6.x ... View more

thanks tony. this is cool, but it's sort of a hack, masking the original problem. i don't know if it has much to do with any appliance-id field. i don't believe a two digit TZ code is standard. just tell the engineers to send a proper strftime variable, then splunk would automatically parse it. use either %z or %Z : https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Configuretimestamprecognition http://docs.splunk.com/Documentation/Splunk/6.2.3/data/Applytimezoneoffsetstotimestamps ... View more

i found this frustrating too. you actually have to POST the option as shown in the CURL examples in the docs. the way i do it is using splunk's rest.simpleRequest method. strangely, i always seem to misplace the link for the SDK, but here's an example: http://answers.splunk.com/answers/144000/modifying-acl-saved-search-permissions-through-rest-api-using-python.html it's not ideal - but perhaps you could first retrieve all the data points in that collection, and then iteratively delete them? query_uri = '/servicesNS/nobody/<app>/storage/collections/data/<kvstore>' serverResponse, serverContent = rest.simpleRequest(query_uri, sessionKey=sessionKey) for result in serverContent: removal_uri = '/servicesNS/nobody/<app>/storage/collections/data/<kvstore>/%s' % result['_key'] serverResponse, serverContent = rest.simpleRequest(removal_uri, sessionKey=sessionKey, method='DELETE') maybe you can also leverage one of the answers from my similar question: https://answers.splunk.com/answers/237859/can-i-delete-all-data-from-a-kv-store-at-once.html ... View more

building on the response from here, elsewhere in answers, we can hardcode the URLs via HTTPS, which are served by a proper Splunk SSL cert: http://download.splunk.com/releases/<version>/splunk/ http://download.splunk.com/releases/<version>/universalforwarder/ so in the end, i use this bit of code in my build script to get both the hash and the download file securely: cd `mktemp -d` VERSION="6.2.1" BUILD="splunk-6.2.1-245427-linux-2.6-amd64" PTYPE="deb" wget "https://download.splunk.com/releases/$VERSION/splunk/linux/$BUILD.$PTYPE" -O $BUILD.$PTYPE wget "https://download.splunk.com/releases/$VERSION/splunk/linux/$BUILD.$PTYPE.sha512" -O $BUILD.$PTYPE.sha512 sha512sum -c $BUILD.$PTYPE.sha512 which outputs: splunk-6.2.1-245427-linux-2.6-amd64.deb: OK note that i had to upgrade my OS to get a fresher copy of openssl , although it might've been my outdated wget install too, as mentioned here. ... View more

apologies @matt but given a recent re-think with a colleague, i'll have to unmark the answer. we are in fact concerned with the validity of the package contents. downloading either the package or the hash, or both via HTTP starts to defeat the purpose. i think the easy answer / request is to ask Splunk to force SSL-enabled downloads for both the packages and the downloads page that serves the hashes, and to sign them with a certificate that matches the download URL. i think it's becoming a de facto standard these days, so shouldn't be an unreasonable one. i've posted a similar response below that uses SSL and checks the hash. ... View more