According to Splunk documentation, you can uses SAML with tokens: "Create authentication tokens to use the REST APIs. Tokens are available for both native Splunk authentication and external authentication through either the LDAP or SAML schemes. To learn more about setting up authentication with tokens, see Set up authentication with tokens  in the Securing Splunk Enterprise manual." There are some SAML side requirements such as (per token doc):  "Single Sign-On (SSO) schemes that use SAML. These schemes must either support Attribute Query Requests (AQR) or provide information through scripted authentication extensions." Hope this helps! ... View more

Hello, Did you get an answer for this? One of my clients is having the same issue. Only the captain is skipping all the searches they have a lot of resources in their environment. Also we set up this captain_is_adhoc_searchhead=true But that didn't help.   Kind Regards. ... View more

Seems like this worked. Thanks! ... View more

thanks @traxxasbreaker. Do we get an option to configure this period for checks? ... View more

Looking into this "tailing process" you listed... Yes, the app is enable. There are (3) files we're monitoring. The other 2 working fine. Definitely a strange issue. The "system_logs" are the ones that mysteriously stopped working. INPUTS.CONF [monitor:///var/logs/cassandra/system.log] disabled = false sourcetype = system_logs index = cassandra [monitor:///var/logs/cassandra/audit/audit.log] disabled = false sourcetype = audit_logs index = cassandra [monitor:///var/logs/cassandra/solrvalidation.log] disabled = false sourcetype = solrvalidation_logs index = cassandra ... View more

Hi did you find out what caused this error? ... View more

I struggled with the same problem for AGES. No proxy. First of all, our SN API wasn't correctly set up. Once we got that set up, I was able to query: https://MYCOMPANY.service-now.com/sys_journal_field.do?JSONv2&sysparm_query=sys_updated_on%3E=2000-01-01+00:00:00^ORDERBYsys_updated_on&sysparm_record_count=50 but kept getting the red ribbon giving the error messages in the SN add-on. Had Splunk specially send me v2.8 of the add-on to try, it gave me this error with https: Encountered the following error while trying to update: Splunkd daemon is not responding: (u"Error connecting to /servicesNS/nobody/Splunk_TA_snow/apps/local/Splunk_TA_snow/setup: ('The read operation timed out',)",) and this with http: Encountered the following error while trying to update: In handler 'localapps': Error while posting to url=/servicesNS/nobody/Splunk_TA_snow/service_now_setup/snow_proxy/snow_proxy I also tried v3.1, which only lets you do https, with the same error. Eventually I got it working with 3.1 by doing this: *Verified my API was queryable correctly *Logged into Splunk with the default admin account instead of my account (even though both have the same admin roles/all permissions) *Set up add-on v3.1, got the error *Installed SN app *Restarted Splunk *Tried to set up add-on again with error message Then I looked at my indexes and the SN indexes were there and receiving data. I'm not sure what fixed it, be it the reboot, or having the app installed as well (which shouldn't make any sense), or what. But bottom line, you're probably going to get a red error message regardless if the setup worked or not. Pretty sure even though it doesn't tell you to, you need to reboot after getting the error message, and that does the trick ... View more

I'm curious as to whether those cold buckets still under the hot/warm path are actually searchable with that configuration, and I strongly suspect that they might not be. Can you please try running | dbinspect and see if it does show those cold buckets? To answer your question about moving the buckets, it's probably better to move the existing cold data rather than just waiting for new data to roll to cold so all the cold buckets will be in the right place, but the fact that it created the directory structures when you applied that configuration is a good sign. You should stop the Splunk instance (one at a time and in maintenance mode if you are using indexer clustering) and move those existing cold buckets to the cold path. Or if you have the filesystem space and are more comfortable with a copy, you can use cp instead then cleanup the originals after you are sure it worked. ... View more

Yes, that is the main idea behind the package manager. ... View more

Could you please share your inputs.conf But my suggestion is first change the CrcSalt text into something else and then push the changes again from the deployment server. If this also doesn't work then delete your fishbucket of your forwarder and check ... View more

Yes, the "L" is causing the "Searching is waiting for input...", change it to "l" makes it go away. However, need to surround the token with " in order to make it work. But it only works with single selected value. In other words: "email_type=foo" works "email_type=foo OR email_type=bar" doesn't work, even in the log both email_type "foo" and "bar" exist ... View more

Which file would those be set in? The [search] stanza sounds like limits.conf but I don't see this setting in the spec. When we hit this problem restarting the deployer seems to help but a solution that doesn't require waiting for it to fail and restarting would be much better. ... View more

Heya @kenharford823, Did you get this to work? ... View more

On the forwarder, it will cause data still present on the filesystem to be reindexed. The fishbucket is what holds the file checkpoints for what the forwarder is monitoring, so it looks like fishbucket corruption is preventing the forwarder from figuring out where to pick up on the files it is monitoring. If the repair command doesn't work, then a reinstall would have the same effect as it wouldn't preserve the fishbucket, so it would cause all the files to be reread from the beginning. ... View more

Hey @danielwan, if they answered your question, please remember to "√Accept" the answer to award karma points and to let other Splunkers know there’s a working solution. We’re hosting a karma point contest, so it’s particularly awesome to up vote on Answers these days. 😄 ... View more

@traxxasbreaker - We've converted that comment to an answer so you can accept it if your issue is handled. ... View more

+1 for starting with search section in the monitoring console! Excellent place to open panels in search and see where it gets it's goodies! It also has a toggle for ad-hoc searches. Otherwise Activity > Jobs in the top right corner of your screen can be a good place if you are digging into particular jobs... Also shout out to the search activity app https://splunkbase.splunk.com/app/2632/#/details ... View more

Hi @traxxasbreaker I changed all of the ownership in local.meta to my username and then I ran server:8000/en-US/debug/refresh, but I got the following errors. Can you help me understand how to fix these and what impact these have? Refreshing admin/collections-conf BadRequest In handler 'collections-conf': Must use user context of 'nobody' when interacting with collection configurations (used user='katz.r') Refreshing admin/indexes InternalServerError In handler 'indexes': reload is not safe since a path has been deleted or modified for an index, or an index has been disabled. You must restart the Splunk Server, for your changes to take effect. Refreshing admin/remote_indexes BadRequest In handler 'remote_indexes': The following required arguments are missing: repositoryLocation. Refreshing admin/search-head-bundles SplunkdConnectionException Splunkd daemon is not responding: ("Error connecting to /servicesNS/katz.r/search/admin/search-head-bundles: ('The read operation timed out',)",) Refreshing admin/transforms-reload ResourceNotFound In handler 'transforms-reload': Invalid action for this internal handler (handler: transforms-reload, supported: list|_reload, wanted: list). I did this for a user that was already removed in the past and object ownership not transferred, but I am still getting this error below. When I search index=_internal sourcetype=splunkd log_level=ERROR I am still getting the error "0400 ERROR UserManagerPro - Failed to get LDAP user=(name of removed user) from any configured servers ... View more

Thank You, I found the folder and file, changed the ownership and I am working. Thanks again. ... View more

Thanks for your answer. This is a standalone instance and not a developer license and it is not expired. It is strange but som days ago it started to go up but not for the original license ). Have You got any other guess? Thanks. ... View more

I hope you found something for the actual routing in the time since you asked this, or would request to see any relevant events in your splunk.d log related to that config, but I also wanted to put a word of warning out there on TCP syslog forwarding from your indexers. If your syslog destination is down, what will happen? Is the IP you put in there actually a VIP that will always point to an active syslog destination? If not what I've seen happen in scenarios when a TCP syslog destination is down, Splunk continues to hold the data destined for it in it's internal queues. Over time, which is relatively short for a high volume of data the queues all fill up and eventually result in the indexer being blocked and unable to return search results. As the forwarders redirect to other indexers, they start taking the rest down too. While I hear there's also some settings that could change the behavior of whether splunk continues to hold the data in queue while waiting on the TCP response, we've also switched to UDP syslog forwarding to prevent a problem on the destination from taking out our indexing cluster again. ... View more

Make sure you also update the sslKeysfilePassword in the sslConfig stanza of the server.conf file in system/local with the default in plain text and let Splunk re-hash it with the new splunk.secret. I think starting with 6.5 the setting changes to sslPassword instead but I found that the upgrade did that conversion. That lets Splunk properly decrypt the passwords for the default certs it ships with that it uses for communications on the management port, which may affect use of the forwarder's REST API and its ability to communicate with the deployment server. ... View more