I've created tokens with local authentication and shared the ID and password with the users of the applications needing access. So a single userid, mapped to a role, works for multiple users running Grafana, in this case. I'd like to do the same thing using SAML. Is it possible (and reasonable) to create a "generic token", for lack of a better phrase, and provide that to the custom dashboard users? It may still be Grafana, or maybe some other mechanism to yet be determined, passing a search string to splunk.  For example, if I had 10 users to configure to use API access, it would seem that I'd need to generate 10 tokens, based on what I get out of the documentation. Can anyone direct me? Thank you in advance.   ... View more

I need help troubleshooting an issue where I am missing events being forwarded from a linux syslog daemon to my heavy forwarders. Beginning the first day of each month, for three or four days, this feed drops from ~50,000 indexed events per hour to maybe ~150. Then, magically, the feed resumes ~50,000 events per hour for the remainder of the month. Only this one index source is affected. All traffic is UDP. To troubleshoot: I've removed the load balancer from the equation and send directly to one heavy forwarder We can see the syslog events leaving the source server  Using tcpdump, I can see events from the source server hitting port 514 on the heavy forwarder I have a dashboard showing blocking on agg, index, parsing and typing queues. There is none. I tested the regexs in my transforms on the actual events captured with tcpdump. All test correctly. While this event was in progress Opened a support case with diag logs from the HWF and one of my indexer servers (nothing yet) There are no errors or warnings in the internal logs for the heavy forwarder used in this test.  I've looked at all the log channels on the HWF (1236 of them) but I don't know which one(s) to elevate the logging level for I tried starting splunk with --debug but I do not see any additional internal logging. I may not have done this correctly. (splunk start --debug) More strange There are two syslog feeds from the source server being used in this troubleshooting effort. The second feed is unaffected. There are 78 source servers in this group. All exhibit the same behavior making it seem that splunk is the common denominator.  I do use a props and transforms configuration for port 514 to parse the index name and sourcetype for a multitude of incoming syslog feeds bound for different indexes. This configuration has not changed for a very long time (and does not change at the first of the month - for a few days). Frankly I'm lost. There must be a way to expose what is happening to these events either at the heavy forwarder or on the indexers but I'm out of ideas. Does anyone have a thought about how I might capture the information I need to diagnose whatever is happening? At this time, the feed has returned to normal i.e. ~50,000 indexed events per hour. Thank you in advance for any advice you have.      ... View more

My security people want me to use an openJDK product from Azul Systems. I get an error when saving the configuration settings {'jre_need': 'Need Oracle Corporation JRE version 1.8 or OpenSDK 1.8', 'jre_using': 'Using Azul Systems, Inc. JRE version 1.8, OpenJDK 64-Bit Server VM', 'message': 'Unsupported JRE detected'} validate java command: /opt/splunk-nexusJDK/zulu8.44.0.10-sa-jdk8.0.242-linux_x64/jre/bin/java. Azul suggested this override in JVM options "-XX:+OverrideVMProperties -Djava.vm.vendor="Oracle Corporation" ". That didn't work either. DB_Connect works fine with standard Oracle Java and Oracle's openJDK Has anyone been successful using an alternative java engine like this? Thanks in advance. ... View more

I have a need to determine, over time, the total size of the buckets rolling to cold regardless of which index. What I think I would like to show is a simple timechart by hour or minute. I've been looking for examples to ride other folks coattails and looked at what is in the metrics log but have not found anything useful yet. Has anyone done this and could you give me a hand? Thank You. ... View more

One of my splunk users has set up an alert that utilizes the "send to new relic" alert action. When it runs, I see the following --in ascending sequence. Can anyone tell me what error code 200 is? Thank you for your consideration. INFO sendmodalert - Invoking modular alert action=alerts_to_newrelic for search=...... INFO sendmodalert - action=alerts_to_newrelic - Alert action script completed in duration=5462 ms with exit code=200 WARN sendmodalert - action=alerts_to_newrelic - Alert action script returned error code=200 ... View more

I'm testing the rest api modular input with Omnicenter. Simple GET requests are successful but I'm trying to use POST now and get a "Failure: Missing required information" response. I see nothing in the log in splunk or omnicenter to help me determine what is missing. Anyone know if there Is a way to see what is being issued on the request? a cURL with this same POST returns the information I am looking for. ... View more

Today, I use CA signed certificates on all search head cluster members. These members are behind a load balancer. The load balancer DNS name and the unique host name (per server) are present in the subject alternative name. The server certificate is part of a chain along with Intermediate and Root certificates. This all seems to work fine. I need to configure SAML, and have, for a single member. I can't make heads or tails of the "configuring SAML in a search head cluster" doc. According to the doc, there is a common "signing certificate" I need to copy to the other members. What is this? Has anyone had experience with SAML configuration in a search head cluster? Your thoughts are appreciated. ... View more