if you're seeing the corrupt only on hot buckets, you need to change the SPL to retrieve. |dbinspect index=* corruptonly=true | search state!=hot Or |dbinspect index=_internal corruptonly=true | search NOT state=hot The reason we exclude hot buckets is because  the status of those hot buckets are in transient and they are still being updated(written). ... View more

To become a certified Splunk developer, ​you have to complete ​CreatingDashboards​, A​dvancedDashboards&Visualizations​ ,​BuildingSplunkApps​, and Developing with Splunk’s REST API​ courses in order to be eligible for the certification exam. https://www.splunk.com/en_us/training/learning-path/courses-for-app-developers/overview.html ​The prerequisite exams for this certification are Splunk Core Certified User, Splunk Core Certified Power User, and Splunk Enterprise Certified Admin. Once you get the authorization for the Developer exam from Splunk Certification team, then you'll be able to apply for the Exam via Pearson VUE Additional information can be found here related to the online proctored option: https://www.splunk.com/pdfs/training/Online-Proctored-Exam-Delivery-Overview.pdf Hope it helps. ... View more

Based on the description, the sessionTimeout should be configured longer than the search runtime. Can you try sessionTimeout=2h? Under server.conf [general] sessionTimeout = [s|m|h|d] * The amount of time before a user session times out, expressed as a search-like time range. * Examples include "24h" (24 hours), "3d" (3 days), "7200s" (7200 seconds, or two hours) * Default: "1h" (1 hour) Also, It's worth while considering to increase timeout value below: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Webconf server.socket_timeout = * The timeout, in seconds, for accepted connections between the browser and Splunk Web * Default: 10 export_timeout = * When exporting results, the number of seconds the server waits before closing the connection with splunkd. * If you do not set a value for export_timeout, Splunk Web uses the value for the 'splunkdConnectionTimeout' setting. * Set 'export_timeout' to a value greater than 30 in normal operations. * No default. Splunk will be logging the REST API export activities in the splunkd_access.log Hope it help. ... View more

If the Splunkd Service is running but unable to start web service in Windows or Linux Check if the web-port configured is open and listening on the instance(sometimes another process is bound to the port) Check if any firewall rule has been enabled to block Check if Certificates and SSL is working as expected then try to restart with disabled WebSSL under web.conf to isolate the issue [settings] enableSplunkWebSSL = false Check if any AV(Antivirus) product is running on the server. https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/RunningSplunkalongsideWindowsantivirusproducts When you run Splunk (Enterprise or forwarder) on a host that has an anti-virus product (such as McAfee VirusScan on Windows or Trend Micro ServerProtect on Linux) installed, Splunk strongly recommends that you exclude all Splunk processes as well as the Splunk installation directory from any kind of on-access scanning. Splunkd is running but once you try to restart the web service via CLI, you might come across below response: C:\Program Files\Splunk\bin>splunk restart splunkweb 502 Couldn't complete HTTP request: Winsock error #10061 Or A server is busy message on HttpListener channel in splunkd log Hope it helps. ... View more

If one of the Indexers ran out-of-space, It may result in incomplete writes to .bucketSummaryManifest in the ../your_index/summary Or ../your_index/datamodel_summary paths. The crash could happen while Splunk reads the corrupt file. As a workaround: Run below command to identify possible corrupted files/lines (run from the splunk/var/lib/splunk): grep -vPn '\"[^\"]+\",\"[^\"]+\",\"[^\"]+\",\d+,\d+,\d+' */summary/.bucketSummaryManifest grep -vPn '\"[^\"]+\",\"[^\"]+\",\"[^\"]+\",\d+,\d+,\d+' */datamodel_summary/.bucketSummaryManifest Move the .bucketSummaryManifest file to temp folder outside Splunk then restart the indexer Confirm that a new .bucketSummaryManifest file is created and keep eyes on your indexer in a problem to see for any additional crash. Also, we have been fixed that issue 7.0.0 onwards via SPL-141877 Hope it helps. ... View more

What is your H/W specification for the search head? Have you checked if the search head is running into Out-Of-Memory once the issue happens? If you still see below Error: "StateStoreError: 'Batch save to KV store failed with code 400. Error details: Request exceeds API limits - see limits.conf for details. (Batch save size=52439798 too large)' " You may need to increase the max_size_per_result_mb for the [kvstore] in limits.conf https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkvstore.5D max_size_per_result_mb = * The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection. * Default: 50 If you see the error during ITSI operation, Please see our doc regarding the size limit: https://docs.splunk.com/Documentation/ITSI/latest/Configure/CreateKPIbasesearches#Increase_the_KV_store_bulk_get_limit Hope it help and see how it works for you. ... View more

Hi euimok, The enhancement request does not be published as It's an internal tracking so you won't be able to find it on the public, unfortunately. but if you have a right entitlement, you will be able to chase it up with your account manager how it goes. Hope it helps. ... View more

After enabling auto-start under systemd : no issue here $SPLUNK_HOME/bin/splunk enable boot-start -user splunk $SPLUNK_HOME/bin/splunk start Starting splunk via systemctl from the root user works as expected Starting splunk as per the doco ($SPLUNK_HOME/bin/splunk start) as below, https://docs.splunk.com/Documentation/Splunk/latest/Admin/RunSplunkassystemdservice#Configure_systemd_using_enable_boot-start To start splunkd. [sudo] $SPLUNK_HOME/bin/splunk start This starts splunkd as a systemd service. Getting into the following: Stopped helpers. Removing stale pid file... done. Splunk> The Notorious B.I.G. D.A.T.A. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-7.2.3-06d57c595b80-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to start 'Splunkd.service'. Authenticating as: root Password: Once you're running into the issue, you'll be able to get the Splunk started with below workaround: Under the path for SLES 12, /etc/polkit-1/rules.d, making a rule for Splunk user and org.freedesktop.systemd1.manage-units as below: cat /etc/polkit-1/rules.d/10-splunk.rules polkit.addRule(function(action, subject) { if(action.id == "org.freedesktop.systemd1.manage-units" && subject.user == "splunk") { return polkit.Result.YES; } }); It would allow the Splunk service to start as normal. In addition, Splunk will be working further under SPL-164816, which systemd configuration on SLES prompts root password when starting for the fix. Stay tuned. ... View more

If you see below errors from the output of health check scripts, Errors: kubelet journalctl: kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach kubelet_node_status.go:82] Attempting to register node YOUR_FQDN_SERVERNAME kubelet_node_status.go:106] Unable to register node "YOUR_FQDN_SERVERNAME" with API server: nodes "YOUR_FQDN_SERVERNAME" is forbidden: node "short_hostname" cannot modify node "YOUR_FQDN_SERVERNAME" kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach kubelet_node_status.go:82] Attempting to register node YOUR_FQDN Output of health check: concern summary: Checking YOUR_FQDN_SERVERNAME first ... McAfee_ePO | Splunk | Processing | | SPLUNK/DIRECT | 2019-01-15 05:18:10 | 0 | 0 | 0 | 949874 | 0 | 485746 | 49 | <== significant failed/skipped events; review datasource SPL eps: 0 <== no response from redis 'YOUR_FQDN' status not OK ... <== check UI system health monitor for errors splunkuba analyticsaggregator-rc-v6csh 0/1 Pending 0 14m <none> <none> <== pod 'analyticsaggregator-rc-v6csh' is 'Pending'; not 'Running' splunkuba analyticsviewsbuilder-rc-nwffg 0/1 Pending 0 14m <none> <none> <== pod 'analyticsviewsbuilder-rc-nwffg' is 'Pending'; not 'Running' splunkuba analyticswriter-rc-jscjf 0/1 Pending 0 14m <none> <none> <== pod 'analyticswriter-rc-jscjf' is 'Pending'; not 'Running' splunkuba anomalyaggregationmodel-rc-j968d 0/1 Pending 0 14m <none> <none> <== pod 'anomalyaggregationmodel-rc-j968d' is 'Pending'; not 'Running' splunkuba devicetopic-modelgroup01-rc-d5ljq 0/1 Pending 0 14m <none> <none> <== pod 'devicetopic-modelgroup01-rc-d5ljq' is 'Pending'; not 'Running' splunkuba devicetopic-modelgroup01-rc-z5mzc 0/1 Pending 0 14m <none> <none> <== pod 'devicetopic-modelgroup01-rc-z5mzc' is 'Pending'; not 'Running' splunkuba domaintopic-modelgroup01-rc-8pvcx 0/1 Pending 0 14m <none> <none> <== pod 'domaintopic-modelgroup01-rc-8pvcx' is 'Pending'; not 'Running' splunkuba domaintopic-modelgroup01-rc-mmzhw 0/1 Pending 0 14m <none> <none> <== pod 'domaintopic-modelgroup01-rc-mmzhw' is 'Pending'; not 'Running' . . . The problem would be the way the hostnames are set with FQDN: i.e. hostnamectl status: Static hostname: YOUR_FQDN_SERVERNAME e.g: /etc/hosts: (on Master Node) 127.0.0.1 localhost Ip_address YOUR_FQDN_SERVERNAME(e.g uba1.splunk.com) Ip_address YOUR_FQDN_SERVERNAME(e.g uba2.splunk.com) Ip_address YOUR_FQDN_SERVERNAME(e.g uba3.splunk.com) To resolve or rectify the issue: 1.Check the current status of the containers sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes -o wide --all-namespaces sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -o wide --all-namespaces 2.Stop containers and services /opt/caspida/bin/Caspida stop-containers /opt/caspida/bin/Caspida stop-container-services #command not in 4.1, added in 4.2 3.Check the current status of the containers sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes -o wide --all-namespaces sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -o wide --all-namespaces 4.Stop kubelet and docker on all nodes sudo service kubelet stop && sudo service docker stop ssh uba1 "sudo service kubelet stop && sudo service docker stop" ssh uba2 "sudo service kubelet stop && sudo service docker stop" 5.Check the current status of the containers sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes -o wide --all-namespaces sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -o wide --all-namespaces 6.Update hostnames to have shortnames sudo hostnamectl set-hostname uba1 ssh uba2 "sudo hostnamectl set-hostname uba2" ssh uba3 "sudo hostnamectl set-hostname uba3" 7.Restart kubelet and docker on all nodes sudo service docker start && sudo service kubelet start ssh uba2 "sudo service docker start && sudo service kubelet start" ssh uba3 "sudo service docker start && sudo service kubelet start" 8.Start containers and services /opt/caspida/bin/Caspida start-container-services #command not in 4.1, added in 4.2 /opt/caspida/bin/Caspida start-containers 9.Check the current status of the containers sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes -o wide --all-namespaces sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -o wide --all-namespaces You'll be able to check the status pending to running now. ... View more

Tested on SHC which has 3 members: To change the default kvstore path to a different mount point in SHC, below steps can be doable. Stop the service on one of search head member(not kvstore primary) Repoint the kvstore path to a newly created empty directory with write permission (http://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverconf?utm_source=answers&utm_medium=in-answer&utm_term=server.conf&utm_campaign=refdoc#KV_Store_configuration) Start the service Check if resync is working. then repeat 1-4 for the next member Above steps worked. ... View more

In case the search head is also the part of indexer clusters and needs talk to multiple cluster masters which has lower version than the search head on it, it could happen. e.g the higher version of search-head(7.1) cannot be talking to 7.0.2 cluster masters as expected or properly. The search head thinks a rolling restart is happening because the messages it gets from the 7.0.2 cluster masters is missing crucial data. We will work in improving the error message the search head reports(…skipped during the searchable rolling restart or upgrade) so we can have better messages or clear understanding on it. http://docs.splunk.com/Documentation/Splunk/7.1.0/Indexer/Systemrequirements#Splunk_Enterprise_version_compatibility Splunk Enterprise version compatibility Interoperability between the various types of cluster nodes is subject to strict compatibility requirements. In brief: The master node must run the same or a later version from the peer nodes and search heads. The search heads must run the same or a later version from the peer nodes. The peer nodes must all run exactly the same version, down to the maintenance level. ... View more

How CC certification evaluation goes? Has Splunk Enterprise 6.4.0 got the certification? It's gone away from NIAP website ... View more

Has there been any progress for the CC certification since then ... View more