Hey, does anyone know any best practice or clever way of removing orphaned Knowledge Objects in a Search Head cluster when it is already too late for reassignment? For each orphaned object we are doing manual job like checking if AD accounts still exist, emailing the users and asking if they still need Splunk etc. For non-existing accounts, we delete /opt/splunk/etc/users<user_id> catalogue from each SH separately (there are 4 SHs in our cluster), but we are looking for more clever solution Unfortunately, there is no option in our case to be informed by the users that they are going to leave the company in order to react in advance and avoid orphaned KOs at all... Greetings, Justyna   ... View more

Hello All, We keep getting some errors from Splunk Add-on for Java Management Extensions 5.3.0: splunkd.log 1. ERROR ExecProcessor [20951 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" INFO: Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml 2. ERROR ExecProcessor [20951 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Mar 01, 2023 11:59:59 PM org.exolab.castor.mapping.Mapping loadMapping jmx.log 3. 2023-03-02 06:37:18,168 - com.splunk.modinput.ModularInput -1968729474 [Thread-11] INFO [] - Failed connection with 'service:jmx:rmi:///jndi/rmi://server1.domain.com:port/jmxrmi', trying to collect data with a short URL: 'service:jmx:rmi://server1.domain.com:port/jndi/jmxrmi' . 2023-03-02 06:37:18,168 - com.splunk.modinput.ModularInput -1968729474 [Thread-11] ERROR [] - Exception@checkConnector, e= java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[?:1.8.0_51] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[?:1.8.0_51] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229) ~[?:1.8.0_51] at com.splunk.jmx.ServerTask.connect(Unknown Source) ~[jmxmodinput.jar:?] at com.splunk.jmx.ServerTask.checkConnector(Unknown Source) ~[jmxmodinput.jar:?] at com.splunk.jmx.Scheduler.run(Unknown Source) ~[jmxmodinput.jar:?] Caused by: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662) ~[?:1.8.0_51] at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[?:1.8.0_51] at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350) ~[?:1.8.0_51] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_51] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957) ~[?:1.8.0_51] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924) ~[?:1.8.0_51] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_51] ... 5 more 2023-03-02 06:37:18,168 - com.splunk.modinput.ModularInput -1968729474 [Thread-11] WARN [] - Server misconfiguration already notified once for jmx://_Splunk_TA_jmx_:server1 stanza with message: Failed to connect with the JMX server. Review the configuration of the jmx://_Splunk_TA_jmx_:server1 stanza, and try again. 2023-03-02 06:37:18,174 - com.splunk.modinput.ModularInput -1968729480 [Thread-11] INFO [] - 1 servers found in stanza jmx://_Splunk_TA_jmx_:server1 Add-on has been configured on a Heavy Forwarder Our jmx_servers.conf is: [hostname] description = <description> destinationapp = Splunk_TA_jmx host = <FQDN> jmxport = <port> protocol = rmi stubSource = jndi disabled = 0 lookupPath = /jmxrmi Does anyone have any idea on how to fix it?  Greetings, Justyna ... View more

Hello Community, We are configuring TA-ms-teams-alert-action to let the customer publish Splunk alerts in their MS Teams channel. The alert config is as follows: When we try opening Webhook URL in browser for testing it says:  Invalid webhook request - GET not supported While debugging TA-ms-teams-alert-action the logs say: 2022-12-12 13:17:05,698 DEBUG pid=106480 tid=MainThread file=cim_actions.py:message:292 | sendmodaction - signature="json data for final rest call:={ "@type": "MessageCard", "@context": "http://schema.org/extensions", "themeColor": "0076D7", "summary": "Alert", "sections": [ { "activityTitle": "Alert", "activitySubtitle": "", "activityImage": "https://myimage.png", "facts": [ { "name": "host", "value": "hostname1" }, { "name": "sourcetype", "value": "splunkd" } ], "markdown": false } ], "potentialAction": [ { "@type": "OpenUri", "name": "View in Splunk", "targets": [ { "os": "default", "uri": "https://splunk.mydomain.com/app/TA-ms-teams-alert-action/@go?sid=scheduler__username123_1233456" } ] } ] }" action_name="ms_teams_publish_to_channel" search_name="test_alert" sid="scheduler__2username123_1233456" rid="0" app="TA-ms-teams-alert-action" user="username123" action_mode="saved" Does anyone know how to troubleshoot this case? Best regards, Justyna ... View more

Hello, We have noticed that in Monitoring Console-> Indexing-> Indexes and Volumes -> Indexes and Volumes: Deployment Dashboard, the "Oldest Data Age (days)" value for a few indexes is extremely high (e.g. 1959 days). Retention time for those indexes is 180 days (frozenTimePeriodInSecs = 15552000). We have checked the data and it really shows very old events (e.g. from 2017) although ignoreOlderthan = 14d parameter has been added to inputs.conf during data onboarding. We have already deleted the very old data (older than 400 days) with a "delete" command, but the Indexes and Volumes: Deployment Dashboard keeps showing very old values for the Anyone knows what else to check and how to solve this issue? BR, Justyna ... View more

Hello All, we have just found some pretty old catalogues and delta files in /opt/splunk/var/run/searchpeers on our indexers (3 indexers in the cluster, Splunk v. 8.2.9.). We have never deleted anything from that location as it always seemed to be a bit risky for us. At the same time the catalogues were modified in 2017 and seem to contain some unneeded and outdated stuff from already decommissioned servers. Is that safe to delete old catalogues in /opt/splunk/var/run/searchpeers? If so, should splunkd be stopped prior to deletion or should splunkd rolling restart be initiated after deletion as this is indexer cluster? Any hints would be much appreciated! BR, Justyna ... View more

Hello Splunkers, Splunk crashes on our Linux core servers (Master, Search Heads and Heavy Forwarders, indexers are not affected) with "ENGINE: Bus STOPPED" errors.  When I check splunkd status on host, it turns out that: se1234@z1il1234:~> /opt/splunk/bin/splunk status splunkd 130820 was not running. Stopping splunk helpers...Done. Stopped helpers. Removing stale pid file... done. It is an intermittent issue. Today in the morning we had this issue on SH1, yesterday on another SH2 and a week before on SH4. We have checked Resource usage and there is nothing shown in DMC that may cause splunkd to crash (e.g. high CPU usage) Any idea what is worth checking or how to troubleshoot? Greetings, Dzasta ... View more

Hello Splunk Admins, What solutions you use to get notified on mobile about internal Splunk issues in out of office hours? I mean when e.g. splunkd goes down on indexers, data is not indexed anymore for any reason etc. We need something free of charge. There is no other team except of us who needs to be notified about the issue. I have heard about Splunk On Call solution but seems to be a bit complex. Anyone having any experience with it? Hope to get some inspirations Many greetings, Justyna ... View more

Hello, We need to configure TA-mailclient, but having a separate account (username and password) to the mailbox. Do you know by any chance what parameter should we add to our inputs.conf?  Unfortunately, changing the format of mail to  [mail://<username>\<mailaddress>] or adding new parameter to the input stanza do not help.  W tried: [mail://thisisourmail.com] attach_message_primary = False host = host1 include_headers = True index = mail mailbox_cleanup = readonly mailserver = mailserver.zz maintain_rfc = False username = mail_username password = encrypted protocol = IMAP disabled = 0 and  [mail://mail_username\thisisourmail.com] attach_message_primary = False host = host1 include_headers = True index = mail mailbox_cleanup = readonly mailserver = mailserver.zz maintain_rfc = False password = encrypted protocol = IMAP disabled = 0 Best regards, Justyna   ... View more

Hello All, We are getting "Failed to load source for MC KPI Indicator visualization" error in Splunk Monitoring Console.  We tried to access it in another browser, cleared browser cache, restarted splunkd, but still some panels in DMC work and some show the above error. Does anyone have any idea on how to solve it? Many greetings, Justyna ... View more