Hello Splunk Admins, What solutions you use to get notified on mobile about internal Splunk issues in out of office hours? I mean when e.g. splunkd goes down on indexers, data is not indexed anymore for any reason etc. We need something free of charge. There is no other team except of us who needs to be notified about the issue. I have heard about Splunk On Call solution but seems to be a bit complex. Anyone having any experience with it? Hope to get some inspirations Many greetings, Justyna ... View more

Hello, We need to configure TA-mailclient, but having a separate account (username and password) to the mailbox. Do you know by any chance what parameter should we add to our inputs.conf?  Unfortunately, changing the format of mail to  [mail://<username>\<mailaddress>] or adding new parameter to the input stanza do not help.  W tried: [mail://thisisourmail.com] attach_message_primary = False host = host1 include_headers = True index = mail mailbox_cleanup = readonly mailserver = mailserver.zz maintain_rfc = False username = mail_username password = encrypted protocol = IMAP disabled = 0 and  [mail://mail_username\thisisourmail.com] attach_message_primary = False host = host1 include_headers = True index = mail mailbox_cleanup = readonly mailserver = mailserver.zz maintain_rfc = False password = encrypted protocol = IMAP disabled = 0 Best regards, Justyna   ... View more

Hello All, We are getting "Failed to load source for MC KPI Indicator visualization" error in Splunk Monitoring Console.  We tried to access it in another browser, cleared browser cache, restarted splunkd, but still some panels in DMC work and some show the above error. Does anyone have any idea on how to solve it? Many greetings, Justyna ... View more

Hello Splunkers, We configured Splunk Add-on for VMware ESXi Logs on one of our Heavy Forwarders as in: https://docs.splunk.com/Documentation/AddOns/released/VMWesxilogs/Install However, we can see a huge bunch of wrongly extracted sourcetypes: e.g.: vmware:esxlog:-- vmware:esxlog:ERROR vmware:esxlog:INFO vmware:esxlog:NoneZ vmware:esxlog:WARNING vmware:esxlog:a vmware:esxlog:a-cli-info-python vmware:esxlog:a-dabc vmware:esxlog:a-e vmware:esxlog:a-vsan-task-tracker vmware:esxlog:ab vmware:esxlog:abc vmware:esxlog:af vmware:esxlog:althSystemImpl We tried to add additional regex in set_syslog_sourcetype in transforms.conf, but then the events stopped coming in at all. Our config files are as follows (all on Heavy Forwarder): inputs.conf [monitor:///opt/splunk/var/log/remote/syslog-tlxfr*.log] disabled = false index = vmware-esxilog sourcetype = vmw-syslog props.conf [vmw-syslog] TRANSFORMS-vmsysloghost=set_host TRANSFORMS-vmsyslogsourcetype = set_syslog_sourcetype MAX_TIMESTAMP_LOOKAHEAD = 20 transforms.conf [set_host] REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+([^ ]+)\s+) DEST_KEY = MetaData:Host FORMAT = host::$1 [set_syslog_sourcetype] REGEX = ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<\d+>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+([A-Za-z\-]+)(?:[^:]*)[:\[] DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::vmware:esxlog:$1 [esx_hostd_fields_6x] REGEX = ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<(\d+)>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+([^\[\:]+):\s(?:(?:[\d\-:TZ.]+)\s*)?(\w+)\s*(?:\S+\[\S+\])?\s*\[(?:[^\s\]]+)\s*(?:sub=([^\s\]]+))?\s*(?:opID=([^\s\]]+))?(?:[^]]+?)?\]\s*(.*)$ FORMAT = Pri::$1 Offset::$2 Application::$3 Level::$4 Object::$5 opID::$6 Message::$7 Does anyone have any idea how to solve it? Seems to be simple, but we are stuck Greetings, Justyna ... View more

Hello, After Splunk_TA_snow upgrade from 6.0.0. to 6.4.1 we have an issue with data collection. When we try to reenter password for snow account in the add-on, we keep getting:  “Unable to reach server at https://xyz.service-now.com. Check configurations and network settings.” We upgraded the add-on to the newest version (7.0.0) and tried out some solutions from https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Troubleshooting#Troubleshoot_the_Splunk_Add-on_for_ServiceNow but still cannot update password in snow account. Also, we have noticed following socket timeout errors: 2021-06-29 16:47:03,365 ERROR pid=67397 tid=Thread-2 file=snow_data_loader.py:_do_collect:298 | Failure occurred while connecting to https://xyz.service-now.com/api/now/table/cmdb_ci?sysparm_display_value=all&sysparm_offset=0&sysparm_limit=1000&sysparm_exclude_reference_link=true&sysparm_query=sys_updated_on>=2020-12-15+21:08:41^sys_updated_on<2021-06-29+14:46:01^ORDERBYsys_updated_on,sys_id. The reason for failure=Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/snow_data_loader.py", line 254, in _do_collect "Authorization": "Basic %s" % credentials File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1959, in request File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1622, in _request File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1528, in _conn_request File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1309, in connect socket.timeout: timed out Do you have any idea how to solve that? ... View more

Hi Guys, we keep getting following errors in Splunk 8.1.2: ERROR ApplicationUpdater - Error reloading <appname> : handler for server ( http_post /replication/configuration/whitelist-reload ) : Configuration from app=<appname>   does not support reload: server.conf/ [ clustering ] /master_uri The app was created manually long time ago and defines basic configuration for our Search Head Cluster. It is installed on the Master server that we use to deploy changes to our Search Head Cluster. We hoped it is just a typo so we double checked server name in [ clustering ]  master_uri but that is not the case. The whole server.conf in <appname> is as follows:  [license] master_uri = https://<license_master_server>:8089 [clustering] master_uri = https://<master_server>:8089 mode = searchhead pass4SymmKey = xyz [replication_port://8815] [shclustering] conf_deploy_fetch_url = https://<master_server>:8089 disabled = 0 pass4SymmKey = xyz replication_factor = 2 shcluster_label = Splunk_AB id = xyz Do you have any idea what may is wrong here? Your help will be much appreciated!  Greetings!         ... View more

Hello everyone, After adding some data to Splunk and running apply cluster bundle & apply shcluster bundle commands, splunkd stopped working on one of our Search Heads (out of 2 in the cluster). We turned it on manually, but then it switched itself to a restart loop. We have managed to solve the case with adjusting  'conf_deploy_fetch_mode' as suggested here:   https://community.splunk.com/t5/Deployment-Architecture/Why-is-the-Search-Head-Cluster-getting-Caught-in-the-Restart/m-p/337615  However, we are still looking for the exact reason of this issue.  We have found the following internal errors on the affected SH: 04-15-2021 13:15:17.244 + 0200  ERROR TcpInputProc - Error encountered for connection from src=myipaddress:port. Local side shutting down 04-15-2021 13:15:30.187 + 0200 ERROR IndexAdminHandler - already reloading or shutting down , will not reload Could someone please help me out interpreting them? Thank you!  ... View more

Hello everyone, Could you please help me out with the following query? We have a TA-Okta_Identity_Cloud_for_Splunk installed on a Heavy Forwarder. Our customer receives   “skinny_user”  rate limit warnings from  /api/v1/apps endpoint. Following Okta documentation, it is suggested to change a limit to the default value of 20: https://www.okta.com/integrate/documentation/security-enforcement-integrations/security-analytics/#apps-26    However, when I check limits in our Add-On, the ranges of  User, Group, App and Log Limits (min/max values) are different from the ones in Okta documentation.  Could you please help me finding the right limit which I should adjust in Splunk? Thank you! ... View more

We want to anonymize the usernames in the following event using sed script. Raw event: {" externalId " : null , " statusChanged " : " 2021-02-09T09:51:42.000Z ", " userName " : " thisisatestuser@123.pl  ", " appid " : " 0oa97rn2ymPrbgM430x612344 ", " lastUpdated " : " 2021-02-09T09:51:42.000Z ", " scope " : "", " userid " : " 00u1a2mn8ouOvSO2A0x71234 ", " created " : " 2021-02-09T09:51:42.000Z ", " status " : " ACTIVE "} However, either the stanza in props.conf is incorrect or we did it in the wrong place. Current config: 1. input is specified on a Heavy Forwarder in an app local folder (we just activated inputs after installing an app in the FrontEnd and that is how this entry was created) 2. we added the following SEDCMD stanza in /opt/splunk/etc/system/local/props.conf on a Heavy Forwarder:  [testapp:appUser] SEDCMD-TestApp_username1=s/userName:(\d{6})/userName:XXXXXX/1 Does anyone have any idea on how to solve it? ... View more