I am running some diags to isolate some problems but I'm getting the following error message:   LSOF   Failed Result Message Could not detect `lsof` for the scottj1 user. Process splunkd - splunkd server (32362)   The application is on the host and it's in my .bashrc file.  When I run the command "which lsof" over ssh it correctly finds it.  What might be the problem here? ... View more

I just set up a heavy forwarder with the AWS add-on.  I launched the app and went to the configuration page but all I get is the spinning 'Loading' icon. I looked over the previous question like this: https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-Add-on-for-AWS-Hangs/m-p/316801   But I'm not using any Google services at all nor do I have any file "/etc/boto.cfg ".   When I look through the Splunkd logs I see this:   +0000 ERROR AdminManagerExternal [1021 TcpChannelThread] - Stack trace from python handler: Traceback (most recent call last): File "/home/y/var/splunk/lib/python3.7/site-packages/splunk/admin.py", line 114, in init_persistent hand.execute(info) File "/home/y/var/splunk/lib/python3.7/site-packages/splunk/admin.py", line 637, in execute if self.requestedAction == ACTION_LIST: self.handleList(confInfo) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_rh_settings.py", line 58, in handleList entity = client.Entity(self._service, uri % (service, LOGGING_ENDPOINTS[service])) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 900, in __init__ self.refresh(kwargs.get('state', None)) # "Prefresh" File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 1039, in refresh self._state = self.read(self.get()) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 1009, in get return super(Entity, self).get(path_segment, owner=owner, app=app, sharing=sharing, **query) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/client.py", line 766, in get **query) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 290, in wrapper return request_fun(self, *args, **kwargs) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 71, in new_f val = f(*args, **kwargs) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 680, in get response = self.http.get(path, all_headers, **query) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 1184, in get return self.request(url, { 'method': "GET", 'headers': headers }) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 1242, in request response = self.handler(url, message, **kwargs) File "/home/y/var/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunklib/binding.py", line 1383, in request connection.request(method, path, body, head) File "/home/y/var/splunk/lib/python3.7/http/client.py", line 1277, in request self._send_request(method, url, body, headers, encode_chunked) File "/home/y/var/splunk/lib/python3.7/http/client.py", line 1323, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/home/y/var/splunk/lib/python3.7/http/client.py", line 1272, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/y/var/splunk/lib/python3.7/http/client.py", line 1032, in _send_output self.send(msg) File "/home/y/var/splunk/lib/python3.7/http/client.py", line 972, in send self.connect() File "/home/y/var/splunk/lib/python3.7/http/client.py", line 1439, in connect super().connect() File "/home/y/var/splunk/lib/python3.7/http/client.py", line 944, in connect (self.host,self.port), self.timeout, self.source_address) File "/home/y/var/splunk/lib/python3.7/socket.py", line 728, in create_connection raise err File "/home/y/var/splunk/lib/python3.7/socket.py", line 716, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused Which looks like it's trying to autoconfigure itself like mentioned in:https://docs.splunk.com/Documentation/AddOns/released/AWS/Setuptheadd-on#Find_an_IAM_role_within_your_Splunk_platform_instance   But this is not running in AWS so I planned to manually configure it.  However I cannot because of the above problem.  How do I fix this?   Splunk-8.2.2 Splunk AWS add-on 5.2.0 ... View more

Is there a way to configure the management port, which is being used to access the REST API, to use the TLS certs we have from DigiCert?   I have server.conf set up with serverCert pointing to the location of that file.  However, when I check using openssl on that port we get the certificate that ships with Splunk despite setting up server.conf to use the certificate from DigiCert.   Note, web access does correctly use the DigiCert certificate.  It's just access over the managment port doesn't. ... View more

Why am I getting this error message "Signature mismatch between license slave and this License Master" where the "license slave" is the same host?  And I have the machine already configured to use a remote license server.  I've checked the server.conf file and it's set up correctly and matches all the other hosts that are not showing this error. ... View more

I have a similar situation as the question "Splunk Offline command - running for hours" however in my case I have several indexers which have been running the offline --enforce-counts command for days. One was started last Friday so it's been a week for it. When I check splunkd.log I can still see it copying buckets. For example, 05-29-2020 14:02:01.562 +0000 INFO DatabaseDirectoryManager - idx=main Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/main/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1' There are also a huge number of entries like this: 05-29-2020 14:45:05.923 +0000 WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user (In splunkd.log 1911 entries for 1st host, 1256 entries for 2nd host, 1277 for 3rd host that has been running for a week, 1226 entries for 4th host) 05-29-2020 14:45:53.476 +0000 ERROR SearchProcessRunner - launcher_thread=0 runSearch exception: PreforkedSearchProcessException: can't create preforked search process: Cannot send after transport endpoint shutdown ( In splunkd.log 19962 entries for 1st host, 20273 entries for 2nd host, 1829 for 3rd host that has been running for a week, 19101 entries for 4th host) And on the one where it's been running for a week: 05-29-2020 14:43:33.464 +0000 WARN DistBundleRestHandler - Failed to find data processor for endpoint=full-bundle 05-29-2020 14:44:26.520 +0000 WARN ReplicatedDataProcessorManager - Failed to find processor with key=delta-bundle since no such entry exists. 05-29-2020 14:44:26.520 +0000 WARN BundleDeltaHandler - Failed to find data processor for endpoint=delta-bundle (3092 total entries for both in splunkd.log) I see in the master Indexer Clustering dashboard that they are still decommissioning (although I don't know what the Buckets entry indicates. The number of buckets left to replicate?) All the indexers are running version 8.0.1 with the exception of a handful in the cluster that are not being decommissioned that have been upgraded to 8.0.3. The Master indexer is still 8.0.1 What do I do to speed this up? There was no solution posted in the other question. ... View more

I was wondering if anyone had a good solution for a proper source type for dmesg? Or failing that some way of handling the fact it is different than most other logs in that entries aren't always single lines, and the timestamps are relative to system boot. That makes it difficult for the indexers to assign a time stamp for the entries. ... View more

I've got a search head cluster running and have a host that I've set as the cluster captain. Other than the configuration settings for it being the cluster captain it is set up like all the other hosts in the cluster. But I cannot load the web page for it and I can load it for every other search head. Instead I get a "404" error despite the boot showing that the web interface has come up just fine. Splunk> Winning the War on Error Checking prerequisites... Checking http port [4443]: open Checking mgmt port [5500]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-7.3.1.1-7651b7244cf2-linux-2.6-x86_64-manifest' All installed files intact. Done Checking replication_port port [8080]: open All preliminary checks passed. Starting splunk server daemon (splunkd)... Done [ OK ] Waiting for web server at https://127.0.0.1:4443 to be available............. Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at https://splunk-search-lead:4443 Any idea why this would be happening? I've deleted the installation twice and reinstalled from scratch and the same outcome happens each time. This is the first time using the latest version of Splunk. Our previous installations ran 7.1.2 and we never had this problem at all using the same deployment steps. This is also on new hosts so there's no previous installation on these hosts either. ... View more