We have a set of UF in a private network that is totally isolated from the Deployment server. For forwarder to indexer traffic we will use intermediate forwarders however we would also like to utilize the deployment server. Is it possible to configure a UF to point to a deployment server through a proxy? ... View more

I'll start out by saying the collection of logs from eventhub via this add-on works fine. I am seeing events in the azure index and they seem to be coming in just fine, however there is a significant amount of errors in splunkd.log around the TA-MS-AAD app. Errors:   07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/async_ops/client_async.py", line 835, in _client_run_async 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" await self._connection.work_async() 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/async_ops/connection_async.py", line 139, in work_async 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" self._conn.do_work() 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/receiver.py", line 239, in _message_received 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" delivery_no=message_number) 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/message.py", line 99, in __init__ 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" self._parse_message_body(message) 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Message: 'Deallocating %r' 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Arguments: ('ArrayValue',) 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" --- Logging error --- 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Traceback (most recent call last): 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/lib/python3.7/logging/handlers.py", line 69, in emit 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" if self.shouldRollover(record): 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/lib/python3.7/logging/handlers.py", line 186, in shouldRollover 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" self.stream.seek(0, 2) #due to non-posix-compliant Windows feature 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" RuntimeError: reentrant call inside <_io.BufferedWriter name='/opt/splunk/var/log/splunk/ta_ms_aad_azure_event_hub.log'> 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Call stack: 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py", line 4, in <module> 07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" import azure_event_hub_core   I can't really pin-point what the errors may be, but this is driving up the amount of logs we're ingesting significantly, up to the point where its actually affecting out licenses.   Any idea? ... View more

I am trying to use invalid_replication_address to tell a cluster master running in front of an ELB to contact the indexer on a different address. However when i try to add the peer I get the following error on the CM: REST_Calls - app=search POST cluster/master/peers/ id=526D8BF5-7412-4934-AC47-08C699290CC9: active_bundle_id -> [14310A4AABD23E85BBD4559C4A3B59F8], add_type -> [Initial-Add], base_generatio n_id -> [0], batch_serialno -> [1], batch_size -> [2], buckets -> [], forwarderdata_rcv_port -> [9997], forwarderdata_use_ssl -> [0], indexes -> [], last_complete_generation_id -> [0], latest_bundle_id -> [14310A4AABD23E85BBD 4559C4A3B59F8], mgmt_port -> [8089], register_forwarder_address -> [], register_replication_address -> [https://10.0.7.181:8089], register_search_address -> [], replication_port -> [9887], replication_use_ssl -> [0], replicat ions -> [], server_name -> [ip-10-0-7-181.ca-central-1.compute.internal], site -> [default], splunk_version -> [8.0.2], splunkd_build_number -> [a7f645ddaf91], status -> [Up] INFO AdminManager - Setting capability.write=edit_indexer_cluster for handler clustermasterpeers. INFO AdminManager - Setting capability.read=edit_indexer_cluster for handler clustermasterpeers. DEBUG AdminManager - Validating argument values... DEBUG AdminManagerValidation - Validating rule='validate(len(name) < 1024, 'Parameter "name" must be less than 1024 characters.')' for arg='name'. ERROR ClusterMasterPeerHandler - Invalid host name https://10.0.7.181:8089 DEBUG AdminManager - URI /services/cluster/master/peers/?output_mode=json generated an AdminManagerExceptionBase exception in handler 'clustermasterpeers': Invalid host name https://10.0.7.181:80 89 INFO CMSlave - event=addPeer status=failure shutdown=false request: AddPeerRequest: { _id= _indexVec=''active_bundle_id=14310A4AABD23E85BBD4559C4A3B59F8 add_type=Initial-Add base_generation_id=0 batch_serialno=1 batch_size=2 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=14310A4AABD23E85BBD4559C4A3B59F8 mgmt_port=8089 name=526D8BF5-7412-4934-AC47 08C699290CC9 register_forwarder_address= register_replication_address=https://10.0.7.181:8089 register_search_address= replication_port=9887 replication_use_ssl=0 replications= server_name=ip-10-0-7-181.ca-central-1 compute.internal site=default splunk_version=8.0.2 splunkd_build_number=a7f645ddaf91 status=Up } 04-23-2020 02:03:56.478 +0000 ERROR CMSlave - event=addPeer start over and retry after sleep 12800ms reason addType=Initial Add Batch SN=1/2 failed. add_peer_network_ms=5 Notice how it says something regarding the name being less than 1024 characters and it possibly failing validation? The Cluster Master can "resolve" the IP ..although its an IP so see no reason why it should resolve it although the "null" cant resolve is weird.. I added a hostfile..no diffference: ` nslookup: can't resolve '(null)' Name: 10.0.7.181 Address 1: 10.0.7.181 ip-10-0-7-181.ca-central-1.compute.internal The Clustermaster can reach the Indexer on that port: Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 10.0.7.181:8089. ` Any reason why this happens? I've read a few posts and register_replication_address seems to be the solution to my problem however i am unsure why it is "unable to resolve hostname" *** UPDATE *** I also want to add here i've been doing more testing on some nodes that are just two EC2 instances with all traffic allowed between each other. nslookup on AWS for the IPs are fine and I still cannot get this working. If i remove register_replication_address in these cases it will work fine..this is really weird. Im not sure what the issue is or how to troubleshoot further if the log just says "invalid hostname" ... View more

The smartstore documentation says the following: "The amount of local storage available on each indexer for cached data must be in proportion to the expected working set. For best results, provision enough local storage to accommodate the equivalent of 30 days' worth of indexed data." Is this the same as HOT bucket data? or is it ontop of the hot data? e.g assuming the following factors: Intake = 100GB/day Compression ratio = 0.50 Hot Retention = 14 days Using this formula found in another forum post: Global Cache sizing = Daily Ingest Rate x Compression Ratio x (RF x Hot Days + (Cached Days - Hot Days)) Cache sizing per indexer = Global Cache sizing / No.of indexers Cached Days = Splunk recommends 30 days for Splunk Enterprise and 90 days for Enterprise Security Hot days = Number of days before hot buckets roll over to warm buckets. Ideally this will be between 1 and 7 but configure this based on how hot buckets rolls in your environment. 100 * .50 ( 2 x 14 + (30-14)) = 2200? ... View more