Indexer was running normally yesterday. We offlined it, and after maintenance, rebooted it. When it came back up, it had a new IP because reasons, and joined the cluster with the new IP. After realizing what happened, and much troubleshooting with my NOC, they got the right IP in place and I offlined/rebooted again. Everything looked normal, but I'm seeing this error today: Search peer dc1prsplixap08 has the following message: Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=DC1PRSPLDP01:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=11.1.136.166 mgmtport=8089 (reason: Peer with guid=C395587E-CB3A-4492-8662-71AFD3002A89 is already registered and UP). Make sure pass4SymmKey is matching if the peer is running well. [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=F24FD19BC912B3FE530FB3917ED1B287 add_type=Initial-Add base_generation_id=0 batch_serialno=1 batch_size=20 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=F24FD19BC912B3FE530FB3917ED1B287 mgmt_port=8089 name=C395587E-CB3A-4492-8662-71AFD3002A89 register_forwarder_address= register_replication_address= register_search_address= replication_port=9000 replication_use_ssl=0 replications= server_name=dc1prsplixap08 site=default splunk_version=6.6.0 splunkd_build_number=e21ee54bc796 status=Up } ]. Linux, Splunk version 6.6.3 ... View more

Spent all day yesterday trying to figure out why a client's logs weren't indexing. Most of the time I had no access to the server in question, so I was simply troubleshooting from internal logs, configs, and the sporadic logs that would show up briefly after a restart. Finally, when I was just about to throw in the towel, I started poking around directories above the target files. The monitor line had an asterisk at this point in the path, so even though most other dirs didn't match further down the line, Splunk had to check them. Several of them had 100k+ files in them. So Splunk was stuck trying to read these dirs. Even performing just an ls | wc -l took over 10 minutes on a few of them. I can find big directories with something like this and send it into Splunk for alerting: find /path -size +100k -type d Adjusting the size requirement as needed. Is there a better way to avoid these landmines? Thanks, Jon ... View more