Splunk Enterprise Version: 10.2.2 Has anyone encountered an issue where the Deployment Server Agent Management page shows "Agent Management Unavailable"? I have two identical Splunk Deployment Servers deployed in separate environments. Both were configured the same way, but only one of them is experiencing this issue. A few things I've already verified: I did not configure Distributed Search on the Deployment Server during the initial setup. As part of troubleshooting, I explicitly set the Distributed Search option to No, but the Agent Management page still fails to load. I also removed the serverclass.conf file that was originally created through the Splunk Web UI and restarted Splunk, but the issue persists. I have verified that serverclass.conf does not contain any syntax errors. The Agent Management page was working previously, and I only noticed this issue recently. Despite the Agent Management page being unavailable, the Deployment Server itself appears to be functioning normally. Universal Forwarders can still successfully phone home, retrieve deployment apps, and receive configuration updates without any issues. Has anyone experienced something similar or knows what could cause the Agent Management page to become unavailable while the Deployment Server continues to function normally? ... View more

Hi, I understand that ports below 1024 are reserved for root access. Is there any supported way for Splunk to listen on UDP port 514 while still running under the splunk user? I’ve tried starting Splunk with: sudo $SPLUNK_HOME/bin/splunk start --run-as-root as well as changing ownership/permissions of the relevant directories to root, but none of these approaches worked. If anyone has successfully configured Splunk to listen on UDP 514 without using port redirection, I’d really appreciate it if you could share the steps. Thanks ... View more

Hi, I’d appreciate it if someone could confirm whether my understanding is correct. My objective is to forward incoming logs from specific devices (based on IP) via TCP:514 to a third-party system, while still allowing the Heavy Forwarder to forward all logs to the indexer. At the moment, I’m unable to verify the raw logs directly. However, I was informed that the logs include the device’s own IP, which is why I configured a REGEX in transforms.conf to match the IP and forward only those logs to the third-party system. With the current setup, though, all logs are being forwarded to the third-party destination instead of just the targeted IP. Below are the configurations I applied across the three conf files.   #outputs.conf [tcpout] defaultgroup = indexer indexAndForward = 0 [tcpout:indexer] server = indexer_ip:8089 [syslog] defaultGroup = thirdparty [syslog:thirdparty] server = thirdparty_ip:514 # props.conf [source::tcp:514] TRANSFORMS-route = route_to_thirdparty # transforms.conf [route_to_thirdparty] REGEX = 192\.168\.68\.(68|74) DEST_KEY = _SYSLOG_ROUTING FORMAT = thirdparty ... View more

Hi, I recently took over a Splunk environment and would like to clarify the following. My objective is to forward all Splunk instance logs to the indexer through the Heavy Forwarder (HF). However, I noticed that the current setup uses indexerDiscovery, which sends data directly to the indexer. When I reconfigured the Splunk instances to use server = HeavyForwarder_IP:9997 instead, I encountered a "tcpout has been blocked" error on the HF when attempting to connect to the indexer. The logs indicate that the connection to the indexer is established, but immediately followed by the message "tcpout has been blocked." I also discovered that the indexer is configured with [splunktcp-ssl://9997], whereas the HF is using [splunktcp://9997] without any SSL settings. Could this be the root cause? Do I need to configure the HF with sslCertPath, sslRootCAPath, sslPassword, and sslVerifyServerCert in order for it to connect properly? ... View more

Hi,  I have saw there are many recommendations to rebuild and migrate with its existing data and configuration. It abit confusing for me as a new Splunk user, would appreciate if there some guidance for it. The following are the instances. 1x Search Head 3x Indexer 3x Heavy Forwarder 1x License server 1x deployment server Current version: 9.3.2 Assuming the hostname/IP could be the same or different for the rebuild. What is the best way to perform the rebuild and migration with it existing data and configuration? Same hostname/IP: - Copy the entire contents of the $SPLUNK_HOME directory from the old server to the new server - Install all instance for the new Splunk component into new server Different hostname/IP: - Copy the entire contents of the $SPLUNK_HOME directory from the old server to the new server - Install all instance for the new Splunk component into new server - Update individual .conf of instances if using new hostname - Update individual instances to point to their respecitive instances roles And could i install a newer version of Splunk without going to 9.3.2 when rebuilding and migrating? For testing purpose, I'll be trying it at one AIO instances for the rebuilding/migration due to space limitation. ... View more

Hi, After setting up a test index and ingesting a test record, I’m now planning to remove the index from the distributed setup. Could anyone confirm the correct procedure for removing an index in a distributed environment with 3 indexers and a management node? I normally run the following command at an all in one setup. /opt/splunk/bin/splunk clean eventdata -index index_name ... View more

Hi, Unsure what is the root cause as i was trying to do some minor adjustment to ignore the [ ] at the transforms.conf. Previously I'm able to view the fields like Id Name and their value but currently nothing shows. I tried to re-do the props.conf, transforms.conf and inputs.conf by adding parameter by parameter and it still didn't work.     ... View more

Hi, I'm facing an issue where the same data gets indexed multiple times every time the JSON file is pulled from the FTP server. Each time the JSON file is retrieved and placed on my local Splunk server, it overwrites the existing file. I don't have control over the content being placed on the FTP server, it could either be an entirely new entry or an existing entry with new data added, as shown below. I'm monitoring a specific file, as its name, type, and path remain consistent. From what I can observe, every time the file has new entries alongside previously indexed data, it is re-indexed, causing duplication. Example: file.json 2024-04-21 14:00 - row 1 2024-04-21 14:10 - row 2 overwritten file.json 2024-04-21 14:00 - row 1 2024-04-21 14:10 - row 2 2024-04-21 14:20 - row 3 Additionally, I checked the sha256sum of the JSON file after it’s pulled into my local Splunk server. The hash value changes before and after the file is overwritten. file.json: 2217ee097b7d77ed4b2eabc695b89e5f30d4e8b85c8cbd261613ce65cda0b851 /home/ws/logs/###.json overwritten file.json: 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd /home/ws/logs//###.json I've tried using initCrcLength, crcSalt, and followTail, but they don't seem to prevent the duplication, as Splunk still indexes it as new data. Any assistance would be appreciated, as I can't seem to prevent the duplication in indexing. ... View more