With load balancing the Universal Forwarder sends data to all the indexers equally so that no indexer should get all the data and together the indexers holds all the data. It also provide automatic switchover capability incase of an indexer goes down. Load balancing can be setup at UF in outputs.conf file in two ways:   By time By Volume   For time based load balancing we used autoLBFrequency setting and for volume we use autoLBVolume. Let's say I've three indexers on which I want to send data from UF. My outputs.conf file will look like below: [tcpout: my_indexers] server=10.10.10.1:9997, 10.10.10.2:997, 10.10.10.3:9997 Now, to send data for 3 minutes to an indexer, then switch to another indexer and then to another, set the autoLBFrequency like this: autoLBFrequency=180 Based on the above settings the UF will send data to indexer 10.10.10.1 for 3 minutes continuously then it will move towards the other indexers, and this loop will continue. To send data based on the volume. Let's say to configure the UF to send 1MB data to an indexer then switch to another indexer in the list, the setting will look like below autoLBVolume=1048576 In the cases of a very large file, such as a chatty syslog file, or loading a large amount of historical data, the forwarder may become "stuck" on one indexer, trying to reach EOF before being able to switch to another indexer. To mitigate this, you can use the forceTimebasedAutoLB setting on the forwarder. With this setting, the forwarder does not wait for a safe logical point and instead makes a hard switch to a different indexer every AutoLB cycle. forceTimebasedAutoLB = true To guard against loss of data when forwarding to an indexer you can enable indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data that the indexer does not acknowledge as "received". useACK setting is used for this purpose useACK= true The final output.conf will look like below [tcpout] useACK= true autoLBFrequency=180 autoLBVolume=1048576 [tcpout: my_indexers] server=10.10.10.1:9997, 10.10.10.2:997, 10.10.10.3:9997 ... View more

Recently I have encountered an issue while rebuilding data on one of our indexers. During this process I needed to execute the following command: /opt/splunk/bin/splunk _internal call /data/indexes/main/rebuild-metadata-and manifests However upon running, I was prompted for Splunk Username and Password. Typically we used the credentials created at Web GUI. But since the usually the indexers Web GUI is set to false most of the time, so there is no GUI username and password available on them. I tried using my Search Head Username and Password, followed by the OS Username and Password, but neither worked. After some research, I discovered that every Splunk instance includes a default admin user created during installation: Username: admin Password: changeme but it doesn't work for me. Here is the procedure that finally worked for me, so to reset the password for the admin user Access the indexers cli, here the passwd file exist in: /opt/splunk/etc/ Rename that file to passwd.bak Create a new file with the name: user-seed.conf, in location: /opt/splunk/etc/system/local/ In this file use the below configuration: [user_info] USERNAME = admin PASSWORD = <password of your choice> Restart the Splunk service on that indexer using /opt/splunk/bin/splunk restart This will generate a new passwd file. You can now use the admin user with the password you set in step 3.   After the resting the password, I've used the initial command, using the updated admin credentials and it worked. ... View more

Hi, I'm facing an issue with 5 hosts, recently we change the hostname of these machines but it is not reflected in the host field, in the host field the old hostname is shown. Below is a sample log: "LogName=Security EventCode=4673 EventType=0 ComputerName=A0310PMTHYCJH15.tnjhs.com.pk host = A0310PMNIAMT05    source = WinEventLog:Security     sourcetype = WinEventLog " We are receiving logs from these windows hosts through UF and I checked the apps deployed in these hosts and checked the inputs.conf, hostname field is not defined. The new hostname is shown in the logs in the field ComputerName. Any suggestions to this problem would be appreciated. ... View more