Having multiple Forwarders sharing the same Instance GUID

[Na_Kang_Lim]

As the title suggests, I am having multiple Universal Forwarders sharing the same Instance GUID due to the mistake of the owners of the servers, as they were just copying the .conf files from servers to servers.

So my question is, what is the impact of this. I think missing the logs from these machines is one of the impact, as every 10-20 seconds, the IP and Hostname change, but the Instance GUID and Instance Name remains the same.

And of course the obvious question, how do I fix this? Solving from the higher level of the system is prefer, like using the deployment server (which I can SSH into), since I have limited access to the servers, and probably have to TeamView them or write a guide for them.

I read that the GUID is now in the `$SPLUNK_HOME/etc/instance.cfg` file, and there is probably a GUID there, which would be the same across the servers. Can I just delete the GUID line and restart the splunk service, and the deployment server would give it a new one?

Can I delete the record from the Deployment Server UI and it would generated a new one and auto update the instance.cfg file of the Forwarder?

I read the docs instance.cfg.conf - Splunk Documentation and it mention not to edit the file, so I am a bit confused.

And I also saw that the docs mentioned that server.conf also has the GUID value, so do I have to do anything in the server.conf file?

[isoutamo]

If you have so called golden image which are created incorrectly (it contains those critical configurations), You could read this old post https://community.splunk.com/t5/Installation/EC2-from-AMI-having-splunk-installed-stops-working/m-p/...

r. Ismo

[livehybrid]

Hi @Na_Kang_Lim 

You're right in that the GUID is set in $SPLUNK_HOME/etc/instance.cfg

Such as

[general]
guid = 5e7f7ee5-ab7b-4e90-8f9e-b2cf1f86e6f3

Unfortunately the DS isnt designed to write to this location as you would typically distribute apps from it. 

Depending how confident you are feeling, it would be possible to write a script which you could deploy to all the clients using your DS which would write a new GUID to that file, once applied you can then remove the app. 
Below I'll include snippets of an app I've used previously for this problem - Disclaimer - Its generally not advised to do this sort of thing, use extreme caution 🙂 

1. Create an app on your deployment server - something like "regenerate_guid":
2. Create a bin directory within the app and create the following file called regenerate_guid.py

import uuid
import os
import configparser

INSTANCE_CFG_PATH = "/opt/splunkforwarder/etc/instance.cfg"

def generate_guid():
    return str(uuid.uuid4())

def write_guid_to_instance_cfg(guid):
    config = configparser.ConfigParser()
    try:
        config['general'] = {'guid': guid}  # Always write the GUID, overwriting any previous value
        with open(INSTANCE_CFG_PATH, "w") as f:
            config.write(f)
        print(f"Successfully wrote new GUID to {INSTANCE_CFG_PATH}")
    except Exception as e:
        print(f"Error writing to {INSTANCE_CFG_PATH}: {e}")

def main():
    new_guid = generate_guid()
    write_guid_to_instance_cfg(new_guid)
    print("Regenerated and overwrote GUID.")

if __name__ == "__main__":
    main()

3. Create inputs.conf:

Create an inputs.conf file in the default/ directory of your app:

[script://./bin/regenerate_guid.py]
# Run once a day (86400) but in reality the app should be removed after actioned!
interval = 86400
source = regenerate_guid
sourcetype = regenerate_guid
index = main
disabled = 0

• [script://./bin/regenerate_guid.py]: This defines a script input that runs the Python script. The path is relative to the app's directory structure.

• interval: How often the script should run (in seconds). We only want it to run once so Ive set this to 1 day, remove the app once it has run.

• source/sourcetype/index - Not really useful as we dont want the data it outputs but typical for a modular input.

• disabled = 0: Enables the input.

4. Create default/app.conf:

[install]
state = enabled

[launcher]
author = Your Name
description =  DANGEROUS APP: Periodically checks and regenerates the Splunk Instance GUID if it's missing. USE WITH EXTREME CAUTION!  REQUIRES inputs.conf for proper deployment.
version = 1.1.0

[package]
id = regenerate_guid

Workflow:

1. The Deployment Server deploys the regenerate_guid app to the forwarder, configure the DS so the app causes a restart.

2. The input should run straight away once installed and then every 24hrs, check the _internal index to check it has run. 

3. Uninstall the app by removing the app or the client(s) from the serverclass/DS

Important Considerations (Reiterated!)

• This approach is still potentially dangerous if mishandled!

• The script runs periodically. This means it will reset the GUID every 24 hours. Make sure you remove it once it has run.

• Permissions are critical. If the Splunk user does not have write access to instance.cfg, the script will fail.

• Testing is essential! Test in a non-production environment before deploying it to production, or at least deploy to a single host you can directly access to validate etc.

• Monitor the logs! Monitor the splunkd.log file on the forwarders to ensure the script is running correctly.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[PickleRick]

While the solution with a scripted input is a nice trick from the technical point of view, as a seasoned admin I'd advise against using it, especially on environments you have limited/difficult connectivity with.

Any splunkd-spawned solutions which change the general splunkd configuration are prone to leaving your installation in a broken state should anything go wrong.

Of course whether it's important depends on how critical the systems are and whether you can tolerate potential downtime vs. what you can save by doing the "automation".

The risk is yours. You have been warned.

[livehybrid]

In addition to my previous message, I've realised that all the servers might have the same serverName in $SPLUNK_HOME/etc/system/local/server.conf - Is that the case too?

[general]
serverName = <YourServerName>

This also cannot be overwritten with an app config because its in the system/local directory, so if you want to update this then you would need to do something similar to the regenerate_guid script I posted.

Reminder - That script is less than ideal so proceed with caution 🙂 

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[Na_Kang_Lim]

Thank you for your detail automation solution! Then what about the manual solution? If I were to write a guide for the owners of the server, what do I need them to do, and what do I have to do too? And yes, I am having both duplicate Instance GUID and Instance Name! The hostname of course it is not the same, I believe it takes the Computer Name of the Host, but the Instance GUID and Instance Name do duplicate!

[livehybrid]

Okay @Na_Kang_Lim 

So the manual approach is to update the $SPLUNK_HOME/etc/instange.cfg and replace the guid = <oldGuid> with a new random guid.
Then update the $SPLUNK_HOME/etc/system/local/server.conf and change the serverName under the [general] stanza.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[Na_Kang_Lim]

@livehybrid And about the first approach, after updating the values in the .conf files, I still have to restart the splunkd service on the server, right? Or restarting splunk service on the DS is an alternative way?

[PickleRick]

No. Restarting DS won't change anything on the deployment clients (your forwarders). Restarting forwarders requires... well, restarting forwarders. That's why I keep saying that ugly hacks like scripting those restarts from scripted inputs are risky. If something goes wrong, you end up with a downed process and no way to bring it up.

[Na_Kang_Lim]

I see that the serverName already has the default value as $HOSTNAME (the docs of server.conf mentioned it), so it means that I can just delete the key-value pair, right?