Deployment Architecture

Why does Splunk forwarder fail to keep running in Linux environment (variable in server.conf)?

joshnetwolf
New Member

Under /opt/splunkforwarder/etc/system/local/server.conf, we have used the env variable $INSTANCE_ID.

[general]

serverName = $INSTANCE_ID

We then verified that we got the right results by using the command:

./splunk show servername

This then showed the correct id we were looking for. However, when we go to start Splunk using ./splunk start OR ./splunk to restart it will then show it's started.

Right after it shows it's started, I attempt to verify it's running by using ./splunk status. This shows us that the service did not in fact start, and it still shows splunkd is not running.

Why is this?

We verified that if we hard code the result into the servername in server.conf, the service does in fact start.
But, for some reason, by using the env variable something is keeping Splunk from running.

Any help on this would be great!

Labels (2)
0 Karma

PavelP
Motivator

Hello @joshnetwolf

  • what are the last lines in the output of ./splunk start?
  • what does the env variable $INSTANCE_ID contains?

    After any environment variables are expanded, the server name
    (if not an IPv6 address) can only contain letters, numbers, underscores,
    dots, and dashes. The server name must start with a letter, number, or an
    underscore.

0 Karma

joshnetwolf
New Member

Update:
If we disable boot-start I found that we can then start the service with no issues. But if I where to enable boot-start after the splunk service is restarted and then preform a ./splunk restart it will fail to restart and will remain offline until you disable boot-start again and start the service.

Other Details:
Splunk forwarder version: splunkforwarder-7.2.3-06d57c595b80-linux-2.6-x86_64

0 Karma

tsizzlebuffet
Observer

Ever figure this out @joshnetwolf ? I was just hitting the same issue now trying to implement the same w 7.3.x

Docs for 7.3 vs 8.x are the same so - but havent tested it with 8.x install yet.

0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...