Getting Data In
Highlighted

Why running "splunk enable boot-start" did not start the indexing of my log data?

Path Finder

Splunk was installed and run as root.
I did a "splunk enable boot-start" which created a /etc/init.d/splunk script.
Upon system reboot, a "ps" shows that splunkd is running.
However, my logs are not indexed (per Settings/Indexes page).
After i did a manual "./splunk restart" then it started to index data.
Q: what am I missing?
Thanks,

0 Karma
Highlighted

Re: Why running "splunk enable boot-start" did not start the indexing of my log data?

Splunk Employee
Splunk Employee

it sounds like you have more than one instance of Splunk installed and that the init.d is pointing at the wrong one... "start" is "start" so... it seems like when you thought you'd started the right instance, you probably hadn't.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
Highlighted

Re: Why running "splunk enable boot-start" did not start the indexing of my log data?

Path Finder

Thanks rsennett for a quick response.
I'm pretty sure there's only one splunk instance installed, and the init.d/splunk script did spell out the path "/opt/splunk/bin/splunk" correctly.
Is there anything in the logs that i can check?
splunkweb works file but splunkd seems to have problems
The TCP data is indexed but flat file logs are not.

Could this be the issue?
my inputs.conf uses env variable $MYDATAPATH, e.g.
[monitor://$MYDATAPATH/*]

and $MYDATAPATH is defined in /etc/profile.d/MySplunkApps.sh
export MYDATAPATH=/my/data/path

Maybe when splunk started, that env variable was not yet defined.

Thanks for helping.

0 Karma
Highlighted

Re: Why running "splunk enable boot-start" did not start the indexing of my log data?

Legend

If you want that environment variable, put it in the user profile for whatever linux user is running splunk. Even better, make it an absolute path or use a path that is relative to the APP that contains the inputs.conf file. See my answer below - you should see the error in the splunkd.log if this is the problem.

0 Karma
Highlighted

Re: Why running "splunk enable boot-start" did not start the indexing of my log data?

Legend

I would take a look at the internal logs. There are two ways to do that:

1) Run this search

index=_internal sourcetype=splunkd

There will be tons of events, you may want to filter further

2) or take a look at $SPLUNK_HOME/var/log/splunk/splunkd.log

I am not sure what you will find, but I expect that the reason will be in there somewhere. Please update with what you discover.

0 Karma
Highlighted

Re: Why running "splunk enable boot-start" did not start the indexing of my log data?

Path Finder

index=internal sourcetype=splunkd loglevel=ERROR

... ERROR TailingProcessor - Input stanza path, '$MYDATAPATH/' is not absolute ...

So, splunk did not recognize $MYDATAPATH which is defined in profile.d

I move the definition to $SPLUNK_HOME/etc/splunk-launch.conf and splunk can see it. Problem solved for me.
Still not sure if that is the preferred solution though, i.e. put it in splunk-launch.conf

0 Karma