Installation

After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

vryzhko
Path Finder

After upgrading Splunk Enterprise from 6.3.3 to 6.4.0, I see this message:

[root@splunk bin]# $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.


Perform migration and upgrade without previewing configuration changes? [y/n] y

-- Migration information is being logged to '/opt/splunk/var/log/splunk/migration.log.2016-04-07.10-40-46' --

Migrating to:
VERSION=6.4.0
BUILD=f2c836328108
PRODUCT=splunk
PLATFORM=Linux-x86_64

Can't create directory "": No such file or directory


An error occurred: Could not create audit keys (returned 4).
[root@splunk bin]#

As a result Splunk does not start.
Please help me to resolve this issue!
Maybe you have ideas which directions to look.

Thank You!

Labels (1)
0 Karma
1 Solution

sgolyak
Engager

Same error happened.
I was able to overcome this by creating file /opt/splunk/etc/system/local/audit.conf with the following content:

[auditTrail]
privateKey = /opt/splunk/etc/auth/audit/private.pem
publicKey = /opt/splunk/etc/auth/audit/public.pem

View solution in original post

edwardrose
Contributor

I ran strace on splunk start to see if I could see what it was doing but don't see anything in particular.

09:30:40.542721 ioctl(3, SNDCTL_TMR_START or SNDRV_TIMER_IOCTL_TREAD or TCSETS, {B38400 opost isig icanon echo ...}) = 0
09:30:40.542750 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
09:30:40.542767 close(3)                = 0
09:30:40.542793 pipe2([3, 4], O_CLOEXEC) = 0
09:30:40.542832 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb4024d99d0) = 19725
09:30:40.542942 close(4)                = 0
09:30:40.542958 fcntl(3, F_SETFD, 0)    = 0
09:30:40.542978 fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
09:30:40.542997 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb4024f1000
09:30:40.543015 read(3, "[clustering]\naccess_logging_for_"..., 4096) = 1224
09:30:40.561740 read(3, "", 4096)       = 0
09:30:40.562243 close(3)                = 0
09:30:40.562263 wait4(19725, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 19725
09:30:40.562294 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19725, si_status=0, si_utime=1, si_stime=0} ---
09:30:40.562306 munmap(0x7fb4024f1000, 4096) = 0
09:30:40.562338 open("/etc/localtime", O_RDONLY) = 3
09:30:40.562390 fstat(3, {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0
09:30:40.562412 fstat(3, {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0
09:30:40.562426 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb4024f1000
09:30:40.562443 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 2819
09:30:40.562461 lseek(3, -1802, SEEK_CUR) = 1017
09:30:40.562476 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 1802
09:30:40.562496 close(3)                = 0
09:30:40.562510 munmap(0x7fb4024f1000, 4096) = 0
09:30:40.562536 open("/apps/splunk/var/log/splunk/migration.log.2016-04-08.09-30-40", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
09:30:40.562717 fstat(3, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
09:30:40.562740 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb4024f1000
09:30:40.562757 fstat(3, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
09:30:40.562771 lseek(3, 0, SEEK_SET)   = 0
09:30:40.562802 write(2, "\n-- Migration information is bei"..., 112
-- Migration information is being logged to '/apps/splunk/var/log/splunk/migration.log.2016-04-08.09-30-40' --
) = 112
09:30:40.562824 write(1, "\nMigrating to:\n", 15
Migrating to:
) = 15
09:30:40.562853 open("/apps/splunk/etc/splunk.version", O_RDONLY) = 4
09:30:40.562877 fstat(4, {st_mode=S_IFREG|0755, st_size=70, ...}) = 0
09:30:40.562892 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb4024f0000
09:30:40.562908 read(4, "VERSION=6.4.0\nBUILD=f2c836328108"..., 4096) = 70
09:30:40.562925 read(4, "", 4096)       = 0
09:30:40.562939 close(4)                = 0
09:30:40.562952 munmap(0x7fb4024f0000, 4096) = 0
09:30:40.562972 write(1, "VERSION=6.4.0\nBUILD=f2c836328108"..., 70VERSION=6.4.0
BUILD=f2c836328108
PRODUCT=splunk
PLATFORM=Linux-x86_64
) = 70
09:30:40.562989 write(1, "\n", 1
)       = 1
09:30:40.563005 write(3, "\nMigrating to:\nVERSION=6.4.0\nBUI"..., 86) = 86
09:30:40.563031 close(3)                = 0
09:30:40.563044 munmap(0x7fb4024f1000, 4096) = 0
09:30:40.563063 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb4024d99d0) = 19726
09:30:40.563143 wait4(19726, Can't create directory "": No such file or directory


An error occurred: Could not create audit keys (returned 4).
[{WIFEXITED(s) && WEXITSTATUS(s) == 2}], 0, NULL) = 19726
09:30:40.729452 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19726, si_status=2, si_utime=7, si_stime=2} ---
09:30:40.729511 exit_group(2)           = ?
09:30:40.729575 +++ exited with 2 +++
[root@splunk-id1 bin]# 
0 Karma

sgolyak
Engager

Same error happened.
I was able to overcome this by creating file /opt/splunk/etc/system/local/audit.conf with the following content:

[auditTrail]
privateKey = /opt/splunk/etc/auth/audit/private.pem
publicKey = /opt/splunk/etc/auth/audit/public.pem

vryzhko
Path Finder

The issue was resolved in accordance with your recommendations.
Thank you!

0 Karma

christian_clout
Explorer

I've had the same. In my case it was caused by the following configuration, which is part of the Enterprise Security (version 3.1.1) App:

$SPLUNK_HOME/etc/apps/SA-AuditAndDataProtection/default/audit.conf:

[filterSpec:event_whitelist:stashWhitelist]
sourcetype=stash

[filterSpec:event_blacklist:nothingElse]
all=True

[eventHashing]
filters=stashWhitelist,nothingElse

SOLNESS-2268: Disabling auditTrail signing by default

To enable, copy the following stanza to SA-AuditAndDataProtection/local/audit.conf

and swap the empty private/public key values for the populated ones

[auditTrail]
privateKey =

privateKey = $SPLUNK_HOME/etc/auth/audit/private.pem

publicKey =

publicKey = $SPLUNK_HOME/etc/auth/audit/public.pem

I also applied the proposed fix for the upgrade successfully, then reverted back.

0 Karma

edwardrose
Contributor

Were you able to resolve this yet?

0 Karma

vryzhko
Path Finder

No. This problem is actual.

0 Karma

gwobben
Communicator

I might be stating the obvious here but, does the user that runs the start command have write access on the Splunk directory and on /var/log (for the migration log file)?

0 Karma

vryzhko
Path Finder

My splunk always works from user root. I install and run it in Linux console also from root.
As an experiment I attempted to change access to files and folders in 777.
Command: chmod -R 777 /opt/splunk
The result is the same. What do you think?

Thank you!

0 Karma

vryzhko
Path Finder

I think what problem is with audit key, but old keys is available in destination folder. I don't understand why Splunk doesn't can to rebuild their.

[root@splunk audit]# ls -la /opt/splunk/etc/auth/audit/
total 16
drwxrwxrwx. 2 splunk splunk 4096 Apr 6 19:31 .
drwxrwxrwx. 6 splunk splunk 4096 Apr 6 19:43 ..
-rwxrwxrwx 1 splunk splunk 891 Mar 18 2014 private.pem
-rwxrwxrwx 1 splunk splunk 272 Mar 18 2014 public.pem
[root@splunk audit]#

0 Karma

vryzhko
Path Finder

Migration log file is created after each trial run, but it contains very short information.

[root@splunk bin]# cat /opt/splunk/var/log/splunk/migration.log.2016-04-08.11-52-31

Migrating to:
VERSION=6.4.0
BUILD=f2c836328108
PRODUCT=splunk
PLATFORM=Linux-x86_64

[root@splunk bin]#

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...