Splunk Search

Why is there no data being written to the _internal\_audit indexes?

karadikid
Explorer

Hello,

After setting up a brand new standalone server (v 8.2.6) and migrating our data from another server, it seems as we don't see any events in both _internal and _audit indexes...

I've checked file permissions, and a whole lot of config files and could not manage to get this fixed.

Can someone please provide some pointers on what might be the issue?

EDIT:

Forgot to mention that the old server was running the same version and that both of those indexes were working just fine.

Thanks!

Labels (1)
Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

What do you mean by "migrating our data"?

0 Karma

karadikid
Explorer

Copying the $SPLUNK_HOME folder to the new server 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. I assume you did something like https://docs.splunk.com/Documentation/Splunk/8.2.6/Installation/MigrateaSplunkinstance

Does the rest of the splunk functionality work properly? Can you search other data?

0 Karma

karadikid
Explorer

Yes, I used the same approach as described in the provided link.

I'm able to search other indexes just fine and see the results.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

First quick check - is the time on your machine set properly? Including timezone.

If so, you have to check whether the problem is with splunk not writing to log files or if the files are being written to but are not read from and forwarded to indexes.

Check the contents of /opt/splunk/var/log/splunk/splunkd.log file and check timestamps of events contained there.

If the timestamp is current and splunk keeps writing to the files in /opt/splunk/var/log/splunk, you should check if it's monitoring them by issuing

/opt/splunk/bin/splunk list monitor

 If the $SPLUNK_HOME/var/log/splunk is not among monitored directories, there's something wrong with the config (permission issue?) check output of

/opt/splunk/bin/splunk btool inputs list --debug

If the directory is monitored, check the splunkd.log for errors.

0 Karma

karadikid
Explorer

Thanks for following up on this, @PickleRick! I've already verified that splunk is writing data to /opt/splunk/var/log/splunk/splunkd.log with correct times and ensured that the $SPLUNK_HOME/var/log/splunk folder is indeed monitored.

Trying btool showed a whole lot of data but nothing seems out of the ordinary there...

/opt/splunk/bin/splunk btool inputs list --debug

 

I forgot to mention that we have Splunk ES installed on the server, if it matters at all...

The splunkd.log contains many errors, but I could not find anything directly connected to the issue in question.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Then go to your /opt/splunk/var/lib/splunk/_internaldb/db/ and check modification times of the files and directories.

And just to be sure, simply search for "index=_internal", set the timepicker to "All time (realtime)". Wait for some time. If you're getting events it means that they're getting indexed but for some reason you're not getting them returned with normal search. Wrong time?

0 Karma

karadikid
Explorer

Thanks again @PickleRick! Trying the real-time search yielded no results...

My SPLUNK_DB is pointing to a different directory (/datadrive/splunk_db) and the files there were not modified for few days now. Probably since we first started the server...

Any idea what might be the case?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ok. That's interesting. Check the "original" index location - the one within $SPLUNK_HOME. Also check your index location in

splunk btool indexes list _internal --debug
0 Karma

karadikid
Explorer

The files in the original location ($SPLUNK_HOME/var/lib/splunk/_internaldb/) are not modified as well... 

btool shows the following:

/opt/splunk/etc/system/default/indexes.conf 

coldPath = $SPLUNK_DB/_internaldb/colddb
homePath = $SPLUNK_DB/_internaldb/db
thawedPath = $SPLUNK_DB/_internaldb/thaweddb

 

0 Karma

karadikid
Explorer

@PickleRick an update. I've re-applied the permissions ownership on $SPLUNK_DB and restarted the server.

It seems as the timestamp is now updated correctly but still no results when searching (including real-time).

0 Karma

karadikid
Explorer

@PickleRick another update, perhaps it might be of help. I've installed a brand new server and confirmed that everything works as expected (including _internal index events).

Then, I copied over my users, roles, and passwd file from the old server and it stopped working.

Specifically:

  • $SPLUNK_HOME/etc/users
  • $SPLUNK_HOME/etc/system/local/authorize.conf
  • $SPLUNK_HOME/etc/passwd

Permissions on those files are set correctly, server was restarted.

Any idea what might be the issue here?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ha! Never thought about it. Probably because I always work with admin-level privileges.

If copying those files over causes you to stop seeing events from _internal it would most probably mean that the events are getting indexed (although at the beginning they might have not - see the timestamp issue).

It would seem that the user you're searching with doesn't have enough permissions to see inside the _internal index. Did you verify the user's and role's settings?

0 Karma

karadikid
Explorer

Tried with the default Admin and also created a brand new admin user, non works...

I'm pretty clueless here TBH.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

TBH - me too. I think it's that point at which I'd consider calling support.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...