topic Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes? in Splunk Search

After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

kalyanilandge — Sat, 25 Jun 2016 09:12:09 GMT

Hi Team,

I have upgraded Splunk from 6.2 to 6.3.1 version. I restored backup, but still I am not getting any output for searches for any of the indexes.

Thanks

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

martin_mueller — Sat, 25 Jun 2016 12:15:20 GMT

What do you mean by "I restored backup"?

Other than that, check the steps along the way of a search for index=* OR index=_* over all time:

Does your user have read permissions on any index?
Does any index contain events?
Are there any errors in the search UI?
Are there any errors in splunkd.log?

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

kalyanilandge — Tue, 29 Sep 2020 10:02:58 GMT

Restored means I have taken back up for
:-splunk/etc &
:-splunk/var/lib.
:-index=_internal ,_audit I am getting results.
:-I have admin rights.
All the index were contain events previously.
:-There is no error in UI.
:-splunkd logs showing today's logs only.No error.

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

jkat54 — Sat, 25 Jun 2016 14:13:24 GMT

What is your search that is showing zero results?

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

martin_mueller — Sat, 25 Jun 2016 14:13:47 GMT

I'm confused as to why you restored backups after upgrading. That's likely to mess things up, kind of like a partial roll-back.

That being said, check if your non-internal indexes you expect to search actually exist and contain events through Settings -> Indexes.

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

kalyanilandge — Tue, 29 Sep 2020 10:03:01 GMT

.. before upgrading splunk on indexer , from that host i have copied the directories splunk/var/lib/splunk (all the indexes for eg:index_a, index_b) to another machine.. once i upgraded splunk version on indexer , again i copied all these directories to the same location (splunl/var/lib/splunk/) on indexer from that host..

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

woodcock — Sun, 26 Jun 2016 00:42:18 GMT

This question is a duplicate, right?

https://answers.splunk.com/answers/419076/search-query-showing-no-result-found-after-upgradi.html

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

jkat54 — Sun, 26 Jun 2016 01:23:56 GMT

Yeah but each had a few details so I threw my hands in the air and came to the active one 😉

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

martin_mueller — Sun, 26 Jun 2016 08:48:32 GMT

Mkay... so you've backed up etc and var/lib, de-installed splunk, installed newer Splunk, copied back etc and var/lib/splunk?
If that's the case, you now have a mix of 6.2 and 6.3 running. That's a recipe for disaster - instead of new settings in each default directory, you've copied over the old defaults.

To fix, I'd do the following:

make sure your backup still is there
remove the broken hybrid of 6.2 and 6.3
install a fresh 6.4.1
restore var/lib/splunk
restore only custom apps and apps/name/local folders in etc/apps
restore etc/system/local
selectively restore lookup files in etc/apps/name/lookups and etc/system/lookups, make sure you don't blindly overwrite existing things
restoring metadata.default and metadata.local in etc/apps/name/metadata probably is going to be too much effort and risk for little gain
restore any other custom thing in etc, e.g. certificates
don't blindly overwrite all other things in etc with the backup

In the future, I'd recommend the following upgrade procedure to avoid this mess:

make a backup
stop splunk
run the installer to actually upgrade
start splunk
confirm everything works

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

Richfez — Sun, 26 Jun 2016 12:32:39 GMT

I'm not sure that's supported and could very likely have messed up the data.

If I were you, I'd set up Splunk 6.2.1 on another machine temporarily and copy the original data to it and make sure everything that it is searchable and works right.

Once I had that backout plan ready to go, you have a couple of options. Upgrade the 6.2.1 machine you just built following the upgrade procedure, or rebuild the machine you had upgraded to 6.3 back to 6.2.1 and copy the data to it, confirm operation then upgrade it following the upgrade procedure. From 6.2.1 to 6.3 (or even 6.4.1) it's not a complicated procedure.

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

woodcock — Sun, 26 Jun 2016 13:40:13 GMT

According to your other version of this question (now closed as a duplicate), you did these steps in this order:

1: stoped splunk on indexer 
2: Executed rm -rf Splunk
3: Took backup for SPLUNK-HOME/etc/apps & SPLUNK-HOME/var/lib
4: Installed pkg for 6.3.2.
5: Restored etc&lib backups
6: Restart splunk

After this you can see the old index names in UI in setting -> indexes, but you are not able to search the data in search query for index=ac_s.

Unless you had a highly unusual (way non-standard) installation, you are toast because steps 2 and 3 are reversed (actually, step 2 should not even be there). The environment variable $SPLUNK_HOME starts with the Splunk directory (which you just removed) so your backup command copied nothing (indeed, it should have given you an error).

Where did you get these directions? I have never seen any directions anywhere for upgrading splunk that suggested deleting any files or directories. It is not only unnecessary, but possibly disastrous, as in this case.

If by chance you actually do have a good backup (like maybe you said it wrong and you did 1-3-2-4), then I would install whatever version USED to be there originally, restore your files, start splunk and make sure everything looks good (data is searchable), stop splunk, DO NOT REMOVE ANYTHING, install new version, start splunk, answer the questions ( 'Yes' to everything), and it should be fine. But I fear that your backup is empty.

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

kalyanilandge — Sun, 26 Jun 2016 16:01:11 GMT

The server on which I have taken the backup is full.That's the reason Files are 0 kb and I lost data.
Does splunk have any other way to restore the deleted data.

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

martin_mueller — Sun, 26 Jun 2016 18:08:00 GMT

If you deleted data and don't have a working backup, the data is likely gone. Certainly far beyond what Splunk Answers can do for you.

Re: After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

woodcock — Sun, 26 Jun 2016 20:48:28 GMT

Show me the output of these 2 commands on the indexer:

echo $SPLUNK_HOME
df -k

I am certain that I know what I will see and if I do, you are toast.