After upgrading Splunk from 6.2 to 6.3.1, why am I getting no results searching any indexes?

[kalyanilandge]

Hi Team,

I have upgraded Splunk from 6.2 to 6.3.1 version. I restored backup, but still I am not getting any output for searches for any of the indexes.

Thanks

[woodcock]

According to your other version of this question (now closed as a duplicate), you did these steps in this order:

1: stoped splunk on indexer 
2: Executed rm -rf Splunk
3: Took backup for SPLUNK-HOME/etc/apps & SPLUNK-HOME/var/lib
4: Installed pkg for 6.3.2.
5: Restored etc&lib backups
6: Restart splunk

After this you can see the old index names in UI in setting -> indexes, but you are not able to search the data in search query for index=ac_s.

Unless you had a highly unusual (way non-standard) installation, you are toast because steps 2 and 3 are reversed (actually, step 2 should not even be there). The environment variable $SPLUNK_HOME starts with the Splunk directory (which you just removed) so your backup command copied nothing (indeed, it should have given you an error).

Where did you get these directions? I have never seen any directions anywhere for upgrading splunk that suggested deleting any files or directories. It is not only unnecessary, but possibly disastrous, as in this case.

If by chance you actually do have a good backup (like maybe you said it wrong and you did 1-3-2-4), then I would install whatever version USED to be there originally, restore your files, start splunk and make sure everything looks good (data is searchable), stop splunk, DO NOT REMOVE ANYTHING, install new version, start splunk, answer the questions ( 'Yes' to everything), and it should be fine. But I fear that your backup is empty.

[martin_mueller]

If you deleted data and don't have a working backup, the data is likely gone. Certainly far beyond what Splunk Answers can do for you.

[kalyanilandge]

The server on which I have taken the backup is full.That's the reason Files are 0 kb and I lost data.
Does splunk have any other way to restore the deleted data.

[woodcock]

Show me the output of these 2 commands on the indexer:

echo $SPLUNK_HOME
df -k

I am certain that I know what I will see and if I do, you are toast.

[martin_mueller]

Mkay... so you've backed up etc and var/lib, de-installed splunk, installed newer Splunk, copied back etc and var/lib/splunk?
If that's the case, you now have a mix of 6.2 and 6.3 running. That's a recipe for disaster - instead of new settings in each default directory, you've copied over the old defaults.

To fix, I'd do the following:

• make sure your backup still is there
• remove the broken hybrid of 6.2 and 6.3
• install a fresh 6.4.1
• restore var/lib/splunk
• restore only custom apps and apps/name/local folders in etc/apps
• restore etc/system/local
• selectively restore lookup files in etc/apps/name/lookups and etc/system/lookups, make sure you don't blindly overwrite existing things
• restoring metadata.default and metadata.local in etc/apps/name/metadata probably is going to be too much effort and risk for little gain
• restore any other custom thing in etc, e.g. certificates
• don't blindly overwrite all other things in etc with the backup

In the future, I'd recommend the following upgrade procedure to avoid this mess:

• make a backup
• stop splunk
• run the installer to actually upgrade
• start splunk
• confirm everything works

[woodcock]

This question is a duplicate, right?

https://answers.splunk.com/answers/419076/search-query-showing-no-result-found-after-upgradi.html

[jkat54]

Yeah but each had a few details so I threw my hands in the air and came to the active one 😉

[martin_mueller]

I'm confused as to why you restored backups after upgrading. That's likely to mess things up, kind of like a partial roll-back.

That being said, check if your non-internal indexes you expect to search actually exist and contain events through Settings -> Indexes.

[kalyanilandge]

.. before upgrading splunk on indexer , from that host i have copied the directories splunk/var/lib/splunk (all the indexes for eg:index_a, index_b) to another machine.. once i upgraded splunk version on indexer , again i copied all these directories to the same location (splunl/var/lib/splunk/) on indexer from that host..

[Richfez]

I'm not sure that's supported and could very likely have messed up the data.

If I were you, I'd set up Splunk 6.2.1 on another machine temporarily and copy the original data to it and make sure everything that it is searchable and works right.

Once I had that backout plan ready to go, you have a couple of options. Upgrade the 6.2.1 machine you just built following the upgrade procedure, or rebuild the machine you had upgraded to 6.3 back to 6.2.1 and copy the data to it, confirm operation then upgrade it following the upgrade procedure. From 6.2.1 to 6.3 (or even 6.4.1) it's not a complicated procedure.

[martin_mueller]

What do you mean by "I restored backup"?

Other than that, check the steps along the way of a search for index=* OR index=_* over all time:

• Does your user have read permissions on any index?
• Does any index contain events?
• Are there any errors in the search UI?
• Are there any errors in splunkd.log?

[kalyanilandge]

Restored means I have taken back up for
:-splunk/etc &
:-splunk/var/lib.
:-index=_internal ,_audit I am getting results.
:-I have admin rights.
All the index were contain events previously.
:-There is no error in UI.
:-splunkd logs showing today's logs only.No error.

[jkat54]

What is your search that is showing zero results?