According to your other version of this question (now closed as a duplicate), you did these steps in this order:
1: stoped splunk on indexer 2: Executed rm -rf Splunk 3: Took backup for SPLUNK-HOME/etc/apps & SPLUNK-HOME/var/lib 4: Installed pkg for 6.3.2. 5: Restored etc&lib backups 6: Restart splunk
After this you can see the old index names in UI in
indexes, but you are not able to search the data in search query for
Unless you had a highly unusual (way non-standard) installation, you are toast because steps 2 and 3 are reversed (actually, step 2 should not even be there). The environment variable
$SPLUNK_HOME starts with the
Splunk directory (which you just removed) so your backup command copied nothing (indeed, it should have given you an error).
Where did you get these directions? I have never seen any directions anywhere for upgrading splunk that suggested deleting any files or directories. It is not only unnecessary, but possibly disastrous, as in this case.
If by chance you actually do have a good backup (like maybe you said it wrong and you did 1-3-2-4), then I would install whatever version USED to be there originally, restore your files, start splunk and make sure everything looks good (data is searchable), stop splunk, DO NOT REMOVE ANYTHING, install new version, start splunk, answer the questions ( 'Yes' to everything), and it should be fine. But I fear that your backup is empty.
Mkay... so you've backed up etc and var/lib, de-installed splunk, installed newer Splunk, copied back etc and var/lib/splunk?
If that's the case, you now have a mix of 6.2 and 6.3 running. That's a recipe for disaster - instead of new settings in each default directory, you've copied over the old defaults.
To fix, I'd do the following:
In the future, I'd recommend the following upgrade procedure to avoid this mess:
I'm confused as to why you restored backups after upgrading. That's likely to mess things up, kind of like a partial roll-back.
That being said, check if your non-internal indexes you expect to search actually exist and contain events through Settings -> Indexes.
.. before upgrading splunk on indexer , from that host i have copied the directories splunk/var/lib/splunk (all the indexes for eg:index_a, index_b) to another machine.. once i upgraded splunk version on indexer , again i copied all these directories to the same location (splunl/var/lib/splunk/) on indexer from that host..
I'm not sure that's supported and could very likely have messed up the data.
If I were you, I'd set up Splunk 6.2.1 on another machine temporarily and copy the original data to it and make sure everything that it is searchable and works right.
Once I had that backout plan ready to go, you have a couple of options. Upgrade the 6.2.1 machine you just built following the upgrade procedure, or rebuild the machine you had upgraded to 6.3 back to 6.2.1 and copy the data to it, confirm operation then upgrade it following the upgrade procedure. From 6.2.1 to 6.3 (or even 6.4.1) it's not a complicated procedure.
What do you mean by "I restored backup"?
Other than that, check the steps along the way of a search for
index=* OR index=_* over all time:
Restored means I have taken back up for
:-index=_internal ,_audit I am getting results.
:-I have admin rights.
All the index were contain events previously.
:-There is no error in UI.
:-splunkd logs showing today's logs only.No error.