Splunk Search

Source and sourcetype filtering no longer working after upgrade

ufotech
Explorer

After upgrade from Splunk 6.2. to 6.6.3 having large existing indexes, any search by either source or sourcetype does no longer work. I.e. "No results found. Try expanding the time range"

Indeed, both fields are present in all events as can be seen if not filtering in the search-line.
Even statistics work.
If I do " * | stats count by source" , then I get a perfect list of all sources having a count of events.

But sill, clicking on a source and "Add to search" will add it to the search-line and return an empty result.

Any Ideas where it goes wrong?

I do find some errors in log, such as:
WordPositionData - couldn't find tab delim
or warnings
reason='couldn't parse hash code:

can this be a reason?

Thanks

0 Karma

ufotech
Explorer

Thanks for the hint.

I did all I found
- $SPLUNK_HOME/bin/splunk fsck repair --all-buckets-all-indexes
- $SPLUNK_HOME/bin/recover-metadata $SPLUNK_DB/$i/db/$d (looping over index/db)
and it took days....

However, still having the same log-entries and cannot search for either of Source, Sourcetype nor Host. Fortunately I can do this on a test-installation, where the issues are exactly the same as on the productive environment. Once I find a solution, this has to be fixed on the real one.

More ideas?
Thanks

0 Karma

ufotech
Explorer

There seems to be no solution to that problem.

What, if I had started from scratch?
How would I import the existing data into the new DB?
When importing data, how do I define the index where they have to go to?

Any help is appreciated.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...