The Splunk search shows that events exist, but why does the Events tab show "No results found"?

[yasinmoha]

I am trying to list specific events, but I am not able to view them. Splunk shows that events exist, but it comes up with no events found. Screenshot attached

[vgtk4431]

Okay found a solution to a similar problem.
ou limits.conf had the following conf :
[search]
max_count=0

This prevent splunk to store events as search results.

Removing the faulty settings resolve our search issue

[vgtk4431]

We encounter the same error last Week (see my comment)

The issue was on our limits.conf where we'd set a
[search]
max_count=0

which prevents Splunk to store any events.
Removing that settings resolved our issue

[vgtk4431]

Hello,

Did you succeed in solving your issue ?

We've encounter the same problem:
We are using a search index=_internal Error
We got the same result as the main screenshot .

If we use the search index=_internal Error | table *We got a table in statistics with every fields. But still no event.

We are using Splunk 6.6.x on a SHCluster/IdxCluster.
After some tests, the error appears only on SH, not if we launch a search from the indexer.

[briancronrath]

Are those sources quite large in size? I've recently ran into this with a sourcetype that has very large log files. My resolution was that I needed to specify the index I wanted to search. I don't know exactly why this needed to occur but I have a hunch it's due to some sort of max being reached when scanning very large events without an index supplied.

[javiergn]

Could it be you have the No Event Sampling activated?
Take a look at this: http://docs.splunk.com/Documentation/Splunk/6.4.0/Search/Retrieveasamplesetofevents#Specify_a_sampli...

UPDATE

Take a look at the following two values from your inspect job trace:

 resultCount     0
 scanCount     287

Now take a look at their correspondent description:

resultCount - The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.

scanCount - The number of events that are scanned or read off disk

Specifically this bit:

This is the subset of scanned events
(represented by the scanCount) that
actually matches the search terms.

In summary, your search filter does not match any events you are reading off disk.
Hope that makes sense.

[yasinmoha]

Below is the job inspection report I hope its format your looking for.

Execution costs
Duration (seconds)      Component   Invocations     Input count     Output count
    0.00    command.fields  5   287     287
    0.23    command.search  5   -   287
    0.13    command.search.index    6   -   -
    0.00    command.search.filter   1   -   -
    0.00    command.search.calcfields   1   287     287
    0.00    command.search.fieldalias   1   287     287
    0.00    command.search.index.usec_1_8   5,325   -   -
    0.00    command.search.index.usec_512_4096  5   -   -
    0.00    command.search.index.usec_64_512    114     -   -
    0.00    command.search.index.usec_8_64  21  -   -
    0.06    command.search.kv   1   -   -
    0.03    command.search.rawdata  1   -   -
    0.01    command.search.typer    1   287     287
    0.00    command.search.lookups  1   287     287
    0.00    command.search.summary  5   -   -
    0.00    command.search.tags     1   287     287
    0.00    dispatch.check_disk_usage   1   -   -
    0.00    dispatch.createdSearchResultInfrastructure  1   -   -
    0.08    dispatch.evaluate   1   -   -
    0.08    dispatch.evaluate.search    1   -   -
    0.28    dispatch.fetch  6   -   -
    0.23    dispatch.localSearch    1   -   -
    0.00    dispatch.readEventsInResults    1   -   -
    0.23    dispatch.stream.local   5   -   -
    0.01    dispatch.timeline   6   -   -
    0.01    dispatch.writeStatus    6   -   -
    0.03    startup.configuration   1   -   -
    0.07    startup.handoff     1   -   -
Search job properties
canSummarize    0
createTime  2016-04-20T12:48:20.000+05:30
cursorTime  2015-09-28T16:00:00.000+05:30
custom  

{
    "search": "(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*"
}

defaultSaveTTL  604800
defaultTTL  600
delegate    None
diskUsage   122880
dispatchState   DONE
doneProgress    1.0
dropCount   0
eai:acl     

{
    "app": "search", 
    "can_write": "1", 
    "modifiable": "1", 
    "owner": "admin", 
    "perms": {
        "read": [
            "admin"
        ], 
        "write": [
            "admin"
        ]
    }, 
    "sharing": "global", 
    "ttl": "600"
}

earliestTime    2012-07-08T01:30:52.000+05:30
eventAvailableCount     0
eventCount  287
eventFieldCount     0
eventIsStreaming    True
eventIsTruncated    False
eventSearch     search (source="nafx.g1303v00'" OR source="nafx.g1304v00'") opc* AND acf*
eventSorting    desc
indexEarliestTime   1460630991
indexLatestTime     1460631052
isBatchModeSearch   False
isDone  True
isFailed    False
isFinalized     False
isPaused    False
isPreviewEnabled    True
isRealTimeSearch    False
isRemoteTimeline    False
isSaved     False
isSavedSearch   False
isTimeCursored  1
isZombie    False
keywords    acf* opc* source::nafx.g1303v00' source::nafx.g1304v00'
label   None
modifiedTime    2016-04-20T12:48:27.086+05:30
normalizedSearch    litsearch ( source="nafx.g1303v00'" OR source="nafx.g1304v00'" ) opc* AND acf* | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
numPreviews     0
pid     21431
priority    5
remoteSearch    litsearch ( source="nafx.g1303v00'" OR source="nafx.g1304v00'" ) opc* AND acf* | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
reportSearch    None
request     

{
    "adhoc_search_level": "verbose", 
    "auto_cancel": "30", 
    "check_risky_command": "false", 
    "custom.search": "(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*", 
    "earliest_time": null, 
    "indexedRealtime": null, 
    "latest_time": null, 
    "preview": "1", 
    "rf": "*", 
    "sample_ratio": "1", 
    "search": "search (source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*", 
    "status_buckets": "300", 
    "ui_dispatch_app": "search"
}

resultCount     0
resultIsStreaming   True
resultPreviewCount  0
runDuration     0.388
runtime     

{
    "auto_cancel": "30", 
    "auto_pause": "0"
}

sampleRatio     1
sampleSeed  0
scanCount   287
search  search (source="nafx.g1303v00'" OR source="nafx.g1304v00'") opc* AND acf*
searchCanBeEventType    1
searchProviders     

[
    "oc8001872801.ibm.com"
]

searchTotalBucketsCount     13
searchTotalEliminatedBucketsCount   0
sid     1461136700.12
statusBuckets   300
ttl     600
Additional info     timeline search.log 

[yasinmoha]

I deleted the source log file and added it again with a new index instead of the default index and now when I run this command I am able to see the records And I have created separate indexes for each of the log files.

May be I used default index for all the log files which caused this issue.

[javiergn]

Ok, I think I might have the answer.
Take a look at this two values from your inspect job trace:

 resultCount     0
 scanCount     287

Now take a look at their correspondent description:

resultCount - The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.

scanCount - The number of events that are scanned or read off disk

Specifically this bit:

This is the subset of scanned events
(represented by the scanCount) that
actually matches the search terms.

In summary, your search filter does not match any events you are reading off disk.
Hope that makes sense.

[yasinmoha]

I am not able to understand the logic. There are something like 300,000+ events for this log file. So it scanned only 287 events and did not find any matches on these 287 events.

[javiergn]

Yeah, that seems to be case.
One way to verify this is to run all the following searches and compare the number of events you get:

(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") 

--

(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") 
| search opc*

--

(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") 
| search acf*

--

(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") 
| search opc* AND acf*

[yasinmoha]

Event sampling returns me very few records. and not the actual set of records that match. This is a normal problem that I have even with another search that I was performing.

There was another similair query I tried and when I filter based on date month it returns value for one particular month and not the other.

I have splunk installed on my linux laptop. Could that have any implication in indexing.

[javiergn]

Running on a Linux laptop shouldn't make any difference.
Could you try running your query in Smart Mode instead of Verbose?

Do you get any results when running any other query?
For instance, if you run the following:

index=_internal | head 10000

Do you get 10,000 results?

[yasinmoha]

I ran the index command and it returned me 10000 results. The case is sometime when I am adding more filter let me give another example. Where I tried to look up all log entries that had recall in them and it returned me 292 results. When I added recall AND fail*. It showed me 36 entries but no results displayed. When I did an inspect job I found that event count is 36 but available event count is 0.

I guess there might be something wrong with the way I am writing the search query or may be indexing.

[javiergn]

Can you run one of those searches that do not return any results but then click on Job > Inspect Job, copy the report and paste it here as Code Sample (use the button with 1s and 0s above)?

Maybe there's something in the Job Inspector telling us what's going on

[yasinmoha]

Looks like I dont have enough Karma points to add more files.

Scene 1 - This does not work

1st query
source=syslog.txt recall* --- Returns 292 records

2nd query
source=syslog.txt recall fail* -- Returns 36 records in event count but does not display results.

But when I try

source=syslog.txt migrat* fail* --- This query returns me results.

And when I concatenate both the query it does not return any results.

source=syslog.txt (migrat* fail*) AND (recall fail*)