Splunk Search
Highlighted

The Splunk search shows that events exist, but why does the Events tab show "No results found"?

Path Finder

I am trying to list specific events, but I am not able to view them. Splunk shows that events exist, but it comes up with no events found. Screenshot attached
alt text

Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

SplunkTrust
SplunkTrust

Could it be you have the No Event Sampling activated?
Take a look at this: http://docs.splunk.com/Documentation/Splunk/6.4.0/Search/Retrieveasamplesetofevents#Specify_a_sampli...

UPDATE

Take a look at the following two values from your inspect job trace:

 resultCount     0
 scanCount     287

Now take a look at their correspondent description:

resultCount - The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.

scanCount - The number of events that are scanned or read off disk

Specifically this bit:

This is the subset of scanned events
(represented by the scanCount) that
actually matches the search terms.

In summary, your search filter does not match any events you are reading off disk.
Hope that makes sense.

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

Path Finder

Event sampling returns me very few records. and not the actual set of records that match. This is a normal problem that I have even with another search that I was performing.

There was another similair query I tried and when I filter based on date month it returns value for one particular month and not the other.

I have splunk installed on my linux laptop. Could that have any implication in indexing.

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

SplunkTrust
SplunkTrust

Running on a Linux laptop shouldn't make any difference.
Could you try running your query in Smart Mode instead of Verbose?

Do you get any results when running any other query?
For instance, if you run the following:

index=_internal | head 10000

Do you get 10,000 results?

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

Path Finder

I ran the index command and it returned me 10000 results. The case is sometime when I am adding more filter let me give another example. Where I tried to look up all log entries that had recall in them and it returned me 292 results. When I added recall AND fail*. It showed me 36 entries but no results displayed. When I did an inspect job I found that event count is 36 but available event count is 0.

I guess there might be something wrong with the way I am writing the search query or may be indexing.

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

Path Finder

Looks like I dont have enough Karma points to add more files.

Scene 1 - This does not work

1st query
source=syslog.txt recall* --- Returns 292 records

2nd query
source=syslog.txt recall fail* -- Returns 36 records in event count but does not display results.

But when I try

source=syslog.txt migrat* fail* --- This query returns me results.

And when I concatenate both the query it does not return any results.

source=syslog.txt (migrat* fail) AND (recall fail)

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

SplunkTrust
SplunkTrust

Can you run one of those searches that do not return any results but then click on Job > Inspect Job, copy the report and paste it here as Code Sample (use the button with 1s and 0s above)?

Maybe there's something in the Job Inspector telling us what's going on

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

Path Finder

Below is the job inspection report I hope its format your looking for.

Execution costs
Duration (seconds)      Component   Invocations     Input count     Output count
    0.00    command.fields  5   287     287
    0.23    command.search  5   -   287
    0.13    command.search.index    6   -   -
    0.00    command.search.filter   1   -   -
    0.00    command.search.calcfields   1   287     287
    0.00    command.search.fieldalias   1   287     287
    0.00    command.search.index.usec_1_8   5,325   -   -
    0.00    command.search.index.usec_512_4096  5   -   -
    0.00    command.search.index.usec_64_512    114     -   -
    0.00    command.search.index.usec_8_64  21  -   -
    0.06    command.search.kv   1   -   -
    0.03    command.search.rawdata  1   -   -
    0.01    command.search.typer    1   287     287
    0.00    command.search.lookups  1   287     287
    0.00    command.search.summary  5   -   -
    0.00    command.search.tags     1   287     287
    0.00    dispatch.check_disk_usage   1   -   -
    0.00    dispatch.createdSearchResultInfrastructure  1   -   -
    0.08    dispatch.evaluate   1   -   -
    0.08    dispatch.evaluate.search    1   -   -
    0.28    dispatch.fetch  6   -   -
    0.23    dispatch.localSearch    1   -   -
    0.00    dispatch.readEventsInResults    1   -   -
    0.23    dispatch.stream.local   5   -   -
    0.01    dispatch.timeline   6   -   -
    0.01    dispatch.writeStatus    6   -   -
    0.03    startup.configuration   1   -   -
    0.07    startup.handoff     1   -   -
Search job properties
canSummarize    0
createTime  2016-04-20T12:48:20.000+05:30
cursorTime  2015-09-28T16:00:00.000+05:30
custom  

{
    "search": "(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*"
}

defaultSaveTTL  604800
defaultTTL  600
delegate    None
diskUsage   122880
dispatchState   DONE
doneProgress    1.0
dropCount   0
eai:acl     

{
    "app": "search", 
    "can_write": "1", 
    "modifiable": "1", 
    "owner": "admin", 
    "perms": {
        "read": [
            "admin"
        ], 
        "write": [
            "admin"
        ]
    }, 
    "sharing": "global", 
    "ttl": "600"
}

earliestTime    2012-07-08T01:30:52.000+05:30
eventAvailableCount     0
eventCount  287
eventFieldCount     0
eventIsStreaming    True
eventIsTruncated    False
eventSearch     search (source="nafx.g1303v00'" OR source="nafx.g1304v00'") opc* AND acf*
eventSorting    desc
indexEarliestTime   1460630991
indexLatestTime     1460631052
isBatchModeSearch   False
isDone  True
isFailed    False
isFinalized     False
isPaused    False
isPreviewEnabled    True
isRealTimeSearch    False
isRemoteTimeline    False
isSaved     False
isSavedSearch   False
isTimeCursored  1
isZombie    False
keywords    acf* opc* source::nafx.g1303v00' source::nafx.g1304v00'
label   None
modifiedTime    2016-04-20T12:48:27.086+05:30
normalizedSearch    litsearch ( source="nafx.g1303v00'" OR source="nafx.g1304v00'" ) opc* AND acf* | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
numPreviews     0
pid     21431
priority    5
remoteSearch    litsearch ( source="nafx.g1303v00'" OR source="nafx.g1304v00'" ) opc* AND acf* | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
reportSearch    None
request     

{
    "adhoc_search_level": "verbose", 
    "auto_cancel": "30", 
    "check_risky_command": "false", 
    "custom.search": "(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*", 
    "earliest_time": null, 
    "indexedRealtime": null, 
    "latest_time": null, 
    "preview": "1", 
    "rf": "*", 
    "sample_ratio": "1", 
    "search": "search (source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*", 
    "status_buckets": "300", 
    "ui_dispatch_app": "search"
}

resultCount     0
resultIsStreaming   True
resultPreviewCount  0
runDuration     0.388
runtime     

{
    "auto_cancel": "30", 
    "auto_pause": "0"
}

sampleRatio     1
sampleSeed  0
scanCount   287
search  search (source="nafx.g1303v00'" OR source="nafx.g1304v00'") opc* AND acf*
searchCanBeEventType    1
searchProviders     

[
    "oc8001872801.ibm.com"
]

searchTotalBucketsCount     13
searchTotalEliminatedBucketsCount   0
sid     1461136700.12
statusBuckets   300
ttl     600
Additional info     timeline search.log 
0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

SplunkTrust
SplunkTrust

Ok, I think I might have the answer.
Take a look at this two values from your inspect job trace:

 resultCount     0
 scanCount     287

Now take a look at their correspondent description:

resultCount - The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.

scanCount - The number of events that are scanned or read off disk

Specifically this bit:

This is the subset of scanned events
(represented by the scanCount) that
actually matches the search terms.

In summary, your search filter does not match any events you are reading off disk.
Hope that makes sense.

0 Karma
Highlighted

Re: The Splunk search shows that events exist, but why does the Events tab show "No results found"?

Path Finder

I am not able to understand the logic. There are something like 300,000+ events for this log file. So it scanned only 287 events and did not find any matches on these 287 events.

0 Karma