Search returned no results.

[pdgill314]

We had searched that were created and running every night and were outputting results. But lately we noticed that our emails had no results in them. When I looked into it I found these errors:

DEBUG: Disabling timeline and fields picker for reporting search due to adhoc_search_level=smart
DEBUG: base lispy: [ AND host::172.16.20.121 idp ]
DEBUG: search context: user="pdgill314", app="launcher", bs-pathname="L:\Program Files\Splunk\etc"
INFO: No matching fields exist
INFO: Your timerange was substituted based on your search string

The search query we are attempting to use is this:

search host="172.16.20.121" idp earliest="-24h@h" 
| table idp_tdstamp, idp_srcAddr, idp_dstAddr, idp_attackSignature, idp_action 
| rename idp_tdstamp as "TIME STAMP", idp_srcAddr as "SRC ADDR", idp_dstAddr as "DST ADDR", idp_attackSignature as "SIGNATURE", idp_action as "ACTION"

Any thoughts on why this would work before and not work now?

EDIT:
Do I have to go back and "retrain" Splunk?
I ask because I ran this query and got no results:

search sourcetype="cisco_wsa_squid" TCP_DENIED earliest="-24h@h" NOT (facebook.com OR .crl) | top cs_url, proxy_client limit=100 | table count, proxy_client, cs_url, percent | rename count as "Frequency", proxy_client as "Client IP", cs_url as "URL", percent as "Percentage"

Upon inspection it showed:
This search has completed and found 323,136 matching events. However, the transforming commands in the highlighted portion of the following search:

search sourcetype="cisco_wsa_squid" TCP_DENIED earliest="-24h@h" NOT (facebook.com OR .crl) | top cs_url, proxy_client limit=100 | table count, proxy_client, cs_url, percent | rename count as "Frequency", proxy_client as "Client IP", cs_url as "URL", percent as "Percentage"

over the time range:

1/9/13 11:00:00.000 AM – 1/10/13 11:45:47.000 AM

generated no results.

Possible solutions are to:
-check the syntax of the commands
-verify that the fields expected by the report commands are present in the events

When I went back and used the "Interactive field extractor"
It gave me notices like:

Note: most of the values you want may already be extracted in the 'c_ip' field. 

After I did the field extractor, I reran the same search query and this time it generated results.

Thoughts? (also see my comments on the answer)

[lguinn2 · Accepted solution]

Based on the updated question, the other answers and comments:

The field extractions still exist, but the permissions on the field extractions is set so that only certain people and/or certain apps can use them.

As the Splunk admin, go to Manager » Fields » Field extractions

You will find a list of all the fields. They are named in a strange way, so the easiest thing may be to search for them using the search box on the upper right.

For each field, check the Permissions link. Users only need READ permission to use the fields. Also, make sure that the fields are Global if you want them to be available in all apps.

HTH

[pdgill314]

lguinn,
Your logic is the same as mine, right when I saw this answer I was playing with the field extraction permissions. I guess a previous user who was the main administrator of Splunk did not set permissions correctly, so that when left, the permissions left with him.

Thanks for all of your help!

[lguinn2]

Well, my first thought is that the data has changed. Or maybe you are no longer indexing the data that you were before. Or maybe someone has messed with your field extractions.

BTW, none of these messages are errors - they aren't even warnings. Splunk appears to be working properly.

What do you get when you run this search manually?

host="172.16.20.121" idp earliest="-24h@h"

Do you get any results? What if you run the following search? What if you extend the time range to a week? 30 days?

host="172.16.20.121" 

If you do get results for either of these searches, can you find the fields in the gray box on the the left?

Let us know!

[pdgill314]

What's interesting is that it looks like 2 of the same jobs were run that day (11/23) by 2 different users. 1 returned results & the other didn't. They were ran @ 1:29am and 1:30am. So I pulled the query terms out of the email and they have the same query.

Alert was triggered because of: 'Saved Search [IDP Hits Last 24-Hours]: always(0)'  
No results.

Alert was triggered because of: 'Saved Search [IDP Hits Last 24-Hours]: always(91)'
TIME STAMP  SRC ADDR    DST ADDR    SIGNATURE   ACTION
2012/11/23 05:12:48 10.10.10.131    66.235.143.121  HTTP:XSS:HTML-SCRIPT-IN-URL-PRM conn dropped

[Drainy]

Perhaps the syslog format has changed which has broken your field extractions? If nothing on the Splunk side has changed then something external to it must have, e.g. the data or the logging format by whatever it is logging (I'm just assuming syslog here)

[pdgill314]

It would previously produce a table like this:
TIME STAMP SRC ADDR DST ADDR SIGNATURE ACTION
2012/11/23 05:12:48 10.10.10.131 66.235.143.121 HTTP:XSS:HTML-SCRIPT-IN-URL-PRM conn dropped
2012/11/23 05:08:01 10.10.10.131 66.235.143.121 HTTP:XSS:HTML-SCRIPT-IN-URL-PRM conn dropped

[pdgill314]

When just the host is searched I get the following data.

 Jan 10 08:46:17 172.16.20.121 Jan 10 08:46:17 127.0.0.1 20130110, 174, 2013/01/10 13:46:14, 2013/01/10 13:46:13, global, 33, idp1.<redacted>.lcl, 172.16.20.117, predefined, HTTP:XSS:HTML-SCRIPT-IN-URL-PRM, (NULL), (NULL), 10.10.10.132, 52147, 0.0.0.0, 0, (NULL), (NULL), 76.13.34.25, 80, 0.0.0.0, 0, tcp, global, 33, <redacted>-default, idp, 10, 0, conn dropped, major, no, interface=eth11, (NULL), (NULL), (NULL), 0, 0, 0, 0, 0, 0, 0, 0, no, 31, Not Set, idp  

The fields that were searched before no longer exist.