topic Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually? in Splunk Search

Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Thu, 03 Nov 2016 16:56:11 GMT

Hi All,

Currently I am facing an issue with scheduled reports. The scheduled job is getting executed as per the cron job set for every two hours, but we are not seeing any events getting triggered.

On splunk --> settings--> Searches, reports, and alerts, --> saved search --> under "actions" row --> View recent --> could see 0 count under Events. When clicked on the saved search, could see No Result found. But when I ran the search, manually I am getting the result, so not sure where the problem is.

Splunk Version: 6.2.1
Scheduled saved search Details

earliest=-60m source="*dhcpd.log" Host=H*  DHCPACK lease-duration OR RENEW | localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME =if(isnull(LAST_SCAN_DATETIME ),"2000-01-01t00:00:00z", LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME =strptime(LAST_SCAN_DATETIME,"%Y-%m-%dT%H:%M:%SZ")|dedup Host | head 10 | stats last(LAST_SCAN_DATETIME ) as LAST_SCAN_DATETIME by Host IP | table Host IP LAST_SCAN_DATETIME | where LAST_SCAN_DATETIME   < relative_time(now(),"-30d")   | table Host IP LAST_SCAN_DATETIME | stats delim="+" Values(IP) AS scanning | mvcombine scanning | eval scanners="QUALYS" | eval title="Test+Assetscanning+TH" | localop | lookup qualysscanlookup title scanners scanning OUTPUT results

Time Range --> Start time and Finish time are not set
Acceleration --> Not set
Schedule and Alert --> Run on Cron Scheduled

0 */2 * * * Every 2 hours, at the 0th min
Run as --> Owner
Alert Condition --> always
Alert Mode --> Once per search
Throttling --> Not set
Expiration --> set as custom time
Severity --> Medium
Alert action --> Not set for any option under this
Summary Indexing --> Not set

As said in the beginning, for the comment, we are able to get an output when we execute the search manually, but when set in scheduled search it not generating any events. Kindly guide me how/where we are having an issue.

thanks in advance.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

somesoni2 — Thu, 03 Nov 2016 17:20:52 GMT

Few observation/questions:

1) you run the search every 2 hr but the query time range is last 60 min. There is a one hour period which you're not searching. In most cased, the frequency of the search and the time range are set in a way that you don't have gaps or overlap. You may want to look at that.
2) When you're running the search manually, are you removing the earliest=-60m from your search and manually setting the time range same as that alert was considering?

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Thu, 03 Nov 2016 17:42:33 GMT

thanks somesoni for quick response.

Even I had set for every one hour by choosing schedule type as Basic and set for 1 hour but no luck, still faced same issue.

2) When you're running the search manually, are you removing the earliest=-60m from your search and manually setting the time range same as that alert was considering?

No I am not removing it and at the same time keeping the time range to All time. But getting the result.

So not sure where exactly having an issue ? Kindly guide me on how to fix this issue.

Note: Above splunk query should find the IPS that are not scanned for last 30 days and it should add it in a file which python script will initiate the qualys scan against those IP'S**.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

lquinn — Fri, 04 Nov 2016 02:02:56 GMT

Do you have any errors in your search.log? Or any that come up when you click on view results for the saved search?

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Tue, 29 Sep 2020 11:41:21 GMT

thanks lquinn, below are the partial logs taken from search.log

11-04-2016 06:00:28.441 INFO SearchParser - PARSING: search earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10\n| stats last(LAST_SCAN_DATETIME ) as LAST_SCAN_DATETIME by Host IP | table Host IP LAST_SCAN_DATETIME | where LAST_SCAN_DATETIME < relative_time(now(),"-30d") | table Host IP LAST_SCAN_DATETIME | stats delim="+" values(IP) AS scanning | mvcombine scanning | eval scanners="QUALYS" | eval title="Test+Assetscanning+TH" | localop | lookup qualysscanlookup title scanners scanning OUTPUT results

11-04-2016 06:00:28.443 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
11-04-2016 06:00:28.563 INFO UserManager - Setting user context: test001
11-04-2016 06:00:28.564 INFO UserManager - Done setting user context: NULL -> test001
11-04-2016 06:00:28.569 WARN IniFile - /opt/splunk/etc/apps/SA-lookups/default/props.conf, line 1: Cannot parse into key-value pair: **********************************
11-04-2016 06:00:28.569 WARN IniFile - /opt/splunk/etc/apps/SA-lookups/default/props.conf, line 2: Cannot parse into key-value pair: *********** ALL HOST LOOKUP
11-04-2016 06:00:28.569 WARN IniFile - /opt/splunk/etc/apps/SA-lookups/default/props.conf, line 3: Cannot parse into key-value pair: **********************************
11-04-2016 06:00:28.645 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'bytes' in stanza [(?::){0}bro_]': bytes_in + bytes_out
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'duration' in stanza [(?::){0}bro_]': if(isnum(duration),duration,null())
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")
11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1")

And similarly when clicked the saved search, could see no result found .

Under Inspect Job -- Could see below message

This search has completed did not match any event. The term specified in the highlighted potion of the search

search earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10 |

so kindly guide me how to fix this issue.
thanks in advance.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Sat, 05 Nov 2016 16:56:14 GMT

Hi All, Can anyone guide me on this on how to fix this issue ? As we are struggling to get a fix this for very long time ....

thanks in advance

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

mattymo — Sat, 05 Nov 2016 17:10:24 GMT

I would think about starting to walk back all the splunk-fu in that search to see if perhaps one of the fields you expect to exist, doesn't. Also, who is the owner of the search that ran? Do you have a knowledge object that needs to be shared?

I believe you said that when you open the job that ran, it shows no results....what happens if start walking back the pipes to the 'base search' ?

search earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10 |

What stands out to me is:

eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME )

if only due to where splunk will place those results....but I mean could just be my ignorance of the data used in this search...

But maybe lets start by just running this over the same window the sched job did:

 source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW

Side note: what kind of environment is this search taking place in? Standalone? Distributed?

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Tue, 29 Sep 2020 11:41:03 GMT

Hi mmodestino, thanks for your effort on this.

earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10 |

I am getting an output when I ran this search manually but when the same search is scheduled it show no result, i am not sure where is the problem ? Ours is distributed environment but how this will have an impact on this issue and we are using splunk 6.2.1 version.

Results fetched when we ran the search manually :

Scanning result scanner Title
IP address success QUALYS Test+Assetscanning+TH

Kindly guide me how to fix this issue.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

mattymo — Sat, 05 Nov 2016 18:07:09 GMT

What app is the saved search in??

Are you running it manually using Admin?

Does Admin own the alert?

can you show the job and job inspector output of a working one vs a broken one?

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Sat, 05 Nov 2016 18:26:36 GMT

1) What app is the saved search in??

Its in Search app

2) Are you running it manually using Admin?

    Yes using admin privilege, but the scheduled search is saved / run with another user name and he also has the admin privilege.

3) Does Admin Own the alert ?

I am not sure how to check this out

4) Do you want the entire output of Inspector output as it has lots of date into it.

thanks in advance .

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

mattymo — Sat, 05 Nov 2016 18:41:32 GMT

can you login as the user who is running the search and see of they can see the index with the dhcp events...also can they run the lookup u are using?

job inspector output probably has owner info...just share on pastebin... one working one failing

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Sat, 05 Nov 2016 19:21:42 GMT

No I will not be able to login using the user ID who is the owner of the search. I hope he should be able to see the index and lookup detail as he is having the same role as I have and he is the actual owner of the saved search.

just share on pastebin ? i do not have an idea about the paste bin can you tell me how do i share this using paste bin..

thanks

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Tue, 29 Sep 2020 11:41:43 GMT

Job inspector details

For the Job which got the output using my id :

This search has completed and has returned 1 result by scanning 419 events in 50.576 seconds.

The following messages were returned by the search subsystem:

INFO: Assuming implicit lookup table with filename 'qualys_hostlist.csv'.
INFO: Your timerange was substituted based on your search string

For the job which failed using user id :

This search has completed, but did not match any events. The terms specified in the highlighted portion of the search:

search earliest=-60m source="dhcpd.log" Host=H DHCPACK lease-duration OR RENEW | localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME =if(isnull(LAST_SCAN_DATETIME ),"2000-01-01t00:00:00z", LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME =strptime(LAST_SCAN_DATETIME,"%Y-%m-%dT%H:%M:%SZ")|dedup Host | head 10 | stats last(LAST_SCAN_DATETIME ) as LAST_SCAN_DATETIME by Host IP | table Host IP LAST_SCAN_DATETIME | where LAST_SCAN_DATETIME < relative_time(now(),"-30d") | table Host IP LAST_SCAN_DATETIME | stats delim="+" Values(IP) AS scanning | mvcombine scanning | eval scanners="QUALYS" | eval title="Test+Assetscanning+TH" | localop | lookup qualysscanlookup title scanners scanning OUTPUT results
over the time range:

11/5/16 2:00:00.000 PM – 11/5/16 3:00:36.929 PM
did not return any data. Possible solutions are to:

relax the primary search criteria
widen the time range of the search
check that the default search indexes for your account include the desired indexes
This search is an instance of the saved search: Test+Assetscanning+TH.

The following messages were returned by the search subsystem:

INFO: Assuming implicit lookup table with filename 'qualys_hostlist.csv'.
INFO: No matching fields exist
INFO: Your timerange was substituted based on your search string

thanks in advance

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

mattymo — Sat, 05 Nov 2016 19:59:00 GMT

I would suggest proving your assumptions about access, just to be sure.

11/5/16 2:00:00.000 PM – 11/5/16 3:00:36.929 PM
did not return any data. Possible solutions are to:


relax the primary search criteria
widen the time range of the search
check that the default search indexes for your account include the desired indexes
This search is an instance of the saved search: Test+Assetscanning+TH.

Just like the job is telling you...relax the search criteria for that user...can they see output after running this search??? :

source="dhcpd.log" Host=H* DHCPACK lease-duration OR RENEW

*Make sure those examples above are a typo....your host field needs to be host=H* not host=H (im sure they are...probably just answers removing the asteriks)

INFO: Assuming implicit lookup table with filename 'qualys_hostlist.csv' means the lookup table isn't defined explicitly...might be a good idea to set the lookup table as a shared knowledge object.

have the user try this search:

| intputlookup qualys_hostlist.csv

what do they get???

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Sun, 06 Nov 2016 17:31:41 GMT

hi mmodestino thanks for your quick response on this issue.

After executing this command | intputlookup qualys_hostlist.csv , we could see all the fields that are present in the spread sheet available in statistics in table format.

Kindly let me know how to share the search.log in answers.com as its having huge number of line it might occupy most of the characters. So is there a way, where I can share the log details to others or to you, in order to diagnose this problem.

thanks in advance.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Sun, 06 Nov 2016 19:00:03 GMT

Hi mmodestino, can you guide me on this as this issue is there for long time. Need to find the fix for this. Kindly suggest or guide me on this.

thanks in advance.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

mattymo — Sun, 06 Nov 2016 19:52:43 GMT

Hey!

Ok, so you logged in as the other user and they can see the lookup file? Thats good, how about the events??

Does the other user see anything when running source="dhcpd.log" Host=H* DHCPACK lease-duration OR RENEW

Slowly build up the search, pipe by pipe, with the other account till you find the problem...

You can share a link to http://pastebin.com/

Get a free account and paste it there.

Re: Why is my scheduled search producing a count of zero, but get results when I run the search manually?

Hemnaath — Tue, 29 Sep 2020 11:41:59 GMT

Hi mmodestino, yes he can see the details when he execute the below search

source="dhcpd.log" Host=H* DHCPACK lease-duration OR RENEW.

Now I could see the below message popping out in splunk portal --> message

msg="A scripted exited abnormally" input="opt/splunk/etc/apps/TA-Qualys/bin/download_qualys_assets.sh" stanza="default" status

Hey I have registered in pastebin site, but not sure whether I can share the information.

thanks in advance.