thanks Masa, but in case of test environment we will have only one splunk instance (free license), so in this case how do you test. Whereas in case of production we have two instance specifically designed as Splunk search head portal and connected with another server for Shared search head pooling. Going for the splunk professional that is not under my control as it should be decided by management. So could you please provide me the exact steps that I need to follow to do the upgrade as its pending for many months. Kindly guide me on this. thanks in advance. ... View more

Hi mmodestino, yes he can see the details when he execute the below search source="dhcpd.log" Host=H* DHCPACK lease-duration OR RENEW. Now I could see the below message popping out in splunk portal --> message msg="A scripted exited abnormally" input="opt/splunk/etc/apps/TA-Qualys/bin/download_qualys_assets.sh" stanza="default" status Hey I have registered in pastebin site, but not sure whether I can share the information. thanks in advance. ... View more

thanks mmodestino for throwing some lights on this issue. ... View more

Hi mmodestino, can you guide me on this as this issue is there for long time. Need to find the fix for this. Kindly suggest or guide me on this. thanks in advance. ... View more

Taken only particular stanza related to bluecoat_syslogs from Admin-HVY-forwarder app /opt/splunk/etc/apps/Admin-HVY-forwarder/default [monitor:///opt/syslogs/proxy/.../*bluecoat.log] whitelist = .log$ sourcetype = bluecoat_syslog index = net_proxy host_segment = 4 ... View more

thanks mmodestino, but how to figure out whether we are running the debug log level in splunk ? ... View more

hi mmodestino thanks for your quick response on this issue. After executing this command | intputlookup qualys_hostlist.csv , we could see all the fields that are present in the spread sheet available in statistics in table format. Kindly let me know how to share the search.log in answers.com as its having huge number of line it might occupy most of the characters. So is there a way, where I can share the log details to others or to you, in order to diagnose this problem. thanks in advance. ... View more

thanks mmodestino for your quick response on this. I could see two props.conf file for bluecoat_syslog. One is under the app name called TA-Bluecoat and Another app name Admin-HVY-Forwarder. Under TA-Bluecoat app I do not see any inputs.conf file defined, whereas under Admin-HVY-Forwarder could see inputs.conf defined but props.conf is not defined for bluecoat. App name Admin-HVY-Forwarder - Props.conf [host::Tesx*] TZ = GMT [host::TESX*] TZ = GMT [f5_web_server] TIME_PREFIX = f5_time=" TRANSFORM-time = f5_syslog_time Under app name TA-bluecoat, could see this configuration setup Props.conf detail [source::....bluecoat] sourcetype = bluecoat [bluecoat] SHOULD_LINEMERGE=false KV_MODE = none REPORT-0auto_kv_for_bluecoat = auto_kv_for_bluecoat LOOKUP-vendor_info_for_bluecoat = bluecoat_vendor_info_lookup sourcetype OUTPUT vendor,product MAX_TIMESTAMP_LOOKAHEAD = 19 TIME_FORMAT = %Y-%m-%d %T TRANSFORM-main = nullPound TRANSFORMS-bluecoat_host_override = bluecoat_host_override TZ = GMT [bluecoat_syslog] SHOULD_LINEMERGE=false KV_MODE = none REPORT-0auto_kv_for_bluecoat = auto_kv_for_bluecoat_syslog LOOKUP-vendor_info_for_bluecoat = bluecoat_vendor_info_lookup sourcetype OUTPUT vendor,product TIME_PREFIX = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\s\w+.\w+.\w+. TIME_FORMAT = %Y-%m-%d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 TRANSFORM-main = nullPound TRANSFORMS-bluecoat_host_override = bluecoat_host_override thanks in advance. ... View more

Job inspector details For the Job which got the output using my id : This search has completed and has returned 1 result by scanning 419 events in 50.576 seconds. The following messages were returned by the search subsystem: INFO: Assuming implicit lookup table with filename 'qualys_hostlist.csv'. INFO: Your timerange was substituted based on your search string For the job which failed using user id : This search has completed, but did not match any events. The terms specified in the highlighted portion of the search: search earliest=-60m source="dhcpd.log" Host=H DHCPACK lease-duration OR RENEW | localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME =if(isnull(LAST_SCAN_DATETIME ),"2000-01-01t00:00:00z", LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME =strptime(LAST_SCAN_DATETIME,"%Y-%m-%dT%H:%M:%SZ")|dedup Host | head 10 | stats last(LAST_SCAN_DATETIME ) as LAST_SCAN_DATETIME by Host IP | table Host IP LAST_SCAN_DATETIME | where LAST_SCAN_DATETIME < relative_time(now(),"-30d") | table Host IP LAST_SCAN_DATETIME | stats delim="+" Values(IP) AS scanning | mvcombine scanning | eval scanners="QUALYS" | eval title="Test+Assetscanning+TH" | localop | lookup qualysscanlookup title scanners scanning OUTPUT results over the time range: 11/5/16 2:00:00.000 PM – 11/5/16 3:00:36.929 PM did not return any data. Possible solutions are to: relax the primary search criteria widen the time range of the search check that the default search indexes for your account include the desired indexes This search is an instance of the saved search: Test+Assetscanning+TH. The following messages were returned by the search subsystem: INFO: Assuming implicit lookup table with filename 'qualys_hostlist.csv'. INFO: No matching fields exist INFO: Your timerange was substituted based on your search string thanks in advance ... View more

No I will not be able to login using the user ID who is the owner of the search. I hope he should be able to see the index and lookup detail as he is having the same role as I have and he is the actual owner of the saved search. just share on pastebin ? i do not have an idea about the paste bin can you tell me how do i share this using paste bin.. thanks ... View more

1) What app is the saved search in?? Its in Search app 2) Are you running it manually using Admin? Yes using admin privilege, but the scheduled search is saved / run with another user name and he also has the admin privilege. 3) Does Admin Own the alert ? I am not sure how to check this out 4) Do you want the entire output of Inspector output as it has lots of date into it. thanks in advance . ... View more

Hi mmodestino, thanks for your effort on this. earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10 | I am getting an output when I ran this search manually but when the same search is scheduled it show no result, i am not sure where is the problem ? Ours is distributed environment but how this will have an impact on this issue and we are using splunk 6.2.1 version. Results fetched when we ran the search manually : Scanning result scanner Title IP address success QUALYS Test+Assetscanning+TH Kindly guide me how to fix this issue. ... View more

Hi All, Can anyone guide me on this on how to fix this issue ? As we are struggling to get a fix this for very long time .... thanks in advance ... View more

thanks lquinn, below are the partial logs taken from search.log 11-04-2016 06:00:28.441 INFO SearchParser - PARSING: search earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10\n| stats last(LAST_SCAN_DATETIME ) as LAST_SCAN_DATETIME by Host IP | table Host IP LAST_SCAN_DATETIME | where LAST_SCAN_DATETIME < relative_time(now(),"-30d") | table Host IP LAST_SCAN_DATETIME | stats delim="+" values(IP) AS scanning | mvcombine scanning | eval scanners="QUALYS" | eval title="Test+Assetscanning+TH" | localop | lookup qualysscanlookup title scanners scanning OUTPUT results 11-04-2016 06:00:28.443 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered. 11-04-2016 06:00:28.563 INFO UserManager - Setting user context: test001 11-04-2016 06:00:28.564 INFO UserManager - Done setting user context: NULL -> test001 11-04-2016 06:00:28.569 WARN IniFile - /opt/splunk/etc/apps/SA-lookups/default/props.conf, line 1: Cannot parse into key-value pair: ********************************** 11-04-2016 06:00:28.569 WARN IniFile - /opt/splunk/etc/apps/SA-lookups/default/props.conf, line 2: Cannot parse into key-value pair: *********** ALL HOST LOOKUP 11-04-2016 06:00:28.569 WARN IniFile - /opt/splunk/etc/apps/SA-lookups/default/props.conf, line 3: Cannot parse into key-value pair: ********************************** 11-04-2016 06:00:28.645 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'bytes' in stanza [(?::){0}bro_]': bytes_in + bytes_out 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'duration' in stanza [(?::){0}bro_]': if(isnum(duration),duration,null()) 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") 11-04-2016 06:00:28.646 INFO CalcFieldProcessor - Found valid eval expression for field 'idm_flags' in stanza [default]': if(isnull(orig_host), "0", "1").if(isnull(src), "0", "1").if(isnull(dest), "0", "1").if(isnull(dvc), "0", "1").if(isnull(src_user), "0", "1").if(isnull(user), "0", "1") And similarly when clicked the saved search, could see no result found . Under Inspect Job -- Could see below message This search has completed did not match any event. The term specified in the highlighted potion of the search search earliest=-60m source="dhcpd.log" Host=W DHCPACK lease-duration OR RENEW |localop | lookup qualys_hostlist.csv NETBIOS AS Host OUTPUT LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = if(isnull(LAST_SCAN_DATETIME ),"2000-01-01T00:00:00Z",LAST_SCAN_DATETIME ) | mvexpand LAST_SCAN_DATETIME | eval LAST_SCAN_DATETIME = strptime(LAST_SCAN_DATETIME , "%Y-%m-%dT%H:%M:%SZ") | dedup Host | head 10 | so kindly guide me how to fix this issue. thanks in advance. ... View more

thanks somesoni for quick response. 1) you run the search every 2 hr but the query time range is last 60 min. There is a one hour period which you're not searching. In most cased, the frequency of the search and the time range are set in a way that you don't have gaps or overlap. You may want to look at that. Even I had set for every one hour by choosing schedule type as Basic and set for 1 hour but no luck, still faced same issue. 2) When you're running the search manually, are you removing the earliest=-60m from your search and manually setting the time range same as that alert was considering? No I am not removing it and at the same time keeping the time range to All time. But getting the result. So not sure where exactly having an issue ? Kindly guide me on how to fix this issue. Note: Above splunk query should find the IPS that are not scanned for last 30 days and it should add it in a file which python script will initiate the qualys scan against those IP'S**. ... View more

thanks sekar. ... View more

Hi All, Can any one clarify my doubts on Splunk scheduled reports. 1) where/how to find out whether the scheduled query had fetched the results after executing the query. 2) I had configured to schedule a cron job to execute every 2 hours and based on the above comments, like this ++/2 +++ Every 2 hours, on all days of the weeks /months. Kindly correct me if this not the right way to schedule the cron job. Note - Instead of star symbol I have used + to describe cron set up as I am unable to use star symbol in comments. thanks in advance. ... View more

thanks sekar, but I am having few doubts on scheduled saved searches ? 1) where / how to find out whether the scheduled query had fetched the results. 2) I want to schedule a cron job to execute every 2 hours and I had seen your above comments, based on that I had scheduled the cron like this. ++/2 +++ Every 2 hours, on all days of the weeks /months Note - Instead of star symbol I have used + to describe cron set up as I am unable to use star symbol in comments. But it seems it taking it has every 2 min, so kindly correct me on this too if this is not right way to configure the cron job for 2 hours. thanks in advance. ... View more

thanks Giuseppe for providing the query but when I executed the above query I am getting result only for splunk component related to Index instances, not for the other components like search head, Heavy Forwarder. Is it possible to get those details in the same query. thanks in advance. ... View more

Hi Giuseppe, I ran the above query and got no result. Kept the duration for an hour / day, but no luck. Kindly guide me in getting the required details. thanks in advance. ... View more

thanks sekar, I need to set the cron job to execute the script every two hours and I have set the cron job but not sure why its not triggering the alert. Kindly guide we whether cron job is set correctly or not . Schedule Job for every two hours in a day. 00 /2 *** thanks in advance. ... View more