We have a data source that has a default _time variable indexed correctly on UPDATE_TIME, which we can't change, but we have some searches where we want to filter and count by OPEN_TIME. As a result, I want to set up a timerange picker that will produce an epoch value for the token value, so I can compare OPEN_TIME to the token directly. For example here's what I want to do, which would work if all time token values were returned as epoch values: index=sm9 source=incident | eval _time = strptime(OPEN_TIME,"%Y-%m-%d %H:%M:%S") | where _time>$time.earliest$ AND _time<$time.latest$ This above doesn't work because not all timerange picker values return the epoch time, they could be in the form of epoch value (e.g. 1484063893), string representation (e.g., -7d@h), or 'now'. As a result, and based on other Splunk Answers posts, I tried to catch these options with logical statements (if, case), but those approaches did not work. For example: index=sm9_us source=incident | eval _time = strptime(OPEN_TIME,"%Y-%m-%d %H:%M:%S"), earlytime=tonumber(if(isnum($time.earliest$), "$time.earliest$", relative_time(now(), "$time.earliest$"))), lasttime = tonumber(case(isnum($time.latest$), "$time.latest$", "$time.latest$"="now", now(), 1=1, relative_time(now(), "$time.latest$"))) | where _time>earlytime AND _time<lasttime The problem is that the isnum() command will throw a malformed error if the token is a preset string representation (e.g., -7d@h): Error in 'eval' command: The expression is malformed. Expected ). However, if you put quotes around the token (isnum("token")) so it runs, then any token as an epoch number (e.g., 1484067160) will always return FALSE since it'll be a string. Note: I also tried 'addinfo' command to get epoch-based fields (info_min_time, info_max_time), but that only works if you're also filtering the events based on the default indexed _time variable (which I am not; in fact that will return potentially wrong results). So my question: how can I make a time token always return as an epoch-based variable or convert the token to an epoch? Thanks ... View more

We're standing up a new Splunk environment (3 search heads in a cluster, 10 indexers), and for a few of the search-intensive dashboards, we've been getting multiple errors and warnings regarding peers. For example: Reading error while waiting for peer xxxxx004. Search results might be incomplete! Reading error while waiting for peer xxxxx005. Search results might be incomplete! Reading error while waiting for peer xxxxx006. Search results might be incomplete! Unknown error for peer xxxxx004. Search Results might be incomplete. If this occurs frequently, please check on the peer. Unable to distribute to peer named xxxx007 at uri=xxxxx:8089 using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information. Our Splunk architect has checked that the peers are up and running over multiple times, and suspects this is due to sloppy inefficient SPL code (lots of 'dedup' commands), and the PS folks from Splunk in the room suggested optimizing the code too. It only seems to happen on pages that have a good number of searches (8-10 queries). But we're worried there are underlying issues that could be causing this, not just SPL, particularly because we don't see them occur in our small dev box. Note: the data source is a relatively small data source being fed from Splunk DB Connect. And this doesn't happen in our single all-in-one dev box, but is happening in our production environment with a lot more resources. For one page, there are 8 total queries, many of which are relatively normal SPL statements (some dedups, setting _time to different date string fields, filtering dates, and timecharts). This seems to happen frequently if two people are viewing the same search-intensive dashboard, or dashboards that are search intensive hitting the same indices. E.g., I can see if it i have two tabs open on the same page or related pages loading at the the same time. We've also had a lot of saved searches not run that were schedule at the same time (i've since spread them out), suggesting to me in retrospect that we probably had a similar issue with the saved searches. What else could be leading to the issues and is this something that can be fixed with better SPL? Could there be short connectivity issues leading to temporary peer being down? Here's a typical query on the pages where this happens: <query>index=sm9_us source=interaction | fields INTERACTION_ID ASSIGNMENT TERRITORY OPEN_TIME | dedup INTERACTION_ID | search $territory$ ASSIGNMENT="*$assignFilter$*" | search ASSIGNMENT=$AssignGroup$ | eval _time = `utc_to_local_tz(OPEN_TIME)` | eval today = relative_time(now(), "+0d@d" ) | eval diff = (_time - today) / 86400 | search diff>-30 diff<0 | timechart span=1d count fixedrange=false </query> <earliest>$time_tok.earliest$</earliest> <latest>$time_tok.latest$</latest> Thanks ... View more

I have an intensive search populating a dashboard that i'd like to schedule once a day, or as requested by the user --for example, with an update submit button to re-run the search. Is there a way to schedule a saved search to run at a minimum - say every 24hrs, but also have it re-run through a user request action? I also don't think I understand the full utility of saved searches - I can see why you use them from a scheduling perspective where I currently schedule them to either populate lookups, summary indices, or through searches that call 'loadjob savedsearch=xyz'. But what do you use saved searches for if you're not scheduling them? thanks ... View more