topic Re: Why the function "strftime" not working in my search query? in Splunk Search

Why the function "strftime" not working in my search query?

chrismok — Tue, 04 Nov 2014 04:37:51 GMT

alt textIf I use this, no event return

sourcetype=abc source="*"+strftime(now(),"%Y%m%d")+"*"

But when I modify the query to

sourcetype=abc source="*20141104*"

There is a events return.

May I know is that a bug in Splunk?

Re: Why the function "strftime" not working in my search query?

vasanthmss — Tue, 04 Nov 2014 05:09:39 GMT

Hi chrismok,

Its not a bug in splunk,

strftime is a function that takes epoch time as first parameter and format human readable format like YYYYDDMM etc, based on your format string in second param.

you should use those functions in "eval".

As per your requirement this query will help you.

sourcetype=abc [|gentimes start=-1 | eval source="*"+strftime(now(),"%Y%m%d")+"*" | return source]

Thanks,
Vasu

Re: Why the function "strftime" not working in my search query?

chrismok — Tue, 04 Nov 2014 07:13:25 GMT

Thanks vasanthmss. But I found that the performance will be decreased using your approach

sourcetype=abc [|gentimes start=-1 | eval source="*"+strftime(now(),"%Y%m%d")+"*" | return source]

Run time: 3~4 seconds, Retrieve events: 18824

sourcetype=abc source="*20141104*"

Run time: 1 seconds, Retrieve events: 18824

Re: Why the function "strftime" not working in my search query?

MuS — Tue, 04 Nov 2014 07:23:14 GMT

Hi chrismok,

you're absolutly right regarding the performance. Try this instead:

... | eval file_date=strftime(now(), "%Y%m%d") | eval mySource="*" + file_date + "*" |  where source=mySource | ...

hope this helps ...

cheers, MuS

Re: Why the function "strftime" not working in my search query?

davebrooking — Tue, 04 Nov 2014 07:24:16 GMT

What if you try adding the specific field name to the search - something like
sourcetype=abc source=[|gentimes start=-1 | eval source=""+strftime(now(),"%Y%m%d")+"" | return source]

Does that improve the performance?

Re: Why the function "strftime" not working in my search query?

chrismok — Tue, 04 Nov 2014 07:52:11 GMT

Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the left hand side.

Re: Why the function "strftime" not working in my search query?

chrismok — Tue, 04 Nov 2014 07:53:45 GMT

Hi MuS.

Using your script, no record found.....

Regards,
Chris

Re: Why the function "strftime" not working in my search query?

MuS — Tue, 04 Nov 2014 08:28:46 GMT

upps my bad...try this updated command:

... | eval file_date=strftime(now(), "%Y%m%d") | eval mySource="." + file_date + "." | where match(source, mySource) | ...

Re: Why the function "strftime" not working in my search query?

chrismok — Tue, 04 Nov 2014 08:48:00 GMT

worser

Re: Why the function "strftime" not working in my search query?

davebrooking — Tue, 04 Nov 2014 09:34:20 GMT

Sorry, I was using the concept from this answer, but had no Splunk instance to test on.

I've since had a chance to try the technique on a small subset of data and noticed quite sizeable differences in execution time depending on whether you use stats count or gentimes start=-1 at the start of the subsearch with return or table as the last command in the subsearch. I'm getting consistently better execution times using

sourcetype=abc [|stats count | eval source="*"+strftime(now(),"%Y%m%d")+"*" | table source]

Re: Why the function "strftime" not working in my search query?

peter_krammer — Tue, 04 Nov 2014 09:34:53 GMT

Here is what you are looking for

sourcetype=abc [|stats count | eval source = "*"+strftime(now(),"%Y%m%d")+"*" | fields source | format]

Edited Answer to show the better performance solution found by davebrooking, but optimized a little by me.

Re: Why the function "strftime" not working in my search query?

MuS — Tue, 04 Nov 2014 09:57:41 GMT

looking at this picture, it's absolutely clear why your first search is the fastest: using any fields like index or source in the base search will speed up the search. Using a sub search will basically double search times but also speeds up the base search because you can use source in it and mine example simply does not provide any source field in the base search.

Re: Why the function "strftime" not working in my search query?

MuS — Tue, 04 Nov 2014 10:26:29 GMT

another approach just came up my mind:
if you always need today's or yesterday's date in the source name, than you could use an eval based macro containing something like this:

strftime(relative_time(time(), "-d"), "%Y%m%d")

If your macro is named yesterday you can use it like this in your searches:

 sourcetype=depmon_sys_rel_log  source=*`yesterday`* | ...

Re: Why the function "strftime" not working in my search query?

chrismok — Wed, 05 Nov 2014 01:38:37 GMT

It's not worked... as the Splunk macro is not similar to excel macro or function.... The Splunk only copy the macro string and place to the query

for example.

sourcetype=depmon_sys_rel_log source=`get_today`

Macro: get_today

strftime(now(),"%Y%m%d")

After ran the query and click the Search job inspecor. You can see that

search sourcetype=depmon_sys_rel_log source=strftime(now(),"%Y%m%d")

As a result, it is not worked

Re: Why the function "strftime" not working in my search query?

chrismok — Wed, 05 Nov 2014 01:53:13 GMT

Tested for your solution, it can provide the better performance

Re: Why the function "strftime" not working in my search query?

MuS — Wed, 05 Nov 2014 07:17:29 GMT

Well there must be something wrong, because this works for sure. If I run a search like this:

index=main source=*`yesterday`*

it becomes this litsearch

litsearch index=main source=*04-Nov-2014*

and returns events ....

Re: Why the function "strftime" not working in my search query?

chrismok — Wed, 05 Nov 2014 08:29:05 GMT

Please take a look

Re: Why the function "strftime" not working in my search query?

MuS — Wed, 05 Nov 2014 09:15:19 GMT

Make sure you tick the Use eval-based definition? in the macro settings! Then it will work and should be pretty fast as well 😉

hmm, cannot add a screenshot here???

Re: Why the function "strftime" not working in my search query?

peter_krammer — Wed, 05 Nov 2014 12:38:09 GMT

I just tried out your solution and it works, if eval-based definition is checked.
So thank you.

Re: Why the function "strftime" not working in my search query?

MuS — Wed, 05 Nov 2014 13:07:59 GMT

you're welcome 😉