topic Re: Variable File Name in outputcsv in Reporting

Variable File Name in outputcsv

simonattardGO — Mon, 28 Sep 2020 10:23:11 GMT

Hi,

I need to run a scheduled search to export some logs every certain amount of time. The search I am using is this:

outputtext usexml=false | rename _xml as raw | fields raw | fields - _* | outputcsv results.txt

The problem is that each time the search runs, results.txt gets overridden. I would like to automatically append the time and date to the name of the file Eg. results_3-2-12_12-00.txt

Is this possible?

Thanks in advance.

Re: Variable File Name in outputcsv

Ayn — Fri, 03 Feb 2012 11:57:23 GMT

You can do this through some subsearch ugliness (or beauty, I guess it's in the eye of the beholder 🙂 )

Subsearches work much like backticks in most UNIX shells, i.e. they run first of all and then return their results back to the outer query. You can put a subsearch anywhere in your search pipeline, including after outputcsv. By default however, a subsearch returns a string that is formatted for being used by the search command. You can change this behaviour by calling format (http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Format) to make sure the formatting suits your purposes.

The idea here would be to create a dynamic value for the filename in the subsearch, then return that filename to outputcsv.

... | outputcsv [search * | head 1 | eval query="results_".strftime(now(),"%d_%m_%y_%H_%M_%S") | fields query | format "" "" "" "" "" ""]

I don't know your level of Splunk-foo so let me know if you want more explanation on the internal workings of the search. I used now() as a method for getting the date/time that shoul be used when naming the results file - you might want to use another time, but if the current time is OK, just use now().

Re: Variable File Name in outputcsv

simonattardGO — Fri, 03 Feb 2012 12:11:17 GMT

Thanks a lot for your response Ayn.
I tried your suggestion, but I am getting the following error:
This search cannot be parsed when parse_only is set to true

What is the reason for this error?

Re: Variable File Name in outputcsv

Ayn — Fri, 03 Feb 2012 12:14:25 GMT

I don't think that's an actual search error (I'm getting it as well), it's just a message from the search assistant that is used for helping you in some situations with the text you enter into the search field.

Re: Variable File Name in outputcsv

simonattardGO — Fri, 03 Feb 2012 12:28:26 GMT

Wonderful!! 🙂 Thanks a lot for that, it works very well.

The only issue I have is that when the file is outputted, each log line is enclosed in double quotes.

Do you know the reason for that?

Re: Variable File Name in outputcsv

Ayn — Fri, 03 Feb 2012 12:31:47 GMT

outputcsv uses double quotes to enclose some fields. It shouldn't be enclosing complete lines.

Re: Variable File Name in outputcsv

simonattardGO — Mon, 28 Sep 2020 10:23:13 GMT

This is the command I am using:

You see anything wrong?

Re: Variable File Name in outputcsv

Ayn — Fri, 03 Feb 2012 12:45:11 GMT

Well you're writing the raw data to the xml field, so Splunk encloses that whole field in double quotes. That is standard behaviour.

Re: Variable File Name in outputcsv

simonattardGO — Fri, 03 Feb 2012 12:53:09 GMT

Thanks for your help. How can I output the raw text file, without enclosing it in an xml field?

Re: Variable File Name in outputcsv

Ayn — Fri, 03 Feb 2012 12:54:46 GMT

For one, you're taking a detour by using outputtext. Check this thread for some inspiration: http://splunk-base.splunk.com/answers/5757/export-raw-logs-from-splunk

Re: Variable File Name in outputcsv

simonattardGO — Fri, 03 Feb 2012 13:00:54 GMT

Thank you very much! 🙂

Re: Variable File Name in outputcsv

Ayn — Fri, 03 Feb 2012 13:32:45 GMT

np. Could you please mark the answer as accepted? Thanks!

Re: Variable File Name in outputcsv

sdwilkerson — Mon, 11 Jun 2012 19:54:50 GMT

Ayn rocks, and this answer saved my butt. I'd give multiple up-votes if I could. Thanks.

Re: Variable File Name in outputcsv

_d_ — Tue, 12 Jun 2012 03:23:53 GMT

That's good, but here's a less ugly one 🙂

<my search here> | outputcsv [ | stats count | eval filename=strftime(now(), "results_%d_%m_%y_%H_%M_%S") | return $filename]

Enjoy it!

Re: Variable File Name in outputcsv

ShaneNewman — Mon, 28 Sep 2020 13:24:09 GMT

I am trying to use this. It will create a file with the correct file name, it just has no contents... Any Ideas?

my command:
outputcsv [ search * | head 1 | eval query="All_lab_new_".strftime(now(),"%b_%d_%Y") | fields query | format "" "" "" "" "" ""]

When I run the search without the outputcsv command I get results...

Re: Variable File Name in outputcsv

crt89 — Wed, 06 Mar 2013 07:21:14 GMT

Thank you ! I'll be using this one. 🙂

Re: Variable File Name in outputcsv

Lowell — Fri, 18 Apr 2014 16:12:18 GMT

The macro approach. Here's yet another approach. You can use an eval-based macro to return the current timestamp and drop that into your search string. This is especially helpful if you want to use this pattern multiple places.

macros.conf

[timestamp]
definition = strftime(time(), "%Y%m%d_%H%M")
iseval = 1

Then your search would look like this:

<my search here> | outputcsv results_`timestamp`.csv

There's a subtle difference between these answers because of the use of time() vs now(), but for many cases it will not matter. Quoting steveyz,

time() is the wall clock time. now() is the "nominal" start time of the search. For example, the scheduler may run a search that is supposed to start at 2PM but really started at 2:15pm, now() would still be 2pm)

See stevyz answer on Can you use now() in eval based macros? for more details.

Re: Variable File Name in outputcsv

varad_joshi — Wed, 09 Sep 2015 10:40:49 GMT

I used this one 😄

Re: Variable File Name in outputcsv

salbayrak — Tue, 12 Nov 2019 19:45:12 GMT

I also used this one, but replaced "| stats count" with "| makeresults"

Re: Variable File Name in outputcsv

Guarddogmonitor — Thu, 25 Feb 2021 23:21:44 GMT

Part of my snippet folder now! really helpful. thanks