topic Re: Export large search date range in raw format in Reporting

Export large search date range in raw format

kerrj712 — Sat, 28 Aug 2010 03:54:05 GMT

OK,

So I have been at this all day and I cannot see a solution. Part of my frustration is that the documentation says one thing, the wiki says another. e.g. use "quotes" no don't use quotes.

My user wants all messages for the last 60 days for a single host, sent to him in a syslog format so he can forward to the vendor.

From the GUI I can get results but it is greater than 10,000 lines so exporting it is heck! ( sorry folks but those links posted here on how to increment a export suck.)

From the command line I don't get any errors but splunk will not under any circumstances report over 60 days. Doesn't matter if I used starttime="m/d/y:h:m:s or if I use daysago=60 etc. The search will not go back far enough.

Can anyone tell me how to get ./splunk search host="foo" daysago=60 > myfoofile.txt to work?

Re: Export large search date range in raw format

Stephen_Sorkin — Sat, 28 Aug 2010 09:14:05 GMT

By default, the Splunk CLI will output 100 results. In Splunk 4.1, for simple searches, you can export an unlimited number of results from the CLI using -maxout 0. For example, you should search:

./splunk search 'host=foo earliest=-60d' -maxout 0 > myfoofile.txt

Re: Export large search date range in raw format

zscgeek — Sat, 28 Aug 2010 22:35:28 GMT

I would really love an option to do this in the GUI as well. requiring CLI access is highly suboptimal for many use cases.

Re: Export large search date range in raw format

gkanapathy — Sun, 29 Aug 2010 01:09:39 GMT

It would be helpful to know some of the use cases for a 10,000+ event manual export from the GUI, and whether all or part of whatever you need it for can be performed within Splunk itself.

Re: Export large search date range in raw format

zscgeek — Sun, 29 Aug 2010 21:51:39 GMT

Use case is sending logs for a problem report to a vendor. I don't want my users touching CLI on search head boxes AND I don't see being able to give the vendors direct splunk access. Oftin the logs for the problem that they request are far larger then 10k lines

Re: Export large search date range in raw format

kerrj712 — Mon, 30 Aug 2010 20:43:22 GMT

I agree. I won't give CLI access to the sysadmins, (separation of duties). But I would like for them to be able to do this on the GUI instead of having to ask me to pull it for them.

Re: Export large search date range in raw format

kerrj712 — Mon, 30 Aug 2010 20:46:08 GMT

Stephen, Thanks much. This did the trick. For what it is worth, I think a better option is to allow this from the GUI as a submitted batch job that is throttled back so it doesnt break the app.

I don't have a problem with allowing a user to pull large data, as long as they don't want it in 2 minutes. Let it cook low and slow so they get their file for the vendor.

Re: Export large search date range in raw format

Stephen_Sorkin — Mon, 30 Aug 2010 23:13:50 GMT

I really would like this to be in the UI as well. The main problem is how the Python appserver relays data from splunkd, but this is a technical issue that we need to work through.

Re: Export large search date range in raw format

gsawyer1 — Sat, 11 Sep 2010 02:09:21 GMT

I am trying the same thing, attempting to export windows event logs sent to the Indexer via WMI. I narrowed the CLI command down to what I expected would be a simple export of only 73 events, which I verified within the web GUI's search interface. I can see that the file gets created at the file location I specified, but the file's size remains 0KB, and I have yet to get the prompt back; I'm assuming this means that Splunk is still processing the command, but exactly how long should it take to export 73 windows events to a text file, and how will I know when the job is done? Also, what is the syntax for specifying a period of time such as from day x to day y within a range, using a CLI search command?

Re: Export large search date range in raw format

ranjyotiprakash — Wed, 13 Mar 2013 05:07:27 GMT

Thanks a lot Stephen 🙂 +2