topic Re: Indexing Zipped Files in Getting Data In

Indexing Zipped Files

mhtedford — Fri, 21 Jul 2017 20:11:45 GMT

I have about 500 excel files that I need to index into Splunk.

If I upload each file individually, I pick my sourcetype in the Add Data wizard and all the events show up correctly.

If I zip all the files together into a single file, I select the same sourcetype, but I cannot see a preview of the sample events: http://imgur.com/a/Un4xL

Splunk then gets confused when parsing the time stamp from the zipped file, and events show up with the wrong time.

Here are the sourcetype settings I'm trying to use: http://imgur.com/a/5F4bK

Is there a way to make the events load correctly for the zipped file, instead of uploading all 500 files individually?

Re: Indexing Zipped Files

niketn — Fri, 21 Jul 2017 20:50:08 GMT

@mhtedford, is the intent of zipping the file only to upload multiple files to Splunk index in single shot, or the CSV files are created as zip through your existing system/application?

If individual file upload is working fine, and there is not hard and fast need to upload a zip file, then you can choose Monitor folder option instead of Upload file. You can put all the files to the folder, and Splunk should pick them up.

PS: Monitor Folder allows you to select folder from UI (instead of individual file).

Re: Indexing Zipped Files

woodcock — Sat, 22 Jul 2017 00:34:29 GMT

What you need is the add oneshot command from the CLI. Write a small script to shoot each file (do not ZIP them all together) and pass in the sourcetype as a parameter so that your timestamping is done correctly as per your configuraitons for that sourcetype:

https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Data/MonitorfilesanddirectoriesusingtheCLI

Re: Indexing Zipped Files

mhtedford — Mon, 24 Jul 2017 13:28:38 GMT

@niketnilay

The intent of zipping the file is only to upload multiple files to Splunk index in a single shot.

I'm trying to use the Monitor folder option, but I am having trouble finding my folder: http://imgur.com/a/OfZZA

It's currently located on my desktop, but the folder is empty in the Splunk wizard. Please advise

Re: Indexing Zipped Files

niketn — Mon, 24 Jul 2017 17:26:49 GMT

What is the folder name and path? You can also directly set the path using text box in the Splunk UI.

Monitor Folder will should folders and not files since by default it will monitor all the files inside the folder (unless you want to restrict the same through Whitelist and/or Blacklist).

In the screenshot attached you have selected entire c drive. For adding a folder on your desktop you should navigate to Users folder and then to your logged in username folder.

Re: Indexing Zipped Files

mhtedford — Mon, 24 Jul 2017 17:35:06 GMT

This is the error I get when I try to set the path directly: http://imgur.com/a/hAStX

When I navigate to the Users folder and then my username, all the folders are empty. I think the permissions might not allow, and I'm not sure how to fix that.