topic Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations? in Getting Data In

Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

sai_john — Tue, 24 Jan 2017 23:14:31 GMT

Is there a feature in Splunk (like Dropbox) to drop all types of logs from different applications ?

Where can i drop in multiple log files with multiple log types (.csv/.txt/.log) from multiple locations?

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

DalJeanis — Wed, 25 Jan 2017 00:09:13 GMT

Splunk really doesn't care where you put them. You just tell the indexer what directories to watch, and it does the rest.

You can look up details regarding forwarders (that move the data from place to place) and fishbuckets (part of how splunk tracks what it's done already), but really you should just start here -

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Getstartedwithgettingdatain

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Howdoyouwanttoadddata

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

jkat54 — Wed, 25 Jan 2017 01:41:23 GMT

Just be sure you understand that Splunk is not a storage engine. It's a machine data analytics platform.

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

sai_john — Wed, 25 Jan 2017 01:58:01 GMT

yes i know that. we need this kind of dropbox feature to store sample log files in lower environment which is used to play with the data to create some dashboards and others.

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

jkat54 — Wed, 25 Jan 2017 02:13:19 GMT

So let's peel that onion... Are all *.log files going to have the same data formats?

If so you can easily monitor one directory like this

[monitor:///var/dropfolder/*.log]
sourcetype=myLogs
index=myIndex
crcSalt=<SOURCE>

However, if each .log file has different formats, then you'll need to be more specific like this:

[monitor:///var/dropfolder/app1*.log]
sourcetype=myApp1Log
index=myIndex
crcSalt=<SOURCE>

[monitor:///var/dropfolder/otherApp*.log]
sourcetype=myOtherAppLog
index=myIndex
crcSalt=<SOURCE>

You need to do this because later you'll want to perform field extractions on each log type, and the regex or other methods to extract those fields will be different for each data format you have.

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

sai_john — Wed, 25 Jan 2017 03:10:17 GMT

yeah this is the regular process to add data from different data formats.
But i am not looking to get data from different servers/data format since i just want to place my sample log files(.csv / .txt) to play with that sample data. Does Splunk provides an appor some other feature to place/drop sample multiple log files and then play around that by extracting fields?

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

jkat54 — Wed, 25 Jan 2017 03:21:04 GMT

You would create a wild carded monitor stanza

[monitor:///var/dropfolder/*]

If you don't specify a sourcetype, Splunk will attempt to detect one for you. It should be fine for csv, JSON, IIS logs, Apache logs, and a hand full of other known log types.

But I'm telling it's going to get ugly...

There is no app for this because everyone's data is different and people would constantly complain about the app.

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

sai_john — Wed, 25 Jan 2017 03:57:58 GMT

yeah i understood that. you are talking about adding new data from different servers/data formats. you are correct in that perspective. But i am talking about sample files which are in my local desktop.

For example I have exported 10 sample log files(lets talk about only csv files) from splunk from 10 sourcetypes into my local desktop.

Now i want to place/drop these 10 .csv sample files into __(kind of dropbox /some app)__ to search from that sample data which is present in those 10 csv files.

i want to know if there is an app/feature in splunk similar to dropbox to place these 10 csv files and then tweek.

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

gcusello — Wed, 25 Jan 2017 06:36:23 GMT

Hi sai_John,
The only way to drop a file after indexing is to schedule a script that deletes files older than a time.
Bye.
Giuseppe

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

gcusello — Wed, 25 Jan 2017 06:36:23 GMT

Hi sai_John,
The only way to drop a file after indexing is to schedule a script that deletes files older than a time.
Bye.
Giuseppe

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

jkat54 — Wed, 25 Jan 2017 15:48:37 GMT

Yes, you do what I've said.

Create an input that is monitoring a folder on your machine for *, then you drop the files in that folder.

Re: Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

jkat54 — Wed, 25 Jan 2017 15:50:46 GMT

If you're talking about editing the files in the folder, then no. Splunk is not a UI for editing data like Google docs.