Getting Data In

Is there a feature in Splunk (similar to Dropbox) where can i drop multiple log files from multiple locations?

sai_john
New Member

Is there a feature in Splunk (like Dropbox) to drop all types of logs from different applications ?

Where can i drop in multiple log files with multiple log types (.csv/.txt/.log) from multiple locations?

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

So let's peel that onion... Are all *.log files going to have the same data formats?

If so you can easily monitor one directory like this

[monitor:///var/dropfolder/*.log]
sourcetype=myLogs
index=myIndex
crcSalt=<SOURCE>

However, if each .log file has different formats, then you'll need to be more specific like this:

[monitor:///var/dropfolder/app1*.log]
sourcetype=myApp1Log
index=myIndex
crcSalt=<SOURCE>

[monitor:///var/dropfolder/otherApp*.log]
sourcetype=myOtherAppLog
index=myIndex
crcSalt=<SOURCE>

You need to do this because later you'll want to perform field extractions on each log type, and the regex or other methods to extract those fields will be different for each data format you have.

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi sai_John,
The only way to drop a file after indexing is to schedule a script that deletes files older than a time.
Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi sai_John,
The only way to drop a file after indexing is to schedule a script that deletes files older than a time.
Bye.
Giuseppe

0 Karma

jkat54
SplunkTrust
SplunkTrust

So let's peel that onion... Are all *.log files going to have the same data formats?

If so you can easily monitor one directory like this

[monitor:///var/dropfolder/*.log]
sourcetype=myLogs
index=myIndex
crcSalt=<SOURCE>

However, if each .log file has different formats, then you'll need to be more specific like this:

[monitor:///var/dropfolder/app1*.log]
sourcetype=myApp1Log
index=myIndex
crcSalt=<SOURCE>

[monitor:///var/dropfolder/otherApp*.log]
sourcetype=myOtherAppLog
index=myIndex
crcSalt=<SOURCE>

You need to do this because later you'll want to perform field extractions on each log type, and the regex or other methods to extract those fields will be different for each data format you have.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If you're talking about editing the files in the folder, then no. Splunk is not a UI for editing data like Google docs.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yes, you do what I've said.

Create an input that is monitoring a folder on your machine for *, then you drop the files in that folder.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You would create a wild carded monitor stanza

[monitor:///var/dropfolder/*]

If you don't specify a sourcetype, Splunk will attempt to detect one for you. It should be fine for csv, JSON, IIS logs, Apache logs, and a hand full of other known log types.

But I'm telling it's going to get ugly...

There is no app for this because everyone's data is different and people would constantly complain about the app.

0 Karma

sai_john
New Member

yeah i understood that. you are talking about adding new data from different servers/data formats. you are correct in that perspective. But i am talking about sample files which are in my local desktop.

For example I have exported 10 sample log files(lets talk about only csv files) from splunk from 10 sourcetypes into my local desktop.

Now i want to place/drop these 10 .csv sample files into __(kind of dropbox /some app)__ to search from that sample data which is present in those 10 csv files.

i want to know if there is an app/feature in splunk similar to dropbox to place these 10 csv files and then tweek.

0 Karma

sai_john
New Member

yeah this is the regular process to add data from different data formats.
But i am not looking to get data from different servers/data format since i just want to place my sample log files(.csv / .txt) to play with that sample data. Does Splunk provides an appor some other feature to place/drop sample multiple log files and then play around that by extracting fields?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Splunk really doesn't care where you put them. You just tell the indexer what directories to watch, and it does the rest.

You can look up details regarding forwarders (that move the data from place to place) and fishbuckets (part of how splunk tracks what it's done already), but really you should just start here -

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Getstartedwithgettingdatain

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Howdoyouwanttoadddata

0 Karma

jkat54
SplunkTrust
SplunkTrust

Just be sure you understand that Splunk is not a storage engine. It's a machine data analytics platform.

0 Karma

sai_john
New Member

yes i know that. we need this kind of dropbox feature to store sample log files in lower environment which is used to play with the data to create some dashboards and others.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...