topic Re: Is it possible to use Splunk as a static data store with a single log entry per line? in Getting Data In

Is it possible to use Splunk as a static data store with a single log entry per line?

michaelrtesco — Tue, 21 Apr 2015 14:31:37 GMT

We have a process that extracts data from a SQL Server in CSV format.

We want the Splunk agent to pick up that data from disk and mirror it in the Splunk dashboard. We do not want the data appended to an existing set (like a log file). We want Splunk to delete the old snapshot of that file's contents and create a completely new data set each time.

The advantage to us of such a design is that we can use Splunk's powerful graphing facilities on our static data source. I understand that such a use case may be rare though.

I have experimented with the batch:// feature, but this seems to consume the whole file as a single log entry. We need one log entry per line so that the report aggregations behave properly.

Is this something Splunk supports out of the box?

Re: Is it possible to use Splunk as a static data store with a single log entry per line?

aweitzman — Tue, 21 Apr 2015 14:47:32 GMT

Check out both the "CSV lookup" and "external lookup" sections of this documentation page and see whether one or the other (or both) applies to your situation:

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Addfieldsfromexternaldatasources

Re: Is it possible to use Splunk as a static data store with a single log entry per line?

michaelrtesco — Tue, 21 Apr 2015 15:03:21 GMT

Hi, thanks for your reply - I had a good read of the link, but I don't think lookups address our requirements. They seem to be for configuring static lookups that Splunk cross-references with existing data. What we want is to store regularly generated static data in Splunk, so we can report on it.

Example:

CSV data generated on day 1:

Col1,Col2
Testing,123
Test,456
TestTest,789

We just need that raw data accessible in Splunk, so we can graph it.

And on day 2, the data gets regenerated:

Col1,Col2
AnotherTest,999
Test,111
HelloWorld,222

We want that pushed up to Splunk to replace the existing data from day 1.

Is this possible?

Re: Is it possible to use Splunk as a static data store with a single log entry per line?

neelamssantosh — Tue, 21 Apr 2015 15:17:06 GMT

Good, unique requirement but
Hope this can help your requirement,
there is a option called 'logrotate' which will help you to created one log file per log(if this is ur requirement) and we can achieve it.
go through the logrotate concept in unix.

Re: Is it possible to use Splunk as a static data store with a single log entry per line?

aweitzman — Tue, 21 Apr 2015 15:33:32 GMT

If you make your csv file a lookup file, everything should just work. Write your csv file (call it mylookup.csv) to [SplunkHome]/etc/apps/search/lookups. Then in the search bar, if you just type: | inputlookup mylookup you should get the results you want. You should be able to overwrite this file, and the search should still work.