Getting Data In

Pre-processing and save data

cross521
Engager

I want to deal with big data uising Splunk.

To reduce time for searching data, I want to select specific data from original data, pre-process it, and save the output data as csv format. Also I want to make dashboard using out data.

Please let me know about example of query or helpful article.

 

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @cross521,

your question id very vague.

Anyway, in general you have to index data in Splunk to analyze and use them.

The steps to do this are (in general) these:

To save the search results in csv forma theres the outputcsv command (https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/Outputcsv), but anyway you have to index data in Splunk.

If you want to pre-process them, you have to use a script (done in the language you like) to prepare data before ingestion but I'am not an expert in scripting and this isn't a Splunk issue so I cannot help you.

Ciao.

Giuseppe

cross521
Engager

Thank you for answer. 
Here is an example where I would like to process data:
1. There are 3 years of data accumulated every 2 seconds.
2. The value of a particular point is always 0 and only becomes 1 or more when a failure occurs.
3. I would like to retrieve the records of any failures over a period of 3 years, i.e. spikes in the data, and save them as csv format.

Can you help me one more time?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @cross521,

yes the Use Case you describe it's possible and easy to create.

I suppose that you already ingested data and stored them in an index using a sourcetype (item 1).

I suppose also that you already extracted fields associated  to that sourcetype (item 2), if not please share a sample of your logs.

For the item 3, I need to know how to identify failures, in the following example I use the rule that if there's a failure, "status" field has the value "failure", and you have to define the fields to add in the results

A the end, you can download the csv from the GUI or use the outputcsv command (at the end of the search) that saves the csv in $SPLUNK_HOME/var/run/splunk/csv, it isn't possible to use a different location for te saving folder, if you want a different one, you have to create a custom script to move this file.

index=your_index status =failure
| table _time host field1 field2
| outputcsv your_csv.cv

if there are different conditions you can modify my search.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Announcing the General Availability of Splunk Enterprise Security 8.1!

We are pleased to announce the general availability of Splunk Enterprise Security 8.1. Splunk becomes the only ...

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...