topic Re: Pre-processing and save data in Getting Data In

Pre-processing and save data

cross521 — Sun, 05 Nov 2023 23:57:21 GMT

I want to deal with big data uising Splunk.

To reduce time for searching data, I want to select specific data from original data, pre-process it, and save the output data as csv format. Also I want to make dashboard using out data.

Please let me know about example of query or helpful article.

Re: Pre-processing and save data

gcusello — Mon, 06 Nov 2023 06:14:26 GMT

Hi @cross521,

your question id very vague.

Anyway, in general you have to index data in Splunk to analyze and use them.

The steps to do this are (in general) these:

analyze data, finding the relevant ones (out of Splunk),
ingest them using the Splunk features (for more infos see https://lantern.splunk.com/Splunk_Platform/Getting_Started/Getting_data_into_Enterprise),
so you can search and use them.

To save the search results in csv forma theres the outputcsv command (https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/Outputcsv), but anyway you have to index data in Splunk.

If you want to pre-process them, you have to use a script (done in the language you like) to prepare data before ingestion but I'am not an expert in scripting and this isn't a Splunk issue so I cannot help you.

Ciao.

Giuseppe

Re: Pre-processing and save data

cross521 — Fri, 10 Nov 2023 00:16:28 GMT

Thank you for answer.
Here is an example where I would like to process data:
1. There are 3 years of data accumulated every 2 seconds.
2. The value of a particular point is always 0 and only becomes 1 or more when a failure occurs.
3. I would like to retrieve the records of any failures over a period of 3 years, i.e. spikes in the data, and save them as csv format.

Can you help me one more time?

Re: Pre-processing and save data

gcusello — Fri, 10 Nov 2023 07:27:19 GMT

Hi @cross521,

yes the Use Case you describe it's possible and easy to create.

I suppose that you already ingested data and stored them in an index using a sourcetype (item 1).

I suppose also that you already extracted fields associated to that sourcetype (item 2), if not please share a sample of your logs.

For the item 3, I need to know how to identify failures, in the following example I use the rule that if there's a failure, "status" field has the value "failure", and you have to define the fields to add in the results

A the end, you can download the csv from the GUI or use the outputcsv command (at the end of the search) that saves the csv in $SPLUNK_HOME/var/run/splunk/csv, it isn't possible to use a different location for te saving folder, if you want a different one, you have to create a custom script to move this file.

index=your_index status =failure | table _time host field1 field2 | outputcsv your_csv.cv

if there are different conditions you can modify my search.

Ciao.

Giuseppe