topic Re: Trigger to create a new custom _raw before indexing data in Deployment Architecture

Trigger to create a new custom _raw before indexing data

Kozokkon — Fri, 15 Jun 2018 11:46:59 GMT

Good afternoon,
I want to ask if there is a way how to create own _raw data and to fire some kind of SPL query when some new event come to splunk ( the best would be if it is possible to make it before indexing ).

The whole idea is: When a new event comes, splunk will create a brand new event and save it in different index with little set of information from previous event + some extra info (for ex. actual timestamp).

I tried to make this work with alert, by "alerting" each new event to a new index. But the problem is that solution is extremely slow. I tried to upload 20k of new data at once and it took about hour to parse all of these.

Here is the pic of my idea:

That line between server and indexes is splunk server, the best would be if it gonna happen at pre or during indexing time, but if it happens after I wouldn't mind. 🙂

Re: Trigger to create a new custom _raw before indexing data

richgalloway — Fri, 15 Jun 2018 13:19:42 GMT

What problem are you trying to solve? Perhaps there is another way to accomplish your goal.

Re: Trigger to create a new custom _raw before indexing data

somesoni2 — Fri, 15 Jun 2018 15:17:32 GMT

When you say alerting, do you mean alert search with summary index option? If not, have you explored option of summary indexing for this?

Re: Trigger to create a new custom _raw before indexing data

Kozokkon — Mon, 18 Jun 2018 06:49:17 GMT

In short: I'm receiving JSON so everything is parsed and saved as field. I need to anonymized some information and reduce some fields for search optimization.

Re: Trigger to create a new custom _raw before indexing data

Kozokkon — Mon, 18 Jun 2018 06:52:19 GMT

Mine idea was to parse already received information to another index with own redefined structure and variables and make searches on them.

Re: Trigger to create a new custom _raw before indexing data

Kozokkon — Mon, 18 Jun 2018 06:54:32 GMT

That was kinda a hack, i set alert per every new event and made a spl where I totally recreate my _raw output as I want and set it to save to another index.

Re: Trigger to create a new custom _raw before indexing data

deepashri_123 — Mon, 18 Jun 2018 07:26:48 GMT

Hey@Kozokkon,

You can try using summary indexing but this can be done after indexing data for the first time.
http://docs.splunk.com/Documentation/Splunk/7.1.1/Knowledge/Usesummaryindexing

Let me know if this helps!!

Re: Trigger to create a new custom _raw before indexing data

richgalloway — Tue, 19 Jun 2018 00:30:00 GMT

Data should be anonymized before it is indexed. Writing the anonymized data to a different index means the original version is still available in another index.

Consider using a scripted or modular input to process the data as you wish before indexing.