Trigger to create a new custom _raw before indexing data

[Kozokkon]

Good afternoon,
I want to ask if there is a way how to create own _raw data and to fire some kind of SPL query when some new event come to splunk ( the best would be if it is possible to make it before indexing ).

The whole idea is: When a new event comes, splunk will create a brand new event and save it in different index with little set of information from previous event + some extra info (for ex. actual timestamp).

I tried to make this work with alert, by "alerting" each new event to a new index. But the problem is that solution is extremely slow. I tried to upload 20k of new data at once and it took about hour to parse all of these.

Here is the pic of my idea:

That line between server and indexes is splunk server, the best would be if it gonna happen at pre or during indexing time, but if it happens after I wouldn't mind. 🙂

[deepashri_123]

Hey@Kozokkon,

You can try using summary indexing but this can be done after indexing data for the first time.
http://docs.splunk.com/Documentation/Splunk/7.1.1/Knowledge/Usesummaryindexing

Let me know if this helps!!

[somesoni2]

When you say alerting, do you mean alert search with summary index option? If not, have you explored option of summary indexing for this?

[Kozokkon]

That was kinda a hack, i set alert per every new event and made a spl where I totally recreate my _raw output as I want and set it to save to another index.

[richgalloway]

What problem are you trying to solve? Perhaps there is another way to accomplish your goal.

[Kozokkon]

In short: I'm receiving JSON so everything is parsed and saved as field. I need to anonymized some information and reduce some fields for search optimization.

[richgalloway]

Data should be anonymized before it is indexed. Writing the anonymized data to a different index means the original version is still available in another index.

Consider using a scripted or modular input to process the data as you wish before indexing.

[Kozokkon]

Mine idea was to parse already received information to another index with own redefined structure and variables and make searches on them.