topic Re: Splitting event to two indexes in Getting Data In

How to split event to two indexes?

klim — Fri, 06 Jan 2023 22:32:35 GMT

I would like to know if it is possible to be able to inject an event to a heavy forwarder via the hec and then have it be split into two events and sent to different indexes.

For example I have the original log line of:

ID=1 time=”2022-12-29 16:57:41 UTC” name=”person” address=”abc” message=”some note”.

I want the event to be split but the two new events can share similar fields.

So index1 would be:

ID=1 time=”2022-12-29 16:57:41 UTC” name=”person” address=”abc”

And index2 would be:

ID=1 time=”2022-12-29 16:57:41 UTC” message=”some note”

Re: Splitting event to two indexes

richgalloway — Fri, 06 Jan 2023 21:34:06 GMT

I believe it can be done. Use the CLONE_SOURCETYPE setting in transforms.conf to create a copy of the event and give it a new sourcetype. Then each sourcetype should be able to use REGEX and FORMAT settings to select the fields they want.

You'd think that the new Ingest Actions feature would be perfect for this, but it does not offer the type of changes you want to make.

This would be easier using Cribl (http://cribl.io).

Either way, duplicating the events means doubling your ingest, with corresponding effects on your license and storage.

Re: Splitting event to two indexes

klim — Fri, 06 Jan 2023 22:37:02 GMT

thanks. doubling my ingest doesn't sound like a great idea. It would be cheaper to send two requests.