topic Re: change index metadata dynamically during ingestion in Getting Data In

change index metadata dynamically during ingestion

iam_ironman — Mon, 03 Jun 2024 21:11:09 GMT

Hi all,

I'm trying to see if logs can be send to different indexes at index time depending on regex. Is it possible to send logs to index name that is part of Source metadata?

Below are my props.conf and transforms.conf

props.conf:

[test:logs]
TRANSFORMS-new_index = new_index
transforms.conf

[new_index]
SOURCE_KEY = MetaData:Source
REGEX = (?<index>\w+)\-\d+
FORMAT = $1 #This needs to be dynamic
DEST_KEY = _MetaData:Index

Thanks in advance.

Re: change index metadata dynamically during ingestion

PickleRick — Mon, 03 Jun 2024 21:50:20 GMT

What do you mean by "dynamic" here?

Also, you might need WRITE_META = true

Also also, you might want to use ingest actions.

Re: change index metadata dynamically during ingestion

gcusello — Tue, 04 Jun 2024 06:05:54 GMT

Hi @iam_ironman ,

only one question: why?

indexes aren't database tables, indexes are containers where logs are stored, the log categorization is done with sourcetype field.

usually custom indexes are mainly created when there are different requirements about retention and grant accesses and secondary for different log volumes.

So why do you want to create so many indexes, that you have to maintain and that after a retention time, will be empty?

Enyway, the rex you used is wrong, you don't need to extract the index field to assign a dinamic value to this field, you have to identify a group and use it for the index value:

[new_index] SOURCE_KEY = MetaData:Source REGEX = ^(\w+)\-\d+ FORMAT = $1 DEST_KEY = _MetaData:Index

Ciao.

Giuseppe

Re: change index metadata dynamically during ingestion

PickleRick — Tue, 04 Jun 2024 06:52:02 GMT

While the general question is of course valid and needs to be considered properly, I saw similar cases in my experience - splitting data from a single source into separate indexes.

The most typical case is when you have a single solution providing logs for separate business entities (like a central security appliance protecting multiple divisions or even companies from a single business group).

You might want to split events so that each unit has access only to its own events (possibly with some overseeing security team having access to all those indexes).

So there are valid use cases for similar setups 🙂

Re: change index metadata dynamically during ingestion

iam_ironman — Tue, 04 Jun 2024 12:26:54 GMT

What I meant by "dynamic" is that the value for index should be what regex finds and uses it for FORMAT. I know I can use static value but wanted to confirm it that is something possible using regex to dynamically use correct index which is part to Source.

Example of sources : phone-1234 , tablet-23456, pc-45623, pc-79954

[new_index]
SOURCE_KEY = MetaData:Source
REGEX = (\w+)\-\d+
FORMAT = $1 #This needs be either phone, tablet, pc etc. and don't want to make static
DEST_KEY = _MetaData:Index
WRITE_META = true

Re: change index metadata dynamically during ingestion

PickleRick — Tue, 04 Jun 2024 12:29:49 GMT

Yes, that's exactly what that is for. Still, consider what @gcusello already said - multiplying indexes is not always a good practice. There are different mechanisms for data "separation" depending on your use case.

Unless you need

- different access permissions

- different retention period

or you have significantly different data characteristics (cardinatility, volume and "sparsity") you should leave the data in the same index and limit your searches by adding conditions.

Re: change index metadata dynamically during ingestion

gcusello — Tue, 04 Jun 2024 13:15:39 GMT

Hi @iam_ironman ,

does it run in this way?

Ciao.

Giuseppe

Re: change index metadata dynamically during ingestion

iam_ironman — Tue, 04 Jun 2024 13:17:20 GMT

Haven't tried yet, but wanted to confirm if it works for POC.

Thanks.

Re: change index metadata dynamically during ingestion

gcusello — Tue, 04 Jun 2024 15:28:57 GMT

Hi @iam_ironman ,

I usually use this configuration.

Ciao.

Giuseppe