topic Re: how to exclude in _MetaData:Index in Getting Data In

how to exclude in _MetaData:Index

AllenZhang — Sun, 02 Jan 2022 16:41:02 GMT

I noticed in our environment, from many uf, the internal logs were indexed under a different index name. After investigation, I find it's related to some settings in transforms.conf.

So in transforms.conf, it's like:

[test_windows_index]
REGEX =.*
DEST_KEY = _MetaData:Index
FORMAT = rexall_windows

in props.conf, for certain hosts, there're settings like:

[host::testserver1]
TRANSFORMS-Microsoft_AD_1 = test_windows_index, Routing_testCloud

I believe I should try to exclude indexes like "_internal, _audit",

so I changed REGEX=.* to REGEX=[a-zA-Z0-9]+

but it doesn't seem to work.

Appreciate if somebody here can help or provide suggestions.

Re: how to exclude in _MetaData:Index

tscroggins — Sun, 02 Jan 2022 17:43:54 GMT

@AllenZhang

The default value for SOURCE_KEY is _raw, so REGEX matches against the raw event data.

Events headed for internal indexes should already have an index value defined, so you might try matching against _MetaData:Index and reformatting when the current value does not begin with an underscore:

[test_windows_index]
REGEX = ^(?!_).
FORMAT = rexall_windows
DEST_KEY = _MetaData:Index
SOURCE_KEY = _MetaData:Index

This should work for unstructured source types; however, data for events cooked by the forwarder, e.g. with INDEXD_EXTRACTIONS or force_local_processing = true in props.conf, may not have transforms applied by a heavy forwarder or indexer.

Re: how to exclude in _MetaData:Index

AllenZhang — Sun, 02 Jan 2022 22:34:25 GMT

Thanks a lot @tscroggins for your help!

I took it for granted that "REGEX=..." means "index=...", and I was confused by different results from 2 servers I was using for test. now I realized that one of them has "-" in computer name, thus my "REGEX=[a-zA-Z0-9]+" must have excluded.

Please correct me if I am wrong: the regex only works as "index=..." when "SOURCE_KEY = _MetaData:Index" is there.