Getting Data In

EVAL-based index redirection?

PickleRick
Path Finder

We all know that manipulating _MetaData:Index we can redirect some events to another index.

But the question is - can we do it using Ingest-time evals? For example - using lookup() function to split by host field value to several separate indices.

Labels (1)
Tags (2)
0 Karma
1 Solution

PickleRick
Path Finder

OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.

It's perfectly ok to overwrite the index field with ingest-time eval.

 

[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")

 

With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.

In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.

View solution in original post

0 Karma

PickleRick
Path Finder

OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.

It's perfectly ok to overwrite the index field with ingest-time eval.

 

[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")

 

With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.

In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @PickleRick,

you cannot use eval, because it's a command used at search time and you're speaking of index time.

You can build your rules to choose the index based on regexes, how much complicated and how many you like, but not on a lookup.

in other words, you can create two, three, ten one hundred or more regex rules to choose the index but not in a dynamic way, only in conf files.

Only one question: why you want to divide logs in many indexes?

Usually the choice to have more indexes comes from two rules:

  • your data have different retention times (retention time is configured only for indexes);
  • your data have different access rules (access rules are defined only for indexes.

If you don't have the above requisites, you could put all the data in the same index!

Eventually you could divide your logs in more than one index but having many indexes is an additional complication for your data management: remember that Splunk isn't a DB, so you don't need to have different indexes for different logs, the only rules are the above ones!

Data are differentiated in Splunk mainly by sourcetype or eventually by other fields as host or source.

I hope to be clear, otherwise, I'll try to be more detailed.

Ciao.

Giuseppe

0 Karma

PickleRick
Path Finder

I beg to differ. EVAL can be used index-time. See https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/IngestLookups

And it does work - the eval gets evaluated and the value is generated (you have to mind the efficiency of this solution though).

And yes - unfortunately due to some infrastructural limitations we have to split the logs because of access reasons.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @PickleRick,

these are ingest time lookups, not indexes, they are different!

Anyway, consider my hint about indexes.

Ciao.

Giuseppe

0 Karma

PickleRick
Path Finder

Yes, I understand what's the difference between index, lookup and eval.

I'm asking whether I can use the eval mechanism invoked in ingest time (which clearly does work) to redirect event to another index (which can be perfectly well done using regex;  I know that).

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!