Getting Data In

EVAL-based index redirection?

PickleRick
Ultra Champion

We all know that manipulating _MetaData:Index we can redirect some events to another index.

But the question is - can we do it using Ingest-time evals? For example - using lookup() function to split by host field value to several separate indices.

Labels (1)
Tags (2)
0 Karma
1 Solution

PickleRick
Ultra Champion

OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.

It's perfectly ok to overwrite the index field with ingest-time eval.

 

[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")

 

With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.

In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.

View solution in original post

0 Karma

PickleRick
Ultra Champion

OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.

It's perfectly ok to overwrite the index field with ingest-time eval.

 

[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")

 

With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.

In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.

0 Karma

gcusello
Legend

Hi @PickleRick,

you cannot use eval, because it's a command used at search time and you're speaking of index time.

You can build your rules to choose the index based on regexes, how much complicated and how many you like, but not on a lookup.

in other words, you can create two, three, ten one hundred or more regex rules to choose the index but not in a dynamic way, only in conf files.

Only one question: why you want to divide logs in many indexes?

Usually the choice to have more indexes comes from two rules:

  • your data have different retention times (retention time is configured only for indexes);
  • your data have different access rules (access rules are defined only for indexes.

If you don't have the above requisites, you could put all the data in the same index!

Eventually you could divide your logs in more than one index but having many indexes is an additional complication for your data management: remember that Splunk isn't a DB, so you don't need to have different indexes for different logs, the only rules are the above ones!

Data are differentiated in Splunk mainly by sourcetype or eventually by other fields as host or source.

I hope to be clear, otherwise, I'll try to be more detailed.

Ciao.

Giuseppe

0 Karma

PickleRick
Ultra Champion

I beg to differ. EVAL can be used index-time. See https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/IngestLookups

And it does work - the eval gets evaluated and the value is generated (you have to mind the efficiency of this solution though).

And yes - unfortunately due to some infrastructural limitations we have to split the logs because of access reasons.

0 Karma

gcusello
Legend

Hi @PickleRick,

these are ingest time lookups, not indexes, they are different!

Anyway, consider my hint about indexes.

Ciao.

Giuseppe

0 Karma

PickleRick
Ultra Champion

Yes, I understand what's the difference between index, lookup and eval.

I'm asking whether I can use the eval mechanism invoked in ingest time (which clearly does work) to redirect event to another index (which can be perfectly well done using regex;  I know that).

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...