Getting Data In

EVAL-based index redirection?

PickleRick
SplunkTrust
SplunkTrust

We all know that manipulating _MetaData:Index we can redirect some events to another index.

But the question is - can we do it using Ingest-time evals? For example - using lookup() function to split by host field value to several separate indices.

Labels (1)
Tags (2)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.

It's perfectly ok to overwrite the index field with ingest-time eval.

 

[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")

 

With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.

In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.

It's perfectly ok to overwrite the index field with ingest-time eval.

 

[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")

 

With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.

In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @PickleRick,

you cannot use eval, because it's a command used at search time and you're speaking of index time.

You can build your rules to choose the index based on regexes, how much complicated and how many you like, but not on a lookup.

in other words, you can create two, three, ten one hundred or more regex rules to choose the index but not in a dynamic way, only in conf files.

Only one question: why you want to divide logs in many indexes?

Usually the choice to have more indexes comes from two rules:

  • your data have different retention times (retention time is configured only for indexes);
  • your data have different access rules (access rules are defined only for indexes.

If you don't have the above requisites, you could put all the data in the same index!

Eventually you could divide your logs in more than one index but having many indexes is an additional complication for your data management: remember that Splunk isn't a DB, so you don't need to have different indexes for different logs, the only rules are the above ones!

Data are differentiated in Splunk mainly by sourcetype or eventually by other fields as host or source.

I hope to be clear, otherwise, I'll try to be more detailed.

Ciao.

Giuseppe

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I beg to differ. EVAL can be used index-time. See https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/IngestLookups

And it does work - the eval gets evaluated and the value is generated (you have to mind the efficiency of this solution though).

And yes - unfortunately due to some infrastructural limitations we have to split the logs because of access reasons.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @PickleRick,

these are ingest time lookups, not indexes, they are different!

Anyway, consider my hint about indexes.

Ciao.

Giuseppe

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yes, I understand what's the difference between index, lookup and eval.

I'm asking whether I can use the eval mechanism invoked in ingest time (which clearly does work) to redirect event to another index (which can be perfectly well done using regex;  I know that).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...