We all know that manipulating _MetaData:Index we can redirect some events to another index.
But the question is - can we do it using Ingest-time evals? For example - using lookup() function to split by host field value to several separate indices.
OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.
It's perfectly ok to overwrite the index field with ingest-time eval.
[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")
With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.
In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.
OK. It seems I managed to do what I needed. In this case it's based on Computer field of windows event.
It's perfectly ok to overwrite the index field with ingest-time eval.
[test-new-index]
INGEST_EVAL = index=coalesce(json_extract(lookup("test_lookup",json_object("Key",spath(_raw,"Event.System.Computer")),json_array("Value")),"Value"),"default")
With this eval I can redirect between various indices based on a defined lookup with a default value in case it's not included in said lookup.
In case you want an existing value to be used as default, replace "default" with index. I'm not sure if it would work without coalesce (just not performing any assignment in case lookup returns null). Probably would, but I'm too lazy to try and in context of the whole operation it's very "cheap" performance-wise and it's a good safeguard.
Hi @PickleRick,
you cannot use eval, because it's a command used at search time and you're speaking of index time.
You can build your rules to choose the index based on regexes, how much complicated and how many you like, but not on a lookup.
in other words, you can create two, three, ten one hundred or more regex rules to choose the index but not in a dynamic way, only in conf files.
Only one question: why you want to divide logs in many indexes?
Usually the choice to have more indexes comes from two rules:
If you don't have the above requisites, you could put all the data in the same index!
Eventually you could divide your logs in more than one index but having many indexes is an additional complication for your data management: remember that Splunk isn't a DB, so you don't need to have different indexes for different logs, the only rules are the above ones!
Data are differentiated in Splunk mainly by sourcetype or eventually by other fields as host or source.
I hope to be clear, otherwise, I'll try to be more detailed.
Ciao.
Giuseppe
I beg to differ. EVAL can be used index-time. See https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/IngestLookups
And it does work - the eval gets evaluated and the value is generated (you have to mind the efficiency of this solution though).
And yes - unfortunately due to some infrastructural limitations we have to split the logs because of access reasons.
Hi @PickleRick,
these are ingest time lookups, not indexes, they are different!
Anyway, consider my hint about indexes.
Ciao.
Giuseppe
Yes, I understand what's the difference between index, lookup and eval.
I'm asking whether I can use the eval mechanism invoked in ingest time (which clearly does work) to redirect event to another index (which can be perfectly well done using regex; I know that).