Getting Data In

Can you specify the order of operations for sourcetype definitions in props.conf to run an eval after a lookup?

Communicator

From: http://docs.splunk.com/Documentation/Splunk/6.4.1/admin/Propsconf

You cannot use a field added through a lookup in an eval statement for a calculated field.

Will we ever be able to choose the order of operations? I've run into a situation where I need an eval to run AFTER a lookup.

Is there an existing workaround to this, besides including the eval in every search in the environment?

0 Karma

Communicator

Ok, this begs the question; Why?

Why on earth would we not be able to control the order of operations?

0 Karma

SplunkTrust
SplunkTrust

Because there has to be an architecture. Some (not many) parts of an architecture are in essence purely arbitrary initial design decisions, but once those decisions get made, other things follow necessarily from the design... and changing those fixed elements becomes more and more complicated and unwise. (See - any initial Microsoft major release)

I believe, if you ever go to a new splunk shop, you will breathe a sigh of relief that certain orders of execution are fixed, so that when you are researching an issue, or trying to understand what your new system is doing -- a system designed by someone else but now YOUR responsibility to keep tuned and running -- that you can read the conf files in a particular order, and eventually trace down exactly what is happening.

If someone could alter that order -- including someone who did not know all the ramifications of that change -- then if could become quite a nightmare.

It's hard enough when there are local conf files in play in clustered environments...

0 Karma

Splunk Employee
Splunk Employee

Here is the order of search operations:

#Search-Time Operation ORDER
Sourcetype RENAME
EXTRACT-xxx
REPORT-xxx
KV_MODE
FIELDALIAS-xxx
EVAL-xxx
LOOKUP-xxx
MILLISECONDS
FILTER
EVENTTYPING
TAGGING

As you can see EVAL occurs before LOOKUP.

What you might consider is not coding the lookup into the props.conf but doing the lookup as part of your search then doing an eval after the lookup.

If it's something you need to do a lot perhaps a macro would simplify it.

0 Karma

Communicator

Regarding "LOOKUP-xxx";

  1. What order are the "xxx" processed in? Alphanumerical?
  2. If you can order lookups, can you use the first lookup as a field for the second lookup?

This is for a sourcetype that feeds into numerous reports and data models. The raw search must produce results from both lookups. Keeping it in the sourcetype is more logical in the environment than a macro would be unfortunately.

0 Karma

Splunk Employee
Splunk Employee

Props.conf based lookups are processed on the precedence order (alpha sort sequence) as other operations. I've not tried using lookups based on lookups myself but it SEEMS logical that they would work - YMMV.

Esteemed Legend

You cannot change the order of operations but you can change the method of your modification. Many of these operations can be twisted to do the same thing as one of the others and this conversion will move it to a different position in the order. This is the order:

INDEXED_EXTRACTIONS -> SEDCMD -> TRANSFORMS <---###Transition from Index-Time to Search-Time###---> (sourcetype)RENAME -> EXTRACT -> REPORT -> KV_MODE -> FIELDALIAS -> EVAL -> LOOKUP -> MILLISECONDS -> FILTER -> EVENTTYPING -> TAGGING

Actually, I am not absolutely certain about the order of the first 2.

Esteemed Legend
0 Karma

SplunkTrust
SplunkTrust

Hi mcrawford44, have considered to use the eval further down the search pipe after an automatic lookup? This should work fine.

cheers, MuS

0 Karma