I need to add extra field at index time.
The field is "Name of DEV/QA/Prod environment", which never changes during its hosts life-cycle.
What would be the best way to do that?
I don't want to do that at search time, because
it is something that needs to be for all sources.
it is static
According to Splunk Doc, its not recommended:
Caution: Do not add custom fields to the set of default fields that Splunk software automatically extracts and indexes at index time unless absolutely necessary.
I agree with Splunk. If I were you, I would do an automatic lookup only for the sourcetype(s) that need to map the host to it's particular key/value pair for this information. This way, you are not clogging your disk space with static data that is the same for every event.
if you really must, you can do it ...
however, there are many other ways
for example, have the source field have the required environment or create a unique index for each environment data, as if its the same data and carries same sourcetype, you can always search all and split by sourcetype for example.
another option, as you mentioned is in search time, tags are a good candidate for that kind of job