Change sourcetype/index after data is indexed from forwarder

[asarolkar]

We are in a bit of pickle currently trying to disassociate indexed data from a sourcetype that is currently tied to a certain index - both of which we wish to discard.

The data was pushed out from the universal forwarder which was setup in this manner in the inputs.conf:

[monitor://c:\accesslog\access*.log]
disabled=false
followTail=0
index=os
sourcetype=accesslog

I want to change configuration on the indexer (or the forwarder) such that it goes to
sourcetype="access_combined" which is associated with index="default" WITHOUT having to edit the aforementioned segment in the inputs.conf on the forwarder.

How do I do this without having to setup a brand new configuration set (like the one above) and re-indexing EVERYTHING again. Bear in mind, this is a tonne of data and we are attempting to avoid an overage.

[Drainy]

Yanns answer is great to override your bits at index time without modifying inputs, but you won't be able to change the existing data without re-indexing it. Once the metadata values have been written to disc you will need to dump them and re-index to change them. Bear in mind that although it might be a tonne of data, if you have an Enterprise licence you can blow it away up to 5 times without violating it, that safety net is designed for times like these or for batch jobs. Exceeding it once in a 30 day rolling window won't do you much harm 🙂

[yannK]

[edit]
To proceed, you need to setup a special transforms at indextime on the indexer.

see http://docs.splunk.com/Documentation/Splunk/4.3.3/Data/Advancedsourcetypeoverrides
probably something like

props.conf

[accesslog]

TRANSFORMS-changemetadata=replace_accesslog_to_access_combined,replace_os_to_default

transforms.conf (apply the 2 tranforms depending on the sourcetype.

[replace_accesslog_to_access_combined]

REGEX = .

FORMAT = sourcetype::access_combined

DEST_KEY = MetaData:Sourcetype

[replace_os_to_default]
REGEX = .
FORMAT = index::main
DEST_KEY = _MetaData:Index

Be careful, because changing sourcetype and index may not work depending on the order (if sourcetype is changed), you could use other rules like the source.
see http://splunk-base.splunk.com/answers/12098/is-it-possible-to-route-an-overrided-sourcetype-to-other...

[yannK]

thanks I edited the answer to fix

[Adam_Sealey]

You switched the labels for props.conf and transforms.conf. The second code block should be props (which is specifying to apply the specified transforms to the accesslog sourcetype).

Also, as an aside, per http://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf, setting the index requires a DEST_KEY = _MetaData:Index (note the prepended underscore)