Use JSON extracted field to route to a different indexes

[yosoypako]

Hello.

I am trying to route some events to a different index based on a field on the events. The events are JSON formatted. This is an example:

{ 
    "topic": "audits", 
    "events": [ 
        { 
            "admin_name": "john doe john.doe@juniper.net", 
            "device_id": "00000000-0000-0000-1000-5c5b35xxxxxx", 
            "id": "8e00dd48-b918-4d9b-xxxx-xxxxxxxxxxxx", 
            "message": "Update Device \"Reception\"", 
            "org_id": "2818e386-8dec-2562-xxxx-xxxxxxxxxxx", 
            "site_id": "4ac1dcf4-9d8b-7211-xxxx-xxxxxxxxxxxx", 
            "src_ip": "xx.xx.xx.xx", 
            "timestamp": 1549047906.201053 
        } 
    ] 
} 

We are receiving the events into a heavy forwarder. And we forward them the event to an indexer. We want to send the events with the topic audits to a different index than the default one (imp_low). I have tried with these settings in the heavy forwarder:

-Props.conf

---------------------------------------------

[_json-Mist_Juniper]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
pulldown_type = 1
TRANSFORMS-force_index = setindexHIGH

-Transforms .conf:

-------------------------

[setindexHIGH]
SOURCE_KEY = topic
REGEX = (audits)
DEST_KEY = _MetaData:Index
FORMAT = imp_high

But it is not working, all the events are going to the "imp_low" index. 

Thanks

[richgalloway]

"topic" is not a recognized value for the SOURCE_KEY field.  Try using these transforms:

[setindexHIGH]
SOURCE_KEY = _raw
REGEX = ("topic":\s*"audits")
DEST_KEY = _MetaData:Index
FORMAT = imp_high

[yosoypako]

Hello.

Thanks for your help.

I have tried with the regex you suggested and with this configuration.

[setindexHIGH]
SOURCE_KEY = _raw
REGEX = audits
DEST_KEY = _MetaData:Index
FORMAT = imp_high

The same result. It is not working. We are receiving the events on the index imp_low

If we run a search for the events, we can see the field named topic is being indexed. But if we set the view to  raw text of the event. I can not see the words topic or audits on the events raw text. It looks like that info is being removed from the event. Could it be because the props settings?

[yosoypako · Accepted solution]

Hello.

We have made it work. This is the stanza we have configured in transforms.conf on the heavy forwarder:

[setindexHIGH]
SOURCE_KEY = field:topic
REGEX = audits
DEST_KEY = _MetaData:Index
FORMAT = imp_high

Thanks for your help.