index routing with transforms.conf

[jedatt01]

I have a single xml source that I need to go multiple indexes. To complicate things, I need multiple transforms to make sure all events get to the appropriate index. Here are my requirements:

1. If any events contains the string <LogEventTypeCode>SEC_EVENT</LogEventTypeCode> it needs to be routed to an index called Security

2. All other events that do not contain <LogEventTypeCode>SEC_EVENT</LogEventTypeCode> need to be routed to appropriate index based on another field <BusinessDomainId>"businessdomain"</BusinessDomainId>

Requirement #2 already works, and what i'm seeing that requirement #1 only works if it has value for <BusinessDomainId>"businessdomain"</BusinessDomainId> that I do not have an index created for.

Props.conf

[mq]
BREAK_ONLY_BEFORE = \<ELLogInputMessage>
KV_MODE=XML
SHOULD_LINEMERGE = true
MAX_EVENTS = 50000
TIME_PREFIX = <LogEventDateTime>
pulldown_type = 1
TRANSFORMS-route = Security, BusinessDomainId

Transforms.conf

[Security]
SOURCE_KEY = _raw
DEST_KEY = _MetaData:Index
REGEX=<LogEventTypeCode>PI_EVENT</LogEventTypeCode>
FORMAT=Security

[BusinessDomainId]
SOURCE_KEY = _raw
DEST_KEY =_MetaData:Index
REGEX=(?m)\<BusinessDomainId\>(BusinessDomainId1|BusinessDomainId2|BusinessDomainId3)\</BusinessDomainId\>
FORMAT=$1

Sample Event

<ELLogInputMessage> 
        <Header> 
            <LogEventTypeCode>PI_EVENT</LogEventTypeCode> 
            <LogSeverityCode>CRITICAL</LogSeverityCode> 
            <LogEventDateTime>2014-05-06T23:19:59.9999999-05:00</LogEventDateTime> 
        </Header> 
        <SourceInformation> 
            <EAPMId>3</EAPMId> 
            <HostMachineName>HostMachineName3</HostMachineName> 
            <HostEnvironmentName>HostEnvironmentName3</HostEnvironmentName> 
            <ComponentId>ComponentId3</ComponentId> 
            <ComponentName>ComponentName3</ComponentName> 
            <ApplicationEventCorrelationId>ApplicationEventCorrelationId3</ApplicationEventCorrelationId> 
            <UserId>UserId33</UserId> 
            <UserSrc>UserSrc33</UserSrc> 
            <BusinessDomainId>BusinessDomainId33</BusinessDomainId> 
            <BusinessDomainName>BusinessDomainName33</BusinessDomainName> 
        </SourceInformation> 
        <DataAccessInformation> 
            <DataCompId>DataCompId33</DataCompId> 
            <TypeOfAccess>VIEW</TypeOfAccess> 
            <SubjectOfInterest> 
                <SubjectId>SubjectId32</SubjectId> 
                <SubjectName>SubjectName32</SubjectName> 
                <SubjectDomainName>SubjectDomainName32</SubjectDomainName> 
            </SubjectOfInterest> 
            <AccessDateTime>2014-05-06T23:19:59.9999999-05:00</AccessDateTime> 
        </DataAccessInformation> 
        <DetailedLogInformation>anyType</DetailedLogInformation>
    </ELLogInputMessage>

[lguinn2 · Accepted solution]

What if you put the transforms in a different order?

TRANSFORMS-route =BusinessDomainId, Security

[lguinn2]

Yes, I believe that is what is happening here!

[jedatt01]

I was picking my brain trying to figure out how this worked. Is it because every event goes through every transform? Then the last transform it goes through is where the event will end up?

[jedatt01]

I had a thought, in my regex

REGEX=(?m)<BusinessDomainId>(BusinessDomainId1|BusinessDomainId2|BusinessDomainId3)</BusinessDomainId>

is there a way to add a condition for it not to match if it sees the string SEC_EVENT in the same event?