index routing with transforms.conf

jedatt01 — Fri, 09 May 2014 15:34:19 GMT

I have a single xml source that I need to go multiple indexes. To complicate things, I need multiple transforms to make sure all events get to the appropriate index. Here are my requirements:

If any events contains the string <LogEventTypeCode>SEC_EVENT</LogEventTypeCode> it needs to be routed to an index called Security
All other events that do not contain <LogEventTypeCode>SEC_EVENT</LogEventTypeCode> need to be routed to appropriate index based on another field <BusinessDomainId>"businessdomain"</BusinessDomainId>

Requirement #2 already works, and what i'm seeing that requirement #1 only works if it has value for <BusinessDomainId>"businessdomain"</BusinessDomainId> that I do not have an index created for.

Props.conf

[mq]
BREAK_ONLY_BEFORE = \<ELLogInputMessage>
KV_MODE=XML
SHOULD_LINEMERGE = true
MAX_EVENTS = 50000
TIME_PREFIX = <LogEventDateTime>
pulldown_type = 1
TRANSFORMS-route = Security, BusinessDomainId

Transforms.conf

[Security]
SOURCE_KEY = _raw
DEST_KEY = _MetaData:Index
REGEX=<LogEventTypeCode>PI_EVENT</LogEventTypeCode>
FORMAT=Security

[BusinessDomainId]
SOURCE_KEY = _raw
DEST_KEY =_MetaData:Index
REGEX=(?m)\<BusinessDomainId\>(BusinessDomainId1|BusinessDomainId2|BusinessDomainId3)\</BusinessDomainId\>
FORMAT=$1

Sample Event

<ELLogInputMessage> 
        <Header> 
            <LogEventTypeCode>PI_EVENT</LogEventTypeCode> 
            <LogSeverityCode>CRITICAL</LogSeverityCode> 
            <LogEventDateTime>2014-05-06T23:19:59.9999999-05:00</LogEventDateTime> 
        </Header> 
        <SourceInformation> 
            <EAPMId>3</EAPMId> 
            <HostMachineName>HostMachineName3</HostMachineName> 
            <HostEnvironmentName>HostEnvironmentName3</HostEnvironmentName> 
            <ComponentId>ComponentId3</ComponentId> 
            <ComponentName>ComponentName3</ComponentName> 
            <ApplicationEventCorrelationId>ApplicationEventCorrelationId3</ApplicationEventCorrelationId> 
            <UserId>UserId33</UserId> 
            <UserSrc>UserSrc33</UserSrc> 
            <BusinessDomainId>BusinessDomainId33</BusinessDomainId> 
            <BusinessDomainName>BusinessDomainName33</BusinessDomainName> 
        </SourceInformation> 
        <DataAccessInformation> 
            <DataCompId>DataCompId33</DataCompId> 
            <TypeOfAccess>VIEW</TypeOfAccess> 
            <SubjectOfInterest> 
                <SubjectId>SubjectId32</SubjectId> 
                <SubjectName>SubjectName32</SubjectName> 
                <SubjectDomainName>SubjectDomainName32</SubjectDomainName> 
            </SubjectOfInterest> 
            <AccessDateTime>2014-05-06T23:19:59.9999999-05:00</AccessDateTime> 
        </DataAccessInformation> 
        <DetailedLogInformation>anyType</DetailedLogInformation>
    </ELLogInputMessage>

Re: index routing with transforms.conf

jedatt01 — Fri, 09 May 2014 15:53:46 GMT

I had a thought, in my regex

REGEX=(?m)<BusinessDomainId>(BusinessDomainId1|BusinessDomainId2|BusinessDomainId3)</BusinessDomainId>

is there a way to add a condition for it not to match if it sees the string SEC_EVENT in the same event?

Re: index routing with transforms.conf

lguinn2 — Fri, 09 May 2014 16:58:00 GMT

What if you put the transforms in a different order?

TRANSFORMS-route =BusinessDomainId, Security

Re: index routing with transforms.conf

jedatt01 — Fri, 09 May 2014 17:52:53 GMT

I was picking my brain trying to figure out how this worked. Is it because every event goes through every transform? Then the last transform it goes through is where the event will end up?

Re: index routing with transforms.conf

lguinn2 — Fri, 09 May 2014 19:05:43 GMT

Yes, I believe that is what is happening here!

topic Re: index routing with transforms.conf in Getting Data In

index routing with transforms.conf

Re: index routing with transforms.conf

Re: index routing with transforms.conf

Re: index routing with transforms.conf

Re: index routing with transforms.conf