topic Re: How to redirect logs to separate indexes based on host? in Getting Data In

How to redirect logs to separate indexes based on host?

roysoman — Tue, 27 Mar 2018 17:19:07 GMT

Hi.. I have a question

From a heavy forwarder , based on the incoming host, I like to send the logs into a separate index.
Currently I configured to send the logs from a single heavy forwarder into a separate indexer based on the Environment variable we are passing with the logs(eg environment = cert/Dev/test/prod),Which is working fine.
On the top of that, I like to send the logs into a separate index within the indexer based on the host. here is Our conf files

Inputs.conf

[default]
host=Heavyfw.aaaa.com

[udp://514]
index=main
sourcetype=syslog_log
connection_host=ip

props.conf

[syslog_log]
LINE_BREAKER = (\s+)\{|\n
SHOULD_LINEMERGE = false
MUST_BREAK_AFTER = ([\r\n]+)[A-Z]+\s+\|\s+\d+ 
TRANSFORMS-test=test 
TRANSFORMS-cert=cert 
TRANSFORMS-dev=dev 
TRANSFORMS-prod=prod 
TRANSFORMS-local=local 
TRANSFORMS-null=null

[host::(198.*)]
TRANSFORMS = rewrite-DPindex

transforms.conf

[cert]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(cert)["]?
FORMAT=cert
DEST_KEY=_TCP_ROUTING

[local]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(local)["]?
DEST_KEY=_TCP_ROUTING
FORMAT=local

[test]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(test)["]?
DEST_KEY=_TCP_ROUTING
FORMAT=test

[null]
REGEX=^default send string$
FORMAT=nullQueue
DEST_KEY=queue

[dev]
DEST_KEY=_TCP_ROUTING
FORMAT=dev
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(dev)["]?

[prod]
FORMAT=prod
DEST_KEY=_TCP_ROUTING
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(prod)["]?

[rewrite-DPindex]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = Newindex

outputs.conf

[tcpout:Prodserv1]
server=ProdServer.aaaa.com:9997

[tcpout:cert]
server= CertServer.aaaa.com:9997

[default]
defaultGroup=default

[tcpout:test]
server= testServer.aaaa.com:9997

[tcpout:dev]
server= dev.Server.aaaa.com:9997

[tcpout:default]
server= testServer.aaaa.com:9997

[tcpout:prod]
server= ProdServer.aaaa.com:9997

[tcpout:local]
server= ProdServer.aaaa.com:9997

Now all the logs coming from the host 198.* going to Newindex and also to main index. How can I stop going to main index and send only to Newindex .

Re: How to redirect logs to separate indexes based on host?

maciep — Tue, 29 Sep 2020 18:43:14 GMT

you're getting duplicate events across those indexes? Does it matter which indexer they get routed to? Is there anything besides that syslog data getting sent to your hf?

I think what you have looks good, but not sure if it's worth trying to match the host in transforms and just call it from the syslog_log stanza in props?

[rewrite-DPindex]
SOURCE_KEY = MetaData:Host
REGEX = ^198\.
DEST_KEY = _MetaData:Index
FORMAT = Newindex

Side note...I think MUST_BREAK_AFTER is only applicable if SHOULD_LINEMERGE=true.

Re: How to redirect logs to separate indexes based on host?

roysoman — Wed, 28 Mar 2018 17:33:53 GMT

Currently the logs coming from the host 198.* is going to both main index and NewIndex..
I am trying to send only to NewIndex. I dont want the logs from the host 198.* should show up on main index.

Re: How to redirect logs to separate indexes based on host?

deepashri_123 — Mon, 02 Apr 2018 09:00:11 GMT

@roysoman,

Your inputs.conf shows index=main, change it to index=newindex and restart service.

Let me know if this helps!!