topic How can I route logs to a specific index based on host/source combo? in Getting Data In

How can I route logs to a specific index based on host/source combo?

bport15 — Tue, 29 Sep 2020 14:21:38 GMT

We currently have our perf and N1 environments combined and I need to route certain logs to certain indexes based on the host/source combo.

Example:
Host: N1host.company.com
Source - /var/appl/logs/company.log
Source - /var/appl/logs/company_perf.log

Right now, our transforms looks like this and routes all logs from this host to index=company_perf:

[index-company_perf]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(<hostname>)$
DEST_KEY = _MetaData:Index
FORMAT = company_perf

What I need to happen is have logs from that host, N1host, with any source that has perf.log to go to the index=company_perf and everything else from that host go to index=company_n1.

How can I modify this existing transforms to do what I need? Or can I create a new transforms to deal with it?

Can we modify the SOURCE_KEY = fields:host,source? What would the regex look like?

Re: How can I route logs to a specific index based on host/source combo?

micahkemp — Thu, 01 Jun 2017 15:35:54 GMT

It's probably possible to do what you want, but I think you'd be better served changing the inputs in some way to make the end result easier to accomplish.

Can you (or do you already) get the hostname as part of the source path? That would allow you to do one transform to determine host and filename to act on.

Re: How can I route logs to a specific index based on host/source combo?

masonmorales — Thu, 01 Jun 2017 17:10:02 GMT

You can do this by calling your transform from a source specific stanza in props.conf
i.e.

props.conf

[source::/var/appl/logs/*_perf.log]
TRANSFORMS-index-company_perf = index-company_perf

transforms.conf

 [index-company_perf]
 SOURCE_KEY = MetaData:Host
 REGEX = ^host::(<hostname>)$
 DEST_KEY = _MetaData:Index
 FORMAT = company_perf

Re: How can I route logs to a specific index based on host/source combo?

bport15 — Thu, 01 Jun 2017 18:25:15 GMT

That's actually a good point. I hadn't thought of that. I'll give that a shot and report back if it worked!

Re: How can I route logs to a specific index based on host/source combo?

micahkemp — Thu, 01 Jun 2017 18:27:16 GMT

Ooooh, I like this, assuming it works. I had never considered specifying index-time transforms by source instead of sourcetype.

Re: How can I route logs to a specific index based on host/source combo?

bport15 — Mon, 12 Jun 2017 15:47:19 GMT

So after looking at all of our current configurations, it's not our perf logs that are the problem. It's everything else.

When we first got Splunk 4 years ago at our company, perf and N1 were combined and the vast majority of it still is. We are routing logs to specific indexes based on hostname. We did this because the log path for each environment is exactly the same, except for perf, and we didn't want to create several different configuration apps that contained the same log paths. Looking back, I wish we hadn't done that, but it would take a giant effort to rework all of this at this point.

Below is an example of how our current logs are, broken out by host, source and index, and a copy of our transforms.conf:

Host=prod source=/var/appl/logs/mobile-remote_access.log index=company_prod
Host=n1 source=/var/appl/logs/mobile-remote_access.log index=company_perf
Host=n1 source= /var/appl/logs/mobile-remote_access_perf.log index=company_perf
Host=n0 source=/var/appl/logs/mobile-remote_access.log index=company_n
Host=n2a source=/var/appl/logs/mobile-remote_access.log index=company_n2a

Transforms.conf

[index-company_prod]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(prod.[\d]+.*)$
DEST_KEY = _MetaData:Index
FORMAT = company_prod

[index-company_n]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(n0.[\d]+.*)$
DEST_KEY = _MetaData:Index
FORMAT = company_n

[index-company_n2a]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(n2a.[\d]+.*)$
DEST_KEY = _MetaData:Index
FORMAT = company_n2a

[index-company_perf]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(n1.[\d]+.*)$
DEST_KEY = _MetaData:Index
FORMAT = company_perf

In case you're wondering, I have asked our developers to fix this on their end instead, which is just as difficult, if not more difficult, on their end.

Re: How can I route logs to a specific index based on host/source combo?

bport15 — Mon, 12 Jun 2017 15:57:13 GMT

I suppose I could modify my current transforms.conf, as suggested by masonmorales, and create a new one for our n environment, like this:

Remove index-company_perf from each Sourcetype within props.conf and use it on a source reference instead.

[index-company_perf]
 SOURCE_KEY = MetaData:Host
 REGEX = ^host::(n1.[\d]+.*)$
 DEST_KEY = _MetaData:Index
 FORMAT = company_perf



 Props.conf
    [source::/var/appl/logs/*_perf.log]
     TRANSFORMS-index-company_perf = index-company_perf

Create a new stanza in transforms.conf

[index-company_n]
 SOURCE_KEY = MetaData:Host
 REGEX = ^host::(n1.[\d]+.*)$
 DEST_KEY = _MetaData:Index
 FORMAT = company_n

Props.conf
 [my_sourcetype]
TRANSFORMS-index-company_n = index-company_n