topic Re: How can I pull out values in a log path and route it to a specific index? in Getting Data In

How can I pull out values in a log path and route it to a specific index?

bport15 — Mon, 06 Mar 2017 18:17:49 GMT

I am looking to route logs to different indexes based on a specific value identified in the log path. For example:

/var/appl/logs/prod/server.log >> should go to index=prod
/var/appl/logs/dev/server.log >> should go to index=dev
/var/appl/logs/imp/server.log >> should go to index=imp
/var/appl/logs/test/server.log >> should go to index=test

The log path will always be the same except for the unique value in the 4th sub-directory, which is where our developers are modifying the path to tell us what index they'd like the data in.

We have around 180 indexes in our environment so I don't want to create a unique transforms.conf to accommodate for each one. I'd like to have one transforms.conf that can pull the unique value from the path and place that log in the corresponding index.

I haven't been able to find great documentation to help me set this up. Any assistance would be greatly appreciated!

Re: How can I pull out values in a log path and route it to a specific index?

DalJeanis — Mon, 06 Mar 2017 19:04:32 GMT

It's going to look something like this.

 # transforms.conf 
  [overrideindexbylogpath]
  DEST_KEY =_MetaData:Index
  REGEX =\/var\/appl\/logs\/(.*)\/server\.log
  SOURCE_KEY= ... name of extracted log path field ...
  FORMAT = $1




  #props.conf 
  [ ...definition of which records you want to change...]
  TRANSFORMS-index = overrideindexbylogpath

I'm a little vague on what level of escaping will be needed on the regex... conf stanza versus search time versus saved search etc have little tweaky variations... but I think that got it correctly.

Here's a link to a version that works based on the host name field.

https://answers.splunk.com/answers/507081/rename-index-based-on-host-and-index-name.html#comment-506626

Re: How can I pull out values in a log path and route it to a specific index?

bport15 — Tue, 29 Sep 2020 13:07:06 GMT

Thanks @DalJeanis. The actual path we have includes several wildcards since we're pulling logs from containers. How would the format $1 know to pull from the (.*) if we have the following (where prod is in the index)?

Actual log path located in inputs.conf:

[monitor:///var/lib/origin/openshift*/pods/*/volumes/kubernetes*/log-dir/prod/dataportal-deployment*/*/catalina*.log]

This is what the transforms.conf would look like using wildcards, but not sure if it would actually work:

[override_index_by_log_path]
SOURCE_KEY = MetaData:Source
REGEX = \/var\/lib\/origin\/openshift*\/pods\/*\/volumes\/kubernetes*\/log-dir\/(.*)\/*\/*\/*.log
DEST_KEY = _MetaData:Index
FORMAT = $1

Re: How can I pull out values in a log path and route it to a specific index?

DalJeanis — Mon, 06 Mar 2017 20:39:29 GMT

Can you be sure how close to the final name the prod or dev etc will be? is it always two before the last slash? Here's one you can try...

   REGEX=.*\/(prod|dev|imp|test)\/[^\/]*\/[^\/]*\/.*\.log

It works over at regex101.com against this input...

/GEORGE/HENRY/var/lib/origin/openshiftWILMA/pods/LUKE/volumes/kubernetes/log-dir/prod/dataportal-deploymentDANA/BETTY/catalina.log

Re: How can I pull out values in a log path and route it to a specific index?

bport15 — Tue, 29 Sep 2020 13:07:11 GMT

So, our indexes don't have a standard naming convention. Some have "prod", "test", "non prod" in the name, depending on the application (online_prod, online_test, appname_prod, appname_nonprod, appname_n, etc) and it depends on the environment as well. We do, however, have a standard log file path that we've implemented so the name of the index is included in the file path and will always be in the 9th sub-directory.

Re: How can I pull out values in a log path and route it to a specific index?

bport15 — Mon, 06 Mar 2017 21:04:29 GMT

I can definitely try your suggestion above and try to catch all of the various environments we have. I'll keep you posted!

Re: How can I pull out values in a log path and route it to a specific index?

bport15 — Tue, 29 Sep 2020 13:12:45 GMT

I was able to request our developers add "index=" into the path, so this is what my transforms.conf looks like. I'm pushing that out today and will update this thread if everything works as expected.

Transforms.conf
[override_index_by_log_path]
SOURCE_KEY = MetaData:Source
REGEX = .*\/index=.*\/.*
DEST_KEY = _MetaData:Index
FORMAT = $1

Props.conf
TRANSFORMS-route_index_to_log = override_index_by_log_path

Re: How can I pull out values in a log path and route it to a specific index?

DalJeanis — Mon, 13 Mar 2017 15:59:10 GMT

I'm not seeing any parenthesis in the regex to mark the capture group for the index value. That should probably be something like...

REGEX =  .*\/index=([^\/]*)\/.*

... the value of the index needs to stop at the first slash, so we define a character class [^\/] which is any character EXCEPT a slash.

You also need to specify/validate what is going to happen to the index name when no capture is found.

Re: How can I pull out values in a log path and route it to a specific index?

bport15 — Mon, 13 Mar 2017 20:10:45 GMT

Good catch, thank you. Modifying now.......

Re: How can I pull out values in a log path and route it to a specific index?

bport15 — Wed, 22 Mar 2017 03:12:07 GMT

I finally got this working!

This is what I had to put in my transforms.conf. I ended up having the developers put "index=" in the log path and this is the final result:

TRANSFORMS.CONF

    [override_index_by_log_path_2]
    SOURCE_KEY = MetaData:Source
    REGEX = .*\/log-dir\/index=([^\/]*)\/.*
    DEST_KEY = _MetaData:Index
    FORMAT = $1

PROPS.CONF

[servicelog]
TRANSFORMS-route_index_to_log_servicelog = override_index_by_log_path_2