topic Re: Filtering data into different indexes in Getting Data In

Filtering data into different indexes

timmy13 — Mon, 20 Feb 2012 19:16:45 GMT

I would like to send some events from a source to one index, and the rest to another. Can someone point me to a link that will help me accomplish this?

Re: Filtering data into different indexes

bwooden — Mon, 20 Feb 2012 19:21:21 GMT

It is best practice to specify index during input phase (inputs.conf)

If only one source needs to be routed to a specific index, it would be best to do this at input time...
E.g. inputs.conf

[monitor:///var/log]
index=main
sourcetype=syslog

[monitor://.../my_secure_app/secure.log]
index=secure
sourcetype=secure_log

If my_secure_app has many logs but only one needs to go to a different index, we may still do this at inputs.conf

[monitor://.../my_secure_app/*]
blacklist=secure.log$
index=main
sourcetype=log4j 

[monitor://.../my_secure_app/secure.log]
index=secure
sourcetype=secure_log

Failing that, if there are too many stanzas required, we can route later, downstream, based on sourcetype, host, and source, per: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad. This is not as efficient and generally not as easy to 'see what is happening' but may be used when configuration at input level is not practical.

Re: Filtering data into different indexes

bwooden — Mon, 20 Feb 2012 19:35:12 GMT

You would perform routing via props wherever parsing occurs (generally heavy forwarders or indexers). Universal forwarders do not parse but indexes and sourcetypes may still be set at input time, which is more efficient. For more information on 'where in Splunk' to configure something, check out this Wiki article: http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

Re: Filtering data into different indexes

timmy13 — Mon, 20 Feb 2012 22:45:13 GMT

So in the above example, the source type would determine which index the events were written. log4j would go in main, and secure_log would go in to secure?

Re: Filtering data into different indexes

timmy13 — Mon, 28 Sep 2020 11:25:40 GMT

More specifically, as an example. Lets say I wanted to filter all the log_level=ERROR events from the splunkd.log to go into the splunk_errors index, and everything else to go to the splunk_ok index. CAn you show me syntax for this example?

Thanks

Re: Filtering data into different indexes

bwooden — Mon, 28 Sep 2020 11:25:42 GMT

props.conf:

[source::...splunkd.log]
TRANSFORMS-indexRouting = route_all_to_ok_index, route_errors_to_errors_index

transforms.conf:

[route_all_to_ok_index]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = splunk_ok

[route_errors_to_errors_index]
REGEX = log_level=ERROR
DEST_KEY = _MetaData:Index
FORMAT = splunk_errors

In this case, you could actually eliminate need for 'route_all_to_ok_index' by sending that source to splunk_ok index at inputs level and override just for log_level=ERROR

Re: Filtering data into different indexes

timmy13 — Tue, 21 Feb 2012 18:21:05 GMT

Ok, that makes perfect sense. Now a stupid question. The splunkd.log data input is defined in the gui and resides in the filesystem in $splunk_home/etc/apps/learned/local. Is this where I should place the props.conf and transforms.conf? Is this best practice? THanks for all the help.

Re: Filtering data into different indexes

bwooden — Tue, 21 Feb 2012 19:18:11 GMT

For organizational purposes, I would place them in a custom app or $splunk_home/etc/apps/search/local

Re: Filtering data into different indexes

timmy13 — Tue, 21 Feb 2012 20:32:21 GMT

As recommended, I made the changes in $splunk_home/etc/apps/search/local

However,I still see no data going to the splunk_errors index.

Any tips on troubleshooting this issue?

Re: Filtering data into different indexes

bwooden — Tue, 21 Feb 2012 20:42:56 GMT

You've re-started?

Re: Filtering data into different indexes

bwooden — Tue, 21 Feb 2012 23:36:20 GMT

Did you set this up on a forwarder or the indexers? This particular configuration belongs at the parsing stage (usually the indexers)

Re: Filtering data into different indexes

timmy13 — Wed, 22 Feb 2012 15:46:11 GMT

Yes, on the indexers, and I have restarted. Still nothing in splunk_errors.

Re: Filtering data into different indexes

bwooden — Wed, 22 Feb 2012 16:24:23 GMT

Is this a literal exercise or a hypothetical example? There are not key value pairs in splunkd.log so the "REGEX = log_level=ERROR" would actually be something like "REGEX = ^(?:\d{2}-\d{2}-\d{4})\s(?:\d{2}:\d{2}:\d{2}.\d{3})\s(?:\S+)\s(?:ERROR)"

...but it would likely be better to create a saved search to report on errors (vs. stashing them in a separate index) depending on your use case.

Re: Filtering data into different indexes

timmy13 — Wed, 22 Feb 2012 21:47:05 GMT

The literal case is that we are trying to allow those with "a need to know" access to all data, and everyone else just some of the more secure data.

I was going to accomplish this by sending the secure data to another index, and limiting access to that index based on role.