props.conf:
[source::...splunkd.log]
TRANSFORMS-indexRouting = route_all_to_ok_index, route_errors_to_errors_index
transforms.conf:
[route_all_to_ok_index]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = splunk_ok
[route_errors_to_errors_index]
REGEX = log_level=ERROR
DEST_KEY = _MetaData:Index
FORMAT = splunk_errors
In this case, you could actually eliminate need for 'route_all_to_ok_index' by sending that source to splunk_ok index at inputs level and override just for log_level=ERROR
The literal case is that we are trying to allow those with "a need to know" access to all data, and everyone else just some of the more secure data.
I was going to accomplish this by sending the secure data to another index, and limiting access to that index based on role.
Is this a literal exercise or a hypothetical example? There are not key value pairs in splunkd.log so the "REGEX = log_level=ERROR" would actually be something like "REGEX = ^(?:\d{2}-\d{2}-\d{4})\s(?:\d{2}:\d{2}:\d{2}.\d{3})\s(?:\S+)\s(?:ERROR)"
...but it would likely be better to create a saved search to report on errors (vs. stashing them in a separate index) depending on your use case.
Yes, on the indexers, and I have restarted. Still nothing in splunk_errors.
Did you set this up on a forwarder or the indexers? This particular configuration belongs at the parsing stage (usually the indexers)
You've re-started?
As recommended, I made the changes in $splunk_home/etc/apps/search/local
However,I still see no data going to the splunk_errors index.
Any tips on troubleshooting this issue?
For organizational purposes, I would place them in a custom app or $splunk_home/etc/apps/search/local
Ok, that makes perfect sense. Now a stupid question. The splunkd.log data input is defined in the gui and resides in the filesystem in $splunk_home/etc/apps/learned/local. Is this where I should place the props.conf and transforms.conf? Is this best practice? THanks for all the help.
It is best practice to specify index during input phase (inputs.conf)
If only one source needs to be routed to a specific index, it would be best to do this at input time...
E.g. inputs.conf
[monitor:///var/log]
index=main
sourcetype=syslog
[monitor://.../my_secure_app/secure.log]
index=secure
sourcetype=secure_log
If my_secure_app
has many logs but only one needs to go to a different index, we may still do this at inputs.conf
[monitor://.../my_secure_app/*]
blacklist=secure.log$
index=main
sourcetype=log4j
[monitor://.../my_secure_app/secure.log]
index=secure
sourcetype=secure_log
Failing that, if there are too many stanzas required, we can route later, downstream, based on sourcetype, host, and source, per: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad. This is not as efficient and generally not as easy to 'see what is happening' but may be used when configuration at input level is not practical.
More specifically, as an example. Lets say I wanted to filter all the log_level=ERROR events from the splunkd.log to go into the splunk_errors index, and everything else to go to the splunk_ok index. CAn you show me syntax for this example?
Thanks
So in the above example, the source type would determine which index the events were written. log4j would go in main, and secure_log would go in to secure?
You would perform routing via props wherever parsing occurs (generally heavy forwarders or indexers). Universal forwarders do not parse but indexes and sourcetypes may still be set at input time, which is more efficient. For more information on 'where in Splunk' to configure something, check out this Wiki article: http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F