How do I index with regex using props.conf/transforms.conf ?

[brianpreston]

I'm trying to put logs which match a regex into a different index ("audit_private") than the one they come in with ("syslog_general_public").

Yet: My logs are all still in the original index, and not "audit_private".

Q: What am I doing wrong here?
Q: How can I see the regex working or not? Right now I'm changing the .conf files, restarting splunk, and watching the search results.

On the indexer:

All these directories exist, are 700, and owned by splunk:
- /misc/cloud2/splunk/
- /misc/cloud2/splunk/hot
- /misc/cloud2/splunk/warm
- /misc/cloud2/splunk/hot/audit_private
- /misc/cloud2/splunk/warm/audit_private

In /opt/splunk/etc/system/local :

inputs.conf: this is how all the logs come in, with their default index

[tcp://:10514]
index = syslog_general_public
sourcetype = syslog
connection_host = dns

indexes.conf: this defines the indexes and where they are stored

[volume:hot]
path = /misc/cloud2/splunk/hot

[volume:cold]
path = /misc/cloud2/splunk/warm

[audit_private]
homePath = volume:hot/audit_private
coldPath = volume:cold/audit_private
thawedPath = /misc/cloud2/splunk/thawed/audit_private

[syslog_general_public]
homePath = volume:hot/syslog_general_public
coldPath = volume:cold/syslog_general_public
thawedPath = /misc/cloud2/splunk/thawed/syslog_general_public

props.conf: points to the regex/transform

[syslog_audit_log_change_index_transform]
TRANSFORMS-syslog_audit_log_change_index = syslog_audit_log_change_index

transforms.conf: the transform! This has the simplest regex.

[syslog_audit_log_change_index]
REGEX = audit_log
DEST_KEY = _MetaData:Index
FORMAT = audit_private

I've also tried adding

WRITE_META = true

...but it didn't seem to make a difference.

Note: Every time I change the regex or props.conf I restart splunk.

This regex is supposed to work on this line:

2015-12-08T14:28:01.740023-05:00 dev-web-1006 audit_log type=USER_END msg=audit(1449602881.734:8461017): user pid=17294 uid=0 auid=0 ses=1243580 msg='op=PAM:session_close acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

I've also tried more elaborate regex, like

REGEX = ^[^ ]+\s+\S+\s+audit_log\s+type=\S+\s+msg=audit\(\S+\)\:.*msg=\'.*\'

[MuS · Accepted solution]

Your inputs.conf sets the sourcetype to syslog but in your props.conf you're using [syslog_audit_log_change_index_transform]. This should be [syslog] instead and remember to restart Splunk after the change.

[brianpreston]

As MuS mentions, it was definitely my props.conf

• the name of the stanza should match the sourcetype
• the sourcetype was set in inputs.conf

I've since added a second transform, and listed them both in that one props.conf stanza:

props.conf:

[syslog]
TRANSFORMS-syslog_audit_log_change_index = syslog_audit_log_change_index, syslog_haproxy_change_index

transforms.conf:

[syslog_audit_log_change_index]
REGEX = audit_log
DEST_KEY = _MetaData:Index
FORMAT = audit_private
WRITE_META = true

[syslog_haproxy_change_index]
REGEX = haproxy\[\d+\]
DEST_KEY = _MetaData:Index
FORMAT = haproxy_private
WRITE_META = true

[vya9836]

2015-12-08T14:28:01.740023-05:00 dev-web-1006 audit_log type=USER_END msg=audit(1449602881.734:8461017): user pid=17294 uid=0 auid=0 ses=1243580 msg='op=PAM:session_close acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

For this data i am trying to extract res=success' field with some regex but it is not extracting, do you have that regex. if you have please send me.