I've been working through trying to get some search time extractions to apply using prop.conf and transforms.conf with REPORTS. The source is parsed on a Heavy Forwarder and I have verified the index time extractions as being correct. I have created an add-on (TA) to do search time parsing on the indexer and when searching in Splunk cannot get the extractions to apply. Here is what is in my configs:
props.conf
[tool:ssh]
REPORT-authentication = ssh-login-events, ssh-session-close, ssh-disconnect, sshd_authentication_kerberos_success, sshd_authentication_refused, sshd_authentication_tried, sshd_login_restricted, pam_unix_authentication_success, pam_unix_authentication_failure, sudo_cannot_identify, ksu_authentication, ksu_authorization, su_simple, su_authentication, su_successful, wksh_authentication, login_authentication
transforms.conf
[ssh-login-events]
REGEX = (sshd)\[\d+\]\:\s+(\[[^]]+]\s+)?.*?(Accepted|Failed|failure|(?:Invalid user)).*?(\S+)\s+from.*?(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(\s+port\s+(\S+)\s+\w?\s*(ssh\d))?
FORMAT = app::$1 vendor_action::$3 user::$4 src_ip::$5 src_port::$7 sshd_protocol::$8
[ssh-session-close]
REGEX = .* ((?:session|Connection) (?:opened|closed))(?: for user (\w+))?(?: by \(uid=(\d+)\))?(?: by (\d+\.\d+\.\d+\.\d+))
FORMAT = name::$1 user::$2 user_id::$3 src_ip::$4
[ssh-disconnect]
REGEX = .* (Received disconnect) from (\w+):
FORMAT = name::$1 src_ip::$2
[sshd_authentication_kerberos_success]
REGEX = (sshd)\[\d+\]\:\s+(\[[^]]+]\s+)?(Authorized\s+to)\s+([^,]+)\,\s+krb5\s+principal\s+([^@]+)
FORMAT = app::$1 vendor_action::"$3" user::"$4" src_user::"$5"
[sshd_authentication_refused]
REGEX = (sshd)\[\d+\]\:\s+(\[[^]]+]\s+)?(Authentication\s+refused)\:.*?directory\s+\/home\/([^\/]+)
FORMAT = app::$1 vendor_action::"$3" user::"$4"
[sshd_authentication_tried]
REGEX = (sshd)\[\d+\]\:\s+(\[[^]]+]\s+)?(Authentication\s+tried)\s+for\s+([^\s]+)(.*?host\=([^,]+),\s+ip=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))?
FORMAT = app::$1 vendor_action::$3 user::"$4" src_dns::"$6" src_ip::"$7"
[sshd_login_restricted]
REGEX = (sshd)\[\d+\]\:\s+(\[[^]]+]\s+)?(Login\s+restricted)\s+for\s+([^:]+)
FORMAT = app::$1 vendor_action::"$3" user::"$4"
[pam_unix_authentication_failure]
REGEX = pam_unix\(([^:]+):\w+\)\:\s+authentication\s+(failure)\;\s+logname\=([^\s]+)?\s+uid\=([^\s]+)?\s+euid=([^\s]+)?\s+tty=([^\s]+)?\s+ruser=([^\s]+)?\s+rhost=([^\s]+)?\s+user=([^\s]+)?
FORMAT = app::"$1" action::$2 src_user::$7 src_dns::$8 user::$9
[pam_unix_authentication_success]
REGEX = pam_unix\(([^:]+):\w+\)\:\s+(session\s+opened)\s+for\s+user\s+([^\s]+)\s+by\s+(.*?)\(uid=\d+\)
FORMAT = app::"$1" vendor_action::"$2" user::$3 src_user::$4
[sudo_cannot_identify]
REGEX = pam_unix\(([^:]+):\w+\)\:\s+auth\s+(could\s+not\s+identify\s+password)\s+for\s+\[([^]]+)
FORMAT = app::"$1" vendor_action::"$2" user::"$3"
[ksu_authentication]
REGEX = (ksu)\[\d+\]\:\s+(\[[^]]+]\s+)?\'ksu\s+([^']+)\'\s+(authentication\s+failed|authenticated).*?for\s+(\w+)
FORMAT = app::$1 user::"$3" vendor_action::"$4" src_user::$5
[ksu_authorization]
REGEX = (ksu)\[\d+\]\:\s+(\[[^]]+]\s+)?Account\s+([^:]+)\:\s+authorization\s+for\s+([^@]+).*?(failed|successful)
FORMAT = app::$1 user::"$3" src_user::"$4" vendor_action::$5
[login_authentication]
REGEX = (login)\:.*(failure).*from\s+(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(\,\s+(\S+))?
FORMAT = app::$1 action::$2 src_ip::$3 user::$5
default.meta
[]
access = read : [ * ], write : [ admin ]
export=system
Is there something I am missing or a reason these extractions are not appearing in search? Permissions issue? The props and transforms are pulled from the Splunk Add-on for Unix and Linux (Splunk_TA_nix), which is also installed on our indexers. I'm guessing the transforms could be removed in this custom TA since the REPORT will use the same names referenced in the other app, right? Any help is greatly appreciated!
Looks like the settings eventually took. One thing I did notice in the field extractor in the UI was the sourcetype did not appear in the dropdown selector initially. I did not make any changes to the sourcetype and it eventually appeared and the extractions above began working. Not sure why it took so long as I did a "reload deploy-server" to handle the HF configs and a "apply cluster-bundle" to push out the configs to the indexers.
Looks like the settings eventually took. One thing I did notice in the field extractor in the UI was the sourcetype did not appear in the dropdown selector initially. I did not make any changes to the sourcetype and it eventually appeared and the extractions above began working. Not sure why it took so long as I did a "reload deploy-server" to handle the HF configs and a "apply cluster-bundle" to push out the configs to the indexers.