topic Re: Troubleshooting a multi-line, multi-value field extraction in Splunk Search

Troubleshooting a multi-line, multi-value field extraction

skooby — Thu, 24 Jul 2014 09:33:56 GMT

My event is mostly output from java's keytool utility, and looks like this......
(except for the top line which is output by a wrapper script)

CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Your keystore contains 4 entries

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: OU=Root CA, O="DataPower Technology, Inc.", C=US
Issuer: OU=Root CA, O="DataPower Technology, Inc.", C=US
Serial number: 0
Valid from: 11/06/03 19:23 until: 06/06/23 19:23
Certificate fingerprints:
MD5: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89
SHA1: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01

*******************************************
*******************************************

Alias name: root
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: 1234567890abcdef
Valid from: 14/07/11 15:00 until: 10/07/26 15:00
Certificate fingerprints:
MD5: CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB
SHA1: CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23

*******************************************
*******************************************

Alias name: default
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: f1234567890abcde
Valid from: 23/03/13 21:30 until: 23/03/14 21:30
Certificate fingerprints:
MD5: EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD
SHA1: EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:CD

*******************************************
*******************************************

Alias name: default_1
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: ef1234567890abcd
Valid from: 14/06/14 21:30 until: 14/06/15 21:30
Certificate fingerprints:
MD5: 01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF
SHA1: 01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:CD:EF

*******************************************
*******************************************

From this, I want to generate a report or table that looks something like this:
KeyStoreName KeyStoreLocation KeyAlias KeySerial KeyExpiry CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 datapower 0 06/06/23 19:23 CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 root 1234567890abcdef 10/07/26 15:00 CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 default f1234567890abcde 23/03/14 21:30 CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 default_1 ef1234567890abcd 14/06/15 21:30

I have tried two approaches:

(1) this one - multi-value-field-extraction

with this one I have only tried to get the KeyAlias

props.conf:
[middleware_unix_cert_scanner1] DATETIME_CONFIG = CURRENT SHOULD_LINEMERGE = false LINE_BREAKER = (=========================================================================) TRUNCATE = 100000 TRANSFORMS-temp-field = temp-field TRANSFORMS-WASKeyStoreEntryAlias = WASKeyStoreEntryAlias

transforms.conf:
[temp-field] REGEX=Alias name: (.*) FORMAT=temporary_field::$1

[WASKeyStoreEntryAlias]
SOURCE_KEY=temporary_field
REGEX=([a-z_+)
FORMAT=WASKeyStoreEntryAlias::$1
MV_ADD=true

(here I used WASKeyStoreEntryAlias as the field name rather than the shorter KeyAlias)

The top 4 lines in the props.conf stanza are to break the script output into events - these work as desired to create the event I have pasted above.

(2) this one - multi-line-multi-value-key-extraction-issue

I should probably shoot for this one as I do need to get multiple multi-valued fields..... but can't get anywhere with this, either:

props.conf:

[middleware_unix_cert_scanner1] DATETIME_CONFIG = CURRENT SHOULD_LINEMERGE = false LINE_BREAKER = (=========================================================================) TRUNCATE = 100000 REPORT-my_report = report_mv

transforms.conf:
[report_mv] REGEX=Alias name: (\w+) FORMAT=WASKeyStoreEntryAlias::$1 MV_ADD=true

(again, used WASKeyStoreEntryAlias as the field name rather than the shorter KeyAlias)

I thought this meant WASKeyStoreEntryAlias would be defined and I could do a search like:

sourcetype="middleware_unix_cert_scanner1" | table _time WASKeyStoreEntryAlias

and it would come up with some values for WASKeyStoreEntryAlias - but when I do this, the WASKeyStoreEntryAlias column is blank.

A few side questions which may help my understanding:

how do I retrieve the value of the multi-valued field? at the moment I am unable to confirm even if the regex works..
how can I confirm my transforms.conf is being read/processed? I have a history of putting the files in the wrong places! it's in the same app as props.conf but inputs.conf is in a different app under deployment-apps, not apps.
does it need the event to occur in order to parse it per the transforms.conf? the script only runs every 24h and hasn't run since I attempted option (2)

Re: Troubleshooting a multi-line, multi-value field extraction

skooby — Thu, 24 Jul 2014 13:14:35 GMT

Just noticed there's a typo in my regex for approach (1) (missing ]):

REGEX=([a-z_+)

However, approach (2) is probably the most appropriate for me so I'm not going to correct & retest (1) unless anyone has a compelling reason to.

Re: Troubleshooting a multi-line, multi-value field extraction

skooby — Thu, 24 Jul 2014 14:49:16 GMT

Looking at
whats-the-best-way-to-track-down-propsconf-problems
and mainly
Where_do_I_configure_my_Splunk_settings,

I think I should have configured the REPORT* settings under my search app in deployment-apps, not under my parsing app in apps.

Testing it now...

Re: Troubleshooting a multi-line, multi-value field extraction

skooby — Fri, 15 Aug 2014 10:52:35 GMT

In the end (thanks to my local Splunk guru) this was solved by:

changing the scripted input so that the line

"CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12"

was output for each keystore stanza, and breaking on ++++++ (also added some extra text to make it easier to pattern match)

This meant each event looked like this:
WASKeyStoreName: CellDefaultTrustStore WASKeyStoreLocation: /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

rex field=C_O_I_S_From_Until "^(?P\d+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P.*)" | dedup WAS_KeyStore_Location Alias_Name New_Cert_num |
table _time WAS_KeyStore_Location WAS_KeyStore_Name StoreType Alias Alias_Name Creation_Date Entry_Type Cert_Chain_Length New_Cert_num New_Owner New_Issuer New_Serial_number New_ValidUntil

We didn't use reports or transforms in the end but are looking into doing that now to make this easier for non-Splunk gurus to use.

Re: Troubleshooting a multi-line, multi-value field extraction

skooby — Fri, 15 Aug 2014 11:01:06 GMT

Markdown is playing up:
klzzwxh:0124 (etc.) should be underscore. Too frustrated to try and fix it!