Troubleshooting a multi-line, multi-value field extraction

[skooby]

My event is mostly output from java's keytool utility, and looks like this......
(except for the top line which is output by a wrapper script)


CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Your keystore contains 4 entries

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: OU=Root CA, O="DataPower Technology, Inc.", C=US
Issuer: OU=Root CA, O="DataPower Technology, Inc.", C=US
Serial number: 0
Valid from: 11/06/03 19:23 until: 06/06/23 19:23
Certificate fingerprints:
MD5: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89
SHA1: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01

*******************************************
*******************************************

Alias name: root
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: 1234567890abcdef
Valid from: 14/07/11 15:00 until: 10/07/26 15:00
Certificate fingerprints:
MD5: CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB
SHA1: CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23

*******************************************
*******************************************

Alias name: default
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: f1234567890abcde
Valid from: 23/03/13 21:30 until: 23/03/14 21:30
Certificate fingerprints:
MD5: EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD
SHA1: EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:CD

*******************************************
*******************************************

Alias name: default_1
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: ef1234567890abcd
Valid from: 14/06/14 21:30 until: 14/06/15 21:30
Certificate fingerprints:
MD5: 01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF
SHA1: 01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:CD:EF

*******************************************
*******************************************

From this, I want to generate a report or table that looks something like this:

KeyStoreName KeyStoreLocation KeyAlias KeySerial KeyExpiry
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 datapower 0 06/06/23 19:23
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 root 1234567890abcdef 10/07/26 15:00
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 default f1234567890abcde 23/03/14 21:30
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 default_1 ef1234567890abcd 14/06/15 21:30


I have tried two approaches:

(1) this one - multi-value-field-extraction

with this one I have only tried to get the KeyAlias

props.conf:

[middleware_unix_cert_scanner1]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = (=========================================================================)
TRUNCATE = 100000
TRANSFORMS-temp-field = temp-field
TRANSFORMS-WASKeyStoreEntryAlias = WASKeyStoreEntryAlias


transforms.conf:

[temp-field]
REGEX=Alias name: (.*)
FORMAT=temporary_field::$1

[WASKeyStoreEntryAlias]
SOURCE_KEY=temporary_field
REGEX=([a-z_+)
FORMAT=WASKeyStoreEntryAlias::$1
MV_ADD=true

(here I used WASKeyStoreEntryAlias as the field name rather than the shorter KeyAlias)

The top 4 lines in the props.conf stanza are to break the script output into events - these work as desired to create the event I have pasted above.

(2) this one - multi-line-multi-value-key-extraction-issue

I should probably shoot for this one as I do need to get multiple multi-valued fields..... but can't get anywhere with this, either:

props.conf:


[middleware_unix_cert_scanner1]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = (=========================================================================)
TRUNCATE = 100000
REPORT-my_report = report_mv


transforms.conf:

[report_mv]
REGEX=Alias name: (\w+)
FORMAT=WASKeyStoreEntryAlias::$1
MV_ADD=true


(again, used WASKeyStoreEntryAlias as the field name rather than the shorter KeyAlias)

I thought this meant WASKeyStoreEntryAlias would be defined and I could do a search like:

sourcetype="middleware_unix_cert_scanner1" | table _time WASKeyStoreEntryAlias

and it would come up with some values for WASKeyStoreEntryAlias - but when I do this, the WASKeyStoreEntryAlias column is blank.

A few side questions which may help my understanding:

1. how do I retrieve the value of the multi-valued field? at the moment I am unable to confirm even if the regex works..

2. how can I confirm my transforms.conf is being read/processed? I have a history of putting the files in the wrong places! it's in the same app as props.conf but inputs.conf is in a different app under deployment-apps, not apps.

3. does it need the event to occur in order to parse it per the transforms.conf? the script only runs every 24h and hasn't run since I attempted option (2)

[skooby · Accepted solution]

In the end (thanks to my local Splunk guru) this was solved by:

1. changing the scripted input so that the line

"CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12"

was output for each keystore stanza, and breaking on ++++++ (also added some extra text to make it easier to pattern match)

This meant each event looked like this:

WASKeyStoreName: CellDefaultTrustStore
WASKeyStoreLocation: /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: OU=Root CA, O="DataPower Technology, Inc.", C=US
Issuer: OU=Root CA, O="DataPower Technology, Inc.", C=US
Serial number: 0
Valid from: 11/06/03 19:23 until: 06/06/23 19:23
Certificate fingerprints:
MD5: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89
SHA1: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01

---

Then (the exciting bit) we used mvzip and mvexpand...

sourcetype=this |
rex "WASKeyStoreName:\s+(?P.*)" |
rex "WASKeyStoreLocation:\s+(?P.*)" |
rex "StoreType:\s+(?P.*)" |
rex "Alias:\s+(?P.*)" |
rex "Alias name:\s+(?P.*)" |
rex "Creation date:\s+(?P.*)" |
rex "Entry type:\s+(?P.*)" |
rex "Certificate chain length: (?P\d+)" |
rex max_match=0 "Certificate[(?P\d+)]" |
rex max_match=0 "Owner:\s+(?P.*)" |
rex max_match=0 "Issuer:\s+(?P.*)" |
rex max_match=0 "Serial number:\s+(?P.*)" |
rex max_match=0 "Valid from:\s+(?P.*)\s+until:(?P.*)" |
rex max_match=0 "\s+MD5:\s+(?P.*)" |
rex max_match=0 "\s+SHA1:\s+(?P.*)" |
eval C_O=mvzip(Cert_num,Owner,"|") |
eval C_O_I=mvzip(C_O,Issuer,"|") |
eval C_O_I_S=mvzip(C_O_I,Serial_number,"|") |
eval C_O_I_S_From=mvzip(C_O_I_S,ValidFrom,"|") |
eval C_O_I_S_From_Until=mvzip(C_O_I_S_From,ValidUntil,"|") |
mvexpand C_O_I_S_From_Until |

rex field=C_O_I_S_From_Until "^(?P\d+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P.*)" | dedup WAS_KeyStore_Location Alias_Name New_Cert_num |
table _time WAS_KeyStore_Location WAS_KeyStore_Name StoreType Alias Alias_Name Creation_Date Entry_Type Cert_Chain_Length New_Cert_num New_Owner New_Issuer New_Serial_number New_ValidUntil

We didn't use reports or transforms in the end but are looking into doing that now to make this easier for non-Splunk gurus to use.

[skooby]

Markdown is playing up:
klzzwxh:0124 (etc.) should be underscore. Too frustrated to try and fix it!

[skooby]

Looking at
whats-the-best-way-to-track-down-propsconf-problems
and mainly
Where_do_I_configure_my_Splunk_settings,

I think I should have configured the REPORT* settings under my search app in deployment-apps, not under my parsing app in apps.

Testing it now...

[skooby]

Just noticed there's a typo in my regex for approach (1) (missing ]):

REGEX=([a-z_+)

However, approach (2) is probably the most appropriate for me so I'm not going to correct & retest (1) unless anyone has a compelling reason to.