Splunk Search
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

Troubleshooting a multi-line, multi-value field extraction

skooby
Explorer
‎07-24-2014 02:33 AM

My event is mostly output from java's keytool utility, and looks like this......
(except for the top line which is output by a wrapper script)


CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Your keystore contains 4 entries

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: OU=Root CA, O="DataPower Technology, Inc.", C=US
Issuer: OU=Root CA, O="DataPower Technology, Inc.", C=US
Serial number: 0
Valid from: 11/06/03 19:23 until: 06/06/23 19:23
Certificate fingerprints:
MD5: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89
SHA1: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01

*******************************************
*******************************************

Alias name: root
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: 1234567890abcdef
Valid from: 14/07/11 15:00 until: 10/07/26 15:00
Certificate fingerprints:
MD5: CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB
SHA1: CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23

*******************************************
*******************************************

Alias name: default
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: f1234567890abcde
Valid from: 23/03/13 21:30 until: 23/03/14 21:30
Certificate fingerprints:
MD5: EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD
SHA1: EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:CD

*******************************************
*******************************************

Alias name: default_1
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: CN=server.acme.co.uk, OU=mycell, OU=dmgrnode, O=IBM, C=US
Issuer: CN=server.acme.co.uk, OU=Root Certificate, OU=mycell, OU=dmgrnode, O=IBM, C=US
Serial number: ef1234567890abcd
Valid from: 14/06/14 21:30 until: 14/06/15 21:30
Certificate fingerprints:
MD5: 01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF
SHA1: 01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:CD:EF

*******************************************
*******************************************

From this, I want to generate a report or table that looks something like this:

KeyStoreName KeyStoreLocation KeyAlias KeySerial KeyExpiry
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 datapower 0 06/06/23 19:23
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 root 1234567890abcdef 10/07/26 15:00
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 default f1234567890abcde 23/03/14 21:30
CellDefaultTrustStore /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12 default_1 ef1234567890abcd 14/06/15 21:30

I have tried two approaches:

(1) this one - multi-value-field-extraction

with this one I have only tried to get the KeyAlias

props.conf:

[middleware_unix_cert_scanner1]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = (=========================================================================)
TRUNCATE = 100000
TRANSFORMS-temp-field = temp-field
TRANSFORMS-WASKeyStoreEntryAlias = WASKeyStoreEntryAlias

transforms.conf:

[temp-field]
REGEX=Alias name: (.*)
FORMAT=temporary_field::$1

[WASKeyStoreEntryAlias]
SOURCE_KEY=temporary_field
REGEX=([a-z_+)
FORMAT=WASKeyStoreEntryAlias::$1
MV_ADD=true

(here I used WASKeyStoreEntryAlias as the field name rather than the shorter KeyAlias)

The top 4 lines in the props.conf stanza are to break the script output into events - these work as desired to create the event I have pasted above.

(2) this one - multi-line-multi-value-key-extraction-issue

I should probably shoot for this one as I do need to get multiple multi-valued fields..... but can't get anywhere with this, either:

props.conf:


[middleware_unix_cert_scanner1]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = (=========================================================================)
TRUNCATE = 100000
REPORT-my_report = report_mv

transforms.conf:

[report_mv]
REGEX=Alias name: (\w+)
FORMAT=WASKeyStoreEntryAlias::$1
MV_ADD=true

(again, used WASKeyStoreEntryAlias as the field name rather than the shorter KeyAlias)

I thought this meant WASKeyStoreEntryAlias would be defined and I could do a search like:

sourcetype="middleware_unix_cert_scanner1" | table _time WASKeyStoreEntryAlias

and it would come up with some values for WASKeyStoreEntryAlias - but when I do this, the WASKeyStoreEntryAlias column is blank.

A few side questions which may help my understanding:

  1. how do I retrieve the value of the multi-valued field? at the moment I am unable to confirm even if the regex works..

  2. how can I confirm my transforms.conf is being read/processed? I have a history of putting the files in the wrong places! it's in the same app as props.conf but inputs.conf is in a different app under deployment-apps, not apps.

  3. does it need the event to occur in order to parse it per the transforms.conf? the script only runs every 24h and hasn't run since I attempted option (2)

0 Karma
Reply
1 Solution
Solved! Jump to solution
Solution

skooby
Explorer
‎08-15-2014 03:52 AM

In the end (thanks to my local Splunk guru) this was solved by:

  1. changing the scripted input so that the line

"CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12"

was output for each keystore stanza, and breaking on ++++++ (also added some extra text to make it easier to pattern match)

This meant each event looked like this:

WASKeyStoreName: CellDefaultTrustStore
WASKeyStoreLocation: /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: OU=Root CA, O="DataPower Technology, Inc.", C=US
Issuer: OU=Root CA, O="DataPower Technology, Inc.", C=US
Serial number: 0
Valid from: 11/06/03 19:23 until: 06/06/23 19:23
Certificate fingerprints:
MD5: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89
SHA1: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01

Then (the exciting bit) we used mvzip and mvexpand...

sourcetype=this |
rex "WASKeyStoreName:\s+(?P.*)" |
rex "WASKeyStoreLocation:\s+(?P.*)" |
rex "StoreType:\s+(?P.*)" |
rex "Alias:\s+(?P.*)" |
rex "Alias name:\s+(?P.*)" |
rex "Creation date:\s+(?P.*)" |
rex "Entry type:\s+(?P.*)" |
rex "Certificate chain length: (?P\d+)" |
rex max_match=0 "Certificate[(?P\d+)]" |
rex max_match=0 "Owner:\s+(?P.*)" |
rex max_match=0 "Issuer:\s+(?P.*)" |
rex max_match=0 "Serial number:\s+(?P.*)" |
rex max_match=0 "Valid from:\s+(?P.*)\s+until:(?P.*)" |
rex max_match=0 "\s+MD5:\s+(?P.*)" |
rex max_match=0 "\s+SHA1:\s+(?P.*)" |
eval C_O=mvzip(Cert_num,Owner,"|") |
eval C_O_I=mvzip(C_O,Issuer,"|") |
eval C_O_I_S=mvzip(C_O_I,Serial_number,"|") |
eval C_O_I_S_From=mvzip(C_O_I_S,ValidFrom,"|") |
eval C_O_I_S_From_Until=mvzip(C_O_I_S_From,ValidUntil,"|") |
mvexpand C_O_I_S_From_Until |

rex field=C_O_I_S_From_Until "^(?P\d+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P.*)" | dedup WAS_KeyStore_Location Alias_Name New_Cert_num |
table _time WAS_KeyStore_Location WAS_KeyStore_Name StoreType Alias Alias_Name Creation_Date Entry_Type Cert_Chain_Length New_Cert_num New_Owner New_Issuer New_Serial_number New_ValidUntil

We didn't use reports or transforms in the end but are looking into doing that now to make this easier for non-Splunk gurus to use.

View solution in original post

0 Karma
Reply
Solved! Jump to solution
Solution

skooby
Explorer
‎08-15-2014 03:52 AM

In the end (thanks to my local Splunk guru) this was solved by:

  1. changing the scripted input so that the line

"CellDefaultTrustStore - /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12"

was output for each keystore stanza, and breaking on ++++++ (also added some extra text to make it easier to pattern match)

This meant each event looked like this:

WASKeyStoreName: CellDefaultTrustStore
WASKeyStoreLocation: /opt/IBM/WebSphere/AppServer/profiles/dmgr/config/cells/mycell/trust.p12

Keystore type: PKCS12
Keystore provider: IBMJCE

Alias name: datapower
Creation date: 01-Jan-1970
Entry type: trustedCertEntry

Owner: OU=Root CA, O="DataPower Technology, Inc.", C=US
Issuer: OU=Root CA, O="DataPower Technology, Inc.", C=US
Serial number: 0
Valid from: 11/06/03 19:23 until: 06/06/23 19:23
Certificate fingerprints:
MD5: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89
SHA1: AB:CD:EF:01:23:45:67:89:AB:CD:EF:01:23:45:67:89:AB:CD:EF:01

Then (the exciting bit) we used mvzip and mvexpand...

sourcetype=this |
rex "WASKeyStoreName:\s+(?P.*)" |
rex "WASKeyStoreLocation:\s+(?P.*)" |
rex "StoreType:\s+(?P.*)" |
rex "Alias:\s+(?P.*)" |
rex "Alias name:\s+(?P.*)" |
rex "Creation date:\s+(?P.*)" |
rex "Entry type:\s+(?P.*)" |
rex "Certificate chain length: (?P\d+)" |
rex max_match=0 "Certificate[(?P\d+)]" |
rex max_match=0 "Owner:\s+(?P.*)" |
rex max_match=0 "Issuer:\s+(?P.*)" |
rex max_match=0 "Serial number:\s+(?P.*)" |
rex max_match=0 "Valid from:\s+(?P.*)\s+until:(?P.*)" |
rex max_match=0 "\s+MD5:\s+(?P.*)" |
rex max_match=0 "\s+SHA1:\s+(?P.*)" |
eval C_O=mvzip(Cert_num,Owner,"|") |
eval C_O_I=mvzip(C_O,Issuer,"|") |
eval C_O_I_S=mvzip(C_O_I,Serial_number,"|") |
eval C_O_I_S_From=mvzip(C_O_I_S,ValidFrom,"|") |
eval C_O_I_S_From_Until=mvzip(C_O_I_S_From,ValidUntil,"|") |
mvexpand C_O_I_S_From_Until |

rex field=C_O_I_S_From_Until "^(?P\d+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P[^|]+)|(?P.*)" | dedup WAS_KeyStore_Location Alias_Name New_Cert_num |
table _time WAS_KeyStore_Location WAS_KeyStore_Name StoreType Alias Alias_Name Creation_Date Entry_Type Cert_Chain_Length New_Cert_num New_Owner New_Issuer New_Serial_number New_ValidUntil

We didn't use reports or transforms in the end but are looking into doing that now to make this easier for non-Splunk gurus to use.

0 Karma
Reply
Solved! Jump to solution

skooby
Explorer
‎08-15-2014 04:01 AM

Markdown is playing up:
klzzwxh:0124 (etc.) should be underscore. Too frustrated to try and fix it!

0 Karma
Reply
Solved! Jump to solution

skooby
Explorer
‎07-24-2014 07:49 AM

Looking at
whats-the-best-way-to-track-down-propsconf-problems
and mainly
Where_do_I_configure_my_Splunk_settings,

I think I should have configured the REPORT* settings under my search app in deployment-apps, not under my parsing app in apps.

Testing it now...

0 Karma
Reply
Solved! Jump to solution

skooby
Explorer
‎07-24-2014 06:14 AM

Just noticed there's a typo in my regex for approach (1) (missing ]):

REGEX=([a-z_+)

However, approach (2) is probably the most appropriate for me so I'm not going to correct & retest (1) unless anyone has a compelling reason to.

0 Karma
Reply
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...