Can you help me with my multi-field field extraction query?

[jpolcari]

I have a field in an event that contains a number of separate individual fields. What would be the most efficient way to extract these so I can search them? Can this be done at index time using transforms? I'm aware of the makemv command, but these are not values of the same field; they are fields within a field.

I'm trying to extract the individual fields within the auth_source field. So I would like something like auth_source_auth, auth_source_level, auth_source_login_id, etc

Example:

2018-10-18 19:36:47.000, date="10/18/2018 19:36:47", id="166450895", action_at="1539891407534", action="CREATE", permission_id="106965896", customer_id="5495063", role_id="270", active="1", created_at="2018-10-18 19:36:47.0", auth_source="auth:shhkey-v1 | auth-level:internal | login-id:firstlast@company.com-1539295510177 | cus-id:651613 | user-id:65161 | origin-ip:123.123.228.35 | correlation-id:2282777c-25c8-4e1e-a823-4e495fa354ad | zk-group:app-users:api:web:user-traffic:us-east-1", hub_id="61861251", produced_at="1539891407829"

[tiagopeq]

If doing this at index-time really is requirement, you can always use a report on the part of the event that starts with auth_source=

the report option in your props file associated with your sourcetype the extracts at search time the pairs separated by the = sign

Check this answer for reference 🙂

https://answers.splunk.com/answers/9853/multivalue-field-regex-question.html#answer-9875

[cpetterborg · Accepted solution]

Do you want these field extractions to be at index time or search time? (Splunk best practice is at search time, but it can be at index time.)

Given that search time is preferred, here is a rex that will do what you want, and the auto-field-extraction will be similar (this is a run-anywhere search that shows the concept by using your data and showing that it works, the only line you need to use is the rex line for the regular expression to use for the field extraction):

| makeresults 
| eval auth_source="auth:shhkey-v1 | auth-level:internal | login-id:firstlast@company.com-1539295510177 | cus-id:651613 | user-id:65161 | origin-ip:123.123.228.35 | correlation-id:2282777c-25c8-4e1e-a823-4e495fa354ad | zk-group:app-users:api:web:user-traffic:us-east-1"
| rex field=auth_source "auth:(?<auth>[^|]*) \| auth-level:(?<auth_level>[^|]*) \| login-id:(?<login_id>[^|]*) \| cus-id:(?<cus_id>[^|]*) \| user-id:(?<user_id>[^|]*) \| origin-ip:(?<origin_ip>[^|]*) \| correlation-id:(?<correlation_id>[^|]*) \| zk-group:(?<zk_group>[^|]*)"

If you need more help on this, just add your comments.

[jpolcari]

Looks great, thanks! I was thinking index time but search time will work just fine.