I'm struggling to understand how I can use the transforms.conf stanza's to rename dynamic set of field names, ideally using the output of of a separate extraction (or just a regex which may be able to match it).
My problem more specifically is, I receive a series of events from a server which are key'd using the action being performed; the data looks like the following:
name=steve host=xxx download.doc_name=myfile.doc download.doc_id=12345 download.doc_owner=jeff name=jeff host=yyy rename.new_doc_name=renamed.xls rename.old_doc_name=original.xls rename.doc_owner=jeff
and so on, essentially events have static metadata like the user performing the action, and information about the system they are on as regular key=value, however, when it comes to the action taken, anything relating to the action is stored as (action.key)=value.
What I'm looking for is a way to use the transforms and props stanzas to dynamically modify the fields so we can use them in searches as if they were like the following:
name=steve host=xxx action=download doc_name=myfile.doc doc_id=12345 doc_owner=jeff name=jeff host=yyy action=rename new_doc_name=renamed.xls old_doc_name=original.xls doc_owner=jeff
I'm able to get the action= item easily enough, however, I can't seem to find any way to then strip the action & period from the front of the rest of the other key=value pairs. I'm unable to just do this statically, as there are far more actions, with each having potentially their own individual key=value pairs (such as newdocname & olddocname for rename actions).
Ideally I'd like to be able to use this transform with a few different sourcetypes which have similar formatting to this (but we keep separate based on some other factors).
Anyone with more experience know if doing something like I mentioned is possible? Please note that trying to change the data before it indexes into Splunk is not currently an option.
I'd also like to point out I want to avoid just doing a regex which just matches everything after the period, as I'm worried that there may be instances where a legitimate period would appear in the key name or possibly in one of the values (such as IP address) and it would cause problems because of that.
Assuming you've configuration to get key and value separated (means you're able to get newdocname=abc.xls olddocname=xyz.xls ), then (assuming action is same for all fields in the event) you could just extract action using a EXTRACT attribute in props.conf.
[yoursourcetype] EXTRACT-action = \s+(?<Action>[^\.]+)\S+=
Thanks for this!
I do already have the Extract for the action, however my issue now lies with how can I then remove action from the front of the key=value pairs where it exists, and without breaking my original extract (not having to modify any of the _raw data).
How many possible values of action you can have (just the action)?
I'm not positive, it seems like there is around 16 or so but there may be actions which I have not seen yet due to their rarity, as well as any potential new actions which may be added into this data set from new sources.
Yup, maintenance and accuracy will be an issue. If it was not, they you could have created (along with field extraction for action) a FIELDALIAS entries in props.conf for each possible action
[yoursourcetype] FIELDALIAS-alias = download.* AS * rename.* AS * ....
Just add this to the end:
... | foreach *.* [ rename $<<FIELD>>$ AS "<<MATCHSEG2>>" ]
... | foreach *.* [ eval "<<MATCHSEG2>>"=$<<FIELD>>$]
I'm looking to do this with the props & transforms fields, not with actual search commands, though thanks for the suggestions.
I'm struggling understanding why the transforms.conf file seems to be rather ineffective compared with a pure EXTRACT in the props file:
my transform.conf stanza is:
[get_action] REGEX = (\w+)\.action\= FORMAT = action::$1
inside props I have:
[custom_sourcetype] ...standard stuff... REPORT-action = get_action
However it's only returning 5% coverage across my events.
If I just do inside props:
[custom_sourcetype] ....standard stuff... EXTRACT-action = (?<action>\w+)\.action\=
I get 100% coverage in my events.
Can anyone explain this behavior?