topic Re: Json ingestion --> Data models Inquiry around best practices in All Apps and Add-ons

Json ingestion --> Data models Inquiry around best practices

bgagliardi1 — Tue, 02 Jan 2018 21:18:45 GMT

A colleague of mine is reporting that data models don't support {} characters in field names.

1) Is this true?
2) If it is true, what's the best practice for getting around this? Is there an addon available that converts all { and } to another character? I have tons of log sources whose fields include squiggly braces.

Re: Json ingestion --> Data models Inquiry around best practices

micahkemp — Tue, 02 Jan 2018 23:38:10 GMT

You can use FIELDALIAS in props.conf to rename the auto-extracted field names into datamodel compliant (preferably CIM-compliant, if possible) field names:

FIELDALIAS-<class> = (<orig_field_name> AS <new_field_name>)+
* Use this to apply aliases to a field. The original field is not removed.
  This just means that the original field can be searched on using any of
  its aliases.
* You can create multiple aliases for the same field.
* <orig_field_name> is the original name of the field.
* <new_field_name> is the alias to assign to the field.
* You can include multiple field alias renames in the same stanza.
* Field aliasing is performed at search time, after field extraction, but
  before calculated fields (EVAL-* statements) and lookups.
  This means that:
        * Any field extracted at search time can be aliased.
        * You can specify a lookup based on a field alias.
        * You cannot alias a calculated field.

Re: Json ingestion --> Data models Inquiry around best practices

bgagliardi1 — Wed, 03 Jan 2018 20:31:43 GMT

@micahkemp - is there a more programmatic way to go about this? Or do I have to write some kind of script that querys all of my fields, greps for {} and make a new alias for it that doesn't have {}? Also, can I specify an index or sourcetype for the fields/do I have to worry about aliases that have duplicate names?

Re: Json ingestion --> Data models Inquiry around best practices

micahkemp — Wed, 03 Jan 2018 20:43:25 GMT

Datamodels have an explicit (and finite) set of fields already, so scripting something to find all your available fields and use FIELDALIAS to remove the {} wouldn't really accomplish anything for you anyway.

What you'll really have to do is examine your datamodels (homegrown, CIM, or whatever) and determine which fields in your events represent the fields in the datamodel it belongs to. This may sound tedious, but it's the method Splunk uses for normalization of datamodels.

Re: Json ingestion --> Data models Inquiry around best practices

bgagliardi1 — Wed, 03 Jan 2018 21:53:20 GMT

@micahkemp - I don't necessarily want to provide all fields and aliases to every data model. Rather, from the context of any app, for a new data model, if a search is made, there's a pseudonym automatically available for any field that contains {}.

Perhaps leveraging Splunk API with a script, and programmatically updating the config files. Do I have to update each props.conf respective to each app, or could I somehow globally share it, assuming any of my logic works? 🙂

I'm definitely trying to avoid any kind of manual process whatsoever. Maintaining each new field manually wouldn't scale in my environment.

Thanks for your help and input.

Re: Json ingestion --> Data models Inquiry around best practices

micahkemp — Wed, 03 Jan 2018 22:28:23 GMT

I'm not saying what you want to do is wrong, but it's not inline with working with datamodels. Datamodels, by definition, are a predefined and finite set of fields.

Re: Json ingestion --> Data models Inquiry around best practices

bgagliardi1 — Wed, 03 Jan 2018 22:43:16 GMT

This is my workaround for now within datamodels, as an example:

index=proofpoint policyRoutes{}=* | rename policyRoutes{} as policyRoutes

Will have to just be in training unless I can get it to be programmatic.

Re: Json ingestion --> Data models Inquiry around best practices

micahkemp — Wed, 03 Jan 2018 22:44:19 GMT

Which datamodel is the field policyRoutes defined in?

Re: Json ingestion --> Data models Inquiry around best practices

micahkemp — Wed, 03 Jan 2018 22:47:13 GMT

At search time you can try this to simply remove the trailing {} from any field:

| rename *{} AS *

A run anywhere example of this in action is:

| makeresults
| eval field1{} = "value1", field2{} = "value2", field3 = "value3"
| rename *{} AS *

Re: Json ingestion --> Data models Inquiry around best practices

micahkemp — Wed, 03 Jan 2018 22:55:38 GMT

It's also worth noting that datamodel feeds can't contain pipes, so your search above can't be used with datamodels.