Custom Data models Splunk

[IAskALotOfQs]

Hi all, I've got a customer with proprietary logs in their environment and they would like it to be CIM mapped to a data model. The problem is that the logs don't fit any of the data models pre-configured for the CIM Mapping add-on, so I assume I will have to create a custom one that fits with their environment

Problem is, I have never done this before so would need some advice on how to tackle this. One thing that confuses me about their environment is that their custom logs can have different formats for 1 source, this means that 1 event might produce a log with 32 lines, another with 12 lines etc

How would I deal with this?

[richgalloway]

Rather than make a datamodel to fit your data, make the data fit existing datamodels.  Use field aliases and evals to map the proprietary log fields to DM fields.  Not all fields in a datamodel have to be populated so don't worry if you can't get all of them.

Events with different line counts are normal and is not really considered a different format.  For example, an event may contain a traceback, which can have an unpredictable number of lines.  Events within the same source that have different formats (like the timestamp is in a different place) are another matter.  A given log file really should contain a single format (sourcetype, in Splunk terms) for simpler processing.

[IAskALotOfQs]

Thanks for the reply,

Is it normal to see a sourcetype configured for 3 sources where they are files on a linux box, they're got the same timestamps pattern but the logs are different and the lines count is different too

[richgalloway]

Linecount is not a significant factor when comparing event formats.  Most significant are the timestamp format and location, and how fields are delimited (key=value, JSON, etc.).