What is the ideal way of writing log files using log4net?

[maniishpawar]

is there a best practice log4net pattern template that can help SPLUNK index data more efficiently and do field extraction?

[maniishpawar]

I will ask our splunk folks about the CIM. If they had it in place I believe this will solve the purpose.

[skoelpin · Accepted solution]

This is the best part of Splunk! You don't need to customize your logging style to Splunk, but rather customize Splunk to your logging format.

This can be done by setting base configs in your props.conf on the indexer. This will tell Splunk when to break the text into events and also identify the timestamp. An example of base configs will look like this

[sourcetype]
TIME_PREFIX = <Regex showing Splunk where your timestamp starts>
TIME_FORMAT = <striftime of your timestamp>
MAX_TIMESTAMP_LOOKAHEAD <How many characters into the event is your timestamp>
SHOULD_LINEMERGE = false
LINE_BREAKER = <Regex showing how to break the text into events>
TRUNCATE = <How many bytes your event should be before automatically breaking.. This prevents un-usually large events>

Most field extractions happen at search time, so you would add the extractions to your props.conf on the search head. You could also do the extractions inside the Splunk user interface

[micahkemp]

Considering the "enterprise wide" aspect, I think it makes sense for the team running Splunk to specify a format for your logs to come over, considering you are the ones generating the logs.

Why make the Splunk team create props/transforms for each different app's data when it's easier to ask the developers to adhere to a standard they already have configurations for?

[skoelpin]

See my message above about how applying base configs will lessen the load on the indexers.. Also, it would be best to standardize the logging format around the CIM so it limits the need to keep writing props.conf based off new sourcetypes.

[micahkemp]

This was the idea I had, as the entire use case at my company was for SIEM purposes, and if you weren't included in the CIM, your data didn't do anything. I left the company before implementing this, but my initial thought was to allow for generic datamodel adherence in a format like:

<timestamp> <tag1> <tag2> ... <tagN> <key1>=<value1> <tab> <key2>=<value2> ... <tab> <keyN> = <valueN>

This allows anyone to send a log that could conform to the CIM by including the tags that the CIM needs, and the key/value pairs that the CIM wants.

With something like that, a single sourcetype can handle all the parsing and feed any CIM datamodel necessary.

[maniishpawar]

Splunk implementation we had is at enterprise level.
But in our business, we are trying to consolidate logging and enforce a set of format/ best practices so thats why the question. It will be cleaner on the app development team side as well as Splunk team side.

[skoelpin]

Every data source ingested should have base configs applied to them to ease the load off the indexers. If you do not apply them and let Splunk "do the work" and try to make a nicely formatted log, this will add additional overhead to Splunk.

If you look at the link below, it shows 4 pipelines. Applying base configs allows you to skip the merging pipeline, which lessens the load.

https://wiki.splunk.com/Community:HowIndexingWorks

If you want to standardize logging formats, you should look into the common information model

http://docs.splunk.com/Documentation/CIM/4.9.1/User/Overview