Splunk Search

Default Log format for splunk

mclane41
Explorer

I see some post about rules for splunk logs.

But I don't find a list of rules. My applications logs a  lot of lines for splunk (100GB/day) and we prefere use the default integration in splunk (without transformation, extraction...) in order to save time during indexing.

I propose to my developeurs to logs with these constraints.

Where can I find all constraints, or the better constraints ...

Please log like that :

[%m-%d-%Y %H:%M:%S.%Q]key1=value1,key2=value2,...

keys : not begin with number or '_'

values : no spaces or commas else between quote

 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Typically most of extraction is taking place in search time so the most important thing about field extraction is that the format is consistent and can be easily configured (so you don't have cases like escaped characters).

From indexing performance point of view it's most important that the format is consistent across the whole sourcetype, the data breaks easily into separate events and that the timestamp is well-defined and hopefully placed at the beginning of the event.

If you have all this and your sourcetype has the so-called great eight properly configured, you're good to go.

From the practical point of view regarding parsing the data - avoid any nesting - like "real" data as somehow-formatted string within a json structure or the other way around - json structure with a syslog header, any escaped strings within strings and so on - it makes writing extractions and searches a painful experience.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Your constraints look reasonable. You appear to have an easy-to-find timestamp, which presumably will help split your log into separate events, and your field definition appears robust. I suggest you go with what you have.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...