topic Re: Timestamp in every single line in multiline events in Getting Data In

Timestamp in every single line in multiline events

rSteinbrenner — Wed, 18 Jan 2012 11:32:10 GMT

I've configured my splunk to recieve data from syslog via udp. The application uses a SyslogAppender in it's log4j configuration.
I edited my props.conf to get multiline log messages as a single event in splunk.
So far everything works fine.

But there's a timestamp and ip, wich (i believe) is generated by splunk, as prefix of every single line in an multiline event. This timestamp/IP prefix reduces readability of log messages dramaticaly.

Therefore I would like to know if there's a way to make splunk not to display these information in every single line?

here's an example of an event as displayed in splunk:

Jan 18 12:08:26 10.228.45.52 ERROR [18.01.12 12:08:26] RequestCycle - Too many path parts,
please provide sufficient number of path parameter names [thread: http-8080-16]
Jan 18 12:08:26 10.228.45.52 java.lang.IllegalArgumentException: Too many path parts, please      provide sufficient number of path parameter names
Jan 18 12:08:26 10.228.45.52     at   org.apache.wicket.request.target.coding.MixedParamUrlCodingStrategy.decodeParameters(MixedParamUrlCodingStrategy.java:178)
Jan 18 12:08:26 10.228.45.52     at org.apache.wicket.request.target.coding.BookmarkablePageRequestTargetUrlCodingStrategy.decode()

The 'Jan 18 12:08:26 10.228.45.52' timestamp/IP part is the one I would like not to diplay at all, or display only at the very begining of each event.

Thanks in advance for your help!

Re: Timestamp in every single line in multiline events

kristian_kolb — Thu, 19 Jan 2012 12:18:09 GMT

No, that part is created/written by the logging function, it looks to me as standard syslog behaviour, i.e. prepending the log message with a timestamp and the hostname/IP of the sending application/server.

I guess that you should try to configure your SyslogAppender to treat your messages as multi-line - if that is possible.

hope this helps,

Kristian

Re: Timestamp in every single line in multiline events

itinney — Thu, 19 Jan 2012 14:25:11 GMT

Yes you can do it using the following line in props.conf against your source type.

SEDCMD-strip-timestamps = s/^[A-Z][a-z]{2}\s+\d{1,2}\s+\d\d:\d\d:\d\d((?!\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s+[A-Z]+\s+).*)/\1/g

NB: This will leave the first timestamp alone and remove all the others in each event.

$ cat inputs.conf 
[monitor://$SPLUNK_HOME/etc/apps/testApp/logs/testfile.log]
index = test
sourcetype = testfile

$ cat indexes.conf 
[test]
homePath = $SPLUNK_DB/test/db
coldPath = $SPLUNK_DB/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb

$ cat props.conf 
[testfile]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^[A-Z][a-z]{2}\s+\d{1,2}\s+\d\d:\d\d:\d\d\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s+[A-Z]+\s+
TIME_FORMAT = %b %d %T
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 20
SEDCMD-strip-timestamps = s/^[A-Z][a-z]{2}\s+\d{1,2}\s+\d\d:\d\d:\d\d((?!\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s+[A-Z]+\s+).*)/\1/g