Is there anyway to append current time along with milliseconds to incoming syslog feed to indexer

[adityapavan18]

Hi

I am receiving a syslog feed from a server.I am trying to index that data. In syslog feed no milliseconds are being sent in feed.So event datetiime is of format 8/31/12 11:44:18*.000* AM
i.e milliseconds is always .000 for all events indexed.

So while calculating duration after using transaction command if transaction happens in less than a second then the duration shows as O, which is claerly wrong.
So is there a way i can append current date time with milliseconds when getting indexed, so that i get proper milliseconds feild also.

[kristian_kolb]

Well, I'd recommend that you investigate if this can be altered on the sending side. (I'm assuming that you send syslog straight into splunk, not via a syslog receiver that writes files to a directory monitored by splunk).

If you actually are writing files to a file system monitored by splunk, perhaps you can force the syslog server to add an extra timestamp when writing the events to disk.

Another option would be to use the _indextime, which is the time the indexer stored the event in an index. Needless to say, the quality of that timestamp can vary, depending on network congestion, temporarily high load on the indexer etc etc. Also... I don't think _indextime stores sub-seconds... mm..forget that... or try yourself.

If I remebmer correctly you cannot use _indextime directly, but I guess something like;

base search | eval _time = _indextime | transaction blah blah | top duration etc etc

/Kristian

[adityapavan18]

Thnx kristian. but _indextime also doesn't track milliseconds, so without those milliseconds, i am again back to square one, as milliseconds will always show .000 🙂

[MHibbin]

As far as I'm aware, this would be something that you have to try and configure at the source (i.e. the syslog source), as adding milliseconds at Splunk's end would give inaccurate values as it would clearly not be the same time (even if you had the worlds fastest networks and systems it would still be different).

It might be better to look at the data source for options, perhaps this is something that can configured if you ask the questions to the vendor or those in the know with the product. If not a possible solution for having some difference at the millisecond level...

You could possible write the syslog to file and have a date inserted into the file before/after each log. Although this would probably be different to the actual syslog time, it may give you the accurate difference... you could then use this as you index time, and extract the actual time as field for additional information when troubleshooting, etc.

Just some thoughts,

MHibbin

[Jason]

You can use milliseconds in TIME_FORMAT - use %xN, where x is the number of digits of precision below second level. For example, %H:%M:%S.%3N for 10:23:45.447 - use of TIME_FORMAT of course requires the data to be in your event to begin with.

[MHibbin]

Apologies I thought you could use milliseconds in strptime

[adityapavan18]

Hi MHibbin,
Using TIME_FORMAT hw can i pick milliseconds, can you please elaborate on that...

[MHibbin]

Well, your answer has better content... I didn't know about the _indextime field. So I've learnt something new... so I'll upvote for that!

[kristian_kolb]

Damn! I realize that I'm more than a little bit late... spend too much time editing. 😞

Well, at least it seems that we have the same thoughts.

/k

[MHibbin]

I would add that if you are monitoring that port directly with Splunk, best practise is to record to file first... ref: http://wiki.splunk.com/Community:Best_Practice_For_Configuring_Syslog_Input

NOTE Option 1

[MHibbin]

So you are indexing syslog straight to Splunk (i.e. allow Splunk to read on UDP 514)?

Perhaps try looking at props.conf for your source/sourctype/host (whichever is more specific to your syslog data) use that as a stanza in props.conf and then use the "DATETIME_CONFIG = CURRENT" parameter, this may force that data type to index with the system time when being indexed. You can also use the "TIME_FORMAT" option to configure specific time format. See the link below for reference and examples.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

[adityapavan18]

Hi MHibbin,

Thnx for that. It is confirmed at datasource that milliseconds cannot be sent in syslog.

So only other should be writing it in file, and then reading it.

But is there anyway i could append the time of receipt at indexer as the event time, instaed of writing and reading from file.