topic not picking up timestamp in Getting Data In

not picking up timestamp

a212830 — Mon, 28 Sep 2020 14:38:46 GMT

Hi,

I added a feed (with some great help from this site), and it is working, except for the timestamp. Each line begins with an epoch timestamp, and I configured Splunk to recognize it, but it's picking up the time that is gets processed.

props.conf:
[snmpinfo]
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = FALSE
NO_BINARY_CHECK = 1
TIME_PREFIX = ^
LINE_BREAKER = ([\r\n]+)
TIME_FORMAT = %s
TRANSFORMS-set = setnull,setparsing

From what I read, the %s is for epoch time. Each event should have a 5 minute interval, but it's appearing with the timestamp when it got processed.

Also, it appears that the output mechanism doesn't always output a full line when writing. Is there any way to make sure it only pulls in entire events? I've seen half events show up.

Re: not picking up timestamp

sowings — Sun, 25 Aug 2013 23:43:11 GMT

That's an epoch to milliseconds, you'll need TIME_FORMAT = %s%3N.

Re: not picking up timestamp

lcrielaa — Mon, 28 Sep 2020 14:39:00 GMT

In order to make sure that Splunk only indexes full events, you can try 2 solutions.

One is to set the following value in your time_before_close in your inputs.conf:

time_before_close =
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past seconds.
* Defaults to 3.

This will tell Splunk to wait a certain amount of seconds before it assumes that the event has completed.
Another way is to use the LINE_BREAKER option in props.conf to tell Splunk what the end of a line of logging looks like. You'll have to be lucky enough that all your logfiles have a determinable ending. Remember that you can use PCRE here, so you can do a positive lookahead to the start of your next line as well to determine the ending of the previous line.