How to configure Splunk to recognize the correct timestamp from tshark data output and index all remaining fields properly?

[mikefoti]

I'm using tshark to carve out and send specific fields to a txt file, in hopes splunk will index it properly. But not only is the date field misinterpreted, all data after the data is ignored (not indexed).

This is what is what I see in the Data Preview when I attempt to create a new source type:

1 2/3/02 2:13:22.116 PM Apr  2, 2015 13:46:32.116002000 192.168.2.64    59356   123.123.123.1   443

2 2/3/02 2:13:22.398 PM Apr  2, 2015 13:46:32.398707000 192.168.130.207 57524   122.122.122.1   443

3 2/3/02 2:13:22.415 PM Apr  2, 2015 13:46:32.415975000 192.168.231.22  51513   121.121.121.1   443

Note the interpreted date is 2/3/2002 vs the actual date in 2015.

Using notepad++ I can see the data elements are separated by TABs. It seems once splunk thinks it did a fine job on the date, it believes the 1st tab (just after the date/time) is the end of line so, it seems, it ignores everything until it sees the next date.

Having read several questions/answers leads me to believe data from tshark should be easily recognized and indexed.

my tshark command is:

tshark -r myInputFile.raw tcp.flags.syn eq 0x1 -T fields -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport > myOutputFile.txt

My Question - how can I get each data element to be indexed properly?

[esix_splunk · Accepted solution]

Since you have multiple time stamps in the event, you need select the right one for Splunk to recognize.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Configuretimestamprecognition

Your format should be the :

Apr  2, 2015 13:46:32.116002000

Portion from your log.

So in your props, your timestamp format should be something like:

TIME_FORMAT =%b, %Y %H:%M:%S.%9N
TIME_PREFIX= \t

I'm not sure if there is a tab, you could also try

TIME_PREFIX = (AM\s|PM\s)

That looks for AM OR PM (not its followed by a space. That might be a tab in your log..)

[mikefoti]

Esix_splunk,
Thanks... once I added %d after %b, it worked.

TIME_FORMAT =%b %d, %Y %H:%M:%S.%9N