I don't have separate search head, so I installed Stream App on indexer.
I have Universal Forwarder and Indexer on a same box, my mac book. So the locale is the same.
kwchang@MacBook: ~ $ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
kwchang@MacBook: ~ $ sudo systemsetup -gettimezone
Password:
Time Zone: Asia/Seoul
And the raw events of stream:tcp from universal forwarder look like followings.
{"endtime":"2015-11-09T00:40:08.061276Z","timestamp":"2015-11-09T00:39:55.449199Z","src_ip":"172.20.10.2","src_mac":"1A:F6:43:34:EF:AA","src_port":49977,"connection":"54.251.115.138:443","client_rtt":140,"client_rtt_packets":1,"client_rtt_sum":140,"ack_packets_in":3,"bytes_in":276,"data_packets_in":0,"duplicate_packets_in":0,"missing_packets_in":0,"packets_in":4,"app":"unknown","server_rtt":0,"server_rtt_packets":0,"server_rtt_sum":0,"dest_ip":"54.251.115.138","dest_mac":"3A:F6:43:43:54:64","dest_port":443,"ack_packets_out":2,"bytes_out":140,"data_packets_out":0,"duplicate_packets_out":0,"missing_packets_out":0,"packets_out":2,"tcp_status":0,"time_taken":12612217}
{"endtime":"2015-11-09T00:40:08.051195Z","timestamp":"2015-11-09T00:39:57.078105Z","src_ip":"172.20.10.2","src_mac":"1A:F6:43:34:EF:AA","src_port":50008,"connection":"184.73.197.150:443","client_rtt":140,"client_rtt_packets":1,"client_rtt_sum":140,"ack_packets_in":3,"bytes_in":276,"data_packets_in":0,"duplicate_packets_in":0,"missing_packets_in":0,"packets_in":4,"app":"unknown","server_rtt":0,"server_rtt_packets":0,"server_rtt_sum":0,"dest_ip":"184.73.197.150","dest_mac":"3A:F6:43:43:54:64","dest_port":443,"ack_packets_out":2,"bytes_out":140,"data_packets_out":0,"duplicate_packets_out":0,"missing_packets_out":0,"packets_out":2,"tcp_status":0,"time_taken":10973230}
Sometimes, there seems to have issues in event breaking. Following data was captured as a single event.
{"timestamp":"2015-11-09T09:12:114Z","endtime":"2015-11-09T09:14:514Z","src_ip":"95.149.230.218","dest_ip":"55.2.98.115","src_port":"36998","dest_port":"10043","src_mac":"00:50:56:92:4E:26","dest_mac":"00:1B:17:00:01:30","packets_in":13,"packets_out":13,"ack_packets_in":6,"ack_packets_out":7,"missing_packets_in":0,"missing_packets_out":0,"duplicate_packets_in":0,"duplicate_packets_out":0,"data_packets_in":6,"data_packets_out":6,"bytes_in":525,"bytes_out":4137,"time_taken":65596,"ssl_version":"3.0","ssl_session_id":"bd608869f0c629767ea7e3ebf7a63bdcffb0ef58b1b941e6b0c044acb6820a77","ssl_cert_md5":"AE435972651BFE75B5E8547BF7A35AC2","ssl_subject":"OU=Domain Control Validated,CN=*.dropbox.com","ssl_issuer":"C=US,O=\"GeoTrust, Inc.\",CN=RapidSSL CA","ssl_signature_algorithm":"sha1WithRSAEncryption","ssl_publickey_algorithm":"rsaEncryption","ssl_serialnumber":"25196489825037653343","ssl_validity_end":"Apr 10 21:21:59 2016 GMT","ssl_validity_start":"Apr 10 21:21:59 2014 GMT","client_rtt":62,"server_rtt":25593,"client_rtt_sum":186,"server_rtt_sum":25593,"client_rtt_packets":3,"server_rtt_packets":1,"connection":"54.200.187.224:443","tcp_status":0,"app":"ssl"} {"timestamp":"2015-11-09T09:12:374Z","endtime":"2015-11-09T09:14:344Z","src_ip":"82.147.19.83","dest_ip":"184.50.214.228","src_port":"48497","dest_port":"10043","src_mac":"00:50:56:92:4E:26","dest_mac":"00:1B:17:00:01:30","packets_in":13,"packets_out":13,"ack_packets_in":6,"ack_packets_out":7,"missing_packets_in":0,"missing_packets_out":0,"duplicate_packets_in":0,"duplicate_packets_out":0,"data_packets_in":6,"data_packets_out":6,"bytes_in":525,"bytes_out":4137,"time_taken":65596,"ssl_version":"3.0","ssl_session_id":"bd608869f0c629767ea7e3ebf7a63bdcffb0ef58b1b941e6b0c044acb6820a77","ssl_cert_md5":"AE0E54D5E06C387EE6725908B44368E8","ssl_subject":"CN=sc-db-centos2.sv.splunk.com,O=SplunkUser","ssl_issuer":"C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA","ssl_signature_algorithm":"sha1WithRSAEncryption","ssl_publickey_algorithm":"rsaEncryption","ssl_serialnumber":"11712524697123285136","ssl_validity_end":"May 10 21:21:59 2014 GMT","ssl_validity_start":"Apr 10 21:21:59 2014 GMT","client_rtt":62,"server_rtt":25593,"client_rtt_sum":186,"server_rtt_sum":25593,"client_rtt_packets":3,"server_rtt_packets":1,"connection":"54.200.187.224:443","tcp_status":0,"app":"ssl"} {"timestamp":"2015-11-09T09:11:174Z","endtime":"2015-11-09T09:12:214Z","src_ip":"70.146.92.251","dest_ip":"183.20.127.198","src_port":"18737","dest_port":"8443","src_mac":"00:50:56:92:4E:26","dest_mac":"00:1B:17:00:01:30","packets_in":13,"packets_out":13,"ack_packets_in":6,"ack_packets_out":7,"missing_packets_in":0,"missing_packets_out":0,"duplicate_packets_in":0,"duplicate_packets_out":0,"data_packets_in":6,"data_packets_out":6,"bytes_in":525,"bytes_out":4137,"time_taken":65596,"ssl_version":"3.0","ssl_session_id":"bd608869f0c629767ea7e3ebf7a63bdcffb0ef58b1b941e6b0c044acb6820a77","ssl_cert_md5":"7FEB15E4180D15E0196F298554DAEC6F","ssl_subject":"C=US,postalCode=20814,ST=Maryland,L=Bethesda,STREET=Suite 205,STREET=8120 Woodmont Ave,O=The SANS Institute,OU=Network Operations Center (NOC),OU=Unified Communications,CN=isc.sans.org","ssl_issuer":"C=US,ST=Arizona,L=Scottsdale,O=\"GoDaddy.com, Inc.\",CN=Go Daddy Root Certificate Authority - G2","ssl_signature_algorithm":"sha1WithRSAEncryption","ssl_publickey_algorithm":"rsaEncryption","ssl_serialnumber":"40564819207326872660","ssl_validity_end":"Jun 10 21:21:59 2014 GMT","ssl_validity_start":"Apr 10 21:21:59 2014 GMT","client_rtt":62,"server_rtt":25593,"client_rtt_sum":186,"server_rtt_sum":25593,"client_rtt_packets":3,"server_rtt_packets":1,"connection":"54.200.187.224:443","tcp_status":0,"app":"ssl"}
I also changed MAX_TIMESTAMP_LOOKAHEAD=0 but not helpful.
With the search you said, the timeendpos and timestartpos fields were not captured and I got followings.
sourcetype="stream:tcp" | eval raw_timestamp=substr(_raw, timestartpos+1, timeendpos-timestartpos) | table _time, endtime, timestamp, raw_timestamp, timestartpos, timeendpos
_time endtime timestamp
2015-11-09 09:55:43 2015-11-09T00:52:30.039647Z 2015-11-09T00:40:14.815575Z
2015-11-09 09:55:43 2015-11-09T00:52:28.041338Z 2015-11-09T00:44:07.828987Z
2015-11-09 09:55:43 2015-11-09T00:52:28.317332Z 2015-11-09T00:44:25.948623Z
2015-11-09 09:55:32 2015-11-09T00:51:37.838308Z 2015-11-09T00:45:41.403840Z
Thank you.
... View more