topic Re: FIX message protocol with Splunk in Splunk Search

FIX message protocol with Splunk

nathanlhopkins — Sun, 19 May 2013 21:39:58 GMT

Does anyone have any recommendations of how to use Splunk with FIX trading messages logs and in particular is there anything that understand's / translates FIX tags?

Re: FIX message protocol with Splunk

dwaddle — Sun, 19 May 2013 23:07:30 GMT

Some prior responses to the same general question, and there's apparently an app for that:

http://splunk-base.splunk.com/answers/3000/using-delims-to-extract-fix-data
http://splunk-base.splunk.com/answers/887/has-anyone-got-a-method-for-decoding-fix-financial-format-logs
http://splunk-base.splunk.com/apps/22347/financial-information-exchange-fix-log-parsing

Re: FIX message protocol with Splunk

nathanlhopkins — Mon, 20 May 2013 12:56:02 GMT

I've installed FIX Log Parsing by Glenn but am not having much joy:

20/05/2013 10:19:37.826 2013-05-20 10:19:37,826 INFO in.GSFUT_FILCRD - <231 ExecutionReport (8=FIX.4.2\x19=330\x135=8\x149=GSFUT\x156=FILCRD\x1142=FUSNYQAC\x157=A396051\x134=231\x152=20130520-09:19:37\x137=FUSNYQAC15120130516\x111=10301529\x141=10301523\x117=F5193780920130520\x120=0\x1150=4\x139=4\x11=C0795408\x163=0\x155=HCK3\x148=HCEIK3\x122=5\x1167=FUT\x1200=201305\x154=1\x138=13\x140=1\x115=HKD\x159=0\x147=A\x132=0\x131=0\x130=XHKF\x1151=0\x114=0\x16=0\x175=20130516\x160=20130520-09:19:37\x1120=HKD\x121=3\x110=255\x1)

Re: FIX message protocol with Splunk

nathanlhopkins — Mon, 20 May 2013 12:56:19 GMT

Within seach I believe I should just be able to run:

index=test_index Execution* 10:19:37 826 | translatefix

To convert the above into readable tagged format?

Re: FIX message protocol with Splunk

nathanlhopkins — Mon, 20 May 2013 13:46:02 GMT

Found the issue with FIX Log Parser: it turned out to be missing values after the stanza in commands.conf;

[translatefix]
filename = translatefix.py
streaming = true
enableheader = false
retainsevents = true

Re: FIX message protocol with Splunk

anewell — Mon, 20 May 2013 17:36:29 GMT

I'm using translatefix too. To set your expectations: In my experience, the translated fields are not subsequently extracted and indexed. For example, I can search "MsgType=Execution" as a string, but I can't search "MsgType!=Heartbeat" because it's not extracted as a key/value pair. I discussed it with a Splunk Sales Engineer, he had a trick to dump the translated fields back into the raw index (?) but I've lost the notes I took that day (arrrgh!).. I've not had the time or talent to revisit the problem, but I would be grateful for anybody who could.