Splunk Add-on for IPFIX: Why is ipfix_collector.py using 99.5% CPU and 70% Memory on our heavy forwarder? Getting "ParseError="Template not known (yet).""

[season88481]

Hi guys,

We have the Splunk Add-on for IPFIX installed on one of our Heavy Forwarders.

I got noticed that one of the Python scripts is causing a daily crash of that HWF host.

-Path of the .py script: /opt/splunk/splunk/etc/apps/Splunk_TA_IPFIX_UDP_NIX/bin/ipfix_collector.py

-Checked splunkd.log in the heavy-weight forwarder, could not find any information related to ipfix_collector.py.

-Checked appflow.log, log had stopped for more than 20h. Also find some error like this:

TimeStamp="2016-03-02T20:35:33"; Template="265"; Observer="0"; Address=""; Port="<>"; ParseError="Template not known (yet).";

-Checked debug.log, it is full of

Have not implemented parsing for 'None' of length 8 (5951:319) which is needed for template 284.

-Ping the Netscaler, PING OK

-Restarted host and Splunk heavy-weight forwarder, still with no luck.

Has anyone seen this before?

Any advice will be much appreciated!

Thank you very much in advance.

Cheers,
Vincent

[jcoates_splunk · Accepted solution]

hi, you're getting this error because you need to add enterprise information elements for the device you're trying to parse data from.

http://docs.splunk.com/Documentation/AddOns/latest/IPFIX/ConfigureEnterpriseInformationElements

[season88481]

Hi Jcoates,

Thanks for your response. Likely we have not configure template for this Citrix device yet. That partially answers my question.

However,

We have another HWF which is receiving template correctly, but the ipfix_collector.py script still progressively growing in memory usage(Same memory issue as my earlier comment. It used 100% CPU, and memory grow gradually...). Is this expected for this python script?

[dailv1808]

Hi anyone there.
So how to configure template for Citrix device?

[season88481]

Each time restarting Splunk. That python script stop for about 10 mins. then it will starting running again and use 100% of CPU. The memory also keep increasing from 0% to 70%. Not long after, the host will crash.

I had confirmed there is enough HDD space for running the script.

But why the python script use all CPU and the memory ever increasing and eventually crash the host?

Could it be ipfix_collector.py script doesn't have permission to write into the disk?