Truncation of some sourcetypes

[gauthig]

I am getting some strange behaviors for some of the sourcetype transforms. 70% of the events are still showing sourcetype=pfsense. The only 2 that get transformed correctly is pfsense:filterlog, pfsense:dhcpclient.

Also, there seems to be a truncation that occurs that strips out what log the event came from. For example, for all unbound events, here is what happens:

Sent from PFSense as:
Oct 17 19:23:09 unbound: [36942:1] info: validator operate: query semanticlocation-pa.googleapis.com. A IN

Indexed into splunk as (sourcetype if pfsense, not pfsense:unbound):
[36942:1] info: validator operate: query semanticlocation-pa.googleapis.com. A IN

So none of the fields for these are extracted for any sourcetype=pfsense, while sourcetype=filterlog or dhcpclient have all fields extracted properly.

I tested the sourcetyper regex from the transforms.prop on the above raw event and it pulls the correct sourectype "unbound". Does this for nginx and openvpn also.

I verified all files are there including the lookups which must be manually installed now.

PFSense 2.4.4-RELEASE-p3
Splunk Enterprise 7.3.2

[gauthig]

Hi Rich, I am using the stock files and configuration from the publishes TA-pfsense app. I can't attach any files as I do not have enough karma points yet 🙂 . But, the file can be downloaded from the apps or from this github https://github.com/datapunctum/TA-pfsense

This morning I did remove the sed commands (SEDCMD) in the props .conf and it seems to work now . Just not sure what the published app was doing with the two sed lines. Are they needed for something else I am missing?

[richgalloway]

The SEDCMDs appear to be cleaning up some unneeded text in the events. Perhaps your version of PFSense doesn't have that same text. At least you have it working now.

Please accept the answer to help future readers.

[richgalloway]

Please share the inputs.conf settings for that source, including the sourcetype, as well as the props.conf settings for the sourcetype.