I have the same or very similar issue asked about by @token2 in TA-pfsense sourcetyping only catching filterlog on 3-28-2020. As of the posting of this message, there has been no response to their message. Splunk Enterprise Version:8.0.3 Build:a6754d8441bf on host splunk. (This is the Trial Version of Enterprise running on my home network. It is a single host install, no forwarders, separate indexers or anything fancy.) TA-pfsense Version 2.2.1, released. Oct. 29, 2019. pfSense 2.4.5-RELEASE-p1 (host fw) to send all syslog events to splunk on port udp/5140. Splunk has a local data input ingesting pfsense on upd/5140 writing logs to index fw with a sourcetype of "pfsense". Running the query `index=fw | stats count by sourcetype` over all time returned the following sourcetypes: pfsense 1004981 pfsense:dhclient 298 pfsense:filterdns 2240 pfsense:filterlog 564704 pfsense:syslogd 3 However, when running tcpdump on the listening interface, I can see more types of events being sent on the wire: dhcpd, nginx, unbound, etc. When doing a search for these events, 'index=fw dhcpd' or 'index=fw nginx', zero (0) events are returned. So it's not that they are not being sourcetyp-ed incorrectly, but they are being dropped altogether. According to Splunk's transforms.conf documentation for 8.0.4 (note I have 8.0.3), * If the REGEX for a field extraction configuration does not have the
capturing groups specified in the FORMAT, searches that use that
configuration will not return events. This comment does not appear in the transforms.conf documentation for 8.0.3. I don't know if that means this is new behavior, previously undocumented behavior, or something else. Regardless, in order to verify that, I have focused my efforts on troubleshooting the REGEX used by the pfsense_sourcetyper stanza. I've tried using the pfsense_sourcetyper REGEX in transforms.conf from a forked version of TA-pfsense to attempt to get the other events sourcetyp-ed correctly, but this did not work Original (Datapunctum): REGEX = \w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s(?:[\w.]+\s)?(\w+) Forked (Apocrathia): REGEX = ^\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s+(\w+)(\[\d+\])?: I wrote a small python script to validate the regular expressions were capturing the correct data (they are). I hope this post shows that I've tried to do some initial legwork in solving the problem before asking for help. In addition to everything above, I have also done the following: I read this April 2020 post on setting up a dashboard on trenchesofit.com, but it does not address using sourcetypes other than "pfsense:filterlog". I searched the Splunk Community forums for "TA-pfsense"; there were 21 unique posts returned. The most recent & similar to mine I mentioned above (posted by @token2 ). The rest were unhelpful in that they were too old or did not apply to this situation. I've done websearches for troubleshooting TA-pfsense and found a few forked instances of the code on github (where I found the updated REGEX above), but none seemed to work. I've uninstalled TA-pfsense and re-installed it. I've deleted indexes and renamed them. If anyone in the Splunk community has any pointers or can provide any assistance in helping me get TA-pfsense to correctly sourcetype all the events being sent from pfsense, I would be very grateful!
... View more