Why do discarded events on the forwarder keep failing?

[MJA411]

Hello!

I am trying to exclude a specific computer_name from showing up in our carbonblack index in Splunk using a Heavy Forwarder where the data is monitored on. 

Below is an example of my props.conf & transform.conf

props.conf

[source::/var/data/events.json]
TRANSFORMS-null = nullFilter

transform.conf

[nullFilter]
REGEX = (?ms)(.*"computer_name":\s*"test-machine".*)
DEST_KEY = queue
FORMAT = nullQueue

Raw data in Splunk:

{"path":"/usr/sbin/abrt-server","md5":"9F469AA349AA64009C3DB7BE","sha256":"","command_line":"abrt-server -s","parent_path":"/usr/sbin/abrtd","parent_pid":546,"parent_guid":-390649270232,"filtering_known_dlls":false,"parent_md5":"97E3CDA03CB1A8CDF9","expect_followon_w_md5":false,"link_parent":"https://server-name:443/#analyze/00000000-0000-0000-0000-74e9a5a/1","username":"root","parent_create_time":1682147484,"pid":27474,"process_guid":"00000000-0000-0000-0000-0000000e","link_process":"https://server-name:443/#analyze/00000000-0000-0000-0000-99132070551e/0","link_sensor":"https://server-name:443/#/host/518","process_path":"/usr/sbin/abrt-server","cb_server":"server-name","type":"ingress.event.procstart","sensor_id":123,"computer_name":"test-machine","event_type":"proc","timestamp":1686123541}

List format data in Splunk (there are two ways the data is displayed in Splunk but the json file produced the above raw data)

{ [-]
   cb_server: server-name
   command_line: abrt-server -s
   computer_name: test-machine
   event_type: proc
   expect_followon_w_md5: false
   filtering_known_dlls: false
   link_parent: https://server-name:443/#analyze/00000000-0000-0000-0000-74e9a5a/1
   link_process: https://server-name:443/#analyze/00000000-0000-0000-0000-99132070551e/0
   link_sensor: https://sever-name:443/#/host/123
   md5: 9F469AA349AA64009C3DB7BE
   parent_create_time: 1682147484
   parent_guid: -390649270232
   parent_md5: 97E3CDA03CB1A8CDF9
   parent_path: /usr/sbin/abrtd
   parent_pid: 546
   path: /usr/sbin/abrt-server
   pid: 27474
   process_guid: 00000000-0000-0000-0000-99132070551e
   process_path: /usr/sbin/abrt-server
   sensor_id: 123
   sha256:
   timestamp: 1686123541
   type: ingress.event.procstart
   username: root
}

I have tried a few different regex entries but they keep failing. I was using a UF initially then read the Splunk docs and upgraded to a Heavy Forwarder but still keep getting the same problem. Can you please provide any assistance, would be very much appreciated.

My initial aim is to get this working for a single machine then hopefully look to exclude multiple machines sharing a similar naming convention for example "PC123..." grateful if you can provide best way to tackle both scenarios. 

Thanks!

[isoutamo]

Hi

have you try it like

REGEX = (?ms).*"computer_name":\s*"test-machine".*

or even

REGEX = (?ms)"computer_name":\s*"test-machine"

 and if those are not working you could try to escape " as \"

r. Ismo

[MJA411]

I tried both regex as entries but didn't change anything with the data coming into Splunk. I am doing this on the Heavy Forwarder where the data is monitored. Do I need to run this anywhere else? 

I thought it should work on the HF without any further changes elsewhere. 

[isoutamo]

You should run it on HF and after changes you must restart splunkd on HF. No need to change it on IDX as HF is the 1st full splunk instance and it modifying it only there.

Your example is _raw on splunk, but do you have raw on that /var/data/events.json file?

[MJA411]

the /var/data/events.json file produces an identifcal events to the raw data. Splunk then show the data in either "raw" or "list" format as provided above. 

I have restarted splunk on the HF after making the changes by running

./splunk stop

./splunk start

under /opt/splunk/bin

[isoutamo]

Time by time those seems to contains same data, but when you are looking source file there could be some hidden data (like ctrl characters etc.). Those could leads the situation where your regex didn't work. For that reason I always try to use real source file instead of _raw from splunk when creating props + transforms.

[MJA411]

Here is an output from the events.json as per the source in props.conf

{"Protocol":17,"domain":"","direction":"inbound","local_ip":"1.2.3.4","local_port":12345,"remote_ip":"1.2.3.4","remote_port":00,"proxy":false,"proxy_ip":"","proxy_domain":"","md5":"A652C59871009183","pid":10,"process_guid":"0000000e-0000-0000-0000-8e1da","link_process":"https://server-name:443/#analyze/0000003e-0000-0000-0000-8e1da/0","link_sensor":"https://server-name:443/#/host/00","process_path":"/opt/splunk/bin/splunkd","cb_server":"server-name","type":"ingress.event.netconn","sensor_id":00,"computer_name":"test-machine","event_type":"netconn","timestamp":000000000}

The regex works perfectly fine on Regex101.com and match all the events where the computer_name = test-machine. Not sure why the data is still indexed in Splunk. I tried to do a different field for example: event_type="proc" to be excluded but it didn't work either. So there has to be something stopping this from working but I can't seem to get my head around it. I had a few second opinions and they think what I did looks good and it should work. 

[isoutamo]

You are using normal monitor input for it?
Can you share that? 
Also you are sure that this props.conf are in use? You have checked it with splunk btool?

[MJA411]

Here's the inputs.conf monitor part:

[monitor:///var/data/events.json]
sourcetype = bit9:carbonblack:json
index = carbonblack
initCrcLength = 2000

I have confirmed I can see the props line by running the splunk btool so it's visible. 

[isoutamo · Accepted solution]

Last options. What happened when you switch [source::/var/data/events.json] to [bit9:carbonblack:json] on your props.conf?

If it didn't help then I suppose that you still have some precedence issue on your conf files, but it's hard to say without looking those trough with splunk btool <file> list <param> --debug.

[MJA411]

Thanks! 

After running Splunk Btool transforms I noticed the entries are not there. Turned out the transforms.conf was incorrectly named to transform.conf (missing an ‘s’)….

quite a silly mistake but all working as expected now so thanks for your help!