A forwarder is sending only some of the data, we don't know why?

[robertlynch2020]

Hi

We have a forwarder that is sending partial data. We can identify the files that it is not sending (Image below).

However, when we copy the forwarder and change only the host name, it sends the reminding files that were missing, we don’t delete fish buckets we just restart it and give it a new host name…any ideas?

[monitor:///net/dell552srv.fr.murex.com/dell552srv1/apps/AMBER_PSC47_SEC1.../*.log]

disabled = false

host = TEST_CLUSTER1

index = mxtiming_live

whitelist=mxtiming.*\.log$

blacklist=logs_|fixing_|tps-archives|mxtiming_crv_nr.*|mxtiming_437_dell552srv.fr.murex.com_215699.log

crcSalt = <SOURCE>

sourcetype = MX_TIMING2

props

[MX_TIMING2]

FIELD_DELIMITER = |

DATETIME_CONFIG =

NO_BINARY_CHECK = true

category = Custom

description = MX_TIMING

disabled = false

pulldown_type = true

REPORT-MX-TIMING = REPORT-MX-TIMING2

EXTRACT-MX-TIMING = ^(?:[^\|\n]*\|){6} *-*(?P<Elapsed>\d+\.\d+)\w+\| *-*(?P<CPU>\d+\.\d+)s\| *-*(?P<CPU_PER>\d+)%\|

EXTRACT-MX-TIMING2 = ^(?:[^\|\n]*\|){11} *-*(?P<Elapsed_C>\d+\.\d+)\w+\|

EXTRACT-MX-TIMING3 = ^(?:[^\|\n]*\|){9}  *-*(?P<RDB_COM1>\d+\.\d+)s\| *-*(?P<RDB_COM_PER1>\d+)%\s+\|

EXTRACT-MX-TIMING-Memory = \| *(?P<Memory>\d+\.\d+)Mb(\|\s?(?P<VmHWM>\d+\.\d+)Mb)?(\|\s?(?P<Malloc>\d+\.\d+)Mb)?$

TRANSFORMS-set = setnull, setparsing_mxtiming

Transform

[setparsing_mxtiming]

REGEX = (Deal insertion|contract insertion|Realtime Shutdown|SessionCreate|SessionKill|Read SHM|Read_SHM|Updated keys|Portfolio_Load|Viewer|Publishing Config|simulation|BOS|MPC|MXWAREHOUSE|RequestDocument|LOGIN|event|Bulkportfoliomodification|Bulkunwind|unwind|Event_insertion|Deal_input)

DEST_KEY = queue

FORMAT = indexQueue

[robertlynch2020 · Accepted solution]

HI

We got the answer to this by changes a prop in the forwarder in the end.

We increased a prop in server.conf in the forwarder. From 3 to 6.

[general]
parallelIngestionPipelines = 6

Rob