About MichaelM1

MichaelM1 · ‎03-24-2025

This IF is NOT a heavy forwarder. It is only a light forwarder and does not do any indexing. I added "force_local_processing=true" to my props.conf and I am finally making progress!! Now the logs that originate from the IF are tagging with the IntermediateForwarder,GUIDe, Project_ID and the debug1 This is exactly as expected now that the force_local_processing is forcing it to actually use the props and transforms. (it also has debug1=debugonIF) Oddly the logs that the IF is forwarding are still NOT tagged with IntermediateForwarder and debug1 tags that I expect them to be. The logs do have the GUIDe, Project_ID tags since that it configured on the UF clients. These are my latest configs. == props.conf == [default] TRANSFORMS-setCustomMetadata=setCustomMetadata force_local_processing=true [host:: <IFhostname>] TRANSFORMS-setCustomMetadata=setThisHostMetadata == transforms.conf == [setCustomMetadata] INGEST_EVAL = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), debug1:=COALESCE(debug1,"testdebug") [setThisHostMetadata] INGEST_EVAL = = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), GUIDe:=COALESCE(GUIDe,"123456"), Project_ID:=COALESCE(ProjectID,"54321"), debug1:=COALESCE(debug1,"debugonIF")

MichaelM1 · ‎03-24-2025

@livehybrid This is definitely an intermediate forwarder that I want to setup. And it is forwarding logs from many UFs that I have point to it. The UF are tagging their logs just fine from the UF configs that they have. It is just that the IF itself is not tagging its own logs. @PickleRick This intermediate forward is only receiving data UFs and I don’t have any HF in my architecture @livehybrid I don’t think that the transforms are working at all. This is my current config for props and transforms and I have simplified as you and @isoutamo suggested == props.conf == [default] TRANSFORMS-setCustomMetadata=setCustomMetadata [host::macdev] TRANSFORMS-setCustomMetadata=setThisHostMetadata == transforms.conf == [setCustomMetadata] INGEST_EVAL = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), debug1:=COALESCE(debug1,"testdebug") [setThisHostMetadata] INGEST_EVAL = = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), GUIDe:=COALESCE(GUIDe,"123456"), ProjectID:=COALESCE(ProjectID,"54321"), debug1:=COALESCE(debug1,"debugonIF") I get this warning in the splunkd.log that could be related WARN TcpOutputProc [19444 parsing] - Pipeline data does not have indexKey. [_path] = e:\program files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe\n[_stmid]blah\n[Metadata:Source] = source::WinEventLog\n[MetaData:Host] = host::<IFhostname>\n[MetaData:Sourcetype] = sourcetype::WinEvtLog\n[_done] = _done\n[_conf] = source::WinEventLog:host::< IFhostname>:WinEventLog|\n This this config I would expect that the logs forwarding though from the UFs should get the debug1 but it does not. I don’t know why it does not seem to be doing anything at all Thank you for all your help btw

MichaelM1 · ‎03-21-2025

Thank you for the response. I have been testing this all day so far today and to be honest nothing I do with the props and transforms is having any effect. I do know that the IF is sending logs because if I add /etc/system/local/inputs.conf [WinEventLog] _meta = GUIDe::123456 Project_ID::654321 the logs get tagged and indexed. This does not work for me because it will double tag the logs since the client is adding its proper project and guide tags and then the IF add another project and guide tag. Nothing I do in the props and transforms seems to change the fact that the logs are not being tagged from the IF itself the logs still get properly forwarders and indexed since they were already tagged by the client UF. I really don't know why the transform is not working. Even if I do this it still has no effect == props.conf == [default] TRANSFORMS-setCustomMetadata=setThisHostMetadata == transforms.conf == [setThisHostMetadata] INGEST_EVAL = GUIDe:=COALESCE(GUIDe,"999-999-999"), ProjectID:=COALESCE(ProjectID,"MyForwarderLayer")

MichaelM1 · ‎03-20-2025

let me try that again my regex was missing a ! I am also tring to remove the :: as I dont think it is necessary [addprojectid] REGEX = ^(?!.*Project_ID) FORMAT = Project_ID::123456 MV_ADD = true SOURCE_KEY=_meta [addGUIDe] REGEX = ^(?!.*GUIDe) FORMAT = GUIDe::654321 MV_ADD = true SOURCE_KEY=_meta [addIntermediateForwarder] REGEX = .* FORMAT = IntermediateForwarder::XXXXXX MV_ADD = false SOURCE_KEY=_meta

MichaelM1 · ‎03-20-2025

You are correct it was just a post typo I added the SOURCE_KEY=_meta to my transforms and it still is not working as expected. The logs that from the UFs still get forwarded by the IF but he IF its self is no longer tagging its own logs and therefor getting rejected by the indexers. This is my transforms.conf now [addprojectid] REGEX = ^(?.*Project_ID::) FORMAT = Project_ID::123456 MV_ADD = true SOURCE_KEY=_meta [addGUIDe] REGEX = ^(?.*GUIDe::) FORMAT = GUIDe::654321 MV_ADD = true SOURCE_KEY=_meta [addIntermediateForwarder] REGEX = .* FORMAT = IntermediateForwarder::XXXXXX MV_ADD = false SOURCE_KEY=_meta

MichaelM1 · ‎03-20-2025

I have a configuration where I have an intermediate forward that is forwarding logs to central indexer that I do not control. In order to send logs to the indexer they MUST be tagged with a Project_ID and GUIDe in the _META tag otherwise the logs will get rejected. I would like to set up a multi-tenant like configuration where the intermediate forwarder is used by several projects that have different Project_ID and GUIDe. The universal forward configuration that I am sending to the clients have the tagging and will be sending their unique project and guide numbers. The problem that I am having is that the intermediate forwarder (windows) itself is not tagging its own logs when it sends to the indexers. The clients tag and send their logs just fine. If I apply the same tagging as the clients on the on the intermediate forwarder it will tag the logs twice or overwrite the tags. What I want to do is tag only the intermediate forward logs or any logs that are not already tagged. This is what I tried, and it is not working. I was attempting to use regex to only add the tags to logs that are not already tagged using this filter ^(?.*Project_ID::) but this is not working. Any help is appreciated. Intermediate forwarder: etc/apps/projtransforms/local/props.conf [default] TRANSFORMS-projectid = addprojectid TRANSFORMS-IntermediateForwarder = addIntermediateForwarder TRANSFORMS-GUIDe = addGUIDe etc/apps/projtransforms/transforms.conf [addprojectid] REGEX = ^(?.*Project_ID::) FROMAT = Project_ID::123456 MV_ADD = true [addGUIDe] REGEX = ^(?.*GUIDe::) FROMAT = GUIDe::654321 MV_ADD = true [addIntermediateForwarder] REGEX = .* FORMAT = IntermediateForwarder::XXXXXX MV_ADD = false UF Client tagging /etc/system/local/inputs.conf [default] _meta = GUIDe::654321 Project_ID::123456 disabled = 0 [WinEventLog] _meta = GUIDe::654321 Project_ID::123456 disabled = 0 [perfmon] _meta = GUIDe::654321 Project_ID::123456 disabled = 0 index = spl_win

MichaelM1 · ‎02-20-2025

Thanks for the help on this. The final solution for me is the [general] parallelIngestionPipelines = 200 I am not sure I see the benefit of taking the time to find the optimal size for the various Queues a you suggest. I have the available CPU and memory to simply increase the pipelines. I will be adding several IFs and allow them to load balance and ultimately 200 will be way over kill and I may drop this back to something like 50 (or maybe I will not bother with this either 🙂 since everything is working)

MichaelM1 · ‎02-14-2025

I was trying to play around with the [general] pipelineSetAutoScale [queue] autoAdjustQueue maxSize and nothing had an effect Then after some more researching more I added this line to the server.conf [general] parallelIngestionPipelines = 200 The server immediately allowed over 7000 connections and consumed all 32gb of memory and used a huge amount of networking on my IF for a few minutes before settling to about 6800 connections and 22 -30GB of memory. It is currently receiving about 1.5Gbps and sending ~100Mbps. Presumably this is burning off the backlog of all the logs that it was not able to forward before. I am up to 3200 devices reporting and increasing. I am hesitant to say this is fixed and I would like to know if there are any long term issues with keeping the parallelIngestionPipelines = 200. It is also unclear why a single pipeline is limited to 1000 connections as I have not seen that document anywhere.

MichaelM1 · ‎02-13-2025

I set my client test script to connect to 9997 on the IF server that is the and currently has ~1000 client connected to it, and it immediately fails to establish any more connections. I have since built a new server that is using the exact same IF configurations and my client script connect to 9997 up to 16000 connections as expected. I don’t understand this, I guess the splunk process with the 1000 connection knows there is data traversing and that it is too busy to accept more connections. In the metrics.log I see this ever minute: <date time> Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=1217, largest_size=1217,smallest_size=0 I don’t know what this mean or if it is significant and I will start researching this. I also am not sure what you mean by adding pipeline so I will look into that too.

MichaelM1 · ‎02-10-2025

Yes there is at least one firewall between the client network and the Intermediate forward network. I did a quick and dirty test like you did by making a powershell script that ran on the client subnet and simply opened as many connection to the IF as it could. I created a corresponding server script to listen on a port. As expected the server maxed out at 16000 connections. This confirms that there is not a networking device between the client network and the IF network that would limit the total number of connections. The inputs and outputs that you have are effective the same as what I have. I am not doing anything special with them and it is just about as basic as it comes. The next hop from the IF to the indexers needs to go through a NAT as my IF is a private address and the indexers are public. I don't suspect that the IF server would not allow more that 1k connections if the upstream is limiting the connections but I don't have a easy way to verify this. I don't control the indexers and so I cant do a similar end to end connection test with a lot of port. I am still scratching my head on this and like I said I am not satisfied with the suggestion of just building more IF servers and limiting them to 1k clients each.

MichaelM1 · ‎02-05-2025

I have an installation where I am trying to leverage an intermediate forwarder (IF) to send logs to my indexers. I have approximately 3000 Universal Forwarders (UFs) that I want to send through the IF, but something is limiting the IF to around 1000 connections. The IF is a Windows Server 2019. I am monitoring the connections with this PowerShell command: netstat -an | findstr 9997 | measure | select count. I never see more than ~1000 connections, even though I have several thousand UFs configured to connect to this IF. I have already tried increasing the max user ports, but there was no change: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay I have validated the network by creating a simple client and server to test the maximum connections. It reached the expected maximum of 16,000 connections from the client network to the IF. I can also configure a server to listen on port 9997 and see several thousand clients trying to connect to the port. I believe there must be something wrong with the Splunk IF configuration, but I am at a loss as to what it could be. There are no limits.conf configurations, and the setup is generally very basic. My official Splunk support is advising me to build more IFs and limit the clients to less than 1000, which I consider a suboptimal solution. Everything I’ve read indicates that an IF should be capable of handling several thousand UFs. Any help would be greatly appreciated.

Posts	11
Solutions	0
Karma Given	4
Karma Received	0
Member Since	‎02-05-2025

Online Status	Offline
Date Last Visited	‎03-24-2025 04:26 PM

Conditional tagging on an intermediate forwarder

Intermediate Forwarder Limited to 1000 connections

Re: Conditional tagging on an intermediate forward...

Re: Conditional tagging on an intermediate forward...

Re: Conditional tagging on an intermediate forward...

Re: Conditional tagging on an intermediate forward...

Re: Conditional tagging on an intermediate forward...

Conditional tagging on an intermediate forwarder

Re: Intermediate Forwarder Limited to 1000 connect...

Re: Intermediate Forwarder Limited to 1000 connect...

Re: Intermediate Forwarder Limited to 1000 connect...

Re: Intermediate Forwarder Limited to 1000 connect...

Intermediate Forwarder Limited to 1000 connections

Join the Conversation