Is there a way to prioritize inputs?

[pdominicb]

I am about to have a few UFs monitoring some extremely high volume logs. These high volume logs are less critical than some of the current low volume logs we're already monitoring. Its acceptable that the new high volume logs are delayed, but we need the current critical ones in (near) real-time as possible. 

We're already looking at setting maxkbps=0 or increasing concurrent pipelines, but we have concerns on resource consumption. We'd rather not add extra CPUs just for logging. 

So, I am wondering if there is anyway to set some inputs to be a higher priority than others. A few ideas I had are :

1. Use TCPOUT routing and set the maxkbps per destination. But maxkbps is global, so that wont work.
2. Raise concurrent pipelines on the UF and prioritize each pipeline somehow. For example, one pipeline is guaranteed 80% of the load, while another pipeline is only allowed 20% of the load. Then specify the pipeline to use per input. But there doesn't seem to be a way to say one pipeline is prioritized over another. 
3. Install two UFs on the servers. Port conflicts... seems horrible. 

Any ideas here?

[PickleRick]

Generally speaking - no. There is no way to prioritize inputs. And yes, it can have an impact on UFs sometimes. I've had a strange setup with a UF checking huge number of  files from network shares. Every time the UF was restarted it would need about an hour to catch up with the states of all the monitored files. As far as I remember it even lagged ingestion of forwarder's internal events. That was very wrong and luckily has been fixed since. But it shows that you can't prioritize inputs versus each other.

[livehybrid]

Hi @pdominicb 

The only thing that comes to my mind is the maxkbps limits.conf setting which you've mentioned too, and yes this is global therefore I think the only way you could control the limit per input is to run two UF on the same server. This is possible but you would need to update the clashing ports, this shouldnt be too much of a big deal as the UF will only listen on port 8089 (mgmt) plus any input ports configured, so you could set your second UF installation to listen on port 8090 (for example).

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing