Depending on what the forwarder is doing, it can take some time to stop/restart. This is typically due to currently active file monitors and modular inputs running. The forwarder will try and clear the queues before it restarts, meaning it will try and wait till and EOF / end of event before stopping the splunkd process. Similar is true on indexers, they will try and wait till search jobs have completed and indexing queues have emptied.
can you elaborate?
what is the reason for frequent restarts? can you see the console while restarting? any messages during restart?
how do you measure the slowness? how long does a forwarder restart takes? indexer? search head?