topic Re: How do I find out what heavy forwarder a device is sending logs from? in Getting Data In

How do I find out what heavy forwarder a device is sending logs from?

kcooper — Fri, 18 Jan 2019 11:19:55 GMT

We have a DMZ heavy forwarder (HF) that sends logs from the devices on the DMZ environment to our Splunk server. I need to know the name of the devices that are sending the data through the HF.

How can I get that information?

Re: How do I find out what heavy forwarder a device is sending logs from?

dkeck — Fri, 18 Jan 2019 11:25:24 GMT

HI,

try this query : index=_internal sourcetype=splunkd host=<heavyforwardername> group=per_host_thruput | stats values(series)

Re: How do I find out what heavy forwarder a device is sending logs from?

FrankVl — Mon, 21 Jan 2019 10:42:05 GMT

Your title and question are confusing. What do you want to find out? The name of the HF that was involved in processing a certain event, or the name of the original source host from which the event originated?

What kind of data feed are we talking about here? How are they sending through the HF? What kind of Splunk input is used? In case you're interested in the original source host: what does the data look like, does that contain the original host name?

Re: How do I find out what heavy forwarder a device is sending logs from?

dkeck — Tue, 22 Jan 2019 06:38:12 GMT

Please accept the answer if it helped you 🙂 thank you

Re: How do I find out what heavy forwarder a device is sending logs from?

kcooper — Tue, 22 Jan 2019 13:14:39 GMT

Thank you. This worked great.

Re: How do I find out what heavy forwarder a device is sending logs from?

dkeck — Tue, 22 Jan 2019 13:17:14 GMT

Then please hit accept to award points 🙂 thank you

Re: How do I find out what heavy forwarder a device is sending logs from?

FrankVl — Tue, 29 Sep 2020 22:51:59 GMT

I highly doubt that works great, since the per_x_thruput metrics logs are incomplete. As the docs specify:

Note: The per_x_thruput categories are not complete. Remember that by default metrics.log shows the 10 busiest of each type, for each sampling window. If you have 2000 active forwarders, you cannot expect to see the majority of them in this data. You can adjust the sampling quantity, but this will increase the chattiness of metrics.log and the resulting indexing load and _internal index size. The sampling quantity is adjustable in limits.conf, [metrics] maxseries = num.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Aboutmetricslog

Re: How do I find out what heavy forwarder a device is sending logs from?

dkeck — Tue, 22 Jan 2019 14:06:30 GMT

What would you suggest then?

Re: How do I find out what heavy forwarder a device is sending logs from?

kcooper — Tue, 22 Jan 2019 14:25:59 GMT

I have skimmed the list of devices that was returned and it looks good but I am not sure at this time if all the devices are listed.

What would you suggest?

Re: How do I find out what heavy forwarder a device is sending logs from?

FrankVl — Tue, 22 Jan 2019 14:37:58 GMT

As I mentioned in my original comment to your question: that highly depends on how the data is coming in.

One thing you can do to allow filtering for data coming through a certain HF using file monitor inputs is to put the log files that splunk reads in a folder that is named after the HF's hostname. That way, the HF's name shows up in the source field.

Another solution is to add a custom meta data field, to explicitly label each event with the HF it passed through.

Re: How do I find out what heavy forwarder a device is sending logs from?

kcooper — Tue, 22 Jan 2019 14:41:38 GMT

we can't have the all the log files in one. they are all different devices.
I will look into adding a custom meta data field - that is a good idea
thx

Re: How do I find out what heavy forwarder a device is sending logs from?

FrankVl — Tue, 22 Jan 2019 15:07:06 GMT

It doesn't all have to be in one folder. Just have the HF name somewhere in the path. For example, say currently you have:
/opt/logs/typeA/foo.log
/opt/logs/typeB/bar.log

Just create something like /opt/hf.dmz.com/logs/ as a symbolic link to /opt/logs/ and update splunk inputs accordingly. Which results in source values like:

/opt/hf.dmz.com/logs/typeA/foo.log
/opt/hf.dmz.com/logs/typeB/bar.log