Getting Data In

How do I find out what heavy forwarder a device is sending logs from?

Path Finder

We have a DMZ heavy forwarder (HF) that sends logs from the devices on the DMZ environment to our Splunk server. I need to know the name of the devices that are sending the data through the HF.

How can I get that information?

0 Karma

Ultra Champion

Your title and question are confusing. What do you want to find out? The name of the HF that was involved in processing a certain event, or the name of the original source host from which the event originated?

What kind of data feed are we talking about here? How are they sending through the HF? What kind of Splunk input is used? In case you're interested in the original source host: what does the data look like, does that contain the original host name?

0 Karma

Influencer

HI,

try this query : index=_internal sourcetype=splunkd host=<heavyforwardername> group=per_host_thruput | stats values(series)

Influencer

Please accept the answer if it helped you 🙂 thank you

0 Karma

Path Finder

Thank you. This worked great.

0 Karma

Ultra Champion

I highly doubt that works great, since the perxthruput metrics logs are incomplete. As the docs specify:

Note: The per_x_thruput categories are not complete. Remember that by default metrics.log shows the 10 busiest of each type, for each sampling window. If you have 2000 active forwarders, you cannot expect to see the majority of them in this data. You can adjust the sampling quantity, but this will increase the chattiness of metrics.log and the resulting indexing load and _internal index size. The sampling quantity is adjustable in limits.conf, [metrics] maxseries = num.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Aboutmetricslog

0 Karma

Path Finder

I have skimmed the list of devices that was returned and it looks good but I am not sure at this time if all the devices are listed.

What would you suggest?

0 Karma

Ultra Champion

As I mentioned in my original comment to your question: that highly depends on how the data is coming in.

One thing you can do to allow filtering for data coming through a certain HF using file monitor inputs is to put the log files that splunk reads in a folder that is named after the HF's hostname. That way, the HF's name shows up in the source field.

Another solution is to add a custom meta data field, to explicitly label each event with the HF it passed through.

0 Karma

Path Finder

we can't have the all the log files in one. they are all different devices.
I will look into adding a custom meta data field - that is a good idea
thx

0 Karma

Ultra Champion

It doesn't all have to be in one folder. Just have the HF name somewhere in the path. For example, say currently you have:
/opt/logs/typeA/foo.log
/opt/logs/typeB/bar.log

Just create something like /opt/hf.dmz.com/logs/ as a symbolic link to /opt/logs/ and update splunk inputs accordingly. Which results in source values like:

/opt/hf.dmz.com/logs/typeA/foo.log
/opt/hf.dmz.com/logs/typeB/bar.log

0 Karma

Influencer

What would you suggest then?

0 Karma

Influencer

Then please hit accept to award points 🙂 thank you

0 Karma