Deployment Architecture

universal forwarder ipaddress

cdstealer
Contributor

Hi, I think this has been asked a couple times over the past few years, but no real answer. Obviously the forwarder sends the hostname, but it must also send the ipaddress as this exists in the _internal index under the "sourceIp" field. Is there anyway, lightweight or builtin, to have this as a field? I'm asking this as we have a lot of servers that have generic hostnames which is next to useless when we require to know the location and environment details which we can stipulate from the ipaddress of the host.
Using a static solution like a lookup table et al will not work as servers can be built and torn down at will. If there is no solution, would a feature request be an answer to get the ball rolling?

I know we can use "_meta = ::" in inputs.conf on each forwarder. But this means specifying a static value which may cause problems further down the line.

TIA
Steve

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The forwarders sends phonehome to deployment server (if you're using one) which includes both hostname and ip for the forwarder. I would query that information and save as lookup table (or rather merge it if you want to keep the historical forwarders as well).

Query:

index=_internal  sourcetype=splunkd_access phonehome | rex "\/services\/broker\/phonehome\/connection_(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})_\d+_(?<hostname>[^_]+)" | stats count by hostname ip | fields - count

The phonehome interval is 60 sec by default, so you can go low on the timerange of above search (15 mins should be sufficient). Adjust the timerange if you've changed the phonehome interval to something higher.

To save this as lookup with merge:

1)Create a saved search with following search and run every 15 mins.

index=_internal  sourcetype=splunkd_access phonehome | rex "\/services\/broker\/phonehome\/connection_(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})_\d+_(?<hostname>[^_]+)" | stats count by hostname ip | fields - count | inputlookup host_ip.csv append=t | stats count by hostname ip | fields - count  | outputlookup host_ip.csv

2) Once the search is run once, it will create the lookup host_ip.csv under the same app as the saved search and private to the owner of the saved search. Change it's sharing permission to make it app level/global.
3) Use the lookup in your search to get ip from host (or vice versa)

cdstealer
Contributor

Hi Somesoni2, Thanks for the suggestion, but this will not work in our environment. This needs to be transparent to the end user. I appreciate that we could do an automated lookup, but it will just put the search head under unnecessary load. We have quite a few users that have to run very intense queries that return several hundred thousand events. I just don't see a lookup being a viable solution due to the expense to the search head. Also, not all forwarders are under deployment control 😞

Thanks
Steve

0 Karma

gcusello
SplunkTrust
SplunkTrust

I don't know how many events you have, but You could automatically extract a lookup with hostname and IP Address and use it to show IP Address in your searches: I think that there isn't a large update of this information.
so you could run a scheduled search (every hour or by night) like this

index=_internal | dedup host | table host sourceIp | outputlookup perimeter.csv

obviously you have to create this lookup and definitions.

then you can insert in your searches the lookup

your_search | lookup perimeter.csv host OUTPUTNEW sourceIp | ...

Bye.
Giuseppe

0 Karma

cdstealer
Contributor

Hi Giuseppe, Many thanks for the suggestion. We did look into this before I posted and it's not feasible as we will have quite a few hundred servers feeding into Splunk. The thought of a lookup being queried hundreds of times a second is frightening. There has to be a more elegant solution. We also don't want end users looking into the _internal index.

Thanks
Steve

0 Karma

gcusello
SplunkTrust
SplunkTrust

I agree!!
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...