topic Catalogue Log/Data inputs details for each index in Getting Data In

Catalogue Log/Data inputs details for each index

hazardoom — Tue, 05 Mar 2024 13:40:38 GMT

Hi,

We have around 340 indexes and I need to know which universal/heavy forwarder forwards data to which exact index. How can I do that?

Thanks,

Re: Catalogue Log/Data inputs details for each index

gcusello — Tue, 05 Mar 2024 13:52:30 GMT

Hi @hazardoom,

at first 340 indexes are very many, so I hint to redesign your indexes structure to reduce them.

Anyway, if you want to know the hosts in each index, youcan run something like this:

| tstats count values(host) AS host WHERE index=* BY index

If instead you want the heavy Forwarders ,it's more difficoult because , for now, the passing through HFs aren't recorded in the events, I asked to Splunk Ideas to have this feature and it's under development.

Anyway, for the moment, you should create a field at index time in each HF and use it in the search, but it's very long to describe, see at https://docs.splunk.com/Documentation/Splunk/9.2.0/Data/Configureindex-timefieldextraction

Ciao.

Giuseppe

Re: Catalogue Log/Data inputs details for each index

hazardoom — Tue, 05 Mar 2024 15:14:02 GMT

Hi Giuseppe,

Thanks for the fast response. Is it possible if I can recreate the search from the monitoring console for forwarder instance and use it somehow to connect it to each index?

`dmc_get_forwarder_tcpin` hostname=* | eval source_uri = hostname.":".sourcePort | eval dest_uri = host.":".destPort | eval connection = source_uri."->".dest_uri | stats values(fwdType) as fwdType, values(sourceIp) as sourceIp, latest(version) as version, values(os) as os, values(arch) as arch, dc(dest_uri) as dest_count, dc(connection) as connection_count, avg(tcp_KBps) as avg_tcp_kbps, avg(tcp_eps) as avg_tcp_eps by hostname, guid | eval avg_tcp_kbps = round(avg_tcp_kbps, 2) | eval avg_tcp_eps = round(avg_tcp_eps, 2) | `dmc_rename_forwarder_type(fwdType)` | rename hostname as Instance, fwdType as "Forwarder Type", sourceIp as IP, version as "Splunk Version", os as OS, arch as Architecture, guid as GUID, dest_count as "Receiver Count", connection_count as "Connection Count", avg_tcp_kbps as "Average KB/s", avg_tcp_eps as "Average Events/s"

I really need this information for each forwarder as from the query. The issue I see is that it searches dmc_get_forwarder_tcpin which is equal to index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) fwdType=* guid=* and I cannot find the indexes there. How can i connect it to each index?

Re: Catalogue Log/Data inputs details for each index

gcusello — Tue, 05 Mar 2024 15:17:20 GMT

Hi @hazardoom,

this search gives you different information, what do you reaaly need? tho know the hosts that are sending in each index?

if this is your requirement, you can use my previous search.

Ciao.

Giuseppe

Re: Catalogue Log/Data inputs details for each index

hazardoom — Tue, 05 Mar 2024 15:44:18 GMT

it will be best if I have all the info that is in the forwarder query - the type of forwarder, the average KB/s, the os, the IP, the splunk version but the index as well as we'd like to create a detailed report which will be in help when moving to cloud.