Is there a way in Splunk to do a stat count based on part of the fields result?
We have multiple data center sites, with the big ones having their own OSSEC / Splunk server. All OSSEC results are directly sent to the respective Splunk indexers for that site. There are few smaller sites where it was not worth adding a dedicated indexer, and thus based on the location, we just selected either one of the main indexers and used them for these.
There is a central Splunk search head which is used to query all remote Splunk indexers.
I wanted to find out how many agents are reporting per site, but now it becomes little trick because we cannot rely on "Splunk_server" field. e.g. Site 12 has around 100 servers, but all those are sending logs to Splunk indexer in Site 30.
If there is a way to do a grouping or stat count on part of the ip address, e.g. all servers in Site 12 will have first 3 octets "10.11.12", all in Site 15 will have "10.11.15" and so on... then it might be possible to create a chart truly based on the Site and not the reporting Splunk server.
A simple way would be to chop your field as needed:
... | rex field=your_ip_field "^(?<first_three_octets>\d+\.\d+\.\d+)" | stats count by first_three_octets
More complicated examples could be solved by an | eval category : case(...) | stats count by category.
A more neat way would be to tag your fields with the site name and do a stats count by tag::your_ip_field. Might require a bit more work to tag the values than just going by /24s, but that would allow much easier searching and all that.