Deployment Architecture

Summary index does not meet my needs

xsstest
Communicator

Hi.
everyone.Please forgive my English level.I hope my description of the problem is clear enough.I have a lot of indexes to store different types of events, for example :
index=nginxindex=firewallindex=apacheindex=vpnindex=waf

Each index has a very large amount of log.Some indexes have 10 million logs a day.
For different indexes. I created a lot of alert.
For the Apache and nginx logs, I created the Web Attack Alert. for the firewall logs , I created the Port Scan alert. etc..
When trigger alarm. It will output results.
I hope it can output the results into a new index (index=attack_info).
so , I used the summary index to solve the problem. But I found that summary index has a lot of bug。

1、There are a lot of duplicate events in summary index,I don't know why, the summary index will have duplicate events, and I'll make sure that the original event is not repeated ,(For example, there are no duplicate events in indexes such as nginx and Apache)

2、Sometimes, after the alarm is triggered, it doesn't necessarily write exactly to summary index (I'm sure the result is not empty)。That is to say, sometimes it doesn't write the results to summary index

3、The third question is a silly question. I'd like to ask if the summary index is for the report or for the alert? In Web UI - >settings - >Searches, reports, and alert, I choose an alert, and then enable summary indexing。Then I clicked on "Alerts" on the navigation bar,
that alert has disappeared, but that alert appears in “Reports”. why ? Why does it transfer from alerts to reports when I enable the summary index?

ok...So there are a lot of flaws in the summary index, and I don't think it can meet my needs.

I hope that the results of each triggered alert can be output to a new index( for example: index=attack_info). Then I can use index=attack_info to search for all the attack results.Because I need to create monthly report every month,monthly report will show the last month of attack information.So, I can't search from every index (index=nginx or index=apache or index=firewall),Because their log volume is very large, so the search is very, very very very very slow.

If there is two alerts:

index=nginx sourcetype=nginx_access| eval result=case(match(uri,"Jmx-console"),"Jboss Attack",match(uri,"/manager/html"),"Tomcat attack",match(uri,"_memberAccess"),"Struts2 Attack") |where result!=""|table _time scan_ip uri user_agent status_code result

index=firewall "Port scan attack:drop!" |table _time attack_ip dst_ip dst_port

then at the zero of every day,Use the above search statement to filter the events of the previous day.Then it writes the result to the "attack_info" index.

Well, is there a better way to meet my needs, whether using a report or an alert?.

If you can tell me every step, I will thank you very much.Because I'm a Splunk newbie

Tags (1)
0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Ill answer some of these in parts...

1、There are a lot of duplicate events in summary index,I don't know why, the summary index will have duplicate events, and I'll make sure that the original event is not repeated ,(For example, there are no duplicate events in indexes such as nginx and Apache)

SI (Summary Indexes) are populated based on scheduled searches or searches with the collect command. If you have not scheduled your searches that create the summary indexes properly, then you're going to get duplicated events.. E.g., if you have a schedule search that runs every hour searching earliest of -1d, then youre going to have huge overlapping of events. I'd check that first.

2、Sometimes, after the alarm is triggered, it doesn't necessarily write exactly to summary index (I'm sure the result is not empty)。That is to say, sometimes it doesn't write the results to summary index

This could be the way you have the alert setup, if there are no results, it wont write the events... Hard to answer this.

3、The third question is a silly question. I'd like to ask if the summary index is for the report or for the alert? In Web UI - >settings - >Searches, reports, and alert, I choose an alert, and then enable summary indexing。Then I clicked on "Alerts" on the navigation bar,
that alert has disappeared, but that alert appears in “Reports”. why ? Why does it transfer from alerts to reports when I enable the summary index?

That enable option means write to summary index... Im not sure I understand the context full of this... Alerts will age out. But this being said, a SI can be used for reporting or alerts. Some customers aggregate data into SI, then run alerts against it. Additionally, ive seen customers create reports from SI, because its much much faster..

For your use case, you can most definitely use alerts to write events out to a summary index, and then report and alert off of that. You may want to look at the SI related commands such as collect (http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Collect) You can use this in your base searches to populate your SI indexes..

There are some downsides to using summary indexing. Mainly being, sourcetype changes (to stash) and also that SI is reliant on your saved searches to run and populate. This means if you miss a scheduled search, the SI wont populate and you need to back fill for that missing time window. This could lead to duplicate events.

This is where Datamodels have come in.. They automatically backfill jobs, and you can aggregate data as neccesary..

Hope that helps.

0 Karma

xsstest
Communicator

Can anyone answer this question?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...