Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Monitoring Splunk
×
Are you a member of the Splunk Community?
Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Find Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- Re: how make this query with lot of append faster ...
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
how make this query with lot of append faster and shorter?
ashish2110
New Member
11-13-2019
07:46 AM
source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,".") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"-") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,",") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"_") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"$") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"+") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"!") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"*") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"'") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,"(") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
| append[ search source="s3://cgsumbrella/*" Action=Allowed
Destination!=192.168.9.20.
Destination!=*in-addr.arpa*
Destination!=*fp.measure.office.com*
Destination!=*sprint.com*
Destination!=*drift.com*
Destination!=*aoltw.net*
Destination!=*yahoo.com*
Destination!=*easyhost.com*
Destination!=*aol.com*
Destination!=*igodigital.com*
Destination!=*doubleclick.net*
Destination!=*cpesmsp.apsx*
Destination!=*microsoft.com*
| eval Office=if(len(Office) > 2, Office, "_INTERNET")
| lookup officeip.csv IP as "ExternalIP" output Office
| eval temp=split(Destination,")") | stats count as total by Office InternalIP ExternalIP Destination temp | where (temp!="" and temp!="com" and temp!="net") | stats count as abnormal by Office InternalIP ExternalIP temp | sort -count | where count>1]
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yannK
Splunk Employee
11-13-2019
01:13 PM
Can you make a more generic search, that could handle them all at once, instead of using appends?
It would be more efficient and faster.
Or on recent spunk, if you need to append, take a look at the alternate command union.
instead of "| append" try " | union " to run sub searches in parallel instead of serialized.
search 1 | union [ search 2 ]
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Union
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
wmyersas
Builder
11-13-2019
11:24 AM
I see lots of weirdness in this search - not least of which is evaling Office and then promptly overwriting it with a lookup!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
richgalloway
SplunkTrust
11-13-2019
10:12 AM
What is the search trying to do?
---
If this reply helps you, Karma would be appreciated.
If this reply helps you, Karma would be appreciated.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ashish2110
New Member
11-13-2019
11:10 AM
I am trying to append different tables having same format. The only difference in each append query is the value of delimiter in split function.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
richgalloway
SplunkTrust
11-13-2019
12:05 PM
You appear to be appending the same data many times, but counting it differently.
What problem are you trying to solve with this search? Perhaps there is another way.
---
If this reply helps you, Karma would be appreciated.
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!
Observability Unlocked: Kubernetes Monitoring with Splunk Observability Cloud
Ready to master Kubernetes and cloud monitoring like the pros?Join Splunk’s Growth Engineering team for an ...
Wrapping Up Cybersecurity Awareness Month
October might be wrapping up, but for Splunk Education, cybersecurity awareness never goes out of season. ...
🌟 From Audit Chaos to Clarity: Welcoming Audit Trail v2
🗣 You Spoke, We Listened
Audit Trail v2 wasn’t written in isolation—it was shaped by your voices.
In ...
