I have created a search in order to:
Pull traffic log from datamodel "DM_1"
Use src_ip and dest_ip as token to pass map search in a different index= with src=$src_ip$ , answer=$dest_ip$ so that I can pull the domain (URL) name for dest_ip .
Then, use eval within the map search to pass in values for all the fields from first search that don't exist in 2nd search.
Final results would include all the fields from 1st search and domain field from 2nd search, with map command.
| tstats count from datamodel=
by log.src_ip log.dest_ip log.transport log.app log.dest_port log.src_zone log.flags log.log_subtype _time log.session_end_reason log.rule log.action log.packets_in log.packets_out
| rename log.* AS *
| map search="search index= answer=$dest_ip$ src=$src_ip$ | eval dest_ip=$dest_ip$ | eval dest_port=$dest_port$ | eval rule=$rule$ | eval map.transport=$transport$ | eval session_end_reason=$session_end_reason$ | eval action=$action$ | eval app=$app$ | eval flags=$flags$ | eval log_subtype=$log_subtype$ | eval packets_in=$packets_in$ | eval packets_out=$packets_out$ | eval time=$_time$"
| eval time = strftime(_time, "%m/%d/%Y:%H:%M:%S")
| stats values(domain) as domain, values(dest_ip) as dest_ip, values(dest_port) as dest_port, values(rule) as rule, values(packets_out) as packets_out, values(packets_in) as packets_in, values(session_end_reason) as session_end_reason, values(time) as time, values(action) as action, values(flags) as flags, values(log_subtype) as log_subtype, values(transport) as transport, values(app) as app by src_ip
The problem I run into is only some of the fields from the first search that do not exist in the 2nd search index ( index_2 ) would return the value ( packets_out , packets_in , session_end_reason , rule , dest_port ) while others don't return any value ( flags , app , action , map.transport ).
From the job log, I could see all those values were parsed out with correct values:
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval time='_time'
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval packets_out=8
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval packets_in=5
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval log_subtype=deny
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval flags=nat <<<<< value was parsed out correctly but was missing from output
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval app=ssl <<<<< value was parsed out correctly but was missing from output
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval action=allowed
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval session_end_reason="policy-deny"
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval "map.transport"=tcp
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval rule="PERMIT-WEB"
06-02-2020 16:31:26.054 INFO SearchParser - PARSING: | eval dest_port=443
06-02-2020 16:31:26.055 INFO SearchParser - PARSING: | eval dest_ip="y.y.y.y"
06-02-2020 16:31:26.055 INFO SearchParser - PARSING: | search (index=index_2 earliest=05/28/2020:13:32:00 latest=05/28/2020:13:33:00 answer="y.y.y.y" src="x.x.x.x")
However, at the end it has some waring error to say some of the fields were missing value from the results:
06-02-2020 16:31:27.151 WARN StatsProcessor - Specified field(s) missing from results: 'action', 'app', 'flags', 'log_subtype', 'transport' ...snip...
I don't see the different between missing fields with others and wonder why there is such inconsistent output.
... View more
Thanks for your suggestion. Yes, regex is not that complicated but the caveat here is Splunk is only evaluate regex for 1 field only. In my case I want to match traffic from either "source_ip" field OR "destination_ip" field on the same query and I can not do with regex in 1 query.
I have to use 2 different queries with regex: 1 query with regex to match source_ip and 1 query with regex to match destination_ip only.
... View more
I am trying to match IPs from discontiguous mask as follow:
1st octet: Match exactly 10
2nd octet: Match any from range 0-255
3rd octet: Match range 32-63
4th octet: Match range 192-255
A couple solutions I found while searching are:
1. Regex: however, regex is used to match 1 field only (either source_ip or destination_ip), not both at the same time. I'd like to be able to match any traffic with source_ip OR destination_ip within the range.
2. cidrmatch to span across multiple CIDR ranges: This would be a long cidrs list in this case.
I wonder if there is any efficient way to do the match in this case.
... View more
Thank you all for your recommendations. I've been able to do "savedsearch <24K query> | search " to bypass browser limitation and get Splunk to process the search successfully per somesoni2 suggestion.
... View more
I have a long Splunk search that I continue to add more conditions to each day so it keeps growing. Eventually, when my search starts to reach 25K long, it starts to fail. The browsers (both Firefox and Chrome) get disconnected from Splunk search head and fails every time I run the search. My question is whether any limitation in the search length that I can run on Splunk, and what is the limitation if any?
If there is no limitation, then I wonder if this is relating to browser itself, though I've been trying both Chrome and Firefox and they both fails with the same condition above.
... View more