topic Re: Map command subsearch results loosing initial search value from the output in Splunk Search

Map command subsearch results losing initial search value from the output?

sureshtskumar — Fri, 06 Jan 2023 04:09:31 GMT

Here is an example of SPL I am trying to run.

I am getting results, however the ProxyUser1 field is empty. The initial searched value of ProxyUser has been eval'd to a new field named ProxyUser1, within the map command. I have read some other posts where the eval command after the map search should do the trick, but I believe I am doing something wrong here

Any leads would be much appreciated

Re: Map command subsearch results loosing initial search value from the output

gcusello — Fri, 30 Dec 2022 10:17:32 GMT

Hi @sureshtskumar,

why don't you use the first part in a subsearch?

somethng like this:

see my approach and adapt it to your need.

Ciao.

Giuseppe

Re: Map command subsearch results loosing initial search value from the output

sureshtskumar — Fri, 30 Dec 2022 10:24:44 GMT

Hi @gcusello ThankYou so much for the response. Just to give you a little more insight, the ProxyUser parameter values are obtained from the first search in a network index (although I am just using makeresults to simulate those users). This list of users from the network index is then passed on to an EDR index to get the hostname these users use. So ideally, this should look as:

index=proxy domain="somebad.com" earliest=-24h
| stats values(ProxyUser) as ProxyUser
| map maxsearches=0 search="search index=edrlogs SubjectName=*$ProxyUser$ earliest=-24h | eval ProxyUser1=$ProxyUser$"
| fillnull value="N/A"
| table _time SubjectName EndpointName IPAddress ProxyUser1

So the values _time, SubjectName, EndpointName and IPAddress are all coming from EDR logs and I want to retain the original username from the first proxylogs index, which is ProxyUser1. This is where the values are getting empty. Hope this clarifies

Re: Map command subsearch results loosing initial search value from the output

gcusello — Fri, 30 Dec 2022 10:32:33 GMT

Hi @sureshtskumar,

if you need to extract values from two indexes correlating events from both the indexes, you can use the transaction command, but I use this solution only when I haven't any other choice because it's very slow.

Please try this different approach using stats:

(index=proxy domain="somebad.com") OR (index=edrlogs) earliest=-24h | rename ProxyUser AS SubjectName | stats earliest(_time) AS _time values(EndpointName) AS EndpointName values(IPAddress) AS IPAddress dc(index) AS index_count BY SubjectName | where index_count>1

in this way you have the fields of only ProxyUsers that are in both the indexes.

Ciao.

Giuseppe

Re: Map command subsearch results loosing initial search value from the output

sureshtskumar — Fri, 30 Dec 2022 10:40:54 GMT

Hi @gcusello thanks again for your kind response.

That search query is very expensive and my search times out as the edr index is humongous. So this couldnot be tested in the actual environment.

I would rather need a search where the usernames are restricted to those who visited the somebad.com domain as seen in the proxy logs, then pipe it to the EDR logs. The very reason why I am using the map command is that I can add more values from the first search in the proxy logs, such as the earliest/latest time this traffic was seen. This would help in pinpointing to the user especially when a shared machine is being investigated.

Re: Map command subsearch results loosing initial search value from the output

gcusello — Fri, 30 Dec 2022 10:43:12 GMT

Hi @sureshtskumar,

add these additional conditions to the main search to limit results.

I'd like that you understand the approach and adapt it to your needs.

ciao.

Giuseppe

Re: Map command subsearch results loosing initial search value from the output

sureshtskumar — Fri, 30 Dec 2022 11:01:01 GMT

Hi @gcusello Thanks for the reply. Unfortunately that search doesn't return any hits. I will keep troubleshooting and let you know if I find where I am messing up

(index=proxy domain="somebad.com") OR (index=edrlogs) earliest=-24h
| rename ProxyUser AS SubjectName
| stats
earliest(_time) AS _time
values(EndpointName) AS EndpointName
values(IPAddress) AS IPAddress
dc(index) AS index_count
BY SubjectName
| where index_count>1

Re: Map command subsearch results loosing initial search value from the output

gcusello — Fri, 30 Dec 2022 11:56:55 GMT

Hi @sureshtskumar,

you should anakyze the correlation key to understand if there are matching values in the two fields.

If not, you have to find a common part of those fields.

Ciao.

Giuseppe

Re: Map command subsearch results loosing initial search value from the output

sureshtskumar — Fri, 30 Dec 2022 12:02:26 GMT

Hi @gcusello @Will get my head around the suggested query. Meanwhile if you can find out what is breaking my original map command, especially why is the value from the first search not retained in the final output of the second search, it would be really helpful. Have seen others raising similar question with the map command and so far haven’t seems any good reason as to why this happens

Re: Map command subsearch results loosing initial search value from the output

sureshtskumar — Thu, 05 Jan 2023 03:23:34 GMT

This is finally found to be a potential bug or atleast a limitation in the map command. I went ahead and performed some very generic searches in Splunk using the map command to pass three types of data to be searched in the map subsearch; text, numbers, alpanumeric with special characters. The results of the test shows that any values containing pure text strings (no numbers, no special characters) that are passed on to the map command cannot be recovered in the search results.

Here is the proof:

Query Used for test:

| makeresults
| eval field1="TextString", field2="12345", field3="user@12345mail.com"
| table field1, field2, field3
| map maxsearches=300 search="search index=_internal ($field1$ OR $field2$ OR $field3$) earliest=-2h | eval TextString=$field1$, Number=$field2$, Alphanum=$field3$"
| table _time index TextString Number Alphanum

Results:

Notice the TextString field is empty while the number and alphanumeric and special character values are all retained in the output.

I will try and bring this to Splunk's attention, however not sure if this is going to be fixed or left as is. Thanks to all who took the time to read through and helped with suggestions

Re: Map command subsearch results loosing initial search value from the output

gcusello — Thu, 05 Jan 2023 08:25:44 GMT

Hi @sureshtskumar,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Map command subsearch results loosing initial search value from the output

PickleRick — Thu, 05 Jan 2023 09:09:15 GMT

Firstly - ughhhh... Not only you use map which should not be used unless it absolutely cannot be avoided. And it should not be run with big datasets.

If you run your search built properly, offloading as much processing as possible to indexers, you're splitting your work across your whole deployment. If you do "tricks" like subsearches and map, you spawn multiple searches, bounce the data back and forth between search heads and indexers and generally make it as inefficient as can be 😉

And additionally, in your original search you use wildcard at the beginning of your match term. No wonder it's taking forever to finish since it has to scan all events!

But secondly - in your search spawned from the map command the token is effectively substituted "directly" with given string. So you're getting something like

| eval resultfield=TextString

Which most probably is not what you need because you're trying to set value of resultfield to a value of a field called TextString instead of TextString itself.

So you need to do

| eval resultfield=\"$field1$\"

And this does work.

Re: Map command subsearch results loosing initial search value from the output

sureshtskumar — Thu, 05 Jan 2023 11:36:45 GMT

Thanks @PickleRick for taking the time and effort to respond.

I am personally not a fan of map command and am using it as it appeared to be the best option to pass multiple parameters from the main search to multiple other indexes to get the final output. The solution to the issue I was facing with map command wasn’t documented anywhere, and it did appear to be a bug/limitation in handling certain type of data while other types were handled correctly. Hope it clarifies