About gabriel_vasseur

gabriel_vasseur · ‎07-22-2022

This issue arised again for us this week. Users were reporting very poor performance from splunk, although it didn't look like the platform was heavily used. The most obvious symptom was a large amount of ad-hoc searches running modular_action_invocations each taking several minutes to finish. It was interesting to read https://community.splunk.com/t5/Splunk-Enterprise-Security/modular-actions-invocations-macro/m-p/390880 that this search is triggered when a notable is expanded within Incident Review, so that is bound to happen a lot. Looking at the macro definition and running it manually, I couldn't reproduce the slowness: it seemed to complete fine. I checked what @lakshman239 recommended and everything seemed to be in order. That said I did notice that the macro starts with "tstats summariesonly=false" and I know from experience that tstats searches can be surprisingly slow even if data model summaries cover virtually all the search time window, so I changed it to "tstats summariesonly=true". This seems to have helped the issue although it's early to tell. I'm aware that I've introduced the risk of the search not returning results for recent notables, however we don't really use fancy adaptive responses so I don't think it'll impact us. I'm also aware I've made upgrading ES in the future more difficult: if that macro is updated by the upgrade, my local tweak will override the upgraded version and it might break things. But desperate times calls for desperate measures. I hope this helps and/or someone can provide a better solution.

gabriel_vasseur · ‎04-19-2022

You do not want the %60 in: | eval hrs=floor((test/3600)%60) Otherwise a duration of 66 hours will become 06 hours...

gabriel_vasseur · ‎09-06-2021

Turns out the solution is: do not put the cluster master in maintenance mode before upgrading it. I'm pretty sure I saw in the docs on a previous upgrade that if the maintenance mode wasn't on whilst upgrading the cluster master it could cause problems... Every upgrade is different I guess!

gabriel_vasseur · ‎09-06-2021

Hi, I'm trying to upgrade splunk from 8.0.9 to 8.2.2. According to the docs, the upgrade starts with the cluster master. After upgrading the cluster master and removing the maintenance mode, all the indexers are stuck at in the "batchadding" status. Looking at the logs from one indexer, it goes through a cycle of: event=addPeer Batch=1/9 ...success... event=addPeer Batch=2/9 ...success... ... event=addPeer Batch=9/9 ERROR Read Timeout... WARN Master is down! Make sure pass4SymmKey is matching if master is running... WARN Failed to register with cluster master... Master is back up! Rinse and repeat. So basically it talks ok to the cluster master for a while and then get a timeout and starts over. Any idea what's going on? I did check the pass4SymmKey and they are the same everywhere, they haven't changed. Cheers, Gabriel.

gabriel_vasseur · ‎08-10-2021

This should be the proper answer. We had the same issue and it fixed it. According to btool, before fixing it, the "replicate.macros" setting didn't even exist on our search heads. But after setting it to true, suppressions started working in IR. Another quality software update from splunk 🙄

gabriel_vasseur · ‎08-03-2021

Do you have a version compatible with python3?

gabriel_vasseur · ‎07-28-2021

Long shot, but it might be worth checking who owns the alerts. "settings" (in the top right corner) > "Searches, reports, and alerts" > search for your alerts and check the owner If for whatever reason the owner isn't a user that exists, the scheduler will not try to schedule it.

gabriel_vasseur · ‎06-16-2021

I want a way to search the content of all my lookups. If I had just one lookup, I would do something like: | inputlookup myonelookup.csv | foreach * [| eval matching_field_and_value = mvappend( matching_field_and_value, if( match( '<<FIELD>>', "mysearchpattern" ), "<<FIELD>>" . "::". '<<FIELD>>' , null() ) )] | search matching_field_and_value=* | table matching_field_and_value * If I knew I wanted to search a few known lookups I would replace the first line with | inputlookup append=t lookup1.csv | inputlookup append=t lookup2.csv | inputlookup append=t lookup3.csv But what if I want to search through all my lookups? I can get a list of lookups via the right rest commands, and I've been thinking about converting the list in a list of fields and using a foreach loop on these fields but you can't put a "non streaming command" such as inputlookup inside a forearch subsearch... Any other idea? The solution needs to work in cloud, so needs to be pure splunk. On premise I know my way with grep and I know it works really well.

gabriel_vasseur · ‎03-25-2021

Yes, ES 6.0. The problem we have arises when people leave the company and come back later, which happens regularly, especially for contractors. Even if they are given a brand new account and brand new employee number, as they should, if they have the same email address identitymerge will merge the two entries. So suddenly we have employees who exist but splunk thinks they have an endDate in the past. And we want to keep the old entry in identities, as it's useful to check activity from expired accounts or when investigating old events. Splunk didn't think this through obviously. I worked around this issue by massaging the data before ES ingests it.

gabriel_vasseur · ‎03-23-2021

I am aware of this https://docs.splunk.com/Documentation/ES/6.4.0/Admin/Merge however, we have a version of ES older than 6.4 and that feature does not exist there. The behaviour of identitymerge is really unhelpful and has been causing many headaches. Is there a way to turn it off?

gabriel_vasseur · ‎03-15-2021

According to https://docs.splunk.com/Documentation/ES/6.5.0/Admin/Searchpreview "The results are the delta since the last merge. If nothing has changed in the source files since the last merge, you do not see any output." That answers my question. This behaviour is really weird and unexpected! :-$

gabriel_vasseur · ‎03-05-2021

I did try that before and it doesn't help. I had another go today and it only works with these extra steps: run "| outputlookup identity_lookup_expanded" to obliterate the old content. open in search the identity merge search preview, and run it with "| outputlookup identity_lookup_expanded" added at the end. I don't understand why these extra steps are needed. And then of course, I had to change the way we ingest identities and the lookup macro casing is consistent for usernames and email addresses. On top of that, I have a lot of problem with identity merging search. entity_merge doesn't seem to work correctly and consistently and sometimes returns much fewer results than it should. I have no confidence in this framework now.

gabriel_vasseur · ‎03-02-2021

This is the search that merges identities, according to the search preview: | inputlookup append=T "administrative_identity_lookup" | fillnull value="administrative_identities" _source | inputlookup append=T "simple_identity_lookup" | fillnull value="static_identities" _source | eval identity=split(identity, "|"),identity=if(_source=="administrative_identities", mvappend(identity,email,replace(email,"@.*","")), identity),identity=if(_source=="static_identities", mvappend(identity,email,replace(email,"@.*","")), identity),identity=mvjoin(mvdedup(identity), "|") | table "_source","bunit","category","email","endDate","first","last","managedBy","nick","phone","prefix","priority","startDate","suffix","watchlist","work_city","work_country","work_lat","work_long","identity" | eval `iden_mktime_meval(startDate)`,`iden_mktime_meval(endDate)`,identity=mvsort(identity) | sort 0 +identity | inputlookup append=T "identity_lookup_expanded" | entitymerge "identity" If identity_lookup_expanded is empty (because I've flushed it), it produces results. If I pipe these results into an "outputlookup identity_lookup_expanded" it is populating identity_lookup_expanded ok (except all text is lowercase, but see my other question regarding that). Now if I run the above search again, it returns no results. It's ok until it hist entitymerge. What does this mean?

gabriel_vasseur · ‎03-02-2021

We upgraded to enterprise security 6.0.2 and now every single piece of text in identity_lookup_expanded is lowercased. For instance, instead of having 'first' be "Gabriel" it's now "gabriel". Does this for every single fields, including job title ('category') etc. How can I fix it?

gabriel_vasseur · ‎02-22-2021

This has been asked before but the solutions I have seen are only for indexers. The best one I've seen is: | rest /services/cluster/config | fields splunk_server guid But like I said this is only for indexers. I want something for search heads. Also, why not also include the cluster master, license manager, deployment server, search head deployer, and all the data forwarders, that would be quite useful.

gabriel_vasseur · ‎11-23-2020

We've just witnessed the exact same issue for us. Looks like tstats will return results if the first 16 characters are the same, ignoring any change beyond that. Unbelievable.

gabriel_vasseur · ‎05-27-2020

Thanks, this helped me so much! To make this perfect, I would suggest the following: Simplify the javascript so that it only sets submit_trigger . That way it can be used in more than one dashboard and you don't have to keep tweaking it when you develop your dashboard. To prevent the annoyance of the searchWhenChanged=false being ignored again, simply add this to the search that is to be ran when submit is pressed: <search> <done> <unset token="submit_trigger"/> </done> ... That way, after the search is ran once, you can take your time to adjust your inputs and only when you're ready you can click submit to run it again. Works a treat. Finally, unlike what I initially assumed when I saw your solution, the inputs (including the submit button) can be part of the standard <fieldset> at the top of your dashboard, in case you want them to have a grey background as usual, not a white one. Also, it's worth noting that your solution allows to hide the submit button with a "depends" clause so that it only appears if the inputs satisfy some kind of condition. I found that useful. Thanks again!

gabriel_vasseur · ‎02-24-2020

Thanks, that works!

gabriel_vasseur · ‎02-24-2020

I can't seem to set/unset a token based on whether some search result is equal or not to 1. Can anybody point out why this doesn't work: <form> <label>TEMP Endpoint Processes Clone</label> <fieldset submitButton="true" autoRun="false"> <input type="text" token="number" searchWhenChanged="false"> <label>Host count</label> <default>100</default> <initialValue>100</initialValue> </input> </fieldset> <row> <panel> <html> <p>host_count=$host_count$</p> <p>host_count1 is 1=$host_count1$</p> <p>host_count2 catchall=$host_count2$</p> <p>gotuniquehost=$gotuniquehost$</p> </html> </panel> </row> <row> <panel> <table> <search> <done> <condition match="$result.host_count$ == 1"> <set token="gotuniquehost">True</set> <set token="host_count">$result.host_count$</set> <set token="host_count1">$result.host_count$</set> </condition> <condition> <unset token="gotuniquehost"></unset> <set token="host_count">$result.host_count$</set> <set token="host_count2">$result.host_count$</set> </condition> </done> <query>|stats count as host_count | eval host_count=$number$</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form> And yet this is working: <form> <label>TEMP Endpoint Processes Clone</label> <fieldset submitButton="true" autoRun="false"> <input type="text" token="number" searchWhenChanged="false"> <label>Host count</label> <default>100</default> <initialValue>100</initialValue> </input> </fieldset> <row> <panel> <html> <p>host_count=$host_count$</p> <p>host_count1 gt 1=$host_count1$</p> <p>host_count2 lt 1=$host_count2$</p> <p>host_count3 catchall=$host_count2$</p> <p>gotuniquehost=$gotuniquehost$</p> </html> </panel> </row> <row> <panel> <table> <search> <done> <condition match="$result.host_count$ > 1"> <unset token="gotuniquehost"></unset> <set token="host_count">$result.host_count$</set> <set token="host_count1">$result.host_count$</set> </condition> <condition match="$result.host_count$ < 1"> <unset token="gotuniquehost"></unset> <set token="host_count">$result.host_count$</set> <set token="host_count2">$result.host_count$</set> </condition> <condition> <set token="gotuniquehost">True</set> <set token="host_count">$result.host_count$</set> <set token="host_count3">$result.host_count$</set> </condition> </done> <query>|stats count as host_count | eval host_count=$number$</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form> I run splunk enterprise 7.3.3. This seems to be a bug.

gabriel_vasseur · ‎11-08-2019

I did try your suggestions to leave no stones unturned, but as I expected it made no difference. I do know the subsearch does nothing, it's just to illustrate that any subsearch triggers the bug, before or after the first pipe. Even the following triggers the bug: index="_audit" sourcetype=audittrail action=success src!=127.0.0.1 login attempt NOT *uba | foreach blah* [ eval foo_<<MATCHSTR>>=lower(<<FIELD>>) ]"

gabriel_vasseur · ‎11-08-2019

We haven't tried version 8 yet. Adding | noop search_optimization=false has no impact on either bug.

gabriel_vasseur · ‎11-07-2019

The first bug we noticed is triggered when using NOT in the where clause of tstats commands using datamodel summaries. To summarise it: | tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex ) by index sourcetype | tstats summariesonly=t count from datamodel=Authentication where NOT ( sourcetype=mysourcetype ) by index sourcetype | tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex sourcetype=mysourcetype ) by index sourcetype The first 2 searches returns results, as they should, but the last one returns nothing at all. Awful for accuracy! The other bug affects raw searches with subsearches. Any kind of subsearch, even a foreach command. For instance: index="_audit" sourcetype=audittrail action=success src!=127.0.0.1 login attempt NOT *uba | search [| stats count as index | eval index="*" ] This search completely ignores the "action=success " and the "src!=127.0.0.1". This yields a lot more results than it should, also bad for accuracy. I'd be curious if people on different versions of splunk (especially 7.3.*) can reproduce these. If you're thinking to upgrade, make sure you test for it before. As far as I can see, I do not believe that these issues are mentioned in the known issues in the release notes, even though splunk is aware that something is wrong. We have open cases with splunk support and unfortunately, at least for the first bug, there doesn't seem to be a fix coming any time soon.

gabriel_vasseur · ‎10-09-2019

Thanks. Maybe things are not too bad for splunk core. Have you ever used Enterprise Security? It has a health check feature that reveals a LOT of "unshipped" files, and a significant portion of these really do not look like anything the team could ever have created themselves. So I believe they are accumulated junk, except I don't feel confident removing them.

gabriel_vasseur · ‎09-27-2019

The upgrade process on linux is basically to unpack the tgz file over the existing splunk home directory. I understand that will add any new file where they need to be, update any file that needs updating, but what about the files that are no longer needed after the upgrade? Are they ever removed or do we just accumulate rubbish over the years?

gabriel_vasseur · ‎03-26-2019

Interesting, but like you said unlikely to be the case here. We have PS on site this week so if there is time I'll ask them to take a look and will update this question if anything transpires.!

Posts	138
Solutions	8
Karma Given	73
Karma Received	39
Member Since	‎03-14-2016

Online Status	Offline
Date Last Visited	2 weeks ago

Dashboard prebuilt panel not using JS/CSS from the...

How to change a Simple XML table height when no re...

Why is KV store created by lookup editor app, unus...

Is there any way to get Splunk to choose the right...

8.0.9 indexers all stuck "batchadding" after upgra...

How can I search the content of all my lookups?

Disable identitymerge in older enterprise security...

entitymerge for identity_lookup_expanded doesn't p...

Why is every single piece of text in identity_look...

Ultimate Hostname from GUID

Re: Macro : modular_action_invocations(Apps: Splun...

Re: How to format 86401 seconds as 24:00:01

Re: 8.0.9 indexers all stuck "batchadding" after u...

8.0.9 indexers all stuck "batchadding" after upgra...

Re: Enterprise Security 6.6.0 Incident review show...

Re: How to encode a URL for a Hipchat notification...

Re: Next Scheduled Time for all alerts is showing ...

How can I search the content of all my lookups?

Re: Disable identitymerge in older enterprise secu...

Disable identitymerge in older enterprise security...

Re: entitymerge for identity_lookup_expanded doesn...

Re: Why is every single piece of text in identity_...

entitymerge for identity_lookup_expanded doesn't p...

Why is every single piece of text in identity_look...

Ultimate Hostname from GUID

Re: Data Model Acceleration TSTATS where clause NO...

Re: Comprehensive implementation of a WORKING subm...

Re: Why is my very simple dashboard token conditio...

Why is my very simple dashboard token condition no...

Re: Major bugs in searches in splunk 7.3 DO NOT UP...

Re: Major bugs in searches in splunk 7.3 DO NOT UP...

Major bugs in searches in splunk 7.3 DO NOT UPGRAD...

Re: Do splunk upgrades ever remove any files?

Do splunk upgrades ever remove any files?

Re: Why would thruput metrics be missing from some...

Join the Conversation