About hettervik

hettervik · ‎08-08-2025

We have an index with a ton of data. A new use for the data has emerged, so now we want a longer retention time on some of the data in the index. We don't want to simply increase the retention time on the index, because the storage cost is too high. We want to create a new index with a longer retention, pick out the events we need, and copy them to the new index. This is on an indexer cluster. In theory, we could use collect, like this: index=oldindex field=the_events_we_need | collect index=newindex However, because the index is too big, we're having problems running this search. Even though we run it bit-by-bit, still we end up missing events in the new index. Could be due to performance or memory limits, or bucket issues. Is there a better and more reliable way of doing this?

hettervik · ‎11-19-2024

Thanks for the reply. I can confirm that if I set my profile preferences to "theme dark", the the default app pages like "Search", "Alerts", "Reports", etc. in my app becomes dark. So yes, the following link is dark: https://mysplunk.com/en-GB/app/MyCustomApp/search I checked all the 18 different dashboards in the app, and all of then are still in light mode though. I expected all of them to be in dark mode now. There are not custom CSS and such in the dashboards. Is this as intended, do I need to add some setting to all my dashboards to make them dynamically suppoert light and dark mode? I could of course manually set the dashboards to dark mode, but then they would be stuck in dark mode, if I understand correctly. I want the dashboards to change dynamically between dark and light.

hettervik · ‎11-11-2024

Anyone else having trouble implementing this? I have an app, it has the following setting in app.conf, but still when users log into the app, they are forced into light mode, even if their user preference is dark mode. We are using Splunk 9.2.2. [ui] is_visible = 1 label = MyApp supported_themes = light,dark

hettervik · ‎10-21-2024

We have different lookup inputs into the Splunk ES asset list framework. Some values for assets change over time, for example due to DHCP og DNS renaming. When an asset gets a new IP due to e.g. DHCP, the lookup used as input into the asset framework is updated accordingly, but the merged asset lookup "asset_lookup_by_str" will contain both the new and the old IP. So the new IP is appended on the asset, it's not replacing the old IP. Due to "merge magic" that runs under the hood in the asset framework, over time this creates strange assets with many DNS names and many IPs. My question is, how long are asset list field values stored in the Splunk ES asset list framework? Are there any hidden values that keep track of say an IP, and will Splunk eventually remove the IP from the asset in the merged list? Or will the IP stay there forever, and these "multivalue assets" will thus just grow with more and more DNS names and IPs until the mv field limits are reached? And, if I reduce the asset list mv field limits, how does Splunk prioritize what values will be included or not? Does the values already on the merged list have priority, or does any new values have priority? Tried looking for answers in the documentation but could not find answers on my questions there. Hoping someone will share some insights here. Thanks!

hettervik · ‎10-04-2024

We also had some inconsistencies with these field extractions. Figured out that we needed to push the new limits configuration to the indexers, as well as the search head. Only pushing to the search head will work if you have a centralizing command before the spath field extraction, but not for streaming field extractions.

hettervik · ‎05-14-2024

Thanks! We see now, after some digging, that the bug is probably caused by a notable event being too big. The error message is "events are not displayed in the search results because _raw fields exceed the limit". Seems like this one too big event have caused bugs in the "Incident Review - Main" search, which also caused other incidents to fail to load. We are deleting the event and fixing the correlation search now, to add a fail-safe to avoid creating this big notable events in the future. Hope this fixes the issue!

hettervik · ‎05-10-2024

Hi, How do Splunk ES create incidents from notable events? I'm aware that a correlaction search in Splunk ES creates a notable event in the "notable" index, but exactly how does it get from here to the "Incident Review" dashboard in Splunk ES? As far as I know the incidents exists in a KV store collection, and I would then assume that there is some scheduled job that take notable events from the "notable" index, and puts them in the KV store collection. The reason I'm asking is that we are missing incidents in our "Incident Review" dashboard, but the corresponding notable events exists in the notable index. So it looks like the "notable event to incident" job has failed somehow. Is this documented somewhere in more detail?

hettervik · ‎03-14-2024

I also have the same question. The new Outlook client makes links not clickable, I guess for security reasons. I want to make it so that my link in the Splunk email alert becomes clickable again, but cannot find any way of doing so?

hettervik · ‎08-30-2023

We've looked a bit more into this case. The error is coming from the script "identity_manager.py" in the app "SA-IdentityManagement". The error is generated in the following "for" loop. for url, path, size, last_updated in update_times: if path and last_updated: lookup[url] = last_updated else: logger.error('status="Lookup file error, unknown path or update time" name=%s', url) The "update_times" array comes from the method "get_lookup_table_file_update_times", which again comes ultimately from the Python package "importlib.util.spec_from_file_location". We were thinking that this error might be from this package, and not from Splunk per se, but when we look at the actual lookup file CSV in the Linux OS, it is there and has the last modified time value sat, so that is not the cause either. So, still haven't figured this out.

hettervik · ‎08-16-2023

I changed the calculated eval field used in the lookup in the datamodel to an extracted field, and rebuilt the datamodel, and now it works. It very much looks like there is a bug with the calculated eval and lookup fields in datamodels, were the lookup field can get some sort of "mixup" if it is using an calculated eval field. Perhaps some sort of race condition between the calculation of the eval field and the lookup field.

hettervik · ‎08-11-2023

Has anyone found the reason for this error message yet, and how to fix it? We're encountering the same error. Both the lookup file and the lookup definition surely exists, and both are available when using inputlookup in the search bar. Also we've checked that they are available in the Splunk ES app. The identity list in Splunk ES does populate with data, so the "identity lookup merging searches" are in fact working, meaning that surely the "lookup file path exists" and is available for Splunk ES.

hettervik · ‎08-01-2023

I think the way to go for combining tstats searches without limits is using "prestats=t" and "append=true". Note that you maybe have to rewrite the searches quite a bit to get the desired results, but it should be possible. Also note that you maybe have to create a table before using the normal stats command to merge the tstats searches (don't know why, but worked for me, perhaps something with having all the data on the search head instead of distributed on the indexers). See thread here as well: https://community.splunk.com/t5/Splunk-Search/How-do-I-join-two-data-models-in-a-TSTATS-without-using-JOIN-or/m-p/479132

hettervik · ‎08-01-2023

I think the way to go for combining tstats searches without limits is using "prestats=t" and "append=true". Note that you maybe have to rewrite the searches quite a bit to get the desired results, but it should be possible. Something like so: | tstats summariesonly=true prestats=t latest(_time) as _time count AS "Count of Web" dc(Web.src) AS "Distinct Count of src" from datamodel=Web where (nodename = Web) groupby Web.src, Web.dest, Web.url, Web.http_user_agent, sourcetype, Web.user | tstats prestats=t append=true count AS "Count of Allowed Traffic" dc(All_Traffic.src_ip) AS "Distinct Count of src_ip" from datamodel=Network_Traffic where (nodename = All_Traffic.Traffic_By_Action.Allowed_Traffic) (All_Traffic.dest_ip!="10.*") (All_Traffic.direction="out*") groupby All_Traffic.src_ip, All_Traffic.dest_ip, sourcetype, All_Traffic.action, _time | rename Web.* as *, All_Traffic.* as * | table _time index dest url src_ip | stats latest(_time) as _time index values(dest) as dest_ip values(url) as url count by src_ip

hettervik · ‎07-10-2023

Hi, I have an accelerated datamodel. This datamodel have a lookup field based on a KV store lookup, that is, the datamodel takes another calculated numerical field and use the lookup to get a string value for that number. However, after some testing I see that my lookup field the accelerated datamodel does not have the correct value. I cross-checked by using the lookup directly on the raw data, and then it workes fine. I also double checked the expression for the lookup field, making sure it actually uses the correct values. The interesting thing is that the lookup field is indeed calculated, so the KV store lookup works, and it get a "hit" in the lookup, but the row that is returned from the lookup is wrong. It seems to me that this could be some kind of bug, e.g. related to memory on the search head or something, that causes the lookup field accelerated datamodel to bug out. The KV store lookup is quite big. Anyone else have experienced this, and knows a way around? My lookup field in the datamodel: { [-] calculationID: 123456 calculationType: Lookup comment: editable: true lookupInputs: [ [-] { [-] inputField: mynumber lookupField: mynumber } ] lookupName: number_to_string outputFields: [ [-] { [-] comment: displayName: mystring editable: true fieldName: mystring fieldSearch: hidden: false lookupOutputFieldName: mystring multivalue: false owner: mydatamodel required: false type: string } ] owner: mydatamodel } The field "mynumber" is based on an eval expression in the datamodel (which is defined before the lookup field): { [-] calculationID: 456789 calculationType: Eval comment: editable: true expression: 'mynumber_raw' outputFields: [ [-] { [-] comment: displayName: mynumber editable: true fieldName: mynumber fieldSearch: hidden: false multivalue: false owner: mydatamodel required: false type: string } ] owner: mydatamodel }

hettervik · ‎09-16-2022

Do you want a correlation search to add risk score to an object? If so, you have to edit the correlation search in Splunk ES, and then add a "Risk Analysis" response action, all the way at the bottom of the edit page. There you can add risk scores to users and systems from you correlation search.

hettervik · ‎09-15-2022

Hi. Yes, I see the confusion. The fields you add under the response action "Risk Analysis" are not added the the notable event itself (index=notable), they are added to the risk event (index=risk). These risk events are used for Risk-Based Alerting, among other things. If you want the "user" and "app" fields to be added to the notable event, just make sure these fields are present in the final output of your correlation search, and you shoud see them in the incident.

hettervik · ‎09-15-2022

I've done some digging in the documentation. For UFs on Windows it's specified that if you don't specify the "SPLUNKPASSWORD" and "SPLUNKUSERNAME" flags, the UF install without an admin user at all. https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/InstallaWindowsuniversalforwarderfromaninstaller For potential older UFs with default credentials we've concluded that the easiest way to make sure these doesn't exist, is to delete all password files from all Windows UFs, deleting all users if any. For Linux UFs the documentation doesn't specify what happens when you don't create an admin user on install, or if it's even possible to not create an admin user. We could assume it works the same way as for Windows UFs, but have to do some testing. I will comment this on the official documentation, so perhaps it's updated on next release. https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/Installanixuniversalforwarder

hettervik · ‎09-08-2022

We're looking over our environment for potential safety flaws. One question that came up is whether an admin-user is available by default on Splunk Universal Forwarders (UF). I'm not thinking about the user the UF runs as on the OS, but an admin user on the application layer. Earlier Splunk Enterprise had a default admin password "changeme". Did this also apply for UFs? How can we make sure that there is no admin users on our UFs, or that if there is, that they have proper passwords?

hettervik · ‎09-02-2022

Seems like this problem has been addressed in newer versions of the supported Splunk Windows TA. https://docs.splunk.com/Documentation/AddOns/released/Windows/Configuration#Configure_Windows_Update_Logs_in_inputs.conf

hettervik · ‎08-26-2022

In general, as notable events are stored in a separate index (called "notable"), the permission for the indexes for the original events doesn't apply anymore. However, I guess you could add the index field from the original events to the notable event, call it e.g. "original_index", and then create a search filters for different user groups, so that for example users with access to index "foobar" can only see notable events with "original_index=foobar". I'm not sure how this would work for incidents in Splunk ES though, as I understand that viewing incidents in the "Incident Review" in Splunk ES is not the same as looking up events from the notable index per se, and that search filters might not apply here.

hettervik · ‎08-26-2022

If you want index to be included as a field in the incident itself, you could add "index" as an incident review event attribute. Configure > Incident Management > Incident Review Settings > Incident Review - Event Attributes For this to work you would also have to make sure that the index field is included in the output of your correlation searches as well. This would require edits to a lot of the out-of-the-box correlation searches that use the tstats command on data models.

hettervik · ‎08-26-2022

Note that "session_id" is not an eval field in the Network Traffic data model, meaning that it could be null for some entries. If you add session_id to the by clause, these entries would be dismissed. If this is the case, you can either use the "fillnull_value" argument on the tstats command, or instead of adding session_id after the by clause, add it as a values function. | tstats summariesonly=t count values(All_Traffic.session_id) as session_id min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Traffic by All_Traffic.src All_Traffic.dest All_Traffic.dest_port

hettervik · ‎08-26-2022

You haven't activated the alert throttling in the correlation search by chance? That could explain it. Alternatively, if the alert run frequently over a small timespan, e.g. run each 5 minutes and searches in the last 5 minutes, and there's also a delay in the logs to Splunk, the alert could "miss" the logs. The solution to this would be to add a delay to the search, so that it e.g. searches from -10m to -5m instead of -5m to now.

hettervik · ‎08-25-2022

Thanks. Though we have a distributed environment, so if the attacker gets access to CLI on the search head, they still cannot delete the data on the indexers. We have extra layers of protection for the indexers. Therefore it would still help to be able to disable the delete command, if I'm not missing something.

hettervik · ‎08-25-2022

In Splunk there exist a delete command. Any admin in Splunk can give themself the capability to use this command. In theory, if a single admin user in our Splunk environment is compromised, the attacker can delete all data from the Splunk indexers. I know that the data is not actually deleted from disk when using the delete command, but still it is for all practical purposes deleted. Is there any way to securely disable the delete command/capability in Splunk, so that not even administrators can get access to it? Preferably we want to disable the command on the indexer layer, so that even if the OS on the server hosting the search head is compromised the command cannot be used. Alternatively, if the command can be disabled on the search head in a way that it cannot be re-enabled through the web interface, that is better than nothing.

Posts	266
Solutions	21
Karma Given	107
Karma Received	48
Member Since	‎09-25-2015

Online Status	Offline
Date Last Visited	‎08-08-2025 01:35 AM

Is there a way to re-index data from an index to a...

How long are asset list field values stored in the...

Nessus security scans dashboard and drilldown

How do Splunk ES create incidents from notable eve...

Why is my lookup field from KV store in datamodel ...

Is an admin-user available by default on a Splunk ...

Is there any way to securely disable the delete co...

How to make the Splunk ES Risk-Based Alerting risk...

How to add setup parameters to a new shell input i...

Is there any way to add a custom time to a notable...

Is there a way to re-index data from an index to a...

Re: How to add dark theme compatibility to custom ...

Re: How to add dark theme compatibility to custom ...

How long are asset list field values stored in the...

Re: Not all field values are extracted for long JS...

Re: How do Splunk ES create incidents from notable...

How do Splunk ES create incidents from notable eve...

Re: Add hyperlink in email alert

Re: Has anyone encountered this error: Asset and I...

Re: Why is my lookup field from KV store in datamo...

Re: Has anyone encountered this error: Asset and I...

Re: How to combine two tstat search to create one ...

Re: How do I join two data models in a TSTATS with...

Why is my lookup field from KV store in datamodel ...

Re: Risk isn't showing up in the Edit Correlation ...

Re: Risk Score, Risk Event and Risk Object is not ...

Re: Is an admin-user available by default on a Spl...

Is an admin-user available by default on a Splunk ...

Re: How to stop getting duplicate events from Wind...

Re: RBAC for Notable events?

Re: How can I view the source index where Splunk E...

Re: How to exclude events with same session_id - R...

Re: The Notable Generation doesn't produce the exp...

Re: Is there any way to securely disable the delet...

Is there any way to securely disable the delete co...

Are you a member of the Splunk Community?