All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

As I earlier said this should be doable. Just try to keep the mixed mode time as short as possible. Of course you must have enough wide and fast connections between your environments, but currently I ... See more...
As I earlier said this should be doable. Just try to keep the mixed mode time as short as possible. Of course you must have enough wide and fast connections between your environments, but currently I don’t think that this is an issue.
thanks @livehybrid . Upvoted. I almost figured it out, but in a slightly different manner. I'm got an ansible setup for URL interaction and automation. The 'contentctl build' will produce artefact s... See more...
thanks @livehybrid . Upvoted. I almost figured it out, but in a slightly different manner. I'm got an ansible setup for URL interaction and automation. The 'contentctl build' will produce artefact similar to a Splunk app with `savedsearches.conf` and other things like `analyticsstories.conf` contentctl build --path content --app.title MY_DETECT --app.appid DA-ESS-MY_DETECT --app.prefix MY --app.label MY Then i'm using the ansible automation which interacts with saved/searches and other endpoints to insert it back. Two things i'm still figuring out is it is slow once the savedsearches have 50+ searches as it runs one by one contentctl new : this option doesn't accept ALL parameters like search, name which means a user input is required Any chance for automation can detect if a savedsearch is Changed, then only insert   Update:  Able to insert into system after contentctl using REST api "saved/searches".  Though the type is specified as 'ebd' (event-based detection), while it is inserted into Splunk, it becomes a 'saved search' type !! any solutions/recommendations for this?
On-prem Splunk Enterprise Security environment, I just recently upgraded to Enterprise Security 9.4.1 and the ES app to 8.0.3. I was watching a video on using Mission Control, and an investigation w... See more...
On-prem Splunk Enterprise Security environment, I just recently upgraded to Enterprise Security 9.4.1 and the ES app to 8.0.3. I was watching a video on using Mission Control, and an investigation was created from a notable event.  Within the investigation, a search was done, to add it to the Investigation.  I want to do this, but when I select the evetn action drop down, within the Search results, I don't have much there, just the default Splunk Event Actions
Honestly, that tells me completely nothing. If sending a json array is so much cheaper than sending separate items from that array... there's something strange here. BTW, you are aware that you can ... See more...
Honestly, that tells me completely nothing. If sending a json array is so much cheaper than sending separate items from that array... there's something strange here. BTW, you are aware that you can simply send your events in batches? And that it's how it's usually done with high-volume setups? So you don't have to use a separate HTTP request for each event?
For verifying regexes https://regex101.com is usually sufficient Anyway, if you're not using the capture group names for field extraction, don't use capture groups. It makes the regexes easier to... See more...
For verifying regexes https://regex101.com is usually sufficient Anyway, if you're not using the capture group names for field extraction, don't use capture groups. It makes the regexes easier to read and saves a bit of performance because Splunk doesn't have to retain the capture group contents. It's a tiny bit of a difference but it's there. So since you're trying to "recast" the data to a static sourcetype, it's enough to use REGEX = \bssh\b to match your events. And you're misunderstanding the relation between fields and capture groups. If you do | rex "(?<ssh>\bssh\b)" Splunk will create a field named "ssh" because that's what the capture group is named. But it will be matching against the whole raw message because if you don't specify the field for matching it's the default option. You can extract data from a specific field using the field= parameter. Like | rex field=message "(?<ssh>\bssh\b)" This would create a field named "ssh" only if an already existing at this point of your search pipeline (either by default extractions defined for your data or manually extracted or created) field named "message" contained a word "ssh". But anyway, this has nothing to do with transforms. With transforms, it's the SOURCE_FIELD option which decides which field the REGEX will be matched against. One big caveat though (beginners often fall into this trap) - during ingest time processing (and that's what you're trying to do) Splunk has no idea about all search-time extracted fields. You can only use indexed fields here in index-time transforms (and they must have been already extracted if they are custom fields). And again, index-time transforms have nothing to do with searching. (And datamodels are something yet completely different so let's not mix it all ;-)) Your config seems pretty OK at first glance but 1. Naming your sourcetype just "authentication" isn't a very good practice. It's usually better to name your sourcetypes in a more unique way. Usually it's some form of convention using vendor name, maybe product and the "kind" of data. Like "apache:error" or "cisco:ios" and so on. 2. You restarted the HF after pushing this config, didn't you? 3. Is the linux_audit sourcetype the original sourcetype of your data or isn't it also a rewritten sourcetype? (I don't remember that one to be honest). Because Splunk decides just once - at the beginning of the ingestion pipeline - what props and transforms options are relevant for the event. And even if you overwrite the event's metadata "in flight" to recast it to another sourcetype, host or source, it will still get processed till the end of the indexing phase according to the original sourcetype/host/source. 4. Oh, and you applied this config in the right place of your infrastructure? On the first "heavy" component in your events' path?
There are a lot of events and there being sent in chunks to save on the lambda processing cost. 
Rick, Thanks for the reply! Seems like this is much more involved than I initially thought. It's not that I am tryin to use the regex as a means of doing searches. I was only running the search to ... See more...
Rick, Thanks for the reply! Seems like this is much more involved than I initially thought. It's not that I am tryin to use the regex as a means of doing searches. I was only running the search to see if the regex I had was actually hitting the data I'm looking for, so rex is out because I'm really not trying to extract anything. Thanks for that clarification. I ran the search with regex instead of rex and it did come back with what I'm looking for. Like I mentioned, I'm just trying to create a props/transforms set to catch data that matches a certain regex and change it's sourcetype to authentication in attempt to CIM the data. Something like: props.conf [linux_audit] TRANSFORMS-changesourcetype = change_sourcetype_authentication transforms.conf [change_sourcetype_authentication] REGEX=(?<ssh>\bssh\b) FORMAT = sourcetype::authentication DEST_KEY=MetaData:Sourcetype Nothing was coming back when I pushed that to my HF's, so I was trying to search the regex to see if it was even hitting anything. If I understand correctly, the <ssh> field needs to already exist for this to work? With that in mind, to your 4th point, does that mean this approach would not be an ideal one? All my indexes are customer based so organizing datamodels by indexes isn't an option.   Do I just have a typo somewhere I'm missing or am I just going down the wrong lane?
Thank you @livehybrid @yuanliu and @bowesmana! This is my first real post here, so I appreciate you bearing with me as I may not have provided a complete picture. @yuanliu 's answer provided a clear... See more...
Thank you @livehybrid @yuanliu and @bowesmana! This is my first real post here, so I appreciate you bearing with me as I may not have provided a complete picture. @yuanliu 's answer provided a clear example of how I can use mvfind and mvindex to extract the correct data. The only thing I had to add was a \b word boundary to the mvfind regex, so it wouldn't hit the earlier partial match. Here is the query: index=okta "debugContext.debugData.privilegeGranted"="*" | eval type_index = mvfind('target{}.type', "CUSTOM_ROLE\b") | eval "Target Name" = mvindex('target{}.displayName', type_index) | eval "Target ID" = mvindex('target{}.alternateId', type_index) | rename actor.displayName as "Actor", description as "Action", debugContext.debugData.privilegeGranted as "Role(s)" | table Time, Actor, Action, "Target Name", "Target ID", Action, "Role(s)"
I'm trying to do a transaction using an array.  I need to define the transaction by a value in an array.  However, this value could be any value in the array and the value could be in a different arr... See more...
I'm trying to do a transaction using an array.  I need to define the transaction by a value in an array.  However, this value could be any value in the array and the value could be in a different array index number in another event.  Is there an easy command for this in Splunk?
As I love border cases I'm wondering if @yuanliu @bowesmana and @livehybrid 's solutions handle properly the situation where not all event is parsed into events. In this case you cannot simply iterat... See more...
As I love border cases I'm wondering if @yuanliu @bowesmana and @livehybrid 's solutions handle properly the situation where not all event is parsed into events. In this case you cannot simply iterate over all extracted fields because the string may be contained within a part of the _raw event which is not extracted. So if you only manipulate fields, you won't catch it. Another thing to consider - automatic lookups and calculated fields. Nobody said it would be easy
1. The SCS docs are not for your normal Splunk Enterprise or Splunk Cloud searching. Yes, they often pop up in google search results. 2. The rex command is for extracting fields from your data. So y... See more...
1. The SCS docs are not for your normal Splunk Enterprise or Splunk Cloud searching. Yes, they often pop up in google search results. 2. The rex command is for extracting fields from your data. So your  search | rex SPL means that Splunk will search for the events matching the search terms of the search command (if you don't specify any command, Splunk implicitly uses the search one) and then from all those events it will try to extract the fields using the regex you provide. In your case it was a regex capturing a group named ssh so if your data matched the regex a field named ssh wojld be created. But if the event doesn't match the regex then the field simply isn't extracted. Nothing else happens. 3. If you want to filter your data by a regex, you have to use the regex command, not rex. But be aware that regex command doesn't capture anything. It just matches the event or an already extracted field against a regex and based on that filters the event stream. 4. While regex-based filtering can sometimes have its uses, it's very inefficient as a "base" method of searching. In your case, the search for the word "ssh" returned just 28 results. And the search without that word returned 2700 events. If you specify a direct search term, Splunk can check its indexes and only consider for further processing the events which contain the word you searched for. But if you did search | regex "\bssh\b" Splunk would have to first fetch all 2700 events from the index and then try to match every single one of them to see if it fits the regex. That is very inefficient way of searching. You'd still get the same 26 results but the processing overhead on this search would be humongous compared to simply searching for the word "ssh".
Hello, Here is what I have. Lookup file containing 52K rows Fields: DATE, USER, COUNT Require forecasting user access, on Sundays, to sensitive data based on 6 months of events to train (Jan-Ju... See more...
Hello, Here is what I have. Lookup file containing 52K rows Fields: DATE, USER, COUNT Require forecasting user access, on Sundays, to sensitive data based on 6 months of events to train (Jan-Jun) 6 months forecasting (Jul-Dec) Data from 2020, so we know the results, but we want to see how close the forecasting was to the actual data DATE format YYYY-MM-DD beginning with 2020-01-05 and ending on 2020-12-27 (Sundays) 52 values USER 1000 values Lookup file; there are 1000 USER values for every DATE; the COUNT is 0 if they did not attempt access, otherwise it is the number of attempts The original lookup is over 1.5 million events (each containing the USER and TIME of attempt) Original TIME value was YYYY-MM-DD HH"MM:SS. But we are concerned with how many attempts that day. Went to ChatGPT to help code the SPL; however, it "claimed" MLTK needed to be in a count of each user for every Sunday, and could work with the original events. Thanks in advance for your help. God bless, Genesius
Thanks. On prem. All-in-one.
I've never worked with splunk regex before so I'm probably just missing something.  I've been up and down  the  https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides a... See more...
I've never worked with splunk regex before so I'm probably just missing something.  I've been up and down  the  https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides and https://docs.splunk.com/Documentation/SCS/latest/SearchReference/RexCommandOverview pages.  All i'm trying to do is set up some regex for a props/transforms that finds any instance of "ssh" and changes it's sourcetype to "authentication" My search: index=accounting sourcetype=linux_admin | rex field=_raw "(?<ssh>\bssh\b)"  Scoped down to the last 60 minutes, I'm getting 2,700 results and none of them have anything to do with ssh. When I run  "index=accounting sourcetype=linux_admin ssh" - which gives the results I'm actually looking for... I only get 28 results and they're all pertaining to ssh.    What am I missing?    Thanks for the input!!
Hi @anthonyi Are you using an on-premise or Splunk Cloud environment? If you're using Splunk Cloud:  In the Splunk UI, go to Settings -> sourcetypes. Find/click the sourcetype you want to restric... See more...
Hi @anthonyi Are you using an on-premise or Splunk Cloud environment? If you're using Splunk Cloud:  In the Splunk UI, go to Settings -> sourcetypes. Find/click the sourcetype you want to restrict the length of, then click the "Advanced" tab. If there is already a "TRUNCATE" value then update this to 2000 and hit save. If there isnt then click "New Setting" at the bottom and then enter TRUNCATE for the name and 2000 for the value. If you are running on-premise then its unlikely you'll be able to make the changes in this way unless you are using a all-in-one instance, which generally wouldnt be used for production.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hello. I cannot find an answer to this simple question, although I have found other information utilizing props.conf and transforms in more complicated situations. I currently have a single Splunk ... See more...
Hello. I cannot find an answer to this simple question, although I have found other information utilizing props.conf and transforms in more complicated situations. I currently have a single Splunk instance as an all-in-one solution, and I am looking for a simple method to truncate ISE logs to 2000 characters to lower Splunk database size. ISE itself is not capable of this. I am very familiar with Splunk via GUI, but not at all with modifying the configuration files, so step by step instructions would be very helpful. Thanks in advance.
@livehybrid  "makecontinuous" - This is exactly what I need! I didn't know there is such useful command. My question is solved. Thank you!
Great question! In that case, it shouldn't return the result(s).
Apologies @robertlynch2020  - I overlooked this - even we extracted it! Lets try again:     | windbag | head 1 | eval _raw="{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"ser... See more...
Apologies @robertlynch2020  - I overlooked this - even we extracted it! Lets try again:     | windbag | head 1 | eval _raw="{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"trade-blotter\"}},{\"key\":\"service.namespace\",\"value\":{\"stringValue\":\"trade-blotter-public\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"61df555eabf3b66cd8933809f00e409f\",\"spanId\":\"abc123\",\"name\":\"referenceDataBuildClient\",\"startTimeUnixNano\":\"1746628424928265438\",\"endTimeUnixNano\":\"1746628424928267680\"},{\"traceId\":\"61df555eabf3b66cd8933809f00e409f\",\"spanId\":\"def456\",\"name\":\"readLiveQuery\",\"startTimeUnixNano\":\"1746628424928290524\",\"endTimeUnixNano\":\"1746628424928301728\"}]}]}]}" ``` main section of query below ``` | spath resourceSpans{}.resource.attributes{} output=resourceAttributes | eval service_name=mvfilter(match(resourceAttributes,".*service.name\"")) | eval service_namespace=mvfilter(match(resourceAttributes,".*service.namespace")) | foreach service_name* [| eval <<FIELD>>=json_extract(json_extract(<<FIELD>>,"value"),"stringValue") ] | spath resourceSpans{}.scopeSpans{}.spans{} output=scopeSpans | stats count by scopeSpans, service_namespace | spath input=scopeSpans | rename startTimeUnixNano as start | rename endTimeUnixNano as end | eval _time=start/pow(10,9) | eval duration = end -start | eval duration= duration/1000000 | eval duration = round(duration,0) | eval parentSpanId =if(parentSpanId="" ,"0", $parentSpanId$) | rename name as SPAN_TYPE | search traceId = 61df555eabf3b66cd8933809f00e409f | table _time SPAN_TYPE spanId service_namespace duration | sort _time  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi Thanks for your efforts - but I am looking for _time SPANTYPE Service.namespace duration. I think you have given me  | table _time SPAN_TYPE spanId duration.   This is what I have original... See more...
Hi Thanks for your efforts - but I am looking for _time SPANTYPE Service.namespace duration. I think you have given me  | table _time SPAN_TYPE spanId duration.   This is what I have originally, or perhaps I am misreading something. Rob