I suppose that you should try to move those timestamp extractions under each source:: definitions. Then those should work. Anyhow those definitions which you have put on that new sourcetype definitions are working on search time if those can apply on search time. But example those _time settings are working only in indexing phase. ... View more

Are those tables individual sourcetypes on index or results of your SPL queries? If last, can you share it so we can modify it to create your requested result? ... View more

As other already mentioned those configurations must be in first HF/Indexer if there haven't been any HF from path source system (where this input is) to indexers. If input is in heavy forwarder then those must be there.  If there are any intermediate HF between UF and Indexers then those must be in this IHF. If source side is UF and there is nothing else before indexers (or your sandbox) then those must be in your sandbox. And ensure that those KOs are shared as all apps/system. Otherwise it could be that those are not valid in ingestion phase. ... View more

Ok, now I understand your requirements. You can do it with props.conf & transforms.conf too.  MV_ADD = <boolean> * NOTE: This setting is only valid for search-time field extractions. * Optional. Controls what the extractor does when it finds a field which already exists. * If set to true, the extractor makes the field a multivalued field and appends the newly found value, otherwise the newly found value is discarded. * Default: false This parameter add multiple values in mv field.  I haven’t use field extractor so much that I cannot recall if there are any options to do same or not, but I think that this MV_ADD is your solution. ... View more

Hi I think that your issue is setting sourcetype inside source stanza. As splunk has only one linear data pipeline and now it has taken those events based on source:: definition it will apply only those values on indexing phase. You cannot put events back into the start of this pipeline again and start same event manipulation with sourcetype stanza.  Your aws:elb:accesslog definitions are used, but only in search time, not in index time. And as those definitions are affecting only in index time it's obviously that nothing happen for you _time value. https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline Have you try to add those same definitions under every source:: stanzas? Of course as you are using HEC it also mater which endpoint you are using. There are differences what manipulations you can do with props.conf based on endpoint. r. Ismo r. Ismo ... View more

If I recall correctly there was same questions some time ago, but I cannot found it now. Anyhow the answer was same on that time too. Maybe you can use the next solutions as a work around? Is it possible that you will change sourcetype to be e.g. <host>:<original sourcetype> for those events which you are forwarding to AWS's S3 buckets? In that way you full will your requirements to store those based on hostname? ... View more

Wait a minute. You said that you have 1TB SSD + 3TB SAS disks and you are ingesting 600-800GB per day. How many indexers you have? I really hope that you have clusters which contains several indexers. And have you any Splunk premium apps like ES or ITSI running on your environment? So which kind of architecture you currently have to manage your workload? And what kind of nodes you have in resource point of view? Basically your understanding is quite correct. With this daily ingestion amount you definitely must have a cluster or at least several indexers take data in and serving searches at same time. Do you have a MC (monitoring console) up and running? This is excellent tool do get more information what is happening in your environment. Volumes are excellent way to manage your indexers space. I usually define one volume for hot+warm and another for cold. You must remember that you shouldn't allocate all disk space for use there must be some additional free space for disk operations. How much this is, depending on your filesystem. Some kind rule of thumb is 10-20%. Also you shouldn't allocate all filesystem space to Splunk Volume leave also there some room as splunk will need it when if flush data from warm to cold and also from cold to frozen if you have separate frozen space.   ... View more

Hi I did't get why you cannot use that rex which is working? In personally I always prefer to use my own rex than those which are created by field extractor. It's splunk's design decision that if there are multiple matches then those are put in mv fields. You can always expand those into individual events if mv fields are not suitable for your use case. | makeresults | eval _raw = "orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, {'NotSecrets!!.txt': 'fileHash': 'a3b9adaee5b83973e8789edd7b04b95f25412c764c8ff29d0c63abf25b772646'}, {}}, 'Secrets!!.txt': 'fileHash': 'c092a4db704b9c6f61d6a221b8f0ea5f719e7f674f66fede01a522563687d24b'}, {}}} orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc," | rex max_match=0 "(?:'(?<attachments>.*?)'.*?'fileHash': '(?<sha256>\w+)'}.*?\{.*?\}\}[,\}]\s?)" | eval foo = mvzip(attachments,sha256,";-;") | mvexpand foo | eval foo=split(foo,";-;") | eval attachments=mvindex(foo,0) | eval sha256=mvindex(foo,1) | table attachments sha256 r. Ismo ... View more

It replace all existing fields so you don’t need to write everything here. You could also add e.g. values(foo*) as bar* and then it takes only those fields which start with foo and put those as a result fields named bar*. This is quite useful and commonly used feature in SPL. ... View more

On UF you shouldn’t have splunktcp input in inputs.conf. The only exception is when you are using UF as an intermediate forwarder, but this is different story. ... View more

Admin is same as root *nix world. You could try different tricks to restrict what it can do, but there is always a way to avoid those restrictions! To be honest your company must implement policies which are mandatory and if someone doesn’t follow it then there is some consequences for those. Otherwise there will be always some surprises time by time. Of course there should be some other ways to motivate your colleagues first to understand why there is policies and why everyone must following those. ... View more

It’s best to use local splunk user for this even all other users are SAML or SSO users. User needs to have role (unless you can use only predefined reports which are run as owner) and if/when user has role in splunk he/she can always log in. So @richgalloway ‘s solution solve this issue. ... View more

Or you can do it on cli also https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/LicenserCLIcommands There are other license stuff also behind above link too. ... View more

I think that as those are usually ingested directly in windows nodes via correct input method, this is not a normal use case. Those who needs to investigate those files later usually already knows how this can do. ... View more

Which macOS and splunk versions you have? And Intel or Mx series workstation you have? I have never seen this, but neither browse those directories with Finder, I just use cli. ... View more

I totally agree this, but when you must keep business up and running and there is this things we called physic, we haven’t any other option than wait. Here is old story about it https://web.mit.edu/jemorris/humor/500-miles ... View more

Even in cluster you could use different versions if you haven’t any other options. But try to limit this time as short as possible. In practice it means the time which you need to update all nodes in cluster to same version. Same is valid for OS and Splunk versions too. I think that easiest way this can do by adding new servers with a new OS version BUT the same splunk version than you have in your cluster’s other nodes.  Here is an old post how this can do https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platform-Linux-but/m-p/538069#M4823. This was a play how I migrate distributed splunk environment into a new service provider into newer OS without service breaks. The migration took couple of weeks but less than month. ... View more

Can you paste your simple xml dashboard inside code block </> ? Insert at least part where you are getting inputs and how you are generating dropdowns. ... View more

Ok, basically your environment is broken and it doesn’t fulfill any Splunk’s requirements for SHC! I said that sooner or later you must rebuild it from scratch. I suggest that you will do it as soon as possible. Now you have those apps, congratulations, data etc still there and you could utilize those with a new environment without issues. But if your environment will collapse then you probably lose at least some of that data. if you don’t know how to do this, pleas ask some local Splunk partner or Splunk PS to do it. There are good instructions on splunk docs how you could migrate from individual SH to SHC. You could use those as starting point,  but probably you need to modify those somehow depending how your “SHC” is currently working. ... View more

One thing which I “found” (fortunately in test with backups). If your deployer is down enough long time your SHC members lost all apps which have deployed by deployer! Have you lost your deployer or only connection between it and members? If first then it should be enough that you restore the connectivity between those. Of course you must ensure that deployer still has all those apps which have previously deployed to members. If you have lost whole node and you haven’t backup and you must build it from scratch then there are some things which you must check and update before you can put it back online. Ensure that you have those apps there what you have previously deployed with same configuration on place. Ensure that lookups are correct. check that deployment modes are globally and app level correctly setup (how local and default are transferred into members) check that how lookups should push into members override or keen member version? check if members have same splunk.secrets and if copy this into deployer before start it first time then those what you and @gcusello already mentioned  check that all nodes have same time! Maybe something else? If you could do and test this on test environment, do it first and check what are issues which arise after deployer is back online. If you haven’t the definite you must take backup of all those nodes when they are offline! And include kvstore backup too. I would like to hear how this succeed after you have done it! ... View more

I’m not sure if I fully understood your issue. Anyhow sourcetype’s purpose is separate different log formats. If all those apps have exactly same lexical content of logs then they could/should have same sourcetype. But if the e.g. apple and orange creates logs which have different fields etc. then those should have different sourcetypes assigned for those logs. When you have two different apps and those have exactly same log format but you want separate those you could named those like sourcetype=“apple:module:log:1” and sourcetype=“orange:module:log1” or what ever names you want to use. Then just define those equally in props.conf. ... View more

If you want to use current time instead of time of previous event which are different to current time the you could in props.conf DATETIME_CONFIG = [<filename relative to $SPLUNK_HOME> | CURRENT | NONE] Select value CURRENT. But if the event time can be something else than current then probably not. ... View more

Which splunk version and is this issue with all apps? ... View more

Splunk should works on both RHEL 8 or 9, but with 9 there are some additional steps which one must do before splunk can installed there. RHEL 9 have cgroups v2 as default and that version of Splunk support only version 1. There are probably also some security changes which must notice before splunk works correctly. If I recall right at least some of those are already on 8 and maybe some more in 9? ... View more

Here is one old answer for indexing windows .evtx files. https://community.splunk.com/t5/Getting-Data-In/Ingesting-offline-Windows-Event-logs-from-different-systems/m-p/649515 ... View more