Hi, I'm trying to modify the name of the index in the Data Inputs for your Digital Shadows SearchLight App. When I try and save it I get the following error:- Encountered the following error while trying to update: 'ValidationDefinition' object has no attribute '__getitem__' Any ideas how to rectify this? Cheers, Mark. ... View more

Hi all, I'm not sure whether this is a bug or a 'holiday hangover'! I used props.conf and transform.conf to re-sourcetype a specific message to a new sourcetype ( i will, also, be changing the index as well when I am satisfied!). The events show up in a search e.g. index=ba.com.logs DOB_RESULTS (DOB_RESULTS is the same string that I use in my regex in the transforms.conf) There are 270 results returned and when I check the 'Sourcetype' field on the left it does show that all 270 events are now in the new Sourcetype - ba.com:authentication:dob However, when I click on the new sourcetype to add it to my search string it returns zero results? Throwing in a few wildcard's returns the correct results though.... index=ba.com.logs DOB_RESULTS sourcetype="ba.com:authentication*" or index=ba.com.logs DOB_RESULTS sourcetype="*authentication:dob" Am I missing a trick here or is this a bug? Cheers and Happy New Year to you all. Mark ... View more

HI, Apologies if this is answered elsewhere but I can't find a question that fits my situation although I'm sure that I'm not alone in trying to achieve this. I am searching two sourcetypes (and indexes) that have (amongst many other fields) two automatically extracted fields that contain IP addresses. One is called c_ip and the other source_ip. What I am trying to get is a new field called src that merges the values in the existing extracted IP address fields. For example:- search index=a sourcetype=source1 | stats count by c_ip produces: c_ip count 1.1.1.1 9 2.2.2.2 5 and search index=b sourcetype=source2 | stats count by source_ip produces: source_ip count 1.1.1.1 3 3.3.3.3 7 What I would like to get is the above two fields merged into one such that search index=* (sourcetype=source1 OR sourcetype=source2) | {whatever code is required here to merge the data into src} | stats count src produces src count 1.1.1.1 12 2.2.2.2 5 3.3.3.3 7 This is obviously very simplified. In reality I have hundreds of thousands of events and thousands of IP addresses, many of which only appear in one or the other extracted fields. Hopefully my examples will help explain my dilemma, Many thanks, Mark. ... View more

Hi all, I couldn't find any definitive answers, so I'm hoping that the forum members' real life experiences may point me in the right direction. We have lots of logged events (over 300 million per day) from two, distinct sources for the same application. Even though the event message format is different between the two types of source data, there is one major commonality - the raw events are completely unstructured! When we first got Splunk, we only had one data source which made it simple to set up an Index and a Custom Sourcetype. When we started ingesting the second source, there was a huge pressure to get the data in and analysed which left me little time to appraise the actual raw data. In my rush ( and, perhaps, naivety!!) I created a second index and a second sourcetype and, as a result of this, many of our searches start with index=abc OR index=xyz etc! My question is, given the volume of data and the similarity in that it's all unstructured data, would it be best, from an indexing and searching perspective, to move towards one of: 1 index and 2 sourcetypes 2 indexes and 1 sourcetype 1 index and 1 sourcetype Or stick with our current 2 indexes and 2 sourcetypes ? Thanks for any ideas, Mark. ... View more

Hi all, I currently have a scheduled search that runs every minute and filters certain events for the previous minute and then creates a summary index. However, I got to wondering whether this would be better done within the transforms.conf using REGEX and DEST? The only problem that I would have is that the events would be 'split' over the two indexes. What I would, ideally, like to achieve is - all events go to index 1 and REGEX filtered events go to index 1 AND index 2. Is this at all possible? Would another approach be to create multiple Monitor stanzas for the same log file in my universal forwarder inputs.conf to send ALL the events to the two indexes and use transforms.conf to send any events that I don't want in index 2 to a null index? Thanks for any advise, Mark. ... View more

In my defense - it's been a really long day and I apologies if this is the most simple question...... I have a search, a transaction and a few field renames (for readability) and a table. e.g. <search> | transaction Session_ID |eval LogonTime=strftime(_time,"%Y/%m/%d - %H:%M:%S") | rename Logon_Username AS "Account Name", ip_address AS "Source IP", mac_address AS "Network MAC" | table LogonTime, "Account Name", "Source IP", "Network MAC" The problem that I'm struggling with is that I want to filter the search/results so that it only returns results where the count of logon_username > 3 ( or any other number!) and I just can't figure out the way to count and filter the events and keep all the other fields for my table. Any help will be gratefully appreciated and will help me sleep tonight. 🙂 🙂 Mark. ... View more

Hi all, I've got a simple search and filter that gets piped into the collect command to create a Summary index. I've saved it as a report, scheduled it to run every hour on one of my, non-clustered search heads. The data is being extracted from an index that is on two, load balanced indexers. Everything is fine and, for this particular search, subsequent search times have dramatically reduced. My question is, would the scheduled report be better run from the two indexers and spread the summary index load over the two, or must it be stored on the search head? If I can run the report on indexer1, will it pick up the indexed data from indexer2? Or can/should I run it on both? Sorry, I just thought of a new question! If I run the report on a 'don't index and forward' Search head, will it automatically send the summary index data to the indexers? Sorry for so many questions but your help, as always, is gratefully received. Mark. ... View more

Hi, Another regex problem I'm afraid..... I've got a very long event with 37 fields where all the fields are quoted and separated by comma. Also there are no key=value pairs. For the most part my regex works nicely with the event data, but there are occasions where a quote also appears in the actual field data thereby breaking my regex separator character. Working example (extremely simplified regex and event): ^"(?P<dest_ip>[^"]+)","(?P<dest_port>[^"]+)","(?P<uri>[^"]+)","(?P<request>[^"]+)","(?P<response>[^\n]+)"$ Data: "192.0.0.20","80","fl=city,name,code,group=true&group.field=city","GET /solr/lpbm/select?fl=city","Logging rate limit reached" No problem with this, all the fields parse out OK. However, this next event fails - note the additional " in fourth field:- "192.0.0.20","80","fl=city,name,code,group=true&group.field=city","GET /solr/"lpbm"/select?fl=city","Logging rate limit reached" This now breaks the [^"]+)"," part of my regex and distorts the field extractions. Is there a way to do the equivalent of:- ......","(?P<request>[^","]+)","....... I know that this is invalid, but I don't know what the alternative looks like 😞 !! Thanks for any help, Mark. ... View more