All Posts

Top

All Posts

@yuanliu wrote: This ask could have two interpretations.  The simple one is extremely simple.  Let me give you the formula first.   | inputlookup pod_name_lookup where NOT [search index=ab... See more...
@yuanliu wrote: This ask could have two interpretations.  The simple one is extremely simple.  Let me give you the formula first.   | inputlookup pod_name_lookup where NOT [search index=abc sourcetype=kubectl | eval pod_name = mvindex(split(pod_name, "-"), 0) | stats values(pod_name) as pod_name] | stats dc(pod_name) as count values(pod_name) as pod_name by importance   This query gets me really close. The one edge case I did not bring up is that some pods have multiple parts of the expected name that are also split by dashes. For example, I would have this in the lookup: podd-unique-name critical   and need to match podd-unique-name-h98erg-n2439f Running critical from the results.   Yes the "importance" in both will match exactly, but it is only important in the lookup field. The goal of this is to display pods that are not found in the search results compared to the inputlookup, and using the "importance" field from the lookup display the missing pod name and importance.
Something in my solution is not right. It works for only one condition (one or another) but combined produced zero events --------- Events reported  ----------- index=firewall (sourcetype=coll... See more...
Something in my solution is not right. It works for only one condition (one or another) but combined produced zero events --------- Events reported  ----------- index=firewall (sourcetype=collector OR sourcetype=metadata) enforcement_mode=block |table event_type, hostname, ip ------------- Events reported ----------- index=firewall (sourcetype=collector OR sourcetype=metadata) event_type="error" |table event_type, hostname, ip ------------ No events reported index=firewall (sourcetype=collector OR sourcetype=metadata) enforcement_mode=block event_type="error" |table event_type, hostname, ip
@sjringo - what should be my trigger condition ?  Also how your query will identify which date as I don't want alert not to be triggered everyday from 21:00 to 4:00 am. I want just specific date whi... See more...
@sjringo - what should be my trigger condition ?  Also how your query will identify which date as I don't want alert not to be triggered everyday from 21:00 to 4:00 am. I want just specific date which is going to be 23rd april from 9 pm to 24th april 4 am   
Oh, I have put a lot of information about it like the example I gave. I have put the search query, an example of an event, the alert configuration, etc. They are events ingested by the Microsoft secu... See more...
Oh, I have put a lot of information about it like the example I gave. I have put the search query, an example of an event, the alert configuration, etc. They are events ingested by the Microsoft security API, coming from the Defender, and the queries are basic, if the title of the events is x, it is triggered. It is already desperation, because if you run the search normally, it detects the event it should but the alert has not been generated. So the only option I can think of is the indexing time, but I understand that if the search runs every 5 minutes and searches the entire previous hour, there should be no problem and there still is. These alerts are very important to me, and they must appear no matter what. In the example I mentioned at the beginning: TimeIndexed = 2024-04-04 01:01:59 _time=04/04/2024 00:56:08.600
Assuming Invetory is spelled (in)correctly, you could try this - the rex at the end is required because this date has an embedded space and it is the last field in the message | makeresults | eval ... See more...
Assuming Invetory is spelled (in)correctly, you could try this - the rex at the end is required because this date has an embedded space and it is the last field in the message | makeresults | eval _raw="{\"id\":\"0\",\"severity\":\"Information\",\"message\":\"CPWTotal=749860, SEQTotal=1026137, EASTotal=1062804, VRSTotal=238, CPWRemaining=5612, SEQRemaining=32746, EASRemaining=15, VRSRemaining=0, InvetoryDate=4/16/2024 7:34:25 PM\"}" | spath | rename message as _raw | extract | rex "InvetoryDate=(?<InvetoryDate>.*)" If the fields were re-ordered or an extra field was in the message (without an embedded space),  then the rex would not be required | makeresults | eval _raw="{\"id\":\"0\",\"severity\":\"Information\",\"message\":\"CPWTotal=749860, SEQTotal=1026137, EASTotal=1062804, VRSTotal=238, CPWRemaining=5612, SEQRemaining=32746, EASRemaining=15, VRSRemaining=0, InvetoryDate=4/16/2024 7:34:25 PM, Tail=True\"}" | spath | rename message as _raw | extract
The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest metrics, traces, and logs to the Splunk platform using an HEC. If you are a DevOps Engineer or... See more...
The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest metrics, traces, and logs to the Splunk platform using an HEC. If you are a DevOps Engineer or SRE, you may already be familiar with the OTel Collector’s flexibility, but for those less experienced, this blog post will serve as an introduction to routing logs.  The idea of OpenTelemetry as a whole is to unify the data so it's suitable for every input and output and put some processors in between to make it possible to perform operations on data (such as transforming and filtering). You may already see that one of the biggest advantages of OTel Collector is its flexibility - but sometimes figuring out how to use it in practice is a challenge. One of the most common cases in log processing is setting up the event’s index. If you’re familiar with the Splunk HEC receiver, you might recall this configuration snippet: This indicates that every event used by this exporter will be sent to the logs index.  As you may see, the logs index is specific to an exporter, so the intuition is to create as many splunk_hec exporters as you need, and additionally create multiple filelog receivers as well, so that we can filter which files go to which index. Using your imagination, visualize a scenario where all the logs go to the ordinary logs index, but some are only visible to people with higher permissions levels. These logs are gathered by filelog/security receiver and the pipeline structure would look like this one: But is it really the best solution? Let’s consider a few questions here: splunk_hec exporter config seems to be the same, the only difference is the index field. Does it make sense to copy the configuration over and over? filelog receiver gives a way of configuring a place to gather logs. What about other filtering options, like ones based on severity or a specific phrase in the log’s body? Everytime we create a new pipeline, a new process comes to life - doesn’t this consume too many resources? The solution: Dynamic index routing Today I’ll show you how to create a pipeline with dynamic index routing, meaning it is based on incoming logs and not statically set, with a transform processor and Splunk OpenTelemetry Collector for Kubernetes (SOCK). The idea is based on this attribute from Splunk HEC Exporter documentation: This means that we can specify com.splunk.index as a resource attribute for a log, and it will overwrite the default index. Let’s go through a few examples of how we can do it in SOCK. Viewing the pipelines configured in SOCK Before we cover how to overwrite your pipelines, let’s start with how you can view the pipeline. The final config is the result of your configuration in values.yaml, as well as the default configuration that is delivered by SOCK. The config’s yaml file is in the pod’s configmap. As logs are generated by the agent, you can look at the agent’s config, the command is: Where my-splunk-otel-collector-otel-agent is the configmap’s name - it might differ in your case, especially if you chose a different name for an installation versus one from the Getting Started docs. You can take a look at a configmaps you have with the command: An output example for a default namespace would be: After successfully running the describe command, scroll all the way down until you see the pipelines section. For logs, it looks more or less like this: Now you know what components your logs pipeline is made of!  Easy scenarios Now let’s get our hands dirty! Let’s see the easy examples of index routing based on real scenarios. Scenarios based on the log attribute The scenario: Let’s say we want to pass all the events with a log.iostream attribute stderr to error_index This would capture events emitted to the error stream and send them to their own index. The solution: This requires doing two things: Overwriting the default agent config with a new transform processor and adding it to the logs pipeline.  Setting up the new processor’s statement to specify the correct target index. Every transform processor consists of a set of statements. We need to create one that matches our use case, by defining what we need and writing it specifically for OTel. The logical statement here would be: set com.splunk.index value to be error_index for EVERY log from the pipeline whose attribute log.iostream is set to stderr Then the statement in the transform processor’s syntax described here looks like this: Next, we need to append the processor to the logs pipeline. To do that, we need to copy and paste the current processors under the agent.config section then insert our processor at the end. The whole config will be: After applying the config, the stderr events appear in the error_index: Scenarios based on specific log text The scenario Passing an event to a different index when something specific is written in the body of the log, for example, every log that contains [WARNING]: The solution All the keywords used here come from the transform processor documentation. We can use the  transform processor, this time using the following logic: Here are some sources that can be used to learn more about OpenTelemetry Transformation Language and its grammar. Then we repeat the steps described in the previous solutions section. The final config is: And the result in the Splunk Enterprise looks like this: How do I know what attributes I can use? At this point, you might think “Oh right, that looks easy, but how would I know what attributes to use?” The logs in the transform processor can use all the elements described here, but the most useful ones are: body - referring to the log body attributes - referring to the attributes specific to a single log resource.attributes - referring to the attributes specific to multiple logs from the same source You can see them in the Splunk Enterprise event preview: However, there’s no indication as to which dimensions are attributes and which are resource.attributes. You can see how it looks by running  your OTel agent with this config: This will produce all the information about log structure and which attributes are really the resource.attributes: From this snippet, you can see that only logtag and log.iostream are attributes, all the rest are part of the resource.attributes. The transform processor has many options aside from the ones described above, check them out here.  Complex Scenarios Let’s go even deeper and operate on two variables instead of one. Scenarios based on setting index-based annotations You may want to annotate the whole namespace with one splunk.com/index, but want specific pods from this namespace to redirect somewhere else. You can do this by using a transform processor to provide additional annotations to the pod of your choice. Let’s say the annotation is second_index. This is how it looks in kubectl describe of the pod: Transforming annotation into resource.attribute First, redirect logs from the pods according to the  second_index annotation to convert the annotation to a resource.attribute. This can be done with extraAttributes.fromAnnotations config:   tag_name is the identifier of an element in resource.attributes, it is optional. If you don’t configure it your attribute will look like this: k8s.pod.annotations.<key>is the output format. With tag_name you can decide how the name of your attribute, in this example it is the same as the key: Make OTel pass logs to the index Now that we have resource.attribute second_index set up, we can set the index destination for logs. We will use transform processor for this purpose: We will replace the com.splunk.index resource attribute with the second_index attribute, but only when the second_index attribute is present - so it doesn’t affect logs from other pods. Delete unnecessary attributes Once the attribute has been moved to the log's index, we can get rid of it. This requires adding another statement to the transform processor: Scenarios based on labels setting the index  This will work exactly the same as an annotation example from the previous section, the only difference is in how we’re transforming the label into resource.attribute. We now have the  second_index label on a pod: We can make it visible to the OTel collector with this config snippet: Conclusion In this article, I showed you how to route logs to different indexes. It is a commonly used feature and it can be used in many scenarios, as we can see in the examples. We will expand on other SOCK features in later articles, so stay tuned!
Thank you so much for prompt reply. Below is the fixed format of the data. Please help me on this.  {"id":"0","severity":"Information","message":"CPWTotal=749860, SEQTotal=1026137, EASTotal=1062804,... See more...
Thank you so much for prompt reply. Below is the fixed format of the data. Please help me on this.  {"id":"0","severity":"Information","message":"CPWTotal=749860, SEQTotal=1026137, EASTotal=1062804, VRSTotal=238, CPWRemaining=5612, SEQRemaining=32746, EASRemaining=15, VRSRemaining=0, InvetoryDate=4/16/2024 7:34:25 PM"}  Need to extract fields in below format. Your help really appreciated.   CPW Total SEQ Total EAS Total VRS Total CPW Remaining SEQ Remaining EAS Remaining VRS Remaining InvetoryDate 844961 244881 1248892 238 74572 22 62751 0 4/15/2024 6:16:07 AM
@dc17 - I'm not sure what logs you are trying to find in the EventViewer. Is it any known Application logs are you trying to find??
So as I understand,   | append [     | inputlookup server_info.csv     | rename customer as customer_name     | stats values(host) as hostnames by customer_name ] is going to append the hostnam... See more...
So as I understand,   | append [     | inputlookup server_info.csv     | rename customer as customer_name     | stats values(host) as hostnames by customer_name ] is going to append the hostnames from lookup into the results received from the first query. Finally, | stats count by customer_name hostnames  is going to count the value as 1 if it's present only in lookup, otherwise, if it's present in the first part of search and lookup then the count is going to be evaluated as 2? Is that correct? However, in the result there are no values with count as 2. Which is unlikely as there are few hosts which are present in events and lookup as well. Here, we try to fetch the events that contains hostnames(configured to receive application logs) and then compare them with the list of servers present in lookup(if found in lookup only then count=1).  However it seems that the query still isn't performing the required search as there are no values with count as 2. Here I observer count is returned as 1 for the hosts that are received from the events in below index=application sourcetype="application:server:log" | stats values(host) as hostnames by customer_name There is almost no trace of values in the lookup, so I'm not sure if they are even being compared. And this is what the issue earlier was.   How can these two lists be compared and listed?
Hi Hardik, You can use this ADQL as a simple SELECT `wait-state-id`, (`wait-state-id`) FROM dbmon_wait_time This query gathers data based on "wait state id" but you can change this query based on ... See more...
Hi Hardik, You can use this ADQL as a simple SELECT `wait-state-id`, (`wait-state-id`) FROM dbmon_wait_time This query gathers data based on "wait state id" but you can change this query based on "wait state name"  For Example wait state id 59 = Using CPU Thanks Cansel
Try this: | tstats count where index=cts-dcpsa-app sourcetype=app:dcpsa host_ip IN (xx.xx.xxx.xxx, xx.xx.xxx.xxx) by host | eval current_time=strftime(now(), "%H%M") | eval is_maintenance_window=if... See more...
Try this: | tstats count where index=cts-dcpsa-app sourcetype=app:dcpsa host_ip IN (xx.xx.xxx.xxx, xx.xx.xxx.xxx) by host | eval current_time=strftime(now(), "%H%M") | eval is_maintenance_window=if(current_time >= 2100 AND current_time < 0400, 1, 0) | eval is_server_down=if(count == 0, 1, 0) | where is_maintenance window = 0 AND is_server_down=1
Your sample data is inconsistently formatted, e.g. sometimes there is a space before/after the =/, Please confirm the exact pattern your data will take so we don't waste effort on invalid data.
Adding to this in case anyone else is having this issue.  It seems like when Python is executed something is attempting to write to /tmp which ends up with a memory error when /tmp is mounted with no... See more...
Adding to this in case anyone else is having this issue.  It seems like when Python is executed something is attempting to write to /tmp which ends up with a memory error when /tmp is mounted with noexec.   Our solution was to add TMPDIR=<writable path> to splunk-launch.conf. 
Since you appear to have a one-to-one relationship between label and index, just include both in the by clause <query>index IN ({aws_stack02_p,aws_stack01_p,aws_stack01_n}) | eval label = case(index... See more...
Since you appear to have a one-to-one relationship between label and index, just include both in the by clause <query>index IN ({aws_stack02_p,aws_stack01_p,aws_stack01_n}) | eval label = case(index == "aws_stack02_p", "Stack1",index == "aws_stack01_p", "Stack2",index == "aws_stack01_n", "Stack3") |stats count by label,index</query>
In order to do calculations or meaningful comparisons with dates and times, they need to be converted (parsed) to unix-style timestamps. | eval datetime_unix=strptime(DATETIME, "%F %T") | eventstats... See more...
In order to do calculations or meaningful comparisons with dates and times, they need to be converted (parsed) to unix-style timestamps. | eval datetime_unix=strptime(DATETIME, "%F %T") | eventstats max(datetime_unix) as last_datetime | where datetime_unix == last_datetime | stats count by market_code
Firstly, join is not a very friendly command, it has its quirks. In this case I'd rather use either append or inputlookup append=t Another thing - if you do mvexpand on multiple multivalued fields y... See more...
Firstly, join is not a very friendly command, it has its quirks. In this case I'd rather use either append or inputlookup append=t Another thing - if you do mvexpand on multiple multivalued fields you'll get a product of both sets of values. It can escalate quickly for bigger data sets. See the run-anywhere example | makeresults | eval a=split("1,2,3,4,5",",") | eval b=split("1,2,3,4,5",",") | mvexpand a | mvexpand b Of course you can do some fancy set arithmetics on those multivalued fields but it's usually easier done another way - count and filter. This part is OK, it lists which customers use which hosts index=application sourcetype="application:server:log" | stats values(host) as hostnames by customer_name You're gonna get a list of hosts per customer. What is important here is that each host will be listed only once per customer. So we expand our list by the servers defined for each customer | append [     | inputlookup server_info.csv     | rename customer as customer_name     | stats values(host) as hostnames by customer_name ] So now for each customer you have a single value of hostnames field per customer per each host. Nothing easier now than counting what we have | stats count by customer_name hostnames So for each pair of customer_name and hostnames values you will have a count value indicating whether it was only present in the lookup (value of 1) or both in the lookup and the indexed events (value of 2). Now you can easily manipulate the data - filter, make a table, whatever. All this assumes that you don't have any hosts in the event data which are not in the lookup. If you can have such situation, the search is getting a bit more complex because you need to add additional field with two different numerical values, depending on whether the data came from the events or from the lookup and do a sum() instead of count in your final stats so you can see where the data came from.
You are not giving much away! You will need to do some digging! Which events are not being picked up? When do they occur and when do they get indexed? How do these times relate to your alert searches... See more...
You are not giving much away! You will need to do some digging! Which events are not being picked up? When do they occur and when do they get indexed? How do these times relate to your alert searches? How important are these missed alerts? How much effort do you want to spend finding these events?
index=firewall (sourcetype=collector OR sourcetype=metadata) (enforcement_mode=block OR event_type="error") |table event_type, hostname, ip
You haven't told us what you want the search to do so I'm only guessing. Probably your hosts log events which have either enforcement_mode=block field or event_type=error field but no single event ha... See more...
You haven't told us what you want the search to do so I'm only guessing. Probably your hosts log events which have either enforcement_mode=block field or event_type=error field but no single event has both of these fields set. So your "combined" search will not find them because both conditions aren't fulfilled in a single event. That's why you need to correlate multple events by using either transaction or stats (the stats approach is preferred due to transaction command's limitations).
Hi @sphiwee, as I said, I don't know very well Powershell, try it. Ciao. Giuseppe