All Posts

tscroggins · ‎11-19-2023

Hi @Oti47, Edit: This answer applies to Simple XML dashboards. Dashboard Studio may be limited to a single category field, as @richgalloway noted. Edit 2: Snap. The Dashboard Studio scatter visuali...

Hi @Oti47, Edit: This answer applies to Simple XML dashboards. Dashboard Studio may be limited to a single category field, as @richgalloway noted. Edit 2: Snap. The Dashboard Studio scatter visualization is limited to x, y, and category fields in that order. The search fragment for the scatter chart visualization provides a hint: | stats x_value_aggregation y_value_aggregation by name_category [comparison_category] If you're using a report, output from the inputlookup or table command, etc., make sure the fields are in name_category, comparison_category, x_value_aggregation, y_value-aggregation order by applying the stats command: | stats values(category) as category values(value) as value by PartnerId article where PartnerId and article are categorical values and category and value are numerical values. You can re-order the aggregation fields, category and value, and the split-by fields, PartnerId and article, as needed for your intended display: | stats values(value) as value values(category ) as category by article PartnerId Reformatting the output with the stats command adds the user interface field metadata (groupby_rank) used by the visualization to identify the name_category ("groupby_rank": 0) and comparison_category ("groupby_rank": 1) fields. As a rule of thumb, the commands referenced in a visualization's search fragment will produce the desired result. Visualizations that reference chart, stats, timechart, xyseries, etc. most likely use internal metadata to format their output. You can then use drilldown tokens associated with the data. See <https://docs.splunk.com/Documentation/Splunk/latest/Viz/PanelreferenceforSimplifiedXML#Predefined_drilldown_tokens>. $click.value$ should represent the scatter chart name_category value, and $click.value2$ should represent the scatter chart comparison_category value.

m_zandinia · ‎11-19-2023

Yes. That is correct. I forgot about how I must manage the new indexes.conf and push it because I faced many issues when the indexes.conf on indexers doesn't match with indexes.conf on CM. I'll be v...

Yes. That is correct. I forgot about how I must manage the new indexes.conf and push it because I faced many issues when the indexes.conf on indexers doesn't match with indexes.conf on CM. I'll be very grateful If also @woodcock gives me an advice

jialiu907 · ‎11-19-2023

Yes I did not round the max after eventstats and I am able to post stats without rounding. I tested the _raw data from earlier and it does work with this search, showing the min, max and avg properl...

Yes I did not round the max after eventstats and I am able to post stats without rounding. I tested the _raw data from earlier and it does work with this search, showing the min, max and avg properly. | makeresults | eval data=mvappend("{\"dsnames\": [\"read\", \"write\"], \"values\": [123, 234]}", "{\"dsnames\": [\"read\", \"write\"], \"values\": [456, 567]}") | mvexpand data | rename data as _raw | spath | eval data = mvappend(json_object("dsname", mvindex('dsnames{}', 0), "value", mvindex('values{}', 0)), json_object("dsname", mvindex('dsnames{}', 1), "value", mvindex('values{}', 1))) | mvexpand data | spath input=data | stats min(value) as min max(value) as max avg(value) as avg by dsname | eval min=round(min, 2) | eval max=round(max, 2) | eval avg=round(avg, 2) The other way to sort and compare with max does give me results. {"dsname":"read","value":"0"} {"values":[0,23347.1366453364],"dstypes":["derive","derive"],"dsnames":["read","write"],"time":1700387069.996,"interval":10.000,"host":"usorla7sp103x.ad101.siemens-energy.net","plugin":"disk","plugin_instance":"dm-0","type":"disk_octets","type_instance":""} I am still not sure why Max would still be the same as those values should be different just on the basis that the "maximum number of disk operations or disk time for operations or disk traffic" should be different for read and written data, logically speaking.

PickleRick · ‎11-19-2023

1. Why do the | eval dest=lower(dest) ? CIM is for normalizing your data. Do it properly - unify the case of your names. 2. if(isnotnull ... can be expressed more clearly with coalesce(). 3. You're...

1. Why do the | eval dest=lower(dest) ? CIM is for normalizing your data. Do it properly - unify the case of your names. 2. if(isnotnull ... can be expressed more clearly with coalesce(). 3. You're searching from CIM indexes but you're manually doing things like | rename computerDnsName as dest. You should have done that as calculated field to make your data CIM-compliant. So first you should make your data CIM-compliant then tell us what you want to achieve.

PickleRick · ‎11-18-2023

I'm not sure if there was any modification to the copy-pasted config and/or events but your regex doesn't allow for spaces between the semicolon after the key name and the value.

PickleRick · ‎11-18-2023

Moving indexes on a working cluster is a tricky thing to do since: 1) You have to physically move the data 2) You have to push the indexes.conf from the CM 3) The indexes.conf has to be consistent...

Moving indexes on a working cluster is a tricky thing to do since: 1) You have to physically move the data 2) You have to push the indexes.conf from the CM 3) The indexes.conf has to be consistent across the whole cluster. So the issue is very tricky and I'd do a lot of testing before attempting it on prod environment. You could get away with taking the whole cluster down, moving the data around physically and deploying "fixed" indexes.conf both on the CM and on each individual indexer. But again - testing, testing, testing. There are many things that could go wrong here. Generally, the best practice would be to leave those indexes alone and don't move them around - if there is a _new_ requirement, just create a new index on a new storage unit, set the proper size/age constraints and stick with it.

yuanliu · ‎11-18-2023

That's impossible. Mathematically, it is extremely improbable for any real event to have avg value. But min and max must be there. Are you sure you didn't round the max after eventstats as in your...

That's impossible. Mathematically, it is extremely improbable for any real event to have avg value. But min and max must be there. Are you sure you didn't round the max after eventstats as in your initial code? Can you post stats without rounding? index="collectd_test" plugin=disk type=disk_octets plugin_instance=dm-0 | eval data = mvappend(json_object("dsname", mvindex('dsnames{}', 0), "value", mvindex('values{}', 0)), json_object("dsname", mvindex('dsnames{}', 1), "value", mvindex('values{}', 1))) | mvexpand data | spath input=data | stats min(value) as min max(value) as max avg(value) as avg by dsname Another way to examine data is to sort and compare with max. index="collectd_test" plugin=disk type=disk_octets plugin_instance=dm-0 | eval data = mvappend(json_object("dsname", mvindex('dsnames{}', 0), "value", mvindex('values{}', 0)), json_object("dsname", mvindex('dsnames{}', 1), "value", mvindex('values{}', 1))) | mvexpand data | spath input=data | table data _raw | sort - value | eventstats min(value) as min max(value) as max avg(value) as avg by dsname The first row should match one of max. You can also make a comparison of values in data with those in _raw directly.

gcusello · ‎11-18-2023

Hi @SplunkDash, you can use REPORT i you have a list of fields separated by comma or another char. In your case I'd use a regex in props.conf like the following EXTRACT-your_sourcetype = ^(?<time...

Hi @SplunkDash, you can use REPORT i you have a list of fields separated by comma or another char. In your case I'd use a regex in props.conf like the following EXTRACT-your_sourcetype = ^(?<timestamp>\d+-\d+-\d+T\d+:\d+:\d+\.\d+\w)\s+(?<host>[^ ]+)\s+(?<user>[^:]+):\s+UserID:\s+\'(?<UserID>[^\']+)\',\s+UserType:\s+\'(?<UserType>[^\']+)\',\s+System:\s+\'(?<System>[^\']+)\',\s+UAT:\s+\'(?<UAT>[^\']+)\',\s+EventType:\s+\'(?<EventType>[^\']+)\',\s+EventID:\s+\'(?<EventID>[^\']+)\',\s+Subject:\s+\'(?<Subject>[^\']+)\',\s+Scode:\s+\'(?<Scode>[^\']*)\'\s+EventStatus:\s+(?<EventStatus>\d*),\s+TimeStamp:\s*\'(?<TimeStamp>[^\']*)\',\s+Device:\s*\'(?<Device>[^\']*)\',\s+Msg:\s*\'(?<Msg>[^\']*)\',\s+var:\s*\'(?<var>[^\']*)\' You can test the regex at https://regex101.com/r/iQZi9K/1 Ciao. Guseppe

m_zandinia · ‎11-18-2023

Hi Splunkers Currently, I have 8 indexers and about 100 indexes! Here is a sample of my indexes.conf: # volumes [volume:HOT] path = /Splunk-Storage/HOT maxVolumeDataSizeMB = 2650000 [volume:COL...

Hi Splunkers Currently, I have 8 indexers and about 100 indexes! Here is a sample of my indexes.conf: # volumes [volume:HOT] path = /Splunk-Storage/HOT maxVolumeDataSizeMB = 2650000 [volume:COLD] path = /Splunk-Storage/COLD maxVolumeDataSizeMB = 27500000 ### indexes ### [testindex1] repFactor = auto homePath = volume:HOT/testindex1 coldPath = volume:COLD/testindex1 thawedPath = /Splunk-Storage/COLD/testindex1/thaweddb summaryHomePath = /Splunk-Storage/HOT/testindex1/summary frozenTimePeriodInSecs = 47520000 [testindex2] repFactor = auto homePath = volume:HOT/testindex2 coldPath = volume:COLD/testindex2 thawedPath = /Splunk-Storage/COLD/testindex2/thaweddb summaryHomePath = /Splunk-Storage/HOT/testindex2/summary frozenTimePeriodInSecs = 47520000 I don't restrain my indexes by size, only by time. The current median age of all data is about 180 days. Regarding my fstab file: /dev/mapper/mpatha-part1 /Splunk-Storage/HOT xfs defaults 0 0 /dev/mapper/mpathb-part1 /Splunk-Storage/COLD xfs defaults 0 0 Now, for compliance reasons, I want to separate two of my indexes to preserve them for a longer duration (at least two years). I have considered two possible methods to accomplish this: 1. a. Create a different path and volume. b. Stop all indexers. c. Move the two indexes to the new path. d. Start all indexers. If I'm correct, the issue is that I can't move just two indexes because I didn't mount different paths in the OS. Therefore, I would have to move all other indexes to another path. Essentially, this means creating two paths and volumes in both my OS and indexes.conf. 2. a. Decrease the frozenTimePeriod for all indexes except the two to, for example, 150 days. b. Wait for Splunk to free up some disk space. c. Increase the frozenTimePeriod for those two indexes to, for example, 730 days. The second solution may seem more straightforward, but I'm uncertain if it is a best practice or a good idea at all. Could you please guide me on how to implement the first solution with minimal downtime? Thank you in advance for your assistance!

tscroggins · ‎11-18-2023

That got me thinking about generating padded strings of arbitrary length. These use a hash mark for clarity: | makeresults | eval length=100 | eval pad=substr(replace(tostring(round(0, length)),...

That got me thinking about generating padded strings of arbitrary length. These use a hash mark for clarity: | makeresults | eval length=100 | eval pad=substr(replace(tostring(round(0, length)), ".", "#"), 0, length) | makeresults | eval length=100 | eval pad=mvjoin(mvmap(mvrange(0, length, 1), "#"), "") I'd love the convenience of an eval function similar to the classic BASIC SPACE$ and STRING$ functions or something more powerful like a regular expression-based string generator.

tscroggins · ‎11-18-2023

Hi @yuanliu, I would normally use a transpose-sort-transpose pattern for custom column sorting; however, trellis needs the field metadata (name, data_source, splitby_field, and splitby_value) provid...

Hi @yuanliu, I would normally use a transpose-sort-transpose pattern for custom column sorting; however, trellis needs the field metadata (name, data_source, splitby_field, and splitby_value) provided by chart, timechart, and xyseries. To force trellis to sort columns without modifying their appearance, we can exploit Split By display behavior. Trellis trims whitespace from field names at display time, so we can untable the timechart result, sort events as needed, pad the aggregation field value with a number of spaces equal to the sort position, and use xyseries to maintain the metadata trellis requires. | untable _time aggregation value | rex field=aggregation "(?<field>[^:]+): (?<clientip>.+)" | sort 0 - field clientip | streamstats count ``` 40 padding spaces in this example ``` | eval aggregation=substr(" ", 0, count).aggregation | rename aggregation as clientip | xyseries _time clientip value

SplunkDash · ‎11-18-2023

Hello, I have some issues to perform field extractions using transform configuration. It's not giving field value pairs as expected. Sample events and configuration files are given below. Some non-u...

Hello, I have some issues to perform field extractions using transform configuration. It's not giving field value pairs as expected. Sample events and configuration files are given below. Some non-uniformities within the events are also marked in Bold. Any recommendations will be highly appreciated. Thank you so much. My Configuration Files [mypropfConf] REPORT-mytranforms=myTransConf [myTransConf] REGEX = ([^"]+?):'([^"]+?)' FORMAT = $1::$2 Sample Events 2023-11-15T18:56:29.098Z OTESTN097MA4515620 TESTuser20248: UserID: '90A', UserType: 'TempEMP', System: 'TEST', UAT: 'UTA-True', EventType: 'TEST', EventID: 'Lookup', Subject: 'A5367817222', Scode: '' EventStatus: 0, TimeStamp: '2023-11-03T15:56:29.099Z', Device: 'OTESTN097MA4513020', Msg: 'lookup ok', var: 'Sec' 2023-11-15T18:56:29.021Z OTESTN097MB7513020 TESTuser20249: UserID: '95B', UserType: 'TempEMP', System: 'TEST', UAT: 'UTA-True', EventType: 'TEST', EventID: 'Lookup', Subject: 'A516670222', Scode: '' EventStatus: 0, TimeStamp: '2023-11-03T15:56:29.099Z', Device: 'OTESTN097MA4513020', Msg: 'lookup ok', var: 'tec' 2023-11-15T18:56:29.009Z OTESTN097MB9513020 TESTuser20248: UserID: '95A', UserType: 'TempEMP', System: 'TEST', UAT: 'UTA-True', EventType: 'TEST', EventID: 'Lookup', Subject: 'A546610222', Scode: '' EventStatus: 0, TimeStamp: '2023-11-03T15:56:29.099Z', Device: 'OTESTN097MA4513020', Msg: 'lookup ok', var: 'test'

jialiu907 · ‎11-18-2023

I tried to input your search to examine actual data and the results are coming back as 0 events matched and i tried it for min and avg as well, changing "| where value == max" to "| where value == mi...

I tried to input your search to examine actual data and the results are coming back as 0 events matched and i tried it for min and avg as well, changing "| where value == max" to "| where value == min" or "| where value == avg".

yuanliu · ‎11-18-2023

Given that min, max, avg operated on the same field, unless all three give the same value for both groupby values, the only conclusion is that max for both is the same. You can examine actual data. ...

Given that min, max, avg operated on the same field, unless all three give the same value for both groupby values, the only conclusion is that max for both is the same. You can examine actual data. | eval data = mvappend(json_object("dsname", mvindex('dsnames{}', 0), "value", mvindex('values{}', 0)), json_object("dsname", mvindex('dsnames{}', 1), "value", mvindex('values{}', 1))) | mvexpand data | spath input=data | eventstats min(value) as min max(value) as max avg(value) as avg by dsname | where value == max This will show you events that matches max.

north_star · ‎11-18-2023

I have tried to simplify the query for better understanding and removing some unnecessary things. This query is to find out if the same malware has been found on more than 4 hosts (dest) in a give...

I have tried to simplify the query for better understanding and removing some unnecessary things. This query is to find out if the same malware has been found on more than 4 hosts (dest) in a given time span, something like a malware outbreak. Below is the indexed based query that works fine. I am trying to convert this to a data model based query, but not getting the desired results. I am new to writing data model based queries. Thanks for all the help! (`cim_Malware_indexes`) tag=malware tag=attack | eval grouping_signature=if(isnotnull(file_name),signature . ":" . file_name,signature) => trying to create a new field called "grouping_signature" by concatenating signature and file_name fields | stats count dc(dest) as infected_device_count BY grouping_signature => trying to calculate the distinct count of hosts the have the same malware found on them by "grouping_signature" field | where infected_device_count > 4 => trying to find events where number of infected devices is greater than 4 | stats sum(count) AS "count" sum(infected_device_count) AS infected_device_count BY grouping_signature => trying to find the total number of infected hosts by "grouping_signature" field

PickleRick · ‎11-18-2023

Was just about to write that if you unpacked it with sudo, you could get mismatched ownership and permission issues. But apparently you got it on your own. Have fun with your searches

dablab · ‎11-18-2023

I want to say that it was a permissions issue!!! Thanks all!

dablab · ‎11-18-2023

Thanks! So when I do cd /opt/splunk/etc/apps/ and then: sudo tar -xzf botsv1_data_set.tgz It will unload the data in the apps area, but then when I restart and go to search it there is noth...

Thanks! So when I do cd /opt/splunk/etc/apps/ and then: sudo tar -xzf botsv1_data_set.tgz It will unload the data in the apps area, but then when I restart and go to search it there is nothing there? I have all the apps download etc.

jeffland · ‎11-18-2023

For anyone interested, there is an idea for this (current status is "Future Prospect"): https://ideas.splunk.com/ideas/EID-I-486.

isoutamo · ‎11-18-2023

Here is some free eLearnigs which covers basic Splunk stuff. https://education.splunk.com/Saba/Web_spf/NA10P2PRD105/app/shared;spf-url=common%2Fsearchresults%2Fxxemptyxx%2FLEARNINGEVENT%252COFFERINGT...

Here is some free eLearnigs which covers basic Splunk stuff. https://education.splunk.com/Saba/Web_spf/NA10P2PRD105/app/shared;spf-url=common%2Fsearchresults%2Fxxemptyxx%2FLEARNINGEVENT%252COFFERINGTEMPLATE%252CCERTIFICATION%252CCURRICULUM%252CPLAYLIST%252COFFERING%252CPACKAGE%252CLXPCONTENT%252CLEARNINGPATHWAY%3Fadvsearch%3Dtrue&categoryId%3Dcateg000000000003041 r. Ismo

All Posts

All Posts

Re: How to read a fourth value from scatter plot?

Re: Move some indexes to separate volume

Re: How to extract specific field values from JSON format events?

Re: Converting index query to data model query

Re: Field Extraction Using Tranforms Configration

Re: Move some indexes to separate volume

Re: How to extract specific field values from JSON format events?

Re: Field Extraction Using Tranforms Configration

Move some indexes to separate volume

Re: How to fix wandering trellis order?

Re: How to fix wandering trellis order?

Field Extraction Using Tranforms Configration

Re: How to extract specific field values from JSON format events?

Re: How to extract specific field values from JSON format events?

Converting index query to data model query

Re: I cant get Splunk to read Boss Of SOC v1 Data Set

Re: I cant get Splunk to read Boss Of SOC v1 Data Set

Re: I cant get Splunk to read Boss Of SOC v1 Data Set

Re: Should Metrics Indexes support overwriting events instead of duplicating events

Re: Heavy Forwarder Instance

All Posts

Are you a member of the Splunk Community?

All Posts