All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Very strange, since also PickleRick says it's the normal behaviour of a DS not deleting custom addons
If an old admin account is deleted in a Splunk Enterprise distributed environment, any actions or related tasks associated with that account will be affected such as ( usecases, lookups, ...etc)
Hi @verbal_666 , As I said I had  a situation where apps installed on a Deployment Client (a UF) where removed because they weren't in the related ServerClass on the DS. Ciao. Giuseppe P.S.: Karm... See more...
Hi @verbal_666 , As I said I had  a situation where apps installed on a Deployment Client (a UF) where removed because they weren't in the related ServerClass on the DS. Ciao. Giuseppe P.S.: Karma Points are appreciated by all the contributors
For anyone reading along, this is an interesting exercise, but please do not store or log passwords in plain text. It's probably not even a good idea to log whether a particular user's password meets... See more...
For anyone reading along, this is an interesting exercise, but please do not store or log passwords in plain text. It's probably not even a good idea to log whether a particular user's password meets complexity requirements; that should be handled while the user is creating a password.
Hi @lynn140428, We should note the example isn't strictly JSON: "alert_data": {"domain": "abc.com", "csv": {"id": 12345, "name": "credentials.csv", "mimetype": "text/csv", "is_safe": true, "content... See more...
Hi @lynn140428, We should note the example isn't strictly JSON: "alert_data": {"domain": "abc.com", "csv": {"id": 12345, "name": "credentials.csv", "mimetype": "text/csv", "is_safe": true, "content": [{"username": "test@abc.com", "password":"1qaz@WSX#EDC"} The string should start with a left brace ({), and the objects and array should be properly closed: {"alert_data": {"domain": "abc.com", "csv": {"id": 12345, "name": "credentials.csv", "mimetype": "text/csv", "is_safe": true, "content": [{"username": "test@abc.com", "password":"1qaz@WSX#EDC"}]}}} We can validate with the json eval function. Note that quotation marks are escaped with a backslash within the string: | makeresults | eval _raw=json("\"alert_data\": {\"domain\": \"abc.com\", \"csv\": {\"id\": 12345, \"name\": \"credentials.csv\", \"mimetype\": \"text/csv\", \"is_safe\": true, \"content\": [{\"username\": \"test@abc.com\", \"password\":\"1qaz@WSX#EDC\"}") No results are returned. Let's correct the JSON and try again: | makeresults | eval _raw=json("{\"alert_data\": {\"domain\": \"abc.com\", \"csv\": {\"id\": 12345, \"name\": \"credentials.csv\", \"mimetype\": \"text/csv\", \"is_safe\": true, \"content\": [{\"username\": \"test@abc.com\", \"password\":\"1qaz@WSX#EDC\"}]}}}") We now have a valid JSON object in the _raw field, and we can use this object to test eval expressions that we'll apply later in a transform. You should correct the source data before proceeding further. Hint: You can correct the data at ingest using a simple eval expression and a transform similar to what I'm describing here. Testing the length of the password is straightforward: | eval length=len(json_extract(_raw, "alert_data.csv.content{}.password")) | eval is_password_meet_complexity=if(length >= 8, "Yes", "No") You haven't provided a list of special characters, but we'll assume they're drawn from the list of printable ASCII characters. Using PCRE character classes, we have: Numbers or digits: [[:digit:]] Uppercase letters: [[:upper:]] Lowercase letters: [[:lower:]] Punctuation characters: [[:punct:]] We can test the password against these to determine whether it contains a character matching the class: | makeresults | eval _raw=json("{\"alert_data\": {\"domain\": \"abc.com\", \"csv\": {\"id\": 12345, \"name\": \"credentials.csv\", \"mimetype\": \"text/csv\", \"is_safe\": true, \"content\": [{\"username\": \"test@abc.com\", \"password\":\"1qaz@WSX#EDC\"}]}}}") | eval length=len(json_extract(_raw, "alert_data.csv.content{}.password")) | eval digit=if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:digit:]]"), 1, 0) | eval upper=if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:upper:]]"), 1, 0) | eval lower=if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:lower:]]"), 1, 0) | eval punct=if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:punct:]]"), 1, 0) | eval is_password_meet_complexity=if(length >= 8 AND (digit + upper + lower + punct) >= 3, "Yes", "No") We can now combine the tests into a single expression: | makeresults | eval _raw=json("{\"alert_data\": {\"domain\": \"abc.com\", \"csv\": {\"id\": 12345, \"name\": \"credentials.csv\", \"mimetype\": \"text/csv\", \"is_safe\": true, \"content\": [{\"username\": \"test@abc.com\", \"password\":\"1qaz@WSX#EDC\"}]}}}") | eval is_password_meet_complexity=if(len(json_extract(_raw, "alert_data.csv.content{}.password")) >= 8 AND (if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:digit:]]"), 1, 0) + if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:upper:]]"), 1, 0) + if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:lower:]]"), 1, 0) + if(match(json_extract(_raw, "alert_data.csv.content{}.password"), "[[:punct:]]"), 1, 0)) >= 3, "Yes", "No") and use that expression to add the is_password_meet_complexity key to the object: | makeresults | eval _raw="{\"alert_data\": {\"domain\": \"abc.com\", \"csv\": {\"id\": 12345, \"name\": \"credentials.csv\", \"mimetype\": \"text/csv\", \"is_safe\": true, \"content\": [{\"username\": \"test@abc.com\", \"password\":\"1qaz@WSX#EDC\"}]}}}" | eval _raw=json_set(_raw, "alert_data.csv.content{0}.is_password_meet_complexity", if(len(json_extract(json(_raw), "alert_data.csv.content{}.password")) >= 8 AND (if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:digit:]]"), 1, 0) + if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:upper:]]"), 1, 0) + if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:lower:]]"), 1, 0) + if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:punct:]]"), 1, 0)) >= 3, "Yes", "No")) {"alert_data":{"domain":"abc.com","csv":{"id":12345,"name":"credentials.csv","mimetype":"text/csv","is_safe":true,"content":[{"username":"test@abc.com","password":"1qaz@WSX#EDC","is_password_meet_complexity":"Yes"}]}}} With "password":"NotComplex": {"alert_data":{"domain":"abc.com","csv":{"id":12345,"name":"credentials.csv","mimetype":"text/csv","is_safe":true,"content":[{"username":"test@abc.com","password":"NotComplex","is_password_meet_complexity":"No"}]}}} Finally, we can use the eval expression in the INGEST_EVAL setting of a transform: # props.conf [json_sourcetype] TRANSFORMS-password_complexity = password_complexity SEDCMD-password = s/\"password\"\:\s+\"\S{6}([^ ]*)/"password":"******\1/g # transforms.conf [password_complexity] INGEST_EVAL = _raw=json_set(_raw, "alert_data.csv.content{0}.is_password_meet_complexity", if(len(json_extract(json(_raw), "alert_data.csv.content{}.password")) >= 8 AND (if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:digit:]]"), 1, 0) + if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:upper:]]"), 1, 0) + if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:lower:]]"), 1, 0) + if(match(json_extract(json(_raw), "alert_data.csv.content{}.password"), "[[:punct:]]"), 1, 0)) >= 3, "Yes", "No")) Note that your sample includes a single object in the content array. If you have multiple objects in your array, we'll need to refactor the solution to accommodate them. Also note that we've used json(_raw) and json_extract(json(_raw), "alert_data.csv.content{}.password") repeatedly. There may be a way to optimize the expression and reduce the number of times the json and json_extract functions are called per event. Finally note that we don't have to use json functions to analyze the password. If "password":"value" only appears once and is well-formed, we can match against _raw directly; however, escaped quotes as in "password":"val\"ue" pose a challenge. I'll leave all of the above to you as an exercise.
@PickleRick keepin' it real.
Hi @sultanulariff, We can create a small lookup file containing the Bag Type values: | makeresults format=csv data="Bag Type Local Rush Transfer" | outputlookup bag_type_lookup.csv We can then add... See more...
Hi @sultanulariff, We can create a small lookup file containing the Bag Type values: | makeresults format=csv data="Bag Type Local Rush Transfer" | outputlookup bag_type_lookup.csv We can then add the inputlookup command to seed the event stream with all known Bag Type values: | makeresults format=csv data=" Date,Out Airline,Bag Type,Total Processed 01/05/2024,IX,Local,100 01/05/2024,IX,Transfer,120 02/05/2024,BA,Local,140 02/05/2024,BA,Transfer,160 03/05/2024,IX,Local,150" | appendpipe [ inputlookup append=true bag_type_lookup.csv ``` seed Bag Type values ``` | eventstats values("Bag Type") as "Bag Type" ``` get all possible values of Bag Type ``` | stats values("Bag Type") as "Bag Type" by Date "Out Airline" ``` get all combinations of Bag Type by Date and Out Airline ``` | mvexpand "Bag Type" ``` expand all combinations into separate events ``` ] | stats sum(eval(coalesce('Total Processed', 0))) as "Total Processed" by Date "Out Airline" "Bag Type" ``` sum Total Processed, replacing null values with 0 ``` Date Out Airline Bag Type Total Processed 01/05/2024 IX Local 100 01/05/2024 IX Rush 0 01/05/2024 IX Transfer 120 02/05/2024 BA Local 140 02/05/2024 BA Rush 0 02/05/2024 BA Transfer 160 03/05/2024 IX Local 150 03/05/2024 IX Rush 0 03/05/2024 IX Transfer 0   We used makeresults to provide test data. In production, we would optimize our base search to summarize results before the appendpipe command, e.g.: index=foo sourcetype=outbound_airline_bag_type_metrics | stats count as "Total Processed" by Date "Out Airline" "Bag Type" | appendpipe [ inputlookup append=true bag_type_lookup.csv | eventstats values("Bag Type") as "Bag Type" | stats values("Bag Type") as "Bag Type" by Date "Out Airline" | mvexpand "Bag Type" ] | stats sum(eval(coalesce('Total Processed', 0))) as "Total Processed" by Date "Out Airline" "Bag Type"  
Hi @tscroggins , there are only 3 possible values i.e. Bag_Type = "Local","Transfer","Rush".  The requirement is every flight for specific date shall have 3 rows with baggage volumes. If any row is ... See more...
Hi @tscroggins , there are only 3 possible values i.e. Bag_Type = "Local","Transfer","Rush".  The requirement is every flight for specific date shall have 3 rows with baggage volumes. If any row is missing from source data, I need to create the particular row with baggage volume as "0".
That is so wrong and so stupid. If you have a system where each user's home directory is mounted off of a CIFS server on logon (yes, I've seen such systems) should user's home be owned by root? And t... See more...
That is so wrong and so stupid. If you have a system where each user's home directory is mounted off of a CIFS server on logon (yes, I've seen such systems) should user's home be owned by root? And that's just one (yes, a bit extreme) example. If that's supposed to be a "best practice", question is "why". Also general "best practice" might not be the best practice for a given situation.
Hi @tscroggins , many thanks for the response. Looks perfect! Will need to try it with the actual data/query and reconfirm. 
Hi @PickleRick , there are three bag_types 1. Local 2. Transfer and 3. Rush From the actual source data, I do not get all three 'Total processed' baggage counts (i.e. there shall be 3 rows for ever... See more...
Hi @PickleRick , there are three bag_types 1. Local 2. Transfer and 3. Rush From the actual source data, I do not get all three 'Total processed' baggage counts (i.e. there shall be 3 rows for every single Airline/Date) every time. Hence, if any of the three bag_type is not there, I need to create a row for the missing Bag_type with Total_Processed=0. I use above index as base search query and left join further sub queries. For example, if 03/05/2024,IX,Transfer,0 is missing I am not able to join the sub queries related to 03/052024,IX,Transfer. So, I need it as a seperate/new row (if the row is missing)
A best practice for my customer isn't a best practice for your customer and vice versa, but in general, if you need to mount a separate file system, I recommend mounting it outside /opt/splunk and ei... See more...
A best practice for my customer isn't a best practice for your customer and vice versa, but in general, if you need to mount a separate file system, I recommend mounting it outside /opt/splunk and either updating Splunk configuration to reference the new location or using symlinks in the /opt/splunk directory tree. The mount point can be owned by root, but once mounted, the file system objects (including the root directory of the mounted file system) should be owned by the splunk user.
Hi @dokaas_2, Hints should be present in either the search or the search log. For example, if the inputlookup or lookup commands are used in the search, the cause may be the file referenced by the l... See more...
Hi @dokaas_2, Hints should be present in either the search or the search log. For example, if the inputlookup or lookup commands are used in the search, the cause may be the file referenced by the lookup. If the error is caused by an automatic lookup, the source lookup should appear in the search log just prior to the error. Here's a contrived example: 05-25-2024 14:06:52.070 INFO CsvDataProvider [421638 searchOrchestrator] - Reading schema for lookup table='bad_csv_header_lookup', file size=32, modtime=1716659989 05-25-2024 14:06:52.070 WARN SearchResultsCSVSerializer [421638 searchOrchestrator] - Corrupt csv header in CSV file , 2 columns with the same name 'bar' (col #5 and #1, #5 will be ignored) Looking at the definition of bad_csv_header_lookup reveals my purposefully malformed bad_csv_header.csv file. In your error, the column name is empty: ''. If you have access to the Splunk host, you can scan $SPLUNK_HOME for CSV files with an empty column header, which is malformed with or without duplicates: find /opt/splunk -type f -name "*.csv" -exec sh -c "head -n 1 {} | grep -- ',,' >/dev/null" \; -printf "%p\n" Some add-ons in Splunkbase ship with malformed CSV lookups files, but the most likely cause is a CSV file without a header row.
Without using JavaScript, it works as expected but I need to use JavaScript.  I’m just assuming that due to the earliest and latest variables defined in JavaScript, the init block is not working for... See more...
Without using JavaScript, it works as expected but I need to use JavaScript.  I’m just assuming that due to the earliest and latest variables defined in JavaScript, the init block is not working for me. Is there a way to solve this issue?   
This is a new instance being installed by a contractor who posited that it's Linux best practice that root owns all mount points.  Internally, a security architect (not a Splunk admin) supported the ... See more...
This is a new instance being installed by a contractor who posited that it's Linux best practice that root owns all mount points.  Internally, a security architect (not a Splunk admin) supported the 'Linux best practice' argument.  In all of my many years of experience with Splunk, or in training, I have never heard of setting root as the owner of mount points within the /opt/splunk folder tree.  So this is a difference of professional opinion and I'm looking for definitive, authoritative consequences of having root as the owner of the mount points.   Often it takes someone outside the organization to validate an item.
The primary consequence is invalidation of assumptions made by Splunk during development. Splunk assumes the owner of the Splunk process has read, write, and execute access on $SPLUNK_HOME and all su... See more...
The primary consequence is invalidation of assumptions made by Splunk during development. Splunk assumes the owner of the Splunk process has read, write, and execute access on $SPLUNK_HOME and all subdirectories. Invalidating this assumption causes unexpected SplunkWeb exceptions at best and splunkd failures at worst. It may be more appropriate to ask what problem you're trying to solve by making root both the owner of the mount point and the owner of the mounted file system object. If splunk is the owner of /opt/splunk/var, then splunk can delete (unlink) /opt/splunk/var/lib and /opt/splunk/var/run if lib and run are unmounted and empty. (Edit: As a friendly aside, you're asking your Splunk administrator(s) and the Splunk community, none of whom presumably work for Splunk or have access to Splunk source code, to validate your assumption that everything will be fine if root owns subdirectories of $SPLUNK_HOME. If you don't trust either party to provide a satisfactory answer, I recommend contacting your Splunk technical account manager or Splunk support. Distrust is a healthy aspect of risk mitigation, but find a party you do trust and verify everyone's assumptions.)
As a brief addendum, if all possible label values are known in advance, it's more efficient to store the values in a lookup file and use the inputlookup command with append=true before the stats comm... See more...
As a brief addendum, if all possible label values are known in advance, it's more efficient to store the values in a lookup file and use the inputlookup command with append=true before the stats command; however, the introduction of any unknown field value, e.g. Date, requires additional logic to produce combinations of known and unknown values.
Hi @sultanulariff, We can use the appendpipe, eventstats, stats, and mvexpand commands to append a product of the set of Bag Type values by the set of Out Airline and Date values and then use the st... See more...
Hi @sultanulariff, We can use the appendpipe, eventstats, stats, and mvexpand commands to append a product of the set of Bag Type values by the set of Out Airline and Date values and then use the stats command to summarize the Total Processed: | makeresults format=csv data=" Date,Out Airline,Bag Type,Total Processed 01/05/2024,IX,Local,100 01/05/2024,IX,Transfer,120 02/05/2024,BA,Local,140 02/05/2024,BA,Transfer,160 03/05/2024,IX,Local,150" | appendpipe [ eventstats values("Bag Type") as "Bag Type" ``` get all possible values of Bag Type ``` | stats values("Bag Type") as "Bag Type" by Date "Out Airline" ``` get all combinations of Bag Type by Date and Out Airline ``` | mvexpand "Bag Type" ``` expand all combinations into separate events ``` ] | stats sum(eval(coalesce('Total Processed', 0))) as "Total Processed" by Date "Out Airline" "Bag Type" ``` sum Total Processed, replacing null values with 0 ``` Date Out Airline Bag Type Total Processed 01/05/2024 IX Local 100 01/05/2024 IX Transfer 120 02/05/2024 BA Local 140 02/05/2024 BA Transfer 160 03/05/2024 IX Local 150 03/05/2024 IX Transfer 0   This example will only append missing Bag Type values. It can be expanded to append any combination of Date, Out Airline, and Bag Type values. We can also convert Date to a time value and use the makecontinuous command to generate spans of dates prior to appending and summarizing combinations.
Thanks. Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder? (this is a search head in a cluster) As well, what about i... See more...
Thanks. Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder? (this is a search head in a cluster) As well, what about in the /opt/splunk/var/lib folder on an indexer.  Note that buckets are actually stored in /opt/splunk/var/lib/splunk/
Thanks.  Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder?  (this is a search head in a cluster)