@gaurav_maniar Thanks for the reply. Replacing numeric nulls with a default string value is not the way I want to go. If a calculation needs to be performed against this field in the future, I don't want to have to deal with converting from string to number. It is the same inconvenience with formatting currency with commas and a dollar sign. I have revert the string back to a numeric any time I want to add them. I could use a numeric default 0. But again in the future the application owner might be interested in when 0's being entered by the customer or programmer, vs. nulls. Splunk should make nulls similar to every other application out there. Thanks again. God bless, Genesius ... View more

@gcusello Thanks for your suggestions. I just posted a deeper explanation what my requirements are, as well as responding to your suggestions. Thanks and God bless, Genesius ... View more

@woodcock Here is what I need to provide to our *nix SysAdmins. Current process: they open 3 separate windows on their monitors. One is to view audit.log, the other to view /var/log/secure, and the last to view sudo.log. Requirements: a single view of all three on one screen, with the ability to drill down into each individual log if needed. Initially, we installed the Splunk Linux Auditd app. And while a much better app than the Splunk App for Unix and Linux (utterly useless to our SysAdmins), the Auditd app does not return ALL the events associated with an ssh session as I hope to explain below. Before I get into that, I would like to comment on the suggestions from gcusello, and why I did or did not follow his valuable advice. check the number of results of each subsearch because in Splunk there's the limit (configurable but to avoid) of 50,000 results of subsearches; Don’t believe any of the subsearches will return more than 50,000. But this will be dependent upon the Time Picker. Which is a question I raised on another post. you don't need to repeat $Selected_Time_Range.earliest$ AND latest=$Selected_Time_Range.latest$ in every search except when you want a different value because they are taken by default; Removed from the query in all subsearches and the main search. you don't need to use _raw="$userId_tok$" because if you use a string for search it's used in _raw, in other words you should try to forget to think to Splunk using a database approach! Unfortunately, the events in the source="ps" are not clearly defined extractions (see explanation further down on why this search is required). One of the ps events below shows the user=user1 as the first field. Therefore, I know these are events related to user1. However, there is also an event where the user=root, AND the string user1 appears elsewhere in the event. This is why my code includes (USER="root" AND "$userId_tok$"). Performing a deep-dive further into the ps events I found the field ARGS, which Splunk extracts using the ps.sh. Using this field instead of _raw speeds up this subsearch due to the removal of the wildcard at the beginning of the field. Further testing is required to determine if the wildcard at the end of the field ARGS can be removed as well. This SPL line changed from (USER=$userId_tok$ OR (USER="root" AND "$userId_tok$")) to (USER=$userId_tok$ OR (USER="root" AND ARGS="$userId_tok$*")). Sample events from source=ps. From this subsearch I am extracting a list of PIDs associated with user1. root 7267 14 0.0 00:00:00 0.0 4032 104388 ? S 01:01 sshd: user1_[priv] user1 7271 2 0.0 00:00:00 0.0 2712 104720 ? S 01:01 sshd: user1@notty you don't need to use "AND" boolean operator because there's by default; I understand this. Does this slow the SPL down? If not, I like to include it for readability. you don't need to use "+" in | sort +_time it's a default; My experience has been that if I don’t use | sort +_time, the table comes out with latest event first. I require first event first. Can this be modified in a conf file? I see that you're using the same search parameters in all the searches. At the present I don’t see any way around this, which is why I am posting. Is there is a command to state to use the same index, host, acct, addr, and auid? Now that I have answered gcusello’s suggestions, here are the reasons for the complexity of this SPL. There would be several events missing for user1’s entire ssh session without this myriad of subsearches. There are some field-value pairs that are not known for user1’s ssh session until other (later) events are seen. Below is an abridged list of the events from this dashboard. I've inserted a number in front of each event for readability. (1) type=CRYPTO_KEY_USER msg=audit(1569008008.220:10732729): user pid=20626 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=28:6a:cb:a7:ab:67:d4:85:ff:34:99:b7:c9:f5:55:5c direction=? spid=20626 suid=0 exe="/usr/sbin/sshd" hostname=? addr=10.10.10.10 terminal=? res=success' (2) type=CRYPTO_KEY_USER msg=audit(1569008008.220:10732730): user pid=20626 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=48:24:f7:c4:0a:87:94:34:88:22:e1:88:47:1a:15:e2 direction=? spid=20626 suid=0 exe="/usr/sbin/sshd" hostname=? addr=10.10.10.10 terminal=? res=success' (3) type=CRYPTO_SESSION msg=audit(1569008008.220:10732731): user pid=20623 uid=0 auid=4294967295 ses=4294967295 msg='op=start direction=from-client cipher=aes128-ctr ksize=128 spid=20626 suid=74 rport=41507 laddr=10.1.1.1 lport=22 exe="/usr/sbin/sshd" hostname=? addr=10.10.10.10 terminal=? res=success' (4) type=CRYPTO_SESSION msg=audit(1569008008.220:10732732): user pid=20623 uid=0 auid=4294967295 ses=4294967295 msg='op=start direction=from-server cipher=aes128-ctr ksize=128 spid=20626 suid=74 rport=41507 laddr=10.1.1.1 lport=22 exe="/usr/sbin/sshd" hostname=? addr=10.10.10.10 terminal=? res=success' (4) type=USER_AUTH msg=audit(1569008008.350:10732733): user pid=20623 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey_auth rport=41507 acct="user1" exe="/usr/sbin/sshd" hostname=? addr=10.10.10.10 terminal=? res=success' The first 4 events do not have an acct field (selected from the earlier dropdown). The only way to correlate these 4 events with the ssh session is by capturing from the 5th event of the results, which is from audit.log. This event includes the acct field and the value we are searching for, user1, as well as the value of its corresponding addr 10.10.10.10. A new subsearch will be run to include these 4 events, as well as any other events in audit, secure or sudo that contain only addr. (12) type=LOGIN msg=audit(1569008008.358:10732739): pid=20623 uid=0 old auid=4294967295 new auid=1014 old ses=4294967295 new ses=23056 In the 12th event, also from audit.log the auid is set for user1. Prior audit.log events had auid set as the RedHat default auid=4294967295. However, Splunk doesn’t extract this field as new auid. Checking the values for auid on this event are both 4294967295 and 1014. The field auid has become a multivalued field for this event. Not good. A custom extract was created. Now we have values for the 3 main fields required to perform this search: acct=user1; addr=10.10.10.10; and auid=1014. (42) 2019-09-20T15:33:28.536160-04:00 ruby01-s sshd[20623]: subsystem request for sftp In event 42, from /var/log/secure, there are no acct, addr or auid fields. This event was discovered because of the source=ps subsearch performed earlier where the pid was extracted (20623). A subsearch for this PID (and all others associated with user1) needs to be performed across all 3 logs to find any other events. After gathering all the events from the 3 logs based on the acct, auid, or addr fields, as well as the PIDs from source=ps, a final seach is run against all events with the transaction command. (45) type=SYSCALL msg=audit(1569008008.833:10732770): arch=c000003e syscall=2 success=yes exit=8 a0=7fff309e25b0 a1=42 a2=180 a3=8 items=2 ppid=4246 pid=20623 auid=1014 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=23056 comm="sshd" exe="/usr/sbin/sshd" key="logins" type=CWD msg=audit(1569008008.833:10732770): cwd="/" type=PATH msg=audit(1569008008.833:10732770): item=0 name="/var/log/" inode=130451 dev=fd:03 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT type=PATH msg=audit(1569008008.833:10732770): item=1 name="/var/log/lastlog" inode=135218 dev=fd:03 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL In event 45 you can see the last three(sub?) events of the transaction do not include acct, auid, or addr. These events are correlated by running transaction on the eventid field (10732770). This process repeats itself multiple times throughout the logs. I hope this explains our SysAdmins requirements, and what missing events were discovered when the logs were deep-dived. Follow up post will include the updated XML. Thanks and God bless, Genesius ... View more

@woodcock Apologies to you and the other forum users for not responding sooner. The logs being ingested are sourcetype=log4j, with some customization. What I need to accomplish is to have a different set of search commands run depending upon the value of the priority field (INFO, DEBUG, ERROR, etc.). I was thinking about using the case function because I worked with case in my former programming days (Teradata SQL), and seem to remember being able to add a full set SQL (or branch to other SQL commands) based on the value of each case. However, after rereading the Splunk docs, I learned only values can be returned from a case function. Not an entire set of search commands. BTW, when I checked the map command it appears to behave similarly to the for-next loop structures used in other programming languages; i.e., the same set of commands would be repeated for each value of the priority field. However, I need different SPL commands run for each different priority value. Example using 3 of the common values for the priority field. INFO: the required fields for these events are being extracted with no other transformations required on my end. ERROR: new fields need to be extracted requiring the | rex command against _raw. DEBUG: requires calculations be performed against both currently extracted and rex'ed fields using | eval commands. Therefore, I will use subsearches. How do I go about closing this post? Also, how did you format the words subsearch and map in your response above? Thanks and God bless, Genesius ... View more

Anyone have any ideas? Thanks and God bless, Genesius ... View more

Doing more research on the case function, I discovered it is not similar to case commands in other programming languages. It does not permit branching to another of commands. The arguments can only be field values. According to the Splunk docs, https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/ConditionalFunctions (Case) accepts alternating conditions and values. Returns the first value for which the condition evaluates to TRUE. a This function takes pairs of arguments X and Y. The X arguments are Boolean expressions that are evaluated from first to last. When the first X expression is encountered that evaluates to TRUE, the corresponding Y argument is returned. The function defaults to NULL if none are true. Y can only be an argument, and not a returned set of new search commands. Thanks and God bless, Genesius ... View more

@woodcock From what I've seen with the map command (which in the docs is very little; more was available in Answers) the same set of search commands is run over a number of maxsearches. This is not what I am looking for. It looks subsearch is similar in that the same set of search commands will be run. Thanks and God bless, Genesius ... View more

@woodcock Thanks. I updated the post above with actual code. God bless, Genesius ... View more

@mayurr98 Not exactly what I am looking for. Using your example, based on field1, dssd needs to be a full search with index, source(types), hosts, booleans, rex's, evals, etc.... Thanks and God bless, Genesius ... View more

@sandeepmakkena Just so I understand. Although the field (column) has been renamed, and we can no longer use the original name in eval, rex, or other commands. However, the original name is still available to be used as a token which can be passed to other child dashboards, and other panels on the parent dashboard? Thanks and God bless, Genesius ... View more

@jdhunter Thank you. This won't work for me. I know about the rename command. What I want to be able to do is rename the header in the table, not the field name itself. For example. Original field name: userId1, userId2 Both these fields are used in child dashboards. However, in the parent dashboard the column names for these two fields needs to be... Remote Central Office User Remote Branch Office User I don't want to use those names as tokens for the subsequent child dashboards. Is there a way to rename a table header AND NOT the field itself? Thanks and God bless, Genesius ... View more

@C_HIEN Thank you. Here is what I entered, again, I don't know the proper syntax for entering if when evaling a token. <eval token="startTime">if(match($form.Selected_Time_Range.earliest$,"@"),strftime(relative_time(now(),"$form.Selected_Time_Range.earliest$"),"%c"),strftime($form.Selected_Time_Range.earliest$,"%c"))</eval> The value on my dashboard is now $startTime Not sure if earliest would have now() as a possibility. I tried with latest, changing it accordingly. The value on my dashboard was Invalid date. What am I doing wrong? Thanks and God bless, Genesius ... View more

@ololdach Here is your code with my added comments. If you feel this correct, please post as a new answer so I can accept and upvote. `comment("If the cmd field contains quotes in _raw, then the quotes will be added to the field extraction.")` | rex field=_raw "cmd=(?<cmd>[^\s]+)" `comment("Move the character after the first doublequote; capture all the text until the second doublequote.")` | rex field=cmd "\"(?<cleardata>[^\"]*)\"" `comment("Add a % before each hexadecimal digit. If the value of cmd is not all hex, there will be % after every 'supposed' hex digit.")` | rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" `comment("Convert cmd from hex to text. If the value of cmd is not all hex, there will no conversion.")` | eval hexdata=urldecode(cmd) `comment("If cleardata, which is the field representing the true text, is blank, use the hex conversion (hexdata). If not, use the cleardata (text) as the value of cmdConverted")` | eval cmdConverted=if(isnull(cleardata),hexdata,cleardata) Thanks again for all your help. God bless, Genesius ... View more

@DavidHourani Thank you. Here is what I ran, replacing the field names as mentioned above. | eval cmdConverted=if(match(cmd,"^\".+"),cmd, null()) | rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" | eval cmdConverted=if(isnull(cmdConverted),cmd,cmdConverted) Here is what both fields now look like. dmi%deco%de dmi%deco%de %73%75%20%2D %73%75%20%2D The code did not convert and it added % in between each set of hex digits, and before the 2nd and 3rd d in the text. Strange. Thanks and God bless, Genesius ... View more

@ololdach If you could re-post as a new answer Replaced oldfield with cmd; and newfield with cmdConverted. | rex field=_raw "cmd=(?<cmd>[^\s]+)" | rex field=cmd "\"(?<cleardata>[^\"]*)\"" | rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" | eval hexdata=urldecode(cmd) | eval cmdConverted=if(isnull(cleardata),hexdata,cleardata)")` Then I can accept and upvote. Thanks and God bless, Genesius ... View more

@ololdach Replaced oldfield with cmd; and newfield with cmdConverted. | rex field=_raw "cmd=(?<cmd>[^\s]+)" | rex field=cmd "\"(?<cleardata>[^\"]*)\"" | rex mode=sed field=cmd "s/([0-9A-Fa-f]{2})/%\1/g" | eval hexdata=urldecode(cmd) | eval cmdConverted=if(isnull(cleardata),hexdata,cleardata)")` At first I didn't know hexdata and cleardata were other new fields. So I had replaced them with cmdConverted and cmd. 😞 But once I "learned" they were other new fields, this worked. Thanks and God bless, Genesius ... View more

@DavidHourani Replaced oldfield with cmd; and newfield with cmdConverted. | eval cmdConverted=if(match(cmd,^\".+),urldecode(substr(cmd,1)), cmd) Didn't work. Error in 'eval' command: The expression is malformed. An unexpected character is reached at '^\".+),urldecode(substr(cmd,1)), cmd)'. Thanks and God bless, Genesius ... View more

@ololdach Thanks. I'm going to try that. While I do, I amended my question with two sample events. Apologies, I should have done this to begin with. Thanks and God bless, Genesius ... View more

@ololdach Close. Sort of. Extract a new field newfield from some oldfield using rex, but only, if the oldfield has double quotes. BUT, if the oldfield doesn't contain double quotes we KNOW the value is in hex and not text. Therefore, a conversion from hex to text needs to be made on the oldfield and stored in the newfield. Thanks and God bless, Genesius ... View more