I like this option, as it allows calculation of the effective registered domain using a wildcard lookup based on the PublicSuffixList. | makeresults format=json data="[{\"id\": 1, \"domain\": [\"test.com\",\"sample.com\",\"example.co.uk\", \"nxdomain\"]}, {\"id\": 2, \"domain\": [\"www.example.com\"]}]" | spath path="domain{}" output=domain | eval domain=mvappend(domain, "") ``` append empty string to force to mv field ``` | foreach mode=multivalue domain [ eval domain_segment_len=mvappend( domain_segment_len, json_extract( lookup( "wildcard_public_suffix_list", json_object("tld_wildcard", '<<ITEM>>'), json_array("length") ), "length" ) ), domain_split = mvappend(domain_split, split('<<ITEM>>', ".")) ``` Use mvcount, mvappend, case, and mvjoin to create the effective TLD ``` ] Although not a single command, using a macro like this with builtin lookups is more scalable for large inputs than using an external lookup that generates the effective TLD or registered domain. ... View more

I made a couple more bookmarklets to help: 1. SID Only: Strip all URL parameters except the SID to have the search parameters loaded from the saved job (only works if the SID is still saved)   javascript&colon; window.location.href = window.location.href.replace(/\?.*?(\bsid=[^&]+).*/, '?$1');   2. Show Search: Show the search after the error message   javascript&colon; query_str = decodeURIComponent(window.location.href.replace(/.*?\bq=([^&]+).*/, '$1')); document.body.innerHTML += `<pre>${query_str}</pre>`;   3. Strip off different parameters until it works. 1st click removes the display fields list, 2nd click collapses repeated spaces, and 3rd click truncates the query to 3500 characters.   javascript&colon;(function(){if (location.href.indexOf('display.events.fields')>=0) {window.location.href = window.location.href.replace(/\b(display\.events\.fields=[^&]+)/, '');}else if (location.href.indexOf('%'+'0A')>=0) {window.location.href = window.location.href.replaceAll(/(%(20|0A))+/g, ' ');}else{window.location.href = window.location.href.replace(/(\bq=[^&]{100,3500})[^&]*(.*)/, '$1$2');}})();   Again,  replace the "&colon;" in the blocks above with the colon character. ... View more

Yes, you can copy the URL, decode the URL parameters, and paste it into a new search, but clicking on a bookmarklet is more convenient for me. If decoding your query due to the 414 error is a common occurrence, you could also make a CyberChef recipe to help. I don't know how much work it would take to make a bookmarklet that would POST the AST to the server instead. I understand that your search has a large number of calculations, but you can use a macro to make the URL shorter.  index=test example.com | `complex_calculations` | `get_geoip_data(src_ip)` | `multiple_stats_commands` In that case, each macro can contain a very large number of commands. When possible, I create macros that are reusable, but that is not always appropriate. In particular, Splunk Enterprise Security content includes a separate filter macro for each Correlation Search so that false positives can be tuned out without editing the detection core logic. Without access to your search query, it is difficult to know how to make the search smaller. In a Windows browser, you can press Ctrl-Shift-E when writing your search to show the "Expanded Search String" with the content in all of the macros being shown. These are a couple examples of how I've moved long parsing and calculation strings to macros: get_datamodel_desc(1) entropy_digits_lowercase(1)  (the Decrypt2 app is better than this macro) ... View more

A Splunk URI query usually contains a few key/value pairs like these: earliest - epoch time for how far back to search latest - epoch time for when to start search sid - unique search ID q - Query string display.events.fields -  Selected Fields If the path and URL query is over ~4000 characters after URL encoding, it will cause that 414 error. I have only had long query values for q and display.events.fields cause the 414 error. Splunk passes the sid in the URL so that the search doesn't need to be run again. All the search parameters are available on the server if you provide the sid, but if the search is deleted or expired it can fall back to the other URL parameters to re-run the search. The solutions are to edit the search to make it shorter in the URL or to edit the URL afterwards to remove some of the long parameters. Now lets discuss the options I mentioned earlier. These will assume the following search and selected fields. They are not long enough to cause the 414 error, but will work for illustrating the issue. search (265 chars encoded): index=test host=0.example.com OR host=1.example.com OR host=2.example.com OR host=3.example.com OR host=4.example.com OR host=5.example.com OR host=6.example.com OR host=7.example.com OR host=8.example.com OR host=9.example.com fields: host, src, src_ip, src_mac, dest, dest_ip, dest_mac 1. Refactor the Search We can make this search string smaller by using the IN statement, a lookup, or a macro. The IN statement and lookup table makes sense if you have a list of values in a search, the macro makes sense if you pipe the output to multiple subsequent commands (multiple eval, stats, etc.). 1.a. IN statement (166 chars encoded) index=test host IN (0.example.com,1.example.com,2.example.com,3.example.com,4.example.com,5.example.com,6.example.com,7.example.com,8.example.com,9.example.com) 1.b. lookup table (77 chars encoded) index=test [inputlookup example_domains | return 1000 host] 1.c. Search macro (40 chars encoded) index=test `example_domain_search` 2. Edit the URL Here is an example path for the first query above /search?q=search%20index%3Dtest%20host%3D0.example.com%20OR%20host%3D1.example.com%20OR%20host%3D2.example.com%20OR%20host%3D3.example.com%20OR%20host%3D4.example.com%20OR%20host%3D5.example.com%20OR%20host%3D6.example.com%20OR%20host%3D7.example.com%20OR%20host%3D8.example.com%20OR%20host%3D9.example.com&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.events.fields=%5B"host"%2C"src"%2C"src_ip"%2C"src_mac"%2C"dest"%2C"dest_ip"%2C"dest_mac"%5D&sid=1723000000.00000 2.a. Manually edit the URL (not recommended) Go to the address bar and manually remove the longer query parameters 2.a.i:. Remove the display parameters and timeframe /search?q=search%20index%3Dtest%20host%3D0.example.com%20OR%20host%3D1.example.com%20OR%20host%3D2.example.com%20OR%20host%3D3.example.com%20OR%20host%3D4.example.com%20OR%20host%3D5.example.com%20OR%20host%3D6.example.com%20OR%20host%3D7.example.com%20OR%20host%3D8.example.com%20OR%20host%3D9.example.com&sid=1723000000.00000 or 2.a.ii:. Remove the display parameters /search?q=search%20index%3Dtest%20host%3D0.example.com%20OR%20host%3D1.example.com%20OR%20host%3D2.example.com%20OR%20host%3D3.example.com%20OR%20host%3D4.example.com%20OR%20host%3D5.example.com%20OR%20host%3D6.example.com%20OR%20host%3D7.example.com%20OR%20host%3D8.example.com%20OR%20host%3D9.example.com&earliest=-24h%40h&latest=now&sid=1723000000.00000 or  2.a.iii. Leave only the search ID (sid) /search?sid=1723000000.00000 2.b. Edit the URL with a bookmarklet With the bookmarklet shared earlier, you can use a regular expression to remove some of the parameters. You could remove all but the sid like I did, or you could remove only the display.events.fields if that is causing issues for you.  Any of the manual edits made above can be made with a regular expression. If you want a regular expression that provides more fields than the sid, you can use an regular expression tool like regex101 to assist in creating a different bookmarklet. It is probably possible to build a lexer bookmarklet that parses the search query and truncates it to fit within the server's ~4000 character limit, but that's probably a waste of time. ... View more

A search can be longer than the URI allows for opening in a new tab, which causes the 414 Request-URI Too Long error.  There are multiple workarounds: Refactor the search and/or move long portions of the query into an inputlookup command or search macro. Edit the URL to remove the query and only use the SID (as long as the search ID hasn't expired) For the second option, you can make a "Bookmarklet" that removes all of the URL parameters except the SID: javascript&colon; window.location.href = window.location.href.replace(/\?.*?(\bsid=[^&]+).*/, '?$1') Note: Khoros is breaking the bookmarklet; replace &colon; with : If you click on that bookmarklet when you get the error, it will open the search.     ... View more

Yes, it appears that most (if not all) non-ASCII character are major breakers. The lispy I see for a simple search for  тестирование is: [ AND index::main а в е и н о р с т ] This is a bigger issue if the data is ingested in ASCII JSON format. {"data": "\u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435"}  If the data above is ingested, "data="тестирование"  or "тестирование" will not find the data.  An initial search like "u04*" must be included.  A similar issue occurs when the raw JSON includes a newline, as a string like {"data": "line_one\nline_two"} cannot be found with a search for "line_two". ... View more

The sed command is probably too aggressive, as it removes escaped backslashes. The regular expression to extract the message didn't include all the characters. By using an alternative of matching every escaped quote or other characters, it will keep any single JSON string.  REGEX = \"message\":\s*\"((?:\\\"|[^\"]*)\" SEDCMD-unescapetab = s/\\t/    /g SEDCMD-unescapenewline = s/\\n/\n/g SEDCMD-removeescape = s/\\(.)/\1/g   ... View more

This isn't an issue that is limited to liberal use of the wildcard.  I don't believe it is intended behavior. When a search is made in Smart or Verbose mode, the base search returns schema for events that are not included in the result. The number of fields shown to the user in the "All Fields" can be a small subset of the fields exposed in the table command or a data export.  In the search example that @rbechtold provided, the "alert_name" placeholder may have matched 144 of ~2000 events returned from the indexers.  The schema from those other ~2000 events is not shown in the UI.  Because of this, I would consider inclusion of null fields for events that are displayed to be unexpected.   index _raw source Event 1 _internal "alert name", k1=val1 scheduler.log Event 2 _internal "name alert", k1=val1, k2=val2, k3=val3 scheduler.log   If I searched that data with ` index=_internal "alert name" source=schedular.log `,  k2 and k3 would not be shown in the web result GUI.  Including them in the table command or a data export is a surprise, especially if Event 2 included hundreds of fields.  I guess something like this could work, and would also work in Fast mode.    index=_internal "alert_name" source="/opt/splunk/var/log/splunk/scheduler.log" | fields _time, _raw | extract     ... View more

No, you didn't miss anything. The splunk-packaging-toolkit (slim), splunk-appinspect, and the Splunk Python SDK are not perfectly aligned yet. If you want slim to work better, you could update the files in ./venv/Lib/site-packages/slim/config/conf-specs/*.conf.spec to what you find in your target Splunk version's config files. If you don't have a Splunk installation, replace commands.conf.spec and others with a current version from Splunk docs (https://docs.splunk.com/Documentation/Splunk/Latest/Admin/Commandsconf#commands.conf.spec).  Some of the other issues I've seen: Options must be provided before the command documented slim options may not exist (e.g. --quiet) Python SDK doesn't pass splunk-appinspect  splunk-appinspect and slim raise conflicting errors no slim option to set the location of inspect.yml no option to return a non-zero code if any validations fail in splunk-appinspect or slim  I'd recommend packaging the app with slim and using splunk-appinspect  to validate, rather than using the validation in slim. ... View more

I understand the frustration, but the issue may be partially due to the email provider blocking the messages. I also hope that Splunk will help identify the root cause and help fix this issue. Have you tried resetting the password for your instance and using the username above? ... View more

  I agree that mode function is most likely what @HeinzWaescher  was looking for.  Because it is a stats function, you can also use it with eventstats to keep the events.  | eventstats mode(country_code) by user_id For the example initially provided, the top command does not appear any better than the stats mode() command provided by @p3hndrx. But if you the top N values (N>=2), there is no function like values() that returns them sorted by frequency of occurrence.  ... View more

The three macros below calculate the haversine formula that @MuS provided.  [haversine(5)] # Calculate the great circle distance for a sphere with an arbitrary radius args = input_lat1, input_lon1, input_lat2, input_lon2, hav_radius definition = "eval hav_lat1_radians = pi()*$input_lat1$/180, hav_lat2_radians=pi()*$input_lat2$/180, hav_delta_lat_radians = pi()* ($input_lat2$-$input_lat1$)/180, hav_delta_lon_radians= pi()*($input_lon2$-$input_lon1$)/180 | eval hav_intermediate = pow(sin(hav_delta_lat_radians/2), 2) + cos(hav_lat1_radians) * cos(hav_lat2_radians) * pow(sin(hav_delta_lon_radians/2), 2) | eval hav_distance = 2 * $hav_radius$ * atan2(sqrt(hav_intermediate), sqrt(1-hav_intermediate)) | fields - hav_*_radians, hav_intermediate " [haversine(4)] # Calculate the great circle distance for the earth (in kilometers) args = input_lat1, input_lon1, input_lat2, input_lon2 definition = "`haversine($input_lat1$, $input_lon1$, $input_lat2$, $input_lon2$, 6371)` " [haversine(2)] # Calculate the great circle distance between two IPs (in kilometers) args = input_ip1, input_ip2 definition = "iplocation $input_ip1$ prefix=$input_ip1$_ | iplocation $input_ip2$ prefix=$input_ip2$_ | `haversine($input_ip1$_lat, $input_ip1$_lon, $input_ip2$_lat, $input_ip2$_lon)` "  Using streamstats, you can calculate IP location distances between events. With eventstats, you can calculate IP location distances between a common IP location and an events IP location.   The calculated value is returned as hav_distance, to decrease the chances of a field name collision. The haversine formula is not as accurate as Vincenty's formulae, but is much more accurate than a simple chord length calculation.  | makeresults | eval usual_src_ip="8.8.8.8", src_ip="9.9.9.9" | `haversine(usual_src_ip, src_ip)` | where hav_distance > 500   ... View more

Removing the last comment of the following search will create a lookup table of all of the values.    | datamodel | spath output=modelName modelName | search modelName!=Splunk_CIM_Validation `comment("mvexpand on the fields value for this model fails with default settings for limits.conf/[mvexpand]/ max_mem_usage_mb")` | spath output=fields objects{}.fields{} | mvexpand fields | eval _time = now(), _raw = fields `comment("fill the values required for display in the Events viewer")` | spath | rename comment.* AS comment_* | rename *{}* AS ** `comment("| fields - _time, _raw, fields, comment | outputlookup datamodel_info.csv")`   This then permits simpler lookups against each data model's field names, including the description, type, etc.    That doesn't directly answer the initial question. That can be directly extracted with spath as commented previously.   | datamodel | spath output=modelName modelName | spath output=field_names objects{}.fields{}.displayName | stats list(field_names) as field_names by modelName    The field displayName includes the object names, but for CIM datamodels they can be removed relatively easily because they are the only values that include uppercase characters.    If you prefer using the rest command, the following have the same output   | rest /servicesNS/-/-/datamodel/model | rename description AS _raw | fields _raw | spath | datamodel | spath     The datamodelsimple command is an easy way to get basic information from a datamodel, like the field name and lineage.  | datamodelsimple datamodel="Network_Resolution" object=DNS type=attributes  For that example,  it returns  lineage attribute DNS.dest_port dest_port host host DNS.answer_count answer_count _raw _raw ... View more

I would recommend using an eventstats command to exclude  index="idx" earliest=-360s latest=-0s source="server.log" "<Request" | xmlkv | eventstats earliest_time(clientId) as earliest_clientId by clientId | where relative_time(now(), "-60s") > earliest_clientId |`enter code here`   ... View more

The URL is visible from the Instances page, and you can also request a password reset from that page. But the username is not shown on that page.  https://splunkcommunities.force.com/customers/apex/RMEC_InstancePage I never got my the "Welcome to Splunk Cloud!" email, so I registered with a different email address, and that welcome got through. That included a username that worked on both instances. You can try the cloud default usernames, like sc_admin, with the password sent in the password reset. https://community.splunk.com/t5/Splunk-Cloud/sc-admin-cannot-login/td-p/519531 ... View more

I don't know when the change was made, but threading seems to be working now. ... View more

I just posted an "idea" for the return command to be able to do what you would like it to do. https://ideas.splunk.com/ideas/EID-I-532  The following is your command, rewritten with a rex command that should create a simple boolean expression from one subsearch.   (index=winsec sourcetype="WinSecurityEvent") OR index=DNS [ search sourcetype="dns" "specific urls" | dedup src | return 100 src_ip=src OR query | eval search = if(len(search)<3, "NOTHING FOUND") | rex mode=sed field=search "s/ (OR|AND|NOT)=\"\" / \1 /g" ] ) | dedup accountname | stats values(accountname) AS accounts | table query, src_ip, accounts     The string in the eval command should be changed to something that will never be found.  Depending on the use, perhaps "index!=*" might be more efficient.  Otherwise, if nothing is found then the search will return an empty string that matches all events. This may be desired, in which case the eval statement should be removed.   ... View more

That is there, under the "DESCRIPTION FORMATTING" section. The earlier "FORMATTING" section states to use \" to represent a quote. This is not directly aligned with the Description Formatting and does not work in a description. The "\" is also shown as "\\", which is not documented. Based on the descriptions shown above that were written by Splunk, it appears that this behavior was not understood by Splunk authors, or the parsing behavior changed and the descriptions were not updated. The only concern I have with @lstewart_splunk's accurate answer, is that the current behavior breaks the description syntax for stock Splunk commands, as shown above for regex.  The Splunk authors for the regex syntax apparently expected that (\")?string(\")? would be shown as (")?string(")?, but instead, it is strangely highlighted. ... View more

In addition the the good advice from @bmacias84 , I recommend also including a .slimignore file. This will exclude your .git and other directories that shouldn't be published in the app. https://dev.splunk.com/enterprise/docs/releaseapps/packagingtoolkit/pkgtoolkitref/packagingtoolkitcli/#slim-package # Default Splunk Packager exclusions .DS_STORE Thumbs.db *.py[co] default/indexes.conf local/ metadata/local.meta # Additional exclusions .git .gitignore venv*/ .idea/ .vscode/   Here is an example from Splunk's Cisco Webex Meeting App: https://github.com/splunk/cisco_webex_meetings_app_for_splunk/blob/master/.slimignore Another option is to use a subdirectory for the app, as Palo Alto Networks did recently when they consolidated their apps into one GitHub repository. https://github.com/PaloAltoNetworks/Splunk-Apps/ ... View more

To modify @martin_mueller's answer to find where the underscores ("_") are, the "rex" command option, "offset_field", will gather the locations of your match. The "offset_field" option has been available since at least Splunk 6.3.0, but I can't go back farther in the documentation to check when it was introduced. If you only want the first match index, or a limited number of indexed locations, the "max_match" parameters can be changed. ... | rex field=message offset_field=message_offsets max_match=0 "(?P<char_match>_)" | rex field=message_offsets max_match=0 "char_match=(?P<offset_range>[0-9]+-[0-9]+)" | chart values(offset_range) by message  The first "rex" command creates a field named "message_offsets" will contain data like the results of these eval statements, if the character(s) are found. | eval example_message_offsets = "char_match=7" | eval example_message_offsets = "char_match=12-12&char_match=19-19" The second "rex" extracts the index from those values into "offset_range". For one character, the values are the same and separated with a "-".  If there is no match, both the char_match and the message_offsets fields are null(), so the chart will not include field values that don't contain the string of interest. Unless fillnull, coalesce, or another eval statement is used to populate null values. ... View more

For the highlighting, it appears that it has affected Splunk as well.  You can look at the shelper response to see the raw text, so the highlighting appears to be in the browser, not in the backend.     Note: if you want to use the "or" ("|") command regex (field("="|"!="))?((\")?string(\")?)               If you look at the chart command, it does the same to the BY/OVER that is capitalized in the source.   ... View more

I think there is a misunderstanding. This is a discussion on how the Splunk searchbnf.conf parser has undocumented and unwanted behavior, not about any particular app or any 3rd party commands. ... View more

I replicated this issue with Splunk versions 7.2 and 8.0. ... View more

I tried the following, with no luck. AAAA _BBBB_ \CCCC "DDDD" 'EEEE' `FFFF`  I was surprised to see backslashes escaped, and the underscore after BBBB is highlighted but the underscore after is not.    ... View more