About yuanliu

yuanliu

Yes, I agree that this is such a useful solution. The following are screenshots to replicate the exact effect as kristian_kolb's original intent: 0. Edit file knownips.csv as desired Upload CSV file in "Lookups -> Lookup table files -> Add new". (Example file name: knownips.csv) Define lookup in "Looksup -> Lookup definitions -> Add new". Select the file you uploaded, e.g., knownips.csv. Check "Advanced options", scroll down to "Match type", enter CIDR(clientip), clientip being the field name used to match input. (Example lookup name: checkip.) Add automatic lookup in "Lookups -> Automatic lookups -> Add new". Select the lookup name you give above (the prompt is "Lookup table"), then type clientip as the first entry in "Lookup input fields", then type clientip after equal sign (=). Here, the first box is the field used for comparison in the table, the second box is the field used for lookup in input. (Example automatic lookup name: check; example sourcetype: access_combined)

yuanliu

A convenience feature was introduced in 7 (well I noticed it in a Splunk 7 installation and not in 5 and 6) that automatically inserts a new line when a pipe sign ("|") is typed at the end of a line. I no longer see this behavior in 8. Is this feature deprecated? Or is there a setting to enable/disable it?

yuanliu

I came across this and tested with 8.1.2 successfully. Meanwhile, as this is dated, Splunk now has official systemd support; see Run Splunk Enterprise as a systemd service. Specifically, in Additional options for enable boot-start, a highlight panel states Do not use the following properties. These properties can cause splunkd to fail on restart. RemainAfterExit=yes ExecStop I didn't experience problem with restart with ExecStop but it's probably prudent to just use the official guide. Procedure is simple, just run [sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname>

yuanliu

The problem is in my query format. Search query must be submitted in POST but my query was sent via GET.

yuanliu

Using a really basic search like the one illustrated in Example: Create a search, my freshly installed 8.1.2 responds with a lot more unrelated information in a format that is very different from exemplified in the document, i.e., something like <?xml version='1.0' encoding='UTF-8'?> <response> <sid>1258421375.19</sid> </response> (which was also how an older server responded.) Instead, the new server's response is like <?xml version="1.0" encoding="UTF-8"?>  <?xml-stylesheet type="text/xml" href="/static/atom.xsl"?> <feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"> <title>jobs</title> <id>https://myserver:8089/services/search/jobs</id> <updated>2021-03-15T22:56:36+00:00</updated> <generator build="545206cc9f70" version="8.1.2"/> <author> <name>Splunk</name> </author> <opensearch:totalResults>3</opensearch:totalResults> <opensearch:itemsPerPage>0</opensearch:itemsPerPage> <opensearch:startIndex>0</opensearch:startIndex> <entry> <title>| archivebuckets</title> <id>https://myserver:8089/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1</id> <updated>2021-03-15T22:17:01.161+00:00</updated> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1" rel="alternate"/> <published>2021-03-15T22:17:00.000+00:00</published> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/search.log" rel="search.log"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/search_telemetry.json" rel="search_telemetry.json"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/events" rel="events"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/results" rel="results"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/results_preview" rel="results_preview"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/timeline" rel="timeline"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/summary" rel="summary"/> <link href="/services/search/jobs/scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1/control" rel="control"/> <author> <name>splunk-system-user</name> </author> <content type="text/xml"> <s:dict> <s:key name="canSummarize">0</s:key> <s:key name="cursorTime">1970-01-01T00:00:00.000+00:00</s:key> <s:key name="defaultSaveTTL">604800</s:key> <s:key name="defaultTTL">600</s:key> <s:key name="delegate">scheduler</s:key> <s:key name="diskUsage">53248</s:key> <s:key name="dispatchState">DONE</s:key> <s:key name="doneProgress">1.00000</s:key> <s:key name="dropCount">0</s:key> <s:key name="earliestTime">1970-01-01T00:00:00.000+00:00</s:key> <s:key name="eventAvailableCount">0</s:key> <s:key name="eventCount">0</s:key> <s:key name="eventFieldCount">0</s:key> <s:key name="eventIsStreaming">1</s:key> <s:key name="eventIsTruncated">0</s:key> <s:key name="eventSearch">archivebuckets </s:key> <s:key name="eventSorting">none</s:key> <s:key name="isBatchModeSearch">0</s:key> <s:key name="isDone">1</s:key> <s:key name="isEventsPreviewEnabled">0</s:key> <s:key name="isFailed">0</s:key> <s:key name="isFinalized">0</s:key> <s:key name="isPaused">0</s:key> <s:key name="isPreviewEnabled">0</s:key> <s:key name="isRealTimeSearch">0</s:key> <s:key name="isRemoteTimeline">0</s:key> <s:key name="isSaved">0</s:key> <s:key name="isSavedSearch">1</s:key> <s:key name="isTimeCursored">0</s:key> <s:key name="isZombie">0</s:key> <s:key name="keywords"></s:key> <s:key name="label">Bucket Copy Trigger</s:key> <s:key name="latestTime">2021-03-15T22:17:00.000+00:00</s:key> <s:key name="normalizedSearch"></s:key> <s:key name="numPreviews">0</s:key> <s:key name="optimizedSearch">| archivebuckets</s:key> <s:key name="phase0"></s:key> <s:key name="phase1">archivebuckets | timeliner remote=0 partial_commits=0 max_events_per_bucket=500000 fieldstats_update_maxperiod=60 bucket=0</s:key> <s:key name="pid">825113</s:key> <s:key name="priority">5</s:key> <s:key name="provenance">scheduler</s:key> <s:key name="remoteSearch"></s:key> <s:key name="reportSearch"></s:key> <s:key name="resultCount">0</s:key> <s:key name="resultIsStreaming">1</s:key> <s:key name="resultPreviewCount">0</s:key> <s:key name="runDuration">0.89</s:key> <s:key name="sampleRatio">1</s:key> <s:key name="sampleSeed">0</s:key> <s:key name="savedSearchLabel">{"owner":"nobody","app":"splunk_archiver","sharing":"app"}</s:key> <s:key name="scanCount">0</s:key> <s:key name="search">| archivebuckets</s:key> <s:key name="searchCanBeEventType">0</s:key> <s:key name="searchLatestTime">1615846620.000000000</s:key> <s:key name="searchTotalBucketsCount">0</s:key> <s:key name="searchTotalEliminatedBucketsCount">0</s:key> <s:key name="sid">scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1</s:key> <s:key name="statusBuckets">0</s:key> <s:key name="ttl">4825</s:key> <s:key name="performance"> <s:dict> <s:key name="command.archivebuckets"> <s:dict> <s:key name="duration_secs">0.858</s:key> <s:key name="invocations">1</s:key> <s:key name="input_count">0</s:key> <s:key name="output_count">0</s:key> </s:dict> </s:key> <s:key name="command.timeliner"> <s:dict> <s:key name="invocations">1</s:key> <s:key name="input_count">0</s:key> <s:key name="output_count">0</s:key> </s:dict> </s:key> <s:key name="dispatch.createdSearchResultInfrastructure"> <s:dict> <s:key name="duration_secs">0.001</s:key> <s:key name="invocations">1</s:key> </s:dict> </s:key> <s:key name="dispatch.evaluate.archivebuckets"> <s:dict> <s:key name="invocations">2</s:key> </s:dict> </s:key> <s:key name="dispatch.finalWriteToDisk"> <s:dict> <s:key name="invocations">1</s:key> </s:dict> </s:key> <s:key name="dispatch.readEventsInResults"> <s:dict> <s:key name="invocations">1</s:key> </s:dict> </s:key> <s:key name="dispatch.timeline"> <s:dict> <s:key name="invocations">1</s:key> </s:dict> </s:key> <s:key name="dispatch.writeStatus"> <s:dict> <s:key name="duration_secs">0.001</s:key> <s:key name="invocations">4</s:key> </s:dict> </s:key> <s:key name="startup.configuration"> <s:dict> <s:key name="duration_secs">0.02</s:key> <s:key name="invocations">2</s:key> </s:dict> </s:key> <s:key name="startup.handoff"> <s:dict> <s:key name="duration_secs">0.092</s:key> <s:key name="invocations">2</s:key> </s:dict> </s:key> </s:dict> </s:key> <s:key name="messages"> <s:dict/> </s:key> <s:key name="request"> <s:dict> <s:key name="auto_cancel">0</s:key> <s:key name="auto_pause">0</s:key> <s:key name="buckets">0</s:key> <s:key name="earliest_time"></s:key> <s:key name="index_earliest"></s:key> <s:key name="index_latest"></s:key> <s:key name="indexedRealtime"></s:key> <s:key name="indexedRealtimeMinSpan"></s:key> <s:key name="indexedRealtimeOffset"></s:key> <s:key name="latest_time">now</s:key> <s:key name="lookups">1</s:key> <s:key name="max_count">500000</s:key> <s:key name="max_time">0</s:key> <s:key name="reduce_freq">10</s:key> <s:key name="rt_backfill">0</s:key> <s:key name="rt_maximum_span"></s:key> <s:key name="sample_ratio">1</s:key> <s:key name="spawn_process">1</s:key> <s:key name="time_format">%FT%T.%Q%:z</s:key> <s:key name="ui_dispatch_app"></s:key> <s:key name="ui_dispatch_view"></s:key> </s:dict> </s:key> <s:key name="eai:acl"> <s:dict> <s:key name="perms"> <s:dict> <s:key name="read"> <s:list> <s:item>*</s:item> <s:item>splunk-system-user</s:item> </s:list> </s:key> <s:key name="write"> <s:list> <s:item>*</s:item> <s:item>splunk-system-user</s:item> </s:list> </s:key> </s:dict> </s:key> <s:key name="owner">splunk-system-user</s:key> <s:key name="modifiable">1</s:key> <s:key name="sharing">global</s:key> <s:key name="app">splunk_archiver</s:key> <s:key name="can_write">1</s:key> <s:key name="ttl">7200</s:key> </s:dict> </s:key> <s:key name="searchProviders"> <s:list/> </s:key> </s:dict> </content> </entry> <entry> ... </entry> <entry> ... </entry> ... </feed> So instead of one simple <sid/> property in <response/>, the SID is embedded in one of nested <entry><s:dict><s:key/> properties, like <s:key name="sid">scheduler__nobody_c3BsdW5rX2FyY2hpdmVy__RMD5473cbac83d6c9db7_at_1615846620_1</s:key>. (Even SID format is very different from the document.) In fact, the return is a job list instead of a single job. I am not sure if this makes a difference: I am using an authorization token to authenticate with the API. The <author/> of the response, meanwhile, is always splunk-system-user instead of the user that the token belongs to. Additionally, I am not able to get any output when querying results of the returned SID. In Splunk Web, all jobs submitted by splunk-system-user shows in application "splunk_archiver" instead of search which is the default application when I search in Splunk Web. The user to which the authorization token belongs to has role of "user" and default app of "launcher" like any other user.

yuanliu · ‎02-25-2020

I got the exact opposite observation, hence the same question. index=myindex | join field1 [inputlookup table1] | more filters vs index=myindex | lookup table1 field1 as field1 OUTPUTNEW field2 as field2 | where field2 == 'value2' | more filters The first search (join) nearly quadruples the time used by the second (lookup). More interestingly, join itself only consumes a fraction of the extra time. (My lookup table is only a few lines.) To make matter even more interesting, this search (without explicit join) index=myindex [ | inputlookup table1 |fields field1 ] | more filters is about as fast or marginally faster than the second (lookup). All three are functionally identical for my purpose.

yuanliu · ‎11-25-2019

Maybe you can give an example of your desired resultant search? If I understand it correctly, you want some sort of search according to the messageID you click. If so, you can set up a custom search/panel using the token $click.value2$ as value of messageID.

yuanliu · ‎11-22-2019

Delta cites an example using sort - _time . Is there a difference in efficiency between this sort and reverse ?

yuanliu · ‎11-21-2019

Looks like @sabirmgd reported this a year ago in https://answers.splunk.com/answers/610037/my-search-string-is-truncated-after-a-question-mar.html and @mayurr98 diagnosed it correctly. The problem is in question mark, not in parenthesis. (The equal sign encoding is still odd.)

yuanliu · ‎11-21-2019

Obviously this can be worked around by hard encoding, perhaps even hard code time range tokens, too. But this behavior is very undesirable.

yuanliu · ‎11-21-2019

Update: I found this question https://answers.splunk.com/answers/610037/my-search-string-is-truncated-after-a-question-mar.html that correctly identified the culprit as question mark ? . If I want to do regex in drilldown, the link returns "Unbalanced quotes"; search windows shows the search as cut off at the first left parenthesis, even though the URL appears to contain the full string. The only difference I can see is that the URL contains something like q=sourcetype=%22mytype%22 instead of q=sourcetype%3D%22awx_json%22 , i.e., all equal signs in the search is no longer encoded. If I do not have parenthesis after pipe, the drilldown automatically encodes equal sign. (Even so, it is not obvious why the resultant search window shows a cutoff at parenthesis.) Additionally, if I urldecode the string, the resultant search is still invalid because the time range, as specified in tokens, gets attached to the last filter (with ampersands - instead of inserting into the first search command) which renders the last filter invalid.

yuanliu · ‎11-21-2019

It looks like a problem only in Edit mode. I'll close this one although it looks like a bug.

yuanliu · ‎11-21-2019

I have two dashboards with custom drilldown search. One of them works just fine. But the other returns a blank page; looking into the SID I can see this error: error:133 - Masking the original 404 message: 'Unexpected query string parameters: earliest, q, latest' with 'Page not found!' for security reasons This actually happens to most custom drilldown searches that I try to set up, no matter which dashboard. I use a CNAME for my server. If I change the URL to use the canonical name of the server, the "security" error goes away. Although this kind of makes sense, it doesn't explain why some custom searches do not trigger this problem. In the problematic dashboard, it fails even the simplest search like sourcetype = this , i.e., no special character of any type.

yuanliu · ‎10-23-2019

I downvoted this post because the correct syntax should have only one backslash escape.

yuanliu · ‎10-03-2019

I have INDEXED_EXTRACTIONS = json and TIMESTAMP_FIELDS = my_timestamp_field in [my_json_type] stanza. This works when I upload a file and select my_json_type as source type. But when I post the exact same data via REST API's receiver endpoint, no field extraction happens. (Both datasets returned by search sourcetype=my_json_type.) How can I make this work for both file upload and REST API?

yuanliu · ‎09-20-2019

Are you looking for something like rex max_match=0 ? https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Rex max_match Syntax: max_match= Description: Controls the number of times the regex is matched. If greater than 1, the resulting fields are multivalued fields. Default: 1, use 0 to mean unlimited.

yuanliu · ‎09-19-2019

For a data set like this: stage=Cstage1 status=h1_status1 host=host1 _time=time1 stage=Astage2 status=h1_status2 host=host1 _time=time2 stage=Bstage3 status=h1_status3 host=host1 _time=time3 ... I would like to report to something like this: host Cstage1 Astage2 Bstage3 host1 h1_status1 h1_status2 h1_status3 host2 h2_status1 h2_status2 h2_status3 ... It is important that those column names be sorted by time order. If I use xyseries or chart commands, they list dynamic column names by alphabetic order, i.e., host Astage2 Bstage3 Cstage2 host1 h1_status2 h1_status3 h1_status2 host2 h2_status2 h2_status3 h2_status2 ... which looks plain weird and incomprehensible. So far I can only achieve my desired order by manually adding a numeric prefix like | eval stage=case(stage="Cstage1", "1-Cstage1", stage="Astage2", "2-Astage2", stage="Bstage3", "3-Bstage3") Alternatively, I can use eval {stage} like | eval {stage} = status | stats list(Cstage1) list(Astage2) list(Bstage3) by host This is hard and only work in limited cases. (In the past, I probably used some eval + string tricks to work with more "stages". Still awkward nonetheless.) What is the "proper" way to persuade xyseries or chart to do what I wanted?

yuanliu · ‎09-17-2019

Thank you @jacobevans. Override CSS in simple XML is pretty cool. I'll have to dig into this later.

yuanliu · ‎09-16-2019

I have a case when results are few, if any. In dashboard, fewer rows takes up smaller vertical space, but if no result returns, it takes up a taller default. I would rather align the tables with smaller vertical space than forcing users to scroll. Is there a way to achieve this? (Display table header is just one such thought but I don't know how.)

yuanliu · ‎11-07-2017

Without clarity in data constraints, like what does it mean by "different patterns" and "one specific field", it is hard to give any definitive answer. Are you looking for the same type of data embedded in different contexts, or are you looking for a potentially different data type when context is different? Regex is extremely versatile when matching patterns if you are willing to invest in it. It all depends on the actual data, is there an order of appearance for patterns, how regular the rest of the patterns are, and so on. In my experience, there are advantages and disadvantages for one-go vs multiple rex commands. If you know your data well, it is nearly always possible to construct a single rex command to extract every pattern if you work hard enough. But if you want to capture just those combinations that you haven't thought of, multiple runs are easier. One whacky example is this pair of constructs: William is my nephew My auntie is Betty How to extract the relationship in one go? You can use ^((?<name>.+) (is|are) (my|our|your|his|her|their) (?<relation>.+)|(My|Our|Your|His|Her|Their) (?<relation>.+) (is|are) (my|our|your|his|her|their) (?<name>.+)) . Is this doable? Perhaps yes. Is this practical? Probably not. Machine data are probably better than natural language. But if you are parsing application logs, more often than not, different pieces of a pattern may appear in some context and not in others. Nowadays I just use multiple passes unless I have confidence that the data space is limited.

Posts	161
Solutions	12
Karma Given	48
Karma Received	49
Member Since	‎11-12-2013

Online Status	Online
Date Last Visited	Monday

Is automatic new line insertion deprecated?

How to get REST API to respond with simple XML?

Is there a difference in efficiency between this s...

Bug? Custom drilldown cannot handle question mark

Bug? Drilldown search causes "error:113", in dashb...

How to make index-time field extraction work for R...

How to sort dynamic column names by time?

How to display table header in dashboard even with...

Why does numeric input value keep dashboard search...

chart over refuses to show OTHER group

Re: Using CIDR in a lookup table

Is automatic new line insertion deprecated?

Re: Splunk Systemd Service

Re: How to get REST API to respond with simple XML...

How to get REST API to respond with simple XML?

Re: Lookup vs join with inputlookup

Re: Is it possible to display events of multiple _...

Is there a difference in efficiency between this s...

Re: Bug? Custom drilldown cannot handle question m...

Re: Bug? Custom drilldown cannot handle question m...

Bug? Custom drilldown cannot handle question mark

Re: Bug? Drilldown search causes "error:113", in d...

Bug? Drilldown search causes "error:113", in dashb...

Re: Search for result with double quotes

How to make index-time field extraction work for R...

Re: Multiple Results in an alert emails subject

How to sort dynamic column names by time?

Re: How to display table header in dashboard even ...

How to display table header in dashboard even with...

Re: Option to extract one specific field from diff...