About Stephen_Sorkin

Stephen_Sorkin · ‎09-08-2010

I'd suggest running: "splunk cmd btool inputs list-debug monitor 'monitor:///var/log'". I'm curious what btool has to say about the stanzas, and make sure that they look right.

Stephen_Sorkin · ‎09-08-2010

We intend to leave the feature in in its "half-working" mode until we fix it or provide a better technique for extracting fields based on a dynamic condition. You are correct in saying that the message is more accurately stated as that you can't search for eventtype-extracted fields.

Stephen_Sorkin · ‎09-08-2010

Your configuration should work fine. Have you cleaned your index and restarted to reread the files?

Stephen_Sorkin · ‎09-07-2010

This is a fairly simple search, provided that you extract the fields City and Response from a set of events, where each event is one line of your data: ... | chart count by City Response

Stephen_Sorkin · ‎09-07-2010

Your dashboard should just refer to source=boat_telemetry_A. Users visiting the dashboard will see the indexes they have access to, so users with access to (only) index=boat1 will see (only) that source for index=boat1.

Stephen_Sorkin · ‎09-05-2010

You should be able to make field3 a multivalued field that contains its original value as well as the value ALL so that the stats command calculates the global sum and count for every value of field4 , stored as field3=ALL and then aggregated properly later. Let's assume that field3 never has commas in it: ... | eval field3 = field3 . ",ALL" | makemv delim="," field3 | stats sum(field1) as sumf1, count(field2) as countf2 by field3 field4 | eval avgf5 = (sumf1/countf2) | chart first(avgf5) over field4 by field3 If field3 does have commas in it, you can just pick another delimiter.

Stephen_Sorkin · ‎09-04-2010

You can search index=_audit to see which users ran searches. The user activity dashboard in the Search app provides an example of this.

Stephen_Sorkin · ‎09-04-2010

No, Splunk doesn't provide per-result set branching logic in the search language. I would script this using the Python SDK: import time import splunk import splunk.auth as au import splunk.search as se splunk.mergeHostPath('localhost:4001', True) key = au.getSessionKey('admin', 'changeme') d = se.dispatch('search index=_internal | head 10') while not d.isDone: time.sleep(1) if d.resultCount > 0: d.setFetchOption(search='sendemail [email protected] [email protected] server=ip1.splunk.com subject=myresults sendresults=true') r = d.results[0] You can then run this via: splunk cmd python <scriptname>.py A shell script may be even easier.

Stephen_Sorkin · ‎09-04-2010

If security is your most important concern, you should ABSOLUTELY use a separate index per boat and have individual users only have access to their boat. The administrative user can have access to all indexes and hence can report across all boats at the same time, for example: index=boat* | stats ... by index You do not need a separate dashboard per boat. Each user's dashboard will only reference the data in the index they have access to. It is very, very important here to NOT schedule the search for the dashboard. Shared scheduled searches (for dashboards) will run as the owner, but the results will be available to all users with access. In this case, if the administrative user shares the scheduled search, all users will see data about all boats. For bursty and sparse data, the biggest challenge is that a simple timechart . If you know when the bursts of data start, you can use a time-based lookup table to assign a run id (and perhaps start time to data in each burst. This can be done after the fact. Then you can make it easier to retrieve and summarize each batch of data.

Stephen_Sorkin · ‎09-04-2010

I think that you're mistaken about multiple indexes. You can search (and correlate) across many indexes if you have access to several indexes using either index=boat* or index=boat1 OR index=boat2 OR ....

Stephen_Sorkin · ‎09-04-2010

This isn't really accurate if you make a separate index per boat. Typeahead only will access indexes that the user has access to. This is one of the key benefits of using indexes to segregate data.

Stephen_Sorkin · ‎09-04-2010

I would use transaction as gkanapathy suggests in his answer. You can then post-filter those transactions to only include those with BOTH sourcetypes. For example: sourcetype=ips OR sourcetype=*syslog* | transaction maxpause=1s | search sourcetype=ips sourcetype=*syslog*

Stephen_Sorkin · ‎09-02-2010

For something this complex, I'd suggest making a custom python search command. Since the python command is given an authentication token, it should be able to run the first search, render it as desired and run the second and finally render and run the third search.

Stephen_Sorkin · ‎09-01-2010

You should be able to use the span=1w directive for timechart . For example: ... | timechart span=1w count . If this doesn't do it, you can tune the chart properties if you put the chart on a dashboard. See http://www.splunk.com/base/Documentation/latest/Developer/ChartReference as a reference.

Stephen_Sorkin · ‎08-31-2010

date_hour and date_minute are set at index time and are based on what we extracted from the event itself. The timestamp to the left of every event is rendered from the local time of the machine running the splunk searches. To correct the rendered time, you can either change the timezone of the machine to match your expectation, or change it just for splunkd by running (on Unix platforms): TZ=<desired timezone> splunk start

Stephen_Sorkin · ‎08-31-2010

If you set the sourcetype of this data to be "syslog" we will automatically extract the hostname into the host field from the log lines. If you don't want to change your sourcetype, you can configure this in props.conf with: [<sourcetype>] TRANSFORMS-host = syslog-host

Stephen_Sorkin · ‎08-31-2010

Yes, there is a limit and it's configurable in limits.conf in: [search] # the maximum number of concurrent searches per CPU max_searches_per_cpu = 4 # the base number of concurrent searches base_max_searches = 4 # max real-time searches = max_rt_search_multiplier x max historical searches max_rt_search_multiplier = 3 # the total number of concurrent searches is base_max_searches + #cpus*max_searches_per_cpu A separate per-user limit exists in authorize.conf in: [role_<role>] srchDiskQuota = <number> * Maximum amount of disk space (MB) that can be taken by search jobs of a user that belongs to this role * Defaults to '100', for 100 MB. srchJobsQuota = <number> * Maximum number of concurrently running historical searches a member of this role can have (excludes real-time searches, see rtSrchJobsQuota) * Defaults to 3 Saved searches count towards this limit, but they have a stricter quota expressed as a percent of the system-wide quota in limits.conf : [scheduler] # the maximum number of searches the scheduler can run, as a percentage # of the maximum number of concurrent searches max_searches_perc = 25 There is also a per-savedsearch concurrent limit which defaults to 1 and is set per-search in savedsearches.conf : [<search_name>] max_concurrent = <int> * The maximum number of concurrent instances of this search the scheduler * is allowed to run. * Defaults to 1

Stephen_Sorkin · ‎08-31-2010

There is no analogous search command to write a JSON formatted file from within a search itself. You can run a search using the REST API (http://www.splunk.com/base/Documentation/latest/Developer/RESTIntro) and fetch the results in JSON format using the argument output_mode=json from the events, results or results_preview resources.

Stephen_Sorkin · ‎08-31-2010

You should use an encrypted filesystem. Most modern operating systems offer one as a choice.

Stephen_Sorkin · ‎08-31-2010

No, but you can use eventtypes or list lookups. However, if you use any of these knowledge objects, you should note that a user can override or edit them for their local context, which often defeats the purpose in a search filter.

Stephen_Sorkin · ‎08-31-2010

gkanapathy's first search will work, but his seconds will not (as you have to look at the "Aide" messages to know which hosts to exclude). You can also use stats to solve this: tag=pci source=/var/log/messages | eval has_Aide = if(searchmatch("Aide"), 1, 0) | stats values(has_Aide) as has_Aide by host | search has_Aide=0 AND has_Aide!=1

Stephen_Sorkin · ‎08-30-2010

I'd normally recommend a lookup as well, but my guess is that the actual eventtypes have wildcards, which CSV lookups don't play well with.

Stephen_Sorkin · ‎08-30-2010

Yes, you should be able to do this by manually running the typer command after the stats count . For example: index="summary" search_name="si_useragent" | stats count by UserAgent | typer | eval eventtype=mvfilter(match(eventtype, "ua_.*")) | top eventtype

Stephen_Sorkin · ‎08-30-2010

SEDCMD-* is executed before TRANSFORMS-* .

Stephen_Sorkin · ‎08-30-2010

I really would like this to be in the UI as well. The main problem is how the Python appserver relays data from splunkd, but this is a technical issue that we need to work through.

Posts	456
Solutions	178
Karma Given	156
Karma Received	905
Member Since	‎01-14-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Re: separating many similar files into different i...

Re: Extracting fields based on eventtype?

Re: separating many similar files into different i...

Re: Create a Pivot Table

Re: Splunk for a boat race?

Re: Need to calculate weighted average across colu...

Re: Splunk for a boat race?

Re: Sendemail (Splunk CLI) always sends email whet...

Re: Splunk for a boat race?

Re: Splunk for a boat race?

Re: Splunk for a boat race?

Re: Group Events Based Solely on Temporal Proximit...

Re: How can you emulate a sub-subsearch?

Re: How do I set the major unit duration to 1 week

Re: Use date_hour date_minute for charting

Re: change hostname for indexed data dynamically?

Re: Simultaneous queries/jobs limit

Re: Exporting in JSON

Re: Encrypting indexed data on rest

Re: Is it possible to use macros to restrict searc...

Re: Diff of 2 searches

Re: Event Types on a Summary Index

Re: Event Types on a Summary Index

Re: Order of operations? SEDCMD vs TRANSFORMS

Re: Export large search date range in raw format

Join the Conversation