About eddit0r

eddit0r · ‎03-15-2015

You can have more than one transform stanza referenced under a single TRANSFORM heading in props.conf, you just add commas between them. Something like this should work. Transforms.conf [setnull1] REGEX=(?m)EventCode=(515|517|537|538|540|551|552|560|565|566|576|577|578|592|593|600|601|626|627|628|629|633|641|642|645|646) DEST_KEY = queue FORMAT = nullQueue [setnull2] REGEX = (?mi)user\s*?=\s*?(userid1|userid2|userid3) DEST_KEY = queue FORMAT = nullQueue props.conf [WinEventLog:Security] TRANSFORMS-setnullqueue = setnull1, setnull2

eddit0r · ‎01-15-2015

Can you display the output from using eval full_name = given." ".sn ? That should work. http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Eval

eddit0r · ‎01-05-2015

The most efficient way to do the extraction in splunk is to use the REPORT feature and a transforms.conf entry. props.conf [gmail-mbox] REPORT-extract-headers = extract-headers transforms.conf [extract-headers] REGEX = ^([^:]+):([^\r\n]+) FORMAT = $1::$2 Or if you want to make it specific to just those headers mentioned you can make it explicit as such. transforms.conf [extract-headers] REGEX = ^(X-Gmail-Labels|Delivered-To|Subject|From):([^\r\n]+) FORMAT = $1::$2 That should grab the fields and values in one repeatable operation.

eddit0r · ‎01-04-2015

For the LINE_BREAKER to work there needs to be a capture group. You should specify the following in props.conf props.conf SHOULD_LINEMERGE = FALSE LINE_BREAKER = ([\r\n]+)From\s\d+@ That will break where there is a carriage return or new line, followed by From 'space' any number of digits and an @ symbol. See how you go. (It is always preferable to delimit multi-line events with LINE_BREAKER as it has significant benefits to processing speed)

eddit0r · ‎12-11-2014

I have a datasource which includes a timestamp without a timezone forwarded from a single collection source, and some other location data in the event which I can use to categorise the data into their respective timezones. As the data is forwarded from a single source, I cannot set the timezones at input.confs without a script to pre-process the data, which I would like to avoid. The output format of the data is fixed and I have no access to the source. Source: foo Sourcetype: bar Event: AAA,foobar,AUVI,201412121738,324521345,3452345,IVXIJB1,3,0,1,2346,357,8 So we have the following props and transforms: props.conf [bar] SHOULD_LINEMERGE = false TIME_PREFIX = (?:[^,]*,){3} TIME_FORMAT = %Y%m%d%H%M TRANSFORMS-bar_create_host = bar_create_host [host::AUVI*] TZ = Australia/Melbourne [host::AUNS*] TZ = Australia/Sydney [host::AUQL*] TZ = Australia/Brisbane [host::AUSA*] TZ = Australia/Adelaide transforms.conf [bar_create_host] SOURCE_KEY = _raw REGEX = (?^AAA),(?[a-zA-Z0-9]*),(?[A-Z0-9]*) FORMAT = $3 DEST_KEY = MetaData:Host ` The host gets rewritten fine, the timestamp extracts fine, but the timezone will not apply based on the rewritten host stanza in props.conf. the date_zone field is always 'local' (should be 600 if set to Australia/Melbourne / UTC+10:00) I have tried rewriting the source instead of the host field and setting the TZ entry under a source::foo based stanza in props.conf, I have tried setting priority of the associated props stanzas to 1 and 100 in alternating configs to no avail. At the moment I'm stuck on using a heavy forwarder to do the host rewrite then forcing the input of the cooked data to be reparsed at the indexer so the TZ entry is honoured on a per-host or per-source basis. That is really more cumbersome than I would like. Is this a bug or is the parsing of these events working as designed, that is you cannot set the TZ at index/parsing time by a rewritten host or source stanza? Any assistance appreciated. I would like to avoid a scripted input if possible. This is in a distributed install on RHEL/OEL 5 & 6 x64, Splunk version 6.1.5 on forwarders, indexers and search heads.

eddit0r · ‎12-01-2014

Not at the moment, at least, not without something to abstract a hierarchical filesystem in front of it. Most object filesystems will fail Splunk's file locktest. You can use things like amazon S3 for storage of frozen/archived splunk data, but not for the live indexes. http://docs.splunk.com/Documentation/Splunk/6.2.0/Installation/Systemrequirements#Supported_file_systems

eddit0r · ‎11-20-2014

That is definitely a concern for scale, I would be looking at that approach only in the instance of realtime search or summary generation. The kvstore provides some additional flexibility for population/update of the incoming data as compared to very large lookups.

eddit0r · ‎11-20-2014

For the multi-line events you need to configure the linebreaking. For the best performance use SHOULD_LINEMERGE = false & LINE_BREAKER in props.conf See http://docs.splunk.com/Documentation/Splunk/latest/Data/Indexmulti-lineevents When left to its own devices Splunk and SHOULD_LINEMERGE = true, Splunk will attempt to break on datestamps. When using LINE_BREAKER there needs to be a capturing group in the regex - eg ([\r\n]+) the default is any number of new lines or carriage returns. That denotes the end of the event and the start of a new one - the captured data is removed. For you, something like: props.conf SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+).*(?:\[\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2},\d*\]) TRUNCATE = 0 That should break on new lines that contain the date/timestamp in the square brackets. *note (?:xxxx) is a non-capturing regex group, that data is not removed.

eddit0r · ‎11-20-2014

A use case like this would be a good application for the KV store lookups introduced in Splunk 6.2, the nosql based db for the 'no fly list' could be applied to the travellers index on a rt or summary based search. http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/ConfigureKVstorelookups

eddit0r · ‎11-19-2014

I don't think the parentheses are the issue, more the trailing space, I think that is being trimmed by the parser. What happens if you make the final \s in that regex optional (using a ?) My test was the following search: index=_internal | head 1 | eval test="AND (sourcetype=\"\") " | rex field=test mode=sed "s/AND\s$sourcetype=\"\"\s?$/foo123/g" It replaces the string fine with the final \s? but not with \s alone. See: KV_TRIM_SPACES in props.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Posts	10
Solutions	1
Karma Given	7
Karma Received	6
Member Since	‎04-03-2012

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

How to apply a timezone (TZ) props.conf entry to a...

Re: How to filter out multiple fields in the same ...

Re: How to concatenate two fields and display as o...

Re: How to write the regex for field extractions o...

Re: Event break regex - match 19 digit number

How to apply a timezone (TZ) props.conf entry to a...

Re: Does splunk currently support object based sto...

Re: Cross-referencing No-Fly lists with passenger ...

Re: Why splunk logging truncates rather than wrapp...

Re: Cross-referencing No-Fly lists with passenger ...

Re: Why regex in search query is now failing, but ...