Hello,
Having followed instructions given in this link - http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Routeandfilterdatad#Keep_specific_events_an....
I am looking to only index events that match my selection criteria. Here's what my setup is like:
The default configuration brings in a flood of events of which I'm only looking to index a subset - 4 to be precise. I want to index only events with sla=abc, sla=def, sla=ghi, or sla=jkl.
My props.conf and transforms.conf contain this:
props.conf:
[snow:task_sla]
TRANSFORMS-set = setnull,setparsing
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = sla="(abc|def|ghi|jkl)"
DEST_KEY = queue
FORMAT = indexQueue
However, all events are still getting indexed into snow:task_sla. How can I troubleshoot this further to isolate the problem please?
I'm unfamiliar with Regex so have tested my regex expression on regexr.com. The search expressions on such sites are usually prefixed and suffixed by a "/" and "/g" respectively. I tried including that in my search to no avail.
I have also tried using only REGEX = (abc|def|ghi|jkl) in my search but that doesn't help either.
For info I'm using the ServiceNow Splunk addon to index ticket data in Splunk. The ServiceNow addon uses RestAPI calls to pull data from the ServiceNow server. My understanding is that the RestAPI call only pulls information into my Splunk server but the indexation process is the same as for any other data input. Could the RestAPI calls be causing issues here?
Hi
only one additional information: do you have in your logs "sla=abc" as a string or "sla" is an extracted field and "abc" is one of the values?
In both the cases, try with a different Regex:
if first case,
REGEX = sla\=abc|sla\=def|sla\=ghi|sla\=jkl
in second case try
REGEX = abc|def|ghi|jkl
Beware that "=" is a special char for regexes!
Bye.
Giuseppe
Hi Guiseppe,
Thanks for responding. In my case it's the former so I have sla=abc in my logs.
I've nevertheless tried both your options and neither work for me. I still see events with sla=zyx getting indexed.
Thanks,
Amrit
try to use a single condition sla=abc and see if events are filtered.
Bye.
Giuseppe
REGEX = sla=abc
REGEX = abc
Tried both. Still indexes sla=zyx and others. Weird!
Thanks,
Amrit
I might have found something.. Events list abc in double quotes so I will need to include that in my regex condition. I tried the expression below but to no avail again.
REGEX sla="abc"
Thanks,
Amrit
if you have quotes, not try
REGEX = sla\=\"abc\"|sla\=\"def\"|sla\=\"ghi\"|sla\=\"jkl\"
or
REGEX = sla\="abc"|sla\="def"|sla\="ghi"|sla\="jkl"
Bye.
Giuseppe
Both of the above regex searches aren't filtering anything out. Here's a sample event.
endpoint="https://xxx.service-now.com/",business_duration="1970-01-01 00:00:00",start_time="2017-11-21 09:21:32",duration="1970-01-01 00:00:00",sys_updated_on="2017-11-21 09:21:33",has_breached="false",sys_created_on="2017-11-21 09:21:33",sys_id="d6f9a8s7d6f9a78sd6f9a78sd6f9a78s",sys_tags="",end_time="",pause_time="",task="sd9f8g6s9df8g69sd8f7g6sdfgs9df87",planned_end_time="2017-11-28 09:21:32",active="true",pause_duration="",sla="xyz",original_breach_time="2017-11-28 09:21:32",percentage="0",sys_mod_count="0",sys_updated_by="asdf",business_time_left="1970-01-03 02:00:00",stage="in_progress",timezone="Europe/London",schedule="asd9f8asd9f78as6d9f876sd9f786asd",business_percentage="0",time_left="1970-01-08 00:00:00",business_pause_duration="",sys_created_by="asdf"
host = splunk source = https://xxx.service-now.com/ sourcetype = snow:task_sla
Here's a btool snippet of my transforms.conf:
[setnull]
CAN_OPTIMIZE = True
CLEAN_KEYS = True
DEFAULT_VALUE =
DEST_KEY = queue
FORMAT = nullQueue
KEEP_EMPTY_VALS = False
LOOKAHEAD = 4096
MV_ADD = False
REGEX = .
SOURCE_KEY = _raw
WRITE_META = False
[setparsing]
CAN_OPTIMIZE = True
CLEAN_KEYS = True
DEFAULT_VALUE =
DEST_KEY = queue
FORMAT = indexQueue
KEEP_EMPTY_VALS = False
LOOKAHEAD = 4096
MV_ADD = False
REGEX = sla\=\"abc\"
SOURCE_KEY = _raw
WRITE_META = False
Btool snippet of props.conf:
[snow:task_sla]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = AUTO
DATETIME_CONFIG = \etc\datetime.xml
FIELDALIAS-snow:task_sla:tak_sys_id = task AS task_sys_id
HEADER_MODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
LOOKUP-sla_supplier_lookup = sla_supplier_lookup sys_id AS sla OUTPUTNEW name AS sla_definition priority sla_supplier
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRANSFORMS-set = setnull,setparsing
TRUNCATE = 10000
detect_trailing_nulls = auto
maxDist = 100
priority =
sourcetype =
Hi amrit_narsipur,
regex is correct (see https://regex101.com/r/B1V4Wl/1)
try to simplify your transforms.conf removing all row except
[setparsing]
DEST_KEY = queue
FORMAT = indexQueue
REGEX = sla\=\"xyz\"|sla\=\"def\"|sla\=\"ghi\"|sla\=\"jkl\"
Bye.
Giuseppe
Hi Guiseppe,
That didn't work either. I amended things to something more elementary to try and debug this - I tried to filter out certain events just to check if the REGEX search expressions were working. I edited my conf files to below to see if events with sla="abc" were getting filtered out.
props.conf:
[snow:task_sla]
TRANSFORMS-set = setnull
transforms.conf:
[setnull]
REGEX = sla\=\"abc\"
DEST_KEY = queue
FORMAT = nullQueue
Since the REGEX expressions are correct I should ideally see all events except for sla="abc" but that is not happening. For some reason it looks like Splunk is either not running the REGEX condition or they aren't working.
Hi amrit_narsipur,
try a different regex with both the stanzas:
props.conf
[snow:task_sla]
TRANSFORMS-set = setnull,setparsing
transforms.conf
[setnull]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = .
[setparsing]
DEST_KEY = queue
FORMAT = indexQueue
REGEX = (sla\=\"xyz\")|(sla\=\"def\")|(sla\=\"ghi\")|(sla\=\"jkl\")
or simply REGEX = (sla\=\"xyz\")
Bye.
Giuseppe
Hi Guiseppe,
I've tried your option of REGEX = (sla=\"xyz\") and it still doesn't work.
Is there anyway I can debug this - to understand why Splunk is not executing the REGEX check? Because I can't think of any other avenue here at the moment.
Also, from an ease of use perspective, is it necessary for me to restart splunk every time I amend conf files with such changes or would running https://splunkserver:port/debug/refresh suffice?
Thanks,
Amrit
Hi amrit_narsipur,,
One additional information: where do you put props.conf and transforms.conf?
they have to be on the indexer.
then in which app are they?
it's better to put them in the app where are the used sourcetype (snow:task_sla).
Bye.
Giuseppe
Hi Guiseppe,
This is a single system setup so all setups are on the same machine. I have validated the folder and app locations of the conf files and they're all OK.
Thanks,
Amrit
Hi amrit_narsipur,
yesterday I had a similar problem that I solved changing the props.conf and transforms.conf location, putting them in the same app where sourcetype is defined.
Bye.
Giuseppe