topic How to change segmenters to make my data working with PREFIX directive in tstats? in Getting Data In

How to change segmenters to make my data working with PREFIX directive in tstats?

cdaviet — Fri, 11 Aug 2023 07:39:37 GMT

Hi,

I'm trying to use the PREFIX directive in TSTATS (here : https://docs.splunk.com/Documentation/Splunk/9.1.0/SearchReference/Tstats#Use_PREFIX.28.29_to_aggregate_or_group_by_raw_tokens_in_indexed_data).

In the docs, it says that it can work with data that does not contain major breakers such as spaces.

My data contains spaces so I decided to try to change the major breakers this way:

props.conf:

[test_sourcetype] SEGMENTATION = test_segments

segmenters.conf:

[test_segments] MAJOR = \t MINOR = / : = @ . - $ # % \\ _ [ ] < > ( ) { } | ! ; , ' " * \n \r \s & ? + %21 %26 %2526 %3B %7C %20 %2B %3D -- %2520 %5D %5B %3A %0A %2C %28 %29

This way, only the tab (\t) is considered as a major breaker.

I applied this, restarted and tried to ingest a line of log with the sourcetype "test_sourcetype".

Unfortunately, it seems the segmenters.conf does not work because it keeps breaking with a space for example.

I also tried to remove all MINOR and keep only MAJOR, but no luck:

MAJOR = \t
MINOR =

Have I made a mistake? Is it possible to do what I want? I think so because in this .conf presentation (https://conf.splunk.com/files/2020/slides/PLA1089C.pdf) they mention it briefly (page 37).

Should I also use

SEGMENTATION-<segment selection> = <segmenter>

in props.conf ? The docs says it is for SplunkWeb but I am considering all options...

Thanks

Re: How to change segmenters to make my data working with PREFIX directive in tstats?

burwell — Fri, 11 Aug 2023 00:02:44 GMT

I am not expert on this but I guess one thing is to run btool and make sure you are getting the settings that you think you are.

A few months ago I did a quick test to remove double quotes in order that I could use tstats.

props.conf

[my_sourcetype] SEGMENTATION = no_double_quotes

segmenters.conf

[no_double_quotes] MAJOR = [ ] < > ( ) { } | ! ; , ' * \n \r \s \t & ? + %21 %26 %2526 %3B %7C %20 %2B %3D -- %2520 %5D %5B %3A %0A %2C %28 %29

Then I could search with tstats where I had myfield="123"

| tstats count where index=myindex by PREFIX(myfield=)

Re: How to change segmenters to make my data working with PREFIX directive in tstats?

cdaviet — Fri, 11 Aug 2023 07:25:16 GMT

Hi burwell, thanks for taking the time to answer.

I actually ran a test on Splunk Enterprise (was doing it on our SplunkCloud production env) and it works!

So it looks like SplunkCloud does not allow to change this kind of parameters... That's sad.

Anyway, thanks!