About jmartens

jmartens

Thanks, that did the trick. I had it working before w/o AFAICT, but nevertheless.

jmartens

I have a (classical) dashboard that makes use of a base search and several dependant searches. Some of the dependent searches fail to return results, but when I drill down using the Open in search button I get a nice list of events as expected, when I go to the visualization tab I can also see the graph as expected. Here is the relevant section of the dashboard: Here is the base search definition (using time frame and host from the drop down menus): <search id="clinical_faults_and_warnings"> <query>index=index1 host=$linac$ unknown=Fault | transaction host FaultCode endswith=FaultState=removed </query> <earliest>$date.earliest$</earliest> <latest>$date.latest$</latest> </search> The dependant search that is not showing the graph on the dashboard (top) is defined like this: <search base="clinical_faults_and_warnings"> <query>| search node=COL trajectory | timechart useother=f usenull=f count by leaf </query> </search> The working dependant search (bottom) is defined like this: <search base="clinical_faults_and_warnings"> <query>| timechart useother=f span=1d limit=10 count by FaultCode</query> </search> But when I drill down using the Open in Seach it seems to work OK: Any pointers as to what might be the problem or how to solve this issue?

jmartens · ‎01-20-2025

Thanks @gcusello. I am aware that I need to escape stuff, problem is I do not see where I might have missed one, I already escaped a lot, at least what was required on regex101. It seems your solution works, will continue with that. Thanks!

jmartens · ‎01-20-2025

I have the following regex that I (currently) use at search time (it will be a field extraction once I get it ironed out): User\[(?:(?<SignOffDomain>[^\\]+)(?:\\))?(?<SignOffUsername>[^\]]+)[^\[]+\[\"(?<SignOffComment>[^\:]+)\:\s+(?<SignOffPrivilege>[^\"]+) It seems to work OK on regex101: https://regex101.com/r/nGdKxQ/5 but fails when trying to parse in Splunk with the following error: Error in 'rex' command: Encountered the following error while compiling the regex 'User\[(?:(?<SignOffDomain>[^\]+)(?:\))?(?<SignOffUsername>[^\]]+)[^\[]+\["(?<SignOffComment>[^\:]+)\:\s+(?<SignOffPrivilege>[^\"]+)': Regex: missing closing parenthesis. Any clue on what I need to escape additionally perhaps? For testing I created the following sample: | makeresults count=2 | streamstats count | eval _raw=if((count%2) == 1, "2025-01-20 08:43:11 Local0 Info 08:43:11:347 HAL-TRT-SN1701 DOMAIN\firstname0.lastname0|4832|TXA HIPAA [1m]HIPAALogging: User[DOMAIN\firstname0.lastname0], Comment[\"Successfully authenticated user with privilege: A_Dummy_Privilege\"], PatientId[PatientIdX], PlanUID[PlanLabel:PlabnLabelX,PlanInstanceUID:PlanInstanceUIDX", "2025-01-20 07:54:42 Local0 Info 07:54:41:911 HAL-TRT-SN1701 domain\firstanme2.lastname2|4832|TXA HIPAA [1m]HIPAALogging: User[firstname1.lastname1], Comment[\"Successfully authenticated user with privilege: AnotherPrivilege\"], PatientId[], PlanUID[], Right[True]") | rex field="_raw" "User\[(?:(?<SignOffDomain>[^\\]+)(?:\\))?(?<SignOffUsername>[^\]]+)[^\[]+\[\"(?<SignOffComment>[^\:]+)\:\s+(?<SignOffPrivilege>[^\"]+)"

jmartens · ‎10-29-2024

Although removing through REST probably works I find it easier to do it this way: edit the configuration file in SPLUNK_INSTALL_DIR\etc\system\local\authentication.conf Naviate to Settings > Authentication methods > reload authentication configuration

jmartens · ‎09-08-2024

@wwangsa_splunk Thanks very much for the update, but AFAICT the current release is already 9.3.0 Is it also incorporated in that branch?

jmartens · ‎05-07-2024

That is no relevant to my case since I have no orphaned knowledge base items. Just checked with the instruction from the knowledge base and even with all filters set to 'All' the list turns up empty.

jmartens · ‎05-06-2024

No, unfortunately not. It is bothering but can be worked around by using Splunk itself to analyze the logs and ignore that message at search time. This will show all messages w/o INFO and the before mentioned messages: index="_internal" sourcetype=splunkd NOT INFO NOT "AQR and authentication extensions not supported. Or authentication extensions is supported but is used for tokens only"

jmartens · ‎02-26-2024

By now found out that creating the user `nobody` is not allowed by Splunk, it throws the following error when creating such a user: Create user: The user="nobody" is reserved by the splunk system

jmartens · ‎02-25-2024

We have enabled Microsoft SAML for Splunk and out splunkd.log seems to be flooded with warnings like this: WARN UserManagerPro [7456 SchedulerThread] - AQR and authentication extensions not supported. Or authentication extensions is supported but is used for tokens only. user=nobody information not found in cache Found a few threads on AQR: - https://community.splunk.com/t5/Monitoring-Splunk/What-is-quot-AQR-quot-and-why-is-it-throwing-warning-messages-in/m-p/347016/highlight/true#M3064 - https://community.splunk.com/t5/Security/How-do-you-resolve-splunk-log-error-messages-after-switching/m-p/354479/highlight/true#M8882 Also the documentation on authentication.conf does not help me much. It seems the only way is to create a low level user (same as mentioned in the error) to suppress the error, which seems doable but I doubt this is the best way and unsure of side effects? Does any of you know more?

jmartens · ‎02-25-2024

@skalliger wrote: How did I find all of that? I have developed an app which gives an overview of the Splunk environment's mitigation status against current Splunk Vulnerabity Disclosures (SVDs) as well as recommended best practice encryption settings. Nice work @skalliger . Is this app publically available? I would be interested in this functionality.

jmartens · ‎08-03-2023

Still suffering the same in 9.10.2. Is the bug already fixed or do we still need to apply this fix to silence the message? Apart from that does this disable ingestion latency warnings and do we need to monitor it another way? Can you shed some light on this @jswann_splunk ?

jmartens · ‎06-12-2023

After some more searching I stumbled on mvexpand and had to add it to expand the temp field like this: | makeresults | eval DICOMQuery="(0008,0052) Query/Retrieve Level [SERIES] | (0010,0020) Patient ID [1234567] | (0020,000D) Study Instance UID [1.2.34.567890.12.3456.789.123456.7.20230501130047.4915859] | (0020,000E) Series Instance UID [1.2.345.678.90.1.23456789012.34567890.20230508110959]" | eval temp=split(DICOMQuery,"|") | mvexpand temp | rex field=temp "(?<DICOMGroup>([\dA-Fa-f]{4})),\d?(?<DICOMElement>([\dA-Fa-f]{4}))\)\s+(?<DICOMLabel>[^\[]+)\s\[(?<DICOMValue>[^\]]+)\]" | eval {DICOMLabel}=DICOMValue | fields - DICOMQuery DICOMLabel, DICOMValue temp

jmartens · ‎06-12-2023

I am trying to extract multi value fields and set dynamic fields with values based on the extracted data. I am able to extract the data but am unable to do the last part and generate dynamic fields, only the first is generated: | makeresults | eval DICOMQuery="(0008,0052) Query/Retrieve Level [SERIES] | (0010,0020) Patient ID [1234567] | (0020,000D) Study Instance UID [1.2.34.567890.12.3456.789.123456.7.20230501130047.4915859] | (0020,000E) Series Instance UID [1.2.345.678.90.1.23456789012.34567890.20230508110959]" | eval temp=split(DICOMQuery,"|") | rex field=temp "(?<DICOMGroup>([\dA-Fa-f]{4})),\d?(?<DICOMElement>([\dA-Fa-f]{4}))\)\s+(?<DICOMLabel>[^\[]+)\s\[(?<DICOMValue>[^\]]+)\]" | eval {DICOMLabel}=DICOMValue | fields - DICOMQuery DICOMLabel, DICOMValue temp Which yields the following: It seems I am nearly there, but I fail to get them as individual searchable fields which is what I am after. Any pointers?

jmartens · ‎03-20-2023

I had indeed resolved it using in the mean time: | extract pairdelim="," kvdelim="=" clean_keys=t

jmartens · ‎03-17-2023

@richgalloway wrote: We know it can't be a multi-value field because the rex command does not use the max_match option, which means only the first match of the regex will be extracted. I don't see why the regex here is such a proof. The `rex` is used to extract two named fields, of which the latter contains the values I am after and is assigned to values. I see no reason why `max_match` is even relevant here. @richgalloway wrote: What is the end goal of this query? The mvexpand command puts each value of a multi-value field into a new event - is that what is desired? The end goal is to extract all values (as per column headings from the results in my screenshot) from value field extracted using the regular expression and assigning them to the event so I can process and plot these values.

jmartens · ‎03-16-2023

I am trying to expand multiple fields from specific log lines using mvexpand but for some strange reason some fields are not extracted as expected, see screenshot for an example: I would also like to have the key/value pairs for col and gantry.

jmartens · ‎02-24-2021

This is an oversimplification. In searches you can search on any field you like, but you will have to define them if you are not using a default sourcetype. Relying on _raw and extracting all your fields from there at search time is cumbersome and not future proof.

jmartens · ‎02-24-2021

I really suggest you read up on how to get data in. If you continue on this path you will have to redo a lot off work and will not benefit very much from your data. Even though you accepted the answer pointing you to the _raw event using SPL and text filtering is not the wat to get data in robustly for the future. It all starts with getting your data in in a ordered and structured way, hence my link to the primer on Getting data in. You first need to setup your sourcetype correctly as I mentioned earlier, get all your data in to one event if it is concerning one entity. From there define fields to be extracted at index time. These fields can than be (re-)used when searching and reporting. Creating your fields each time in the query window (e.g. in your SPL query) is cumbersome and not future proof. When defining your sourcetype you can then also set other stuff like custom delimiters so key value pair extraction might be done for up you (automatically/automagically), for instance extracting the values after the colon in your screenshots. Once agian, do yourself a favour and invest in a proper start, it will gain you time and joy in the long run. 😄

jmartens · ‎02-24-2021

Did you even try bother reading the documentation section I referred to? The second example in the examples section more or less handles your case of one event running over multiple lines where each line is prepended with a timestamp.

jmartens · ‎02-24-2021

You don't need a regular expression for that, substring will do perfectly well.

jmartens · ‎02-24-2021

I think you also need to revisit your sourcetype configuration as it seems to split you log entry in (at least) three lines judging from your image. I am assuming that all three lines should be in one event and you want to extract fields from there. This way it is hardly impossible to correlate data to one event concerning your image. As a start you might want to take a look at the Getting data in primer, especially the section on Configure event line breaking

jmartens · ‎02-24-2021

Can you post some example data and an example of desired results? I think your current question is kind of cryptic. From what I could make of it you might be better of using one of Splunks text functions for instance substr: As a simple example: | makeresults | eval test=substr("string", 1, 3) Which should create a dummy test event with a field test with the value 'str' (first three characters of the text 'string')

jmartens · ‎02-24-2021

Assuming the field is always starting with 'SourceSystem=' and ending with a comma (,) you could do this: | rex field=_raw "sourceSystem=(?<field_name>[^,]+)" But as additional advice it seems your data is JSON you might be better of using spath to extract your field(s) or even better configure your sourcetype correctly using props.conf and INDEXED_EXTRACTIONS=json or KV_MODE=json For details see the Getting data in primer especially Extract fields from files with structured data. As an example of what you could do here is a run very-where example: | makeresults count=2 | streamstats count | eval _raw=case(count=1, "{\"timestamp\":\"2021-02-24T00:00:46.533+00:00\",\"message\":\"Snapshot event published: SnapshotEvent(status=CREATED, version=SnapshotVersion(sourceSystem=zvkk, source=sdp\/deposits\/zvkk\/2021-02-24\/NextBusinessDays\/Snapshot1, entityType=NEXT_BUSINESS_DAYS, date=2021-02-24, version=1, snapshotSize=5, uuid=8683aa33-3a6c-4087-9cdd-3084d8e70147, holiday=false))\",\"component\":\"com.db.sdda.dc.kafka.service.SnapshotEventNotifyService\",\"thread\":\"scheduling-1\",\"level\":\"INFO\"}", count=2, "{\"timestamp\":\"2021-02-23T20:56:37.797+00:00\",\"message\":\"Snapshot event published: SnapshotEvent(status=CREATED, version=SnapshotVersion(sourceSystem=IDMS-0781, source=sdp\/deposits\/IDMS-0781\/2021-02-23\/FacilityLimit\/Snapshot1, entityType=FACILITY, date=2021-02-23, version=1, snapshotSize=15168, uuid=016cc1ad-8c27-4144-a9d2-c0233cc1e450, holiday=false))\",\"component\":\"com.db.sdda.dc.kafka.service.SnapshotEventNotifyService\",\"thread\":\"scheduling-1\",\"level\":\"INFO\"}") | spath path=message | eval backup=_raw | eval _raw=message | extract | eval _raw=backup | rename backup as _raw The first lines are to mimic your example, then there is a spath extraction (since I did not bother creating a proper sourcetype configuration) to extract the message part using: | spath path=message Then we need to do some backing up of the raw event as extract only operates on the raw event so we copy _raw to backup, assign message to _raw: | rename _raw as backup message as _raw Now we can run an extract command as you data is already in the default key = value pair format: | extract And then revert our rename command to have our original values again: | rename _raw as message backup as _raw

jmartens · ‎02-24-2021

Situation I am trying to parse events with an unrestricted number of key value pairs that might also include empty values at some places. I would like to extract the part between the closing parenthesis and opening square bracket as the field name without spaces (but don't want them replaced by underscores) This is an example of such data: 2021-02-24 10:02:31 Local0 Info 10:02:31:346 VARC-DCM-01.ad.maastro.nl MAASTRO\VARC-DCM-01$|80012|DICOM Service VARC_DCM_SCP_SVC_Export 021556/Export requested for object with key: (0008,0008) Image Type [DERIVED] | (0008,0016) SOP Class UID [1.2.840.10008.5.1.4.1.1.481.1] | (0008,0022) Acquisition Date [20210223] | (0008,0023) Content Date [20210223] | (0008,0032) Acquisition Time [184740.207] | (0008,0033) Content Time [184740.208] | (0008,1150) Referenced SOP Class UID [1.2.840.10008.5.1.4.1.1.481.5] | (0020,0013) Instance Number [1] | (300C,0002) Referenced RT Plan Sequence [Mergecom.MCitem] | (300C,0006) Referenced Beam Number [1] | (300E,0002) Approval Status [] Working solution using SPL Using this SPL expression (inspired by the example in this question on multiple field extraction) : | eval backup=_raw | rex max_match=0 mode=sed "s/(?:(?:\s\|)?\s)$(?<g>[\da-fA-F]{4}),(?<e>[\da-fA-F]{4})$\s+(?<k>(?:\w+(?:\s*))+)\[(?<v>[^\]]*)\]/\3=\"\4\",/g" | rex mode=sed "s/\s//g" | extract pairdelim=":," kvdelim="=" | rename backup AS _raw I am able to translate this to my desired outcome: Image Type Derived SOP Class UID 1.2.840.10008.5.1.4.1.1.481.1 Referenced Beam Number 1 Approval Status Example in SPL (for testing) Here is a working example to help with testing: | makeresults | eval _raw="2021-02-24 10:02:31 Local0 Info 10:02:31:346 VARC-DCM-01.ad.maastro.nl MAASTRO\VARC-DCM-01$|80012|DICOM Service VARC_DCM_SCP_SVC_Export 021556/Export requested for object with key: (0008,0008) Image Type [DERIVED] | (0008,0016) SOP Class UID [1.2.840.10008.5.1.4.1.1.481.1] | (0008,0022) Acquisition Date [20210223] | (0008,0023) Content Date [20210223] | (0008,0032) Acquisition Time [184740.207] | (0008,0033) Content Time [184740.208] | (0008,1150) Referenced SOP Class UID [1.2.840.10008.5.1.4.1.1.481.5] | (0020,0013) Instance Number [1] | (300C,0002) Referenced RT Plan Sequence [Mergecom.MCitem] | (300C,0006) Referenced Beam Number [1] | (300E,0002) Approval Status []" | eval backup=_raw | rex max_match=0 mode=sed "s/(?:(?:\s\|)?\s)$(?<g>[\da-fA-F]{4}),(?<e>[\da-fA-F]{4})$\s+(?<k>(?:\w+(?:\s*))+)\[(?<v>[^\]]*)\]/\3=\"\4\",/g" | rex mode=sed "s/\s//g" | extract pairdelim=":," kvdelim="=" | rename backup AS _raw Question Now I would like to transfer this to configuration files but I am unsure what to add where. I am guessing the regular expression goes in to tokenizer.conf based on this post but not sure when combined with the sed command. Normally SED commands I would put the SED commands into the transforms.conf file but how do I prevent them from applying to all evens? The events like the one processed in the example is only a subset of the events in the index and sourcetypes in there. The pairdelim and kvdelim are overrides to the default ones from the sourcetype configuration, not sure where to put this either. Can someone guide me here? Is there some sort of sequence I can configure like the one in SPL to apply to specific events? How would I go about filtering out these events?

Posts	43
Solutions	3
Karma Given	3
Karma Received	3
Member Since	‎08-01-2015

Online Status	Offline
Date Last Visited	Wednesday

Dashboard not showing results of dependent search,...

Howto escape double quote in regex when using rex

SAML and logs flooded with warnings: AQR and authe...

How to extract multivalue data to dynamic fields?

Why is mvexpand not extracting all key/value pairs...

How to translate mv field extraction from SPL to c...

Extracting key-value pairs using regex at search ...

How to extract multivalue fields and assigning the...

Reuse field extraction in another field extraction...

regex transformation not working in transforms.con...

Re: Dashboard not showing results of dependent sea...

Dashboard not showing results of dependent search,...

Re: Howto escape double quote in regex when using ...

Howto escape double quote in regex when using rex

Re: How to remove users from Splunk registered by ...

Re: SAML and logs flooded with warnings: AQR and a...

Re: SAML and logs flooded with warnings: AQR and a...

Re: SAML and logs flooded with warnings: AQR and a...

Re: SAML and logs flooded with warnings: AQR and a...

SAML and logs flooded with warnings: AQR and authe...

Re: A writeup: KV store / mongo TLS configuration ...

Re: Why are we receiving this ingestion latency er...

Re: Extract multivalue data to dynamic fields

How to extract multivalue data to dynamic fields?

Re: mvexpand not extracting all key/value pairs

Re: mvexpand not extracting all key/value pairs

Why is mvexpand not extracting all key/value pairs...

Re: Extract New Field - First x characters

Re: Extract New Field - First x characters

Re: Extract New Field - First x characters

Re: Extract New Field - First x characters

Re: Extract New Field - First x characters

Re: Extract New Field - First x characters

Re: Unable to extract two different pattern string...

How to translate mv field extraction from SPL to c...

Join the Conversation