About kristian_kolb

kristian_kolb · ‎02-13-2025

TLDR; check your file permissions. A bit late to the party, but this one had me swearing for a a few hours. I work for an MSP, and manage several separate Splunk Enterprise environments. After the latest upgrade to 9.3.2, I noticed that for a few of them the DS was "broken". Checked this post and others and went over configs side-by-side to see if there were any differences in outputs.conf, distsearch.conf, indexes.conf etc. There shouldn't really be many of those, since almost all config is done through puppet, and there are not so many reasons to change settings on an individual server basis. All types of internal logs are forwarded to the indexer cluster. Always. Turns out that the upgrade process (ours or splunk's) had left /opt/splunk/var/log/client_events owned by root, with 700-permissions. No wonder that the files weren't even written to begin with... I suspect on those environments that did work, I had out of habit run a chown -R splunk:splunk to ensure that I hadn't messed up something somewhere. Lesson: check the obvious stuff first.

kristian_kolb · ‎11-09-2021

You seem to be doing the same mistake that the OP did, mixing nullQueueing with index-transformation in general. While there might be more clever ways to solve this HEC-wise or not sending the unwanted stuff in the first place, this should work as a general principle: Set queue = nullQueue for all events Set queue = indexQueue for those events you want to keep, i.e. regex matching file names for any file you want to keep (i.e. the ones in the regexes from the security, application or provisioning parts) Set _MetaData:Index = xxx (basically the existing stuff) Props.conf --- [blablah] TRANSFORMS-dostuff =setnullq, keepsome, setindexsec, setindexapp, setindexprov transforms.conf --- as before but with the addition: [keepsome] REGEX = insert your regex here DEST_KEY = queue FORMAT= indexQueue

kristian_kolb · ‎12-17-2020

Great to hear that it worked. To be honest, though, the app is clearly labeled as "non-supported". It is not an official Splunk product. I sent an email the other day to the person I believe to be the author to suggest this bug. No reply as yet, but I hope it will be fixed in the next release. /Kristian

kristian_kolb · ‎12-17-2020

There is a slight bug(?) in the API calls where a comparison is made related to the checkpoint timestamp. Specifically this is found on line 52 in TA-MS_Defender/bin/input_module_microsoft_defender_atp_alerts.py where it says; atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+ge+%s" % (helper.get_arg("location"), query_date) and I believe that the "ge" (>=) is the culprit here, and that its the reason for the last event from the previous run being indexed again. Changing it to "gt" (>) seemed to fix the problem. atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+gt+%s" % (helper.get_arg("location"), query_date) NB: The same thing is found in inputs_module_microsoft_365_defender_incidents.py in the same directory, but since I don't have that input, I haven't been able to verify that the behaviour is the same.

kristian_kolb · ‎12-30-2019

Sorry for the late reply @pkiripolsky . Got it working, but I can't recall exactly how. Could have been your suggested solution. Thanks.

kristian_kolb · ‎09-06-2018

Yep, this is a slightly old piece of documentation, but it gives a good understanding of what goes where, in terms of configuration items. https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

kristian_kolb · ‎09-04-2018

REGEX = time\"\:\s\"(201[0-7]|2018\-0[0-7]) if you want to have the exclude anything before August 2018.

kristian_kolb · ‎09-04-2018

your subsearch will generate a set of field-value pairs, effectively giving a search like ... | where "Missed Exports Message Alert" NOT in (clearExport=a OR clearExport=b OR clearExport=c) This does not work for two reasons; 1) "NOT in" is not valid syntax. At least not to perform what you wish. 2) "clearExport" is probably not a valid field in the first type of event. on a side-note, I've always used the dot (.) to concatenate strings in eval. I believe that you can alter the subsearch to return the results as values only, which may come closer to what you want to do, i.e. to rename the field to "search" or "query". https://docs.splunk.com/Documentation/Splunk/7.1.2/Search/Changetheformatofsubsearchresults That brings us back to "NOT in" - with the above changes you should probably only need to remove the "in" part. Also, I'm guessing that your search does not really start with "| where", as that would probably not yield any results

kristian_kolb · ‎09-04-2018

Assuming that these strings ("App", "APP", "001" etc) are field values of the same field in both indexes - lets call it 'blah' - you could make a subsearch like so; index=feeindex NOT [search index=di_project_map_merge | dedup blah | table blah] | table blah But perhaps you are looking for something different?

kristian_kolb · ‎09-03-2018

Following a server crash (unknown reason) the collection of S3 logs has partially broken down. Seeing the following in _internal (after enabling DEBUG logging for the S3 generic part). Any ideas on how to get it working again? 2018-09-03 14:38:22,122 level=INFO pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:104 | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 | message="Start processing." 2018-09-03 14:38:22,123 level=DEBUG pid=15104 tid=Thread-4 logger=splunksdc.checkpoint pos=checkpoint.py:build_indexes:169 | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyye31754e" start_time=1535978302 | message="Key was set." pos=221 key="securitylogs/ls.s3.2cf5f2c6-xxxx-4947-bb01-yyyyy96c7e22.2018-08-01T22.30.part11921.txt.gz" some more of the above DEBUG message - different key, but same Job_uid, and then it ends with: 2018-09-03 14:38:22,123 level=ERROR pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:90 | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001" | message="Failed to collect data through generic S3." job_uid="757azzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 85, in index_data self._do_index_data() File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 106, in _do_index_data self.collect_data() File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 139, in collect_data index_store = s3ckpt.S3IndexCheckpointer(self._config) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 141, in __init__ config[asc.data_input], config[tac.checkpoint_dir] File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 80, in get S3CkptPool.S3CkptItem(ckpt_name, ckpt_dir) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 65, in __init__ self.idx_ckpt = LocalKVStore.open_always(ckpt_file_idx) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 156, in open_always indexes = cls.build_indexes(fp) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 163, in build_indexes for flag, key, pos in cls._replay(fp): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 92, in _replay flag, key, _ = umsgpack.unpack(fp) TypeError: 'int' object is not iterable Something seems to have gotten corrupted. Have tried to disable/enable the inputs from within the AWS-addon-app-GUI, but with little success.

kristian_kolb · ‎05-31-2016

Probably the best way is to use EXTRACTs in props.conf with the following regexes; msgno[^<]+(?<msgno>\d+) \s39:\s\]\<(?<field39>\d+) You should probably check the documentation regarding EXTRACT http://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf Also, it may be worth checking out the rex command, which lets you perform regex-based field extractions as part of the search query. At least it is simpler for trying out new extraction patterns before committing them to a config file. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex /k

kristian_kolb · ‎05-25-2016

probably a missing space. table Name Address ... /k

kristian_kolb · ‎05-25-2016

Unless you are batching in your logs somehow to the UF, or have SEVERE file monitor lag on your UF's, the event timestamp (as written by some application) and the UF "read-timestamp" should be fairly close to each other. Within seconds - easily - for most normal situations. The UF program does not maintain its own clock, it uses system time anyway. So if the UF is reading the logs in a timely manner, you would not gain anything. It looks like you are trying to fix the symptom, rather than the problem, as woodcock suggests.

kristian_kolb · ‎04-06-2016

For a role, you can assign access to one or more indexes. However, this has nothing to do with where log data FROM a particular user is stored. The fact that you can see everything is perhaps that you are an administrator, and your role has full access? /k

kristian_kolb · ‎04-06-2016

you can most likely use a simple search to validate your inputs with help of the source attribute, e.g. your_search | timechart span=1d distinct_count(source) values(source) /k

kristian_kolb · ‎03-29-2016

Check this out. https://answers.splunk.com/answers/339930/how-do-forwardedindex-whitelists-and-blacklists-wo.html /K

kristian_kolb · ‎03-29-2016

jmallorquin is right mentioning Heavy Forwarders, that is the only way you can control masking before getting data processed by an indexer (beyond your control). The reason is that the universal forwarder does not read/understand those parts of the configuration in props/transforms, so they are basically ignored by the UF. The deployment server in itself has nothing to do with this, and any app you deploy to a UF through the DS with such configs will do nothing - for the reasons mentioned above. So you should either install a HF (with configs for masking data) and direct your UF's to send to the HF - and of course set the HF to send to your indexer(s). Or you install a HF on each source system instead of a UF, and then you can deploy a masking app via DS. /k

kristian_kolb · ‎03-24-2016

I agree with you that setting up a dedicated syslog server is the better option for the reason of preventing message loss. A syslog daemon is restarted in under a second, whereas restarting Splunk can take a minute or two. Thus, relying to Splunk to do your syslog work can mean that you lose any messages that are sent when Splunk is restarting. Depending on your OS platform, a syslog server may already be installed. RedHat Linux comes with rsyslog by default, and there is no reason to replace it with Syslog-NG. Setting up a dedicated syslog server would also allow you to achieve a higher level of transport security, i.e. RELP (rsyslog) or RLTP (syslog-ng). As for the original question, you should probably try to ensure what the limitations are from the sending side. IIRC I have seen some Nexus versions that could only send to port 514, i.e. destination not configurable. Or if it was that only UDP was supported. Don't remember now, but it's worth checking out before you start configuring the receiving side. /K

kristian_kolb · ‎03-23-2016

That would not differ in effect from coalesce() in case there are non-null user/cause fields in the 'wrong' events.

kristian_kolb · ‎03-23-2016

Then the coalesce function will not be what you want. But since the sample data you posted did not have that limitation ... There are other ways of accomplishing the desired results, e.g. sub-searches, joins etc. If you post some 'real' events, you will probably get better help. /K

kristian_kolb · ‎03-23-2016

You could try the coalesce() function to eval : your_search | eval my_t_id = coalesce(user, cause) | transaction my_t_id However, I think that if this is really your data, you'd need to set up some more constraints (time, number of events, start, stop etc) on the transaction. Otherwise it may be so that your transactions will span too many events. http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/CommonEvalFunctions http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction /K

kristian_kolb · ‎06-05-2015

Define what you mean by "keep"? This evaluation creates a new field on a per-event basis. It is not keeping a state. Remember that a log searching tool is not necessarily the best way for finding out a state, because for whatever timerange you search, you might always miss that important piece of state information that was logged 5 minutes before your search time span...

kristian_kolb · ‎06-05-2015

Ok, I'm guessing that you are trying out the free version of Splunk, and didn't actually buy a license. 🙂 Splunk is a tool for searching, correlating, reporting and alerting on (mainly) log data. Think of it as a powerful analytical tool, which also has capabilities to store data in its own internal format for a specified amount of time. Actually, you can decide to have different retention periods for different types of data. It is not a backup tool, but it is commonly used for fulfilling internal/external requirements for centralized logging, i.e. to have important logs moved away from the systems where they were generated. So in a sense, the logs are 'backed up' to a secondary system, but usually logs would never be restored to the originating system in the event of a crash/attack/whatever. I would guess that you could find the basic introductory videos on splunk.com, and a lot of detailed administrative tips on youtube. /K

kristian_kolb · ‎06-04-2015

Hi, your sample data does not correspond with the event delimiters you specify. (There is no line "Send message to queue..."). Also, this is the user forum, not an official support site - maybe someone will call you on skype, but you shouldn't count on it. In general, it would probably be good to study the documentation sections for props.conf, more specifically around the parameters for breaking the incoming data stream into events (BREAK_ONLY_BEFORE... MUST_NOT_BREAK...), and possibly also the docs on anonymizing data, which could be a means for removing the unwanted lines. http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Indexmulti-lineevents http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Anonymizedatausingconfigurationfiles /k

kristian_kolb · ‎06-04-2015

Just use a regular [monitor] in for inputting the files in the directory (i.e. in inputs.conf) [monitor:///my/dir/*.dat] sourcetype=my_dat index=my_index In props.conf, you might need to specify TIME_FORMAT For the field extraction, use a REPORT in props.conf, and use FIELDS and DELIMS in transforms.conf props.conf [my_dat] TIME_FORMAT = %Y%m%d %H%M%S REPORT-dat = dat_pipes transforms.conf [dat-pipes] DELIMS = "|" FIELDS = field1, field2, field3 ... field20 You should probably read the Getting Data In section of the docs, and check out the documentation on REPORT field extractions. EDIT: typo

Posts	2153
Solutions	395
Karma Given	1033
Karma Received	1584
Member Since	‎05-10-2011

Online Status	Offline
Date Last Visited	‎08-20-2025 03:58 AM

Amazon Web Services Add-on: s3 generic error - Typ...

Splunk Support and End-of-Life

Bug in interactive field extractor (IFX)?

How to handle changing log file formats - BlueCoat...

License Violation Prediction

Duplicate GUIDs for cloned forwarders - how to cor...

date_hour not present in WinEventLogs

Re: List deployment client showing "no clients con...

Re: UF+Indexer+nullQueue/Route = Zero

Re: Microsoft 365 Defender Add-on for Splunk dupli...

Re: Microsoft 365 Defender Add-on for Splunk dupli...

Re: Amazon Web Services Add-on: s3 generic error -...

Re: Why are my props/transforms not taking effect?

Re: Can someone help me with a regex that transfor...

Re: How to implement "NOT IN" in Splunk

Re: How to implement "not in" in splunk?

Amazon Web Services Add-on: s3 generic error - Typ...

Re: Extract field from a complex multi-lines event...

Re: renaming field names from string+number to onl...

Re: How to configure props.conf to set the univers...

Re: Can i restrict splunk users to a particular in...

Re: How to find the missing source files that are ...

Re: TCP Routing and Anonymizing data

Re: TCP Routing and Anonymizing data

Re: Which port can I use to send Syslog traffict t...

Re: Conditional Transaction

Re: Conditional Transaction

Re: Conditional Transaction

Re: How to write a search where if a certain strin...

Re: splunk : what is purpose ?

Re: the beginning and end of the event

Re: Importing Text file with DAT extension separat...

Join the Conversation