I think your question may be better answered by the Cribl community first as Cribl is modifying the JSON events before it hits Splunk.  I recommend reviewing this doc to properly format JSON event for Splunk HEC destination: https://help.splunk.com/en/splunk-cloud-platform/get-data-in/get-started-with-getting-data-in/10.6/get-data-with-http-event-collector/format-events-for-http-event-collector Here are some possibilities to your duplicate field values: Cribl's Splunk HEC destination turns top-level fields into indexed fields. If an event has a _raw, Cribl sends _raw as the HEC event body. Every other top-level field (except _time, host, source, sourcetype, index) goes in the HEC fields object, so Splunk stores it as an index-time field. After Flatten, FileName, SHA256 and the rest are all top-level, so they all become indexed fields. _raw still holds the JSON. Unroll copies the whole event, including _raw, into each new event. Flatten and their Object.assign eval only change event fields; neither touches _raw. So _raw is still JSON, possibly the original Event Hub payload with the whole records array. Search-time JSON extraction (KV_MODE=json, or AUTO_KV_JSON under KV_MODE=auto) pulls the same keys out of _raw again.  Just some questions: - What is the splunk hec endpoint you're using? raw? event? - Can you double check which are indexed fields? | tstats count where index=<idx> sourcetype=<st> by FileName - If this returns results, FileName is an indexed field, which points at the Cribl HEC fields path. | walklex index=<idx> type=field - This lists every indexed field name. - Can you show us what the sourcetype configuration is set to on your heavy forwarder vs what is on the search head? You should have the big 6 props configuration for your sourcetype if you haven't already done so. -  Not sure what you can or can't do in your environment but you're already running an HF, have you tried using Splunk Add-on for Microsoft Cloud Services TA to ingest the data?   ... View more

The simple way would be to use a custom app that has your indexes configuration for splunk cloud, and then have a deployment server deploy the same app to your HFs. Your HFs won't index data locally but you'll be able to see those indexes in the UI. Depending on how fancy your environment is and what your team is capable of, in CI/CD, you can have a pipeline that specifies a list of indexes to manage for Splunk Cloud in a yaml or .tf file. That same pipeline can also deploy the index configuration to your on-prem HFs. Look into these terraform providers if you're interested. You won't need something like Gitlab CI/CD and can run the terraform jobs manually.  splunk TF provider (on-prem/splunk enterprise): https://registry.terraform.io/providers/splunk/splunk/latest scp TF provider (cloud): https://registry.terraform.io/providers/splunk/scp/latest ... View more

You probably already figured it out by now but you will can use the ACS CLI or Terraform splunk/scp provider to manage indexes in Splunk Cloud. Splunk ACS API REST reference: https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Config/ACSREF#Manage_indexes Creating a new index in Splunk Cloud example: curl -X POST 'https://admin.splunk.com/{stack}/adminconfig/v2/indexes' --header 'Authorization: Bearer eyJraWQiOiJzcGx1bmsuc2VjcmV0Iiwi…' \ --header 'Content-Type: application/json' \ --data-raw '{ "name": "testindex" }' ACS CLI: https://docs.splunk.com/Documentation/SplunkCloud/latest/Config/ACSCLI Splunk Splunk Cloud Platform Terraform provider: https://github.com/splunk/terraform-provider-scp ... View more

Any non-internal indexes could be a summary index to be honest. But like @dtburrows3 said, you'll have to take a look at savedsearches.conf to see what search is using the collect command that writes to an index. This isn't guaranteed to identify summary indexes but will help you narrow down what indexes to look into. In our environment, our summary indexes are identified with the "summary_" prefix as best practice. ... View more

Can you clarify what technical addon you're using? Also, couldn't you ask your admin to clarify on the question you have originally? If you're using this addon here, then you can write a search using the LDAP command to write to an index with the collect command. Otherwise, whatever you're doing with the CSV file and then having a file monitoring to ingest the CSV is the long way to do it. ... View more

I have a couple of questions: - Are you trying to get rid of the the metrics data from Splunk's metrics.log? - Can you post the props and transforms config that you tried? What you're trying to do may not "fix" the memory utilization on your container. ... View more

You may need to modify one or both settings below in your inputs.conf to get Splunk to ingest the appended logs. It's kind of hard to say without seeing a sample of your full log - with redacted info; or you can read the config details below and make the determination yourself.  Can you search in index=_internal for the specific host with the search string of the log file name that you're interested in? It should show what the UF is doing when it monitors for that file path. Commonly, folks tend to use crcSalt = <SOURCE> when they have issues with Splunk not ingesting a log file.   crcSalt = <string> * Use this setting to force the input to consume files that have matching CRCs, or cyclic redundancy checks. * By default, the input only performs CRC checks against the first 256 bytes of a file. This behavior prevents the input from indexing the same file twice, even though you might have renamed it, as with rolling log files, for example. Because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers. * If set, <string> is added to the CRC. * If set to the literal string "<SOURCE>" (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC. When 'crcSalt' is invoked, it is usually set to <SOURCE>. * Be cautious about using this setting with rolling log files; it could lead to the log file being re-indexed after it has rolled. * In many situations, 'initCrcLength' can be used to achieve the same goals. * Default: empty string initCrcLength = <integer> * How much of a file, in bytes, that the input reads before trying to identify whether it has already seen the file. * You might want to adjust this if you have many files with common headers (comment headers, long CSV headers, etc) and recurring filenames. * Cannot be less than 256 or more than 1048576. * CAUTION: Improper use of this setting causes data to be re-indexed. You might want to consult with Splunk Support before adjusting this value - the default is fine for most installations. * Default: 256 (bytes)   https://docs.splunk.com/Documentation/Splunk/latest/Admin/inputsconf  ... View more

If you plan on using a deployment server to update your TA or apps, then that would be the easiest route. It's a lot to cover on the deployment server if you haven't used it before, give the link below a read if you can: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Deploymentserverarchitecture Splunk also covers the deployment server part in this training: Splunk Enterprise System Administration https://www.splunk.com/en_us/pdfs/training/splunk-enterprise-system-administration-course-description.pdf https://www.splunk.com/en_us/training/course-catalog.html?filters=filterGroup2SplunkEnterpriseCertifiedAdmin   The gist of a deployment server is: Your non-distributed Splunk instances check into your deployment server (DS) to retrieve any apps you want to deploy. The TAs/apps are all on your DS (etc/deployment-apps) and you manage what app your Splunk instances get with the DS serverclass.conf. ... View more

Try this in your props.conf on your HF - I don't know the time zone of your log file so I assume it's UTC, change as needed:   TIME_PREFIX = ^ MAX_TIMESTAMP_LOOKAHEAD = 30 TIME_FORMAT = %FT%T.%6QZ TZ = UTC   I highly recommend you read through best practices for parsing configurations here: https://kinneygroup.com/blog/splunk-magic-8-props-conf/ Time variables can be found here: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables ... View more

You need to add your index time configurations on the HF and not on the SH or Indexers since the HF is where your data is being parsed. Your TIME_PREFIX configuration could be simpler but we would need to see a full sample log line to help with that; redact any sensitive information by the way if you provide a sample. ... View more

Hi - the numbered list provided step by step instructions on searching for the items I mentioned. In addition, the lookup errors you see in the UI usually tells you the name of the lookup related configuration that's having problems. This doc page should help: https://docs.splunk.com/Documentation/Splunk/9.1.2/Knowledge/Aboutlookupsandfieldactions#Lookup_definitions ... View more

There can be various reasons for this issue but here are the common ways to troubleshoot this error. First and foremost, - you need to track down the automatic lookup definition - record the lookup definition name being referenced - find the lookup definition and record the lookup table name and then go check the following:   Check if your lookup file exist - you can use the Lookup Editor app to check this or go to: Settings > Lookups > Lookup table files Check if your lookup definition exist - you can check this by going to Settings > Lookups > Lookup definition If you are using an automatic lookup check the following: Do you have the correct read permission to the lookup definition and lookup table? If the permissions are correct, check the lookup table size (see step #3) If you are using the lookup command: Do you have permission to the lookup table or lookup definition? Does your lookup definition exist? Does your search runs fine with adding local=true to your lookup command? This means that your lookup isn't being replicated to the indexer cluster, see step #4. Rare that this happens, but check the lookup table size for the lookup listed in the automatic lookup and check if it exceeds the size defined in [replicationSettings] in distsearch.conf. If the lookup table exceeds whatever size is defined there, the lookup error comes up. Check if the lookup being used is in the deny list under distsearch.conf btool distsearch list replicationBlacklist --debug btool distsearch list replicationDenylist--debug Update: - I installed SSE app v3.7.1 on a new Nix host with Splunk v9.1.1 and I didn't see any lookup errors when I run a search. So I recommend you follow the troubleshooting steps above since I can't replicate the issue with a fresh app and Splunk install. ... View more

@niketn wrote: @badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not. In essence you would need to test out stats first then lookup vs lookup first and stats next. Do share your current SPL for community members to assist you better with your use case.   I think there may have been a typo this this original answer as the lookup command has local=false set by default - source: https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/Lookup   local Syntax: local=<bool> Description: If local=true, forces the lookup to run on the search head and not on any remote peers. Default: false ... View more

My go-to fix for most issues with MC displaying incorrect info is to go to: MC app > Settings > General setup > Click apply changes. Then refresh and verify. ... View more

So a few questions: What is the version number of the Windows TA are you using on your search head? What version number of the Windows TA on your UF for this data? What does your inputs.conf look like for the following stanza? [WinEventLog://Security] Like @PickleRick said in his comment, this doesn't look like a standard Windows Event Log. ... View more

Try this - I'm not the best at regex and someone else may come along and provide a more efficient one.   Registrar: (?<registrar>.+[^\s]).+Registrar ID      ... View more

Try adding this to your search | rex field=_raw "Registrar ID: (?<registrar_id>\S+)"  Update: I misread your post, standby for an updated search to include all three field extraction - unless someone else beat me to it. You can also use the "Extract New Fields" or "Event Actions" option when you run your search. ... View more

Hi - can you post name of the sourcetype to the event where EventID=4647 comes up? You can then search for the sourcetype name in Splunk_TA_windows/default/props.conf to see how signature_id field is created. ... View more

Are you still having this issue with the latest SSE app v3.7.1?  ... View more

Even though Splunk allows TCP/UDP inputs, it's best practice not to use it if you can. Lots of unpredictable data can come in and then you'll lose data if you happen to do anything with the Splunk service (restart/os shutdown etc). It's best if you can use rsyslog for these types of inputs if you can.  ... View more

Just to add to this, for the path in the stanza - make sure you use the correct slashes depending which operating system it is (forward slash for Linux and back slash for Windows).   [monitor://<path>] * Configures a file monitor input to watch all files in the <path> you specify. * <path> can be an entire directory or a single file. * You must specify the input type and then the path, so put three slashes in your path if you are starting at the root on *nix systems (to include the slash that indicates an absolute path). https://docs.splunk.com/Documentation/Splunk/latest/Admin/inputsconf  https://docs.splunk.com/Documentation/Splunk/9.1.1/Data/Monitorfilesanddirectorieswithinputs.conf Windows inputs stanza example: [monitor://C:\Windows\System32\WindowsUpdate.log] index = test sourcetype = my_sourcetype   ... View more

This is an old post but I want to post resolutions that worked for us in case someone else runs into the same error. You'll usually see these bundle replication errors with the search below (you'll need to edit the search with your search head and indexer hostnames - wildcard it if you want): Note: the Monitoring Console app has a dashboard for these type of errors in Search > Knowledge Bundle Replication   index=_internal host IN (<YOUR_SH_HOSTNAME>, <YOUR_INDEXER_HOSTNAME>) source=*splunkd.log* (component=BundlesAdminHandler OR component=BundleDataProcessor OR component=BundleDeltaHandler OR component=BundleReplicationProvider OR component=BundleStatusManager OR component=BundleTransaction OR component=CascadePlan OR component=CascadeReplicationReaper OR component=CascadingBundleReplicationProvider OR component=CascadingReplicationManager OR component=CascadingReplicationTransaction OR component=CascadingReplicationStatusActor OR component=CascadingUploadHandler OR component=ClassicBundleReplicationProvider OR component=DistBundleRestHandler OR component=DistributedBundleReplicationManager OR component=GetCascadingReplicationStatusTransaction OR component=RFSManager OR component=RFSBundleReplicationProvider) (log_level=WARN OR log_level=ERROR) component=ClassicBundleReplicationProvider log_level=ERROR   In the error logs, note down the search head reporting the errors and the indexers listed in logs. Verify that the search head can connect to the indexers listed in the error log. The second resolution is log into the search head that is reporting the error and check the timestamp of the content inside $SPLUNK_HOME/var/run/proxy_bundles (IE Linux command: ls -lah). If the timestamp of the files are more than a few days ago, then you would need to move the proxy_bundles directory to a backup location and restart Splunk; this should fix the errors. ... View more

Try going into your monitoring console app > Settings > General Setup - then click on Apply Changes to see if that fixes your issue. Double check this page again to be safe: https://docs.splunk.com/Documentation/Splunk/9.1.0/DMC/Configureindistributedmode   ... View more

Hey - we've ran into some weird issues with the new Lookup Editor app before and it isn't the same issue you're having, but can you try to clear out your browser cache and history to see if it fixes your issue? ... View more

The developer added the configuration below in "default/app.conf" that prevents you from accessing the add-on's web UI. This may be by mistake by the developer. You can fix this by adding your own "local/app.conf" within the addon's directory with the config below and set "is_visible = 1" for the add-on to be navigable.     [ui] is_visible = 0 label = mqinput   is_visible = <boolean> * Indicates if this app is visible/navigable as an app in Splunk Web. * Apps require at least one view to be available in Splunk Web. ... View more