@hiteshdholakiya , That is exactly what I needed.  Karma added.  Wish this was marked a the solution.  Worked perfect for what I needed.  THANK YOU!! ... View more

Going to try and use: MUST_NOT_BREAK_AFTER = \d+ MUST_BREAK_AFTER = [\r\n]+ I just think the request ID will still get ingested at the time it is logged and not later when the request ID event is completed/logged. ... View more

I found this post useful by @joesrepsolc, but it mostly covers entire file ingestion into a single event and was answered nicely by @bandit: https://community.splunk.com/t5/Getting-Data-In/How-to-set-a-large-log-to-ingest-as-one-single-event/m-p/471061 ... View more

@sundareshr Appreciate the response.  Was looking for this one. ... View more

No takers on this question, huh? ... View more

OK, so I figured out what was going on and I'd like to explain.  Then someone can correct me or tell me that I have a defective Splunk or something like that: I was trying to make an ingestion metric by sourcetype. I know some of you will tell me that the metric is already provided by Splunk, but you do not know my environment. I work in Lehi, Utah and those metrics are collected and stored in Boise, Idaho (where I do not have the access to query).  However, the local indexers and heavy forwarders still have the sourcetype=splunkd, group=per_sourcetype_thruput, and source=/opt/splunk/var/log/splunk/metrics.log .  These logs track ingestion by sourcetype and they track it on a fairly frequent nature, so I could actually perform timecharts and other reporting visualizations over time. So, that was the background of what I was trying to accomplish. In the past 15 minutes, for each sourcetype this could give me possible events/records of anywhere from 1 to 500. So, I took this data and broke it out to give me columns of sourcetype as well as the other metrics (kbps, eps, kb, etc.). Keep in mind that a single sourcetype would appear in my table thousands of times depending on the time frame I pull. Then I ran a simple query to return a unique table of sourcetype/index combinations (dedup).  Each index/sourcetype combo appeared in the table only once. On average, I got between 75 and 150 results in this table. When I joined these two tables, the first 75-150 rows in the sourcetype data table gained an index value. The join NEVER went farther in the sourcetype table than the number of records I had in the sourcetype/index table. In my outrageously optomistic mind, I believed that each row in the sourcetype data table would be evaluated against the sourcetype/index table (the way a normal SQL join works - the sane way a join works).  The only way I could get this to work was to create a MASSIVE sourcetype/index table that had excessively more rows in it than the sourcetype/index table, so that each row could be evaluated.  But this added too much overhead to the search and I quickly gave up on the effort. I have tested this many times in our environment and I get the same result over-and-over again, so no one can tell me this is not the way it works using the type of syntax I outlined in my original post. The site I work at is running the 7.3.3 version of Splunk Enterprise. I appreciate responses from @thambisetty and @to4kawa.  Both of you provided new ideas I had to go try and I did learn more from each experiment. Just wanted to send both of you a solid thank you for supporting me. ... View more

OK, here is my last update, I just don't think Splunk is capable of doing a join they way I expect a join to work: Search 1: (designed to get back all of the index/sourcetype combinations I care about, takes less than a second to run, returned back 85 rows, columns of index and sourcetype) | tstats count where (index=BlackPearl OR index=Tortuga OR index=Swashbuckler) by index, sourcetype | table sourcetype, index Search 2: (designed to pull back the license usage by sourcetype from the Splunk internal metrics logs) index=_internal source=*metrics.log group=per_sourcetype_thruput earliest=-5d@d latest=-0d@d host=lelsplunki* | rename series as sourcetype | eval GB=kb/(1024*1024) | bucket _time span=1d | stats sum(GB) as TotGB by _time, sourcetype, index Ran search number 2 once for everything and got back 450 results.  Added "(series=crew OR series=jacksparrow)" to the base search to see how many should be returned and got back 169.  This is the 169 results I want back from this search. Used both of the following searches to try the join option: Search 3: index=_internal source=*metrics.log group=per_sourcetype_thruput earliest=-5d@d latest=-0d@d host=lelsplunki* | rename series as sourcetype | eval GB=kb/(1024*1024) | bucket _time span=1d | join max=0 type=inner sourcetype [ | tstats count where (index=BlackPearl OR index=Tortuga OR index=Swashbuckler) by index, sourcetype | table index, sourcetype ] | stats sum(GB) as TotGB by _time, sourcetype, index Search 4: index=_internal source=*metrics.log group=per_sourcetype_thruput earliest=-5d@d latest=-0d@d host=lelsplunki* | rename series as sourcetype | eval GB=kb/(1024*1024) | bucket _time span=1d | stats sum(GB) as TotGB by _time, sourcetype | join max=0 type=inner sourcetype [ | tstats count where (index=BlackPearl OR index=Tortuga OR index=Swashbuckler) by index, sourcetype | table index, sourcetype ] Both of those return only 16 rows and only ~5 combinations of index/sourcetype.  Append, appendcols has also been tried, but I am willing to try anything else you may suggest (even unique variations of append). ... View more

@thambisetty & @to4kawa, I appreciate your responses, but neither of those methods work. @thambisetty, Your method gets me the closest, but I tried this same method before I submitted this help message ("I tried to use join with the max=0 and type=inner and it only returned a handful of rows and only for a few of the index/sourcetype combinations.").  Using your "join type=left" search suggestion resulted in hundreds of sourcetypes I did not want because they were not part of the type=inner join that I was trying to achieve. @to4kawa, Your solution returned only 5 rows (when it should have returned a minimum of 45,000 rows.  And it was also missing most of the index/sourcetype combinations. Both options presented here give me at most 5 combinations of index/sourcetype in the results.  Neither one provides the full list I was trying to achieve. I've simplified the search to try and make more sense out of this and drastically reduce the number of rows I am returning into the result set: Search 1: (returns less than 500 rows) index=_internal source=*metrics.log group=per_sourcetype_thruput earliest=-5d@d latest=-0d@d host=lelsplunki* | rename series as sourcetype | eval GB=kb/(1024*1024) | bucket _time span=1d | stats count by _time, sourcetype This returns 67 of sourcetypes I do not want.  I want only a set of sourcetypes that match my index types (~40 sourcetypes). _time sourcetype count 2020-08-14T00:00:00.000-0600 ship 3 2020-08-14T00:00:00.000-0600 rum 583 2020-08-14T00:00:00.000-0600 jacksparrow 623 2020-08-14T00:00:00.000-0600 crew 38 2020-08-14T00:00:00.000-0600 splunkd 11136 2020-08-14T00:00:00.000-0600 splunkd_access 1278   Charting this data with " | chart sum(GB) as TotGB by _time, sourcetype useother=false usenull=false limit=100" instead of the stats command gives results like this: _time ship rum jacksparrow crew splunkd splunkd_access 2020-08-14T00:00:00.000-0600 3.01609E-05 0.105190552 0.007976488 0.028933661 30.11121383 0.577985538 2020-08-15T00:00:00.000-0600 2.77823E-05 0.08146602 9.0844E-05 0.032564973 29.98200507 0.536031151 2020-08-16T00:00:00.000-0600 4.0628E-05 0.085774285 8.4443E-06 0.044780319 29.95757868 0.540789116 2020-08-17T00:00:00.000-0600 3.3265E-05 0.105401807   0.042801711 29.97104058 0.515818447 2020-08-18T00:00:00.000-0600 3.45539E-05 0.09992009 0.000168551 0.030914299 32.72855597 0.816412885 The index for all of these results is _internal, so I want to tie the sourcetype back to its actual index.  In my previous example I was using: Search 2: (returns less than 100 rows) | tstats count where (index=BlackPearl OR index=Tortuga OR index=Swashbuckler) by index, sourcetype | table sourcetype, index Results: (example - roughly 86 rows returned) sourcetype     index ship                    BlackPearl crew                  BlackPearl rum                    Tortuga wench              Tortuga willturner        Swachbuckler jacksparrow  Swashbuckler Now that the data has been reduced from 500,000 events to the summary of the stats, I'm hoping that there is a way to join the two small result sets to make a single set of data that has the _time bucket, sourcetype, index, and Sum of the Total GB for each sourcetype over the time bucket window. I'm going to try your solutions again with this new data set, but I don't know who flagged the reply provided by @thambisetty as a solution... it simply does not work.  Sorry. ... View more

@thambisetty , You are 100% correct.  I screwed up (even after reading it so carefully).  The first search should have been this (need to watch what I copy/paste more carefully): index=_internal source=*metrics.log group=per_sourcetype_thruput host=lelsplunkix* | rename series as sourcetype | eval GB=kb/(1024*1024) | table _time, _raw, sourcetype, GB ... View more