Splunk Search

Applying Field Extractions to Several Sourcetypes

niall_munnelly
Path Finder

Hi,
Per a policy I've inherited, we're separating our business groups' web server logs into separate sourcetypes. It seems it would be easier if I could create a transform and use tags, but the dude running the show is averse to these solutions.

So we have maybe 20 sourcetypes, named

  • foo_apache_access
  • bar_apache_access
  • baz_apache_access
  • quux_apache_access

...and so on. This has made creating a standard collection of extracted fields tricky; it seems like I have to create the extractions within every sourcetype, which is manual and slow. I'm also not getting consistent results for some of the extracted fields in one sourcetype vs another, so I may have to break my combined regex patterns into several individual extractions, making the whole process way slower and sloppier, still.

Wildcard characters appear to be parsed literally, so *_apache_access didn't work as hoped. Have I missed something? Is there a wildcard or UI command I can use to effect this? Thanks.

1 Solution

jhowkins
Path Finder

If you would rather create an extraction by sourcetype and using a wildcard, you can do it but it requires a little regex trickery...

To capture the sourcetypes you provided, you can use the following as your sourcetype;

(?::){0}*_apache_access

View solution in original post

jhowkins
Path Finder

If you would rather create an extraction by sourcetype and using a wildcard, you can do it but it requires a little regex trickery...

To capture the sourcetypes you provided, you can use the following as your sourcetype;

(?::){0}*_apache_access

HattrickNZ
Motivator

What if the soucetype is not in the text where you want to do the extraction?

2 source types
XBSN
QSN

need 2 extractions on each sourctype:
^[^/\n]/\w+:(?P[^"]+)
^[^/\n]
/\w+:(?P[^"]+)

Where do I specify the sourcetype?

0 Karma

UCOP
New Member

I have the same problem. I need a regular expression to identify several sourcetypes. Examples of the source types are as follows:

application-ucop-topcop-pub:default-2
application-ucop-bft-nais-sub:default
application-ucop-bft-pub:default

I know that "application-ucop" and ":default" will remain constant, but in between them will be different one word or two words separated by a "-". Also some of them have nothing after :default and some have a "-" followed by a number.

I have never worked with regular expressions and I am trying to simplify my field extractions. Thank you.

0 Karma

ss026381
Communicator

have you ever find out the answer for this?

0 Karma

niall_munnelly
Path Finder

This totally worked! Thanks!

0 Karma

niall_munnelly
Path Finder

Thanks for your replies, JHowkins; I'll see how i can make this fit. With some 20-odd sourcetypes and a projected 5 or 6 extractions per sourcetype, anything I can do to avoid creating them manually will be blessing.

0 Karma

jhowkins
Path Finder

To answer your question, yes there is a way to use wildcards to use a single extraction against multiple sourcetypes. See my answer below.

0 Karma

jhowkins
Path Finder

I ran into a similar issue where I have 60 different sourcetypes all belonging to the same host.

Depending on the contents of your files, you could create a transform and/or extraction that captures similar things contained within each file. When you create your regex pattern, just make sure it qualifies something within the data that distinguishes it from everything else.

If you can provide an example of the file contents and your intentions, I can provide a better answer.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...