Starting a new project with Adobe's CQ5...
I'm starting with the access log, as it is straight forward.
I've done field extractions before for another custom log type, worked great. Now, I can't seem to get any of my extractions appear in the Search.
Walkthrough:
Created an index called adobe_cq5.
Created various "file" type data inputs using various CQ5 log files, setting the index to adobe_cq5. The one I started with is called cq5-access.
Go into Search and verify that I'm getting good data, which I am.
Select the drop down next to the first log line and click Extract Fields.
Under Generated Pattern, click Edit.
Put in a basic regex: ^(?P<FIELDNAME>\d+\.\d+\.\d+\.\d+?)
Click "Apply"
Check several lines to make sure the IP addresses are selected.
Click "Save"
In "Save Field Extraction" enter ip_address for the field name.
Click "Save"
Click "Close" on "Successfully Saved" dialog.
Reload the Search page.
Note that ip_address is not appearing in the log line list as it has for past projects.
Click "Pick fields"
Note that ip_address does not appear in Available Fields.
Go back to Extract Fields
Enter in: ^(?P<FIELDNAME>\d+\.\d+\.\d+\.\d+?)
Get two errors:
--Note: the values you want may already be extracted in the 'ip_address' field.
--Note: This regex already extracts ip_address for cq5-access.
Close out of Extract Fields
Browse to Manager » Fields » Field extractions
Verify extraction: cq5-access : EXTRACT-ip_address
Click Permissions give Everyone Read Permission and set Object should appear in This app only (search)
Click Save
Re-Check the search page, ip_address still does not appear.
Open up terminal
cat: /opt/splunk/etc/apps/search/local/props.conf
Verify extraction: EXTRACT-ip_address = ^(?P<ip_address>\d+\.\d+\.\d+\.\d+?)
For my last project, I simply entered the Extract Fields tool, entered my regex, saved and the data appeared right in the Search.
props.conf for modified extraction
[cq5-access]
EXTRACT-ip_address = ^(?P<ip_address>\d+\.\d+\.\d+\.\d+?)
props.conf with original full extraction
[cq5-access]
EXTRACT-ip_address-username-day-month-year-hour-minute-second-http_type-http_request-http_code-referer-user_agent = ^(?P<ip_address>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s.+?\s(?P<username>.+?)\s(?P<day>\d\d)/(?P<month>\w\w\w)/(?P<year>\d\d\d\d):(?P<hour>\d\d):(?P<minute>\d\d):(?P<second>\d\d)\s.+?\s"(?P<http_type>\w+?)\s(?P<http_request>.+?)\sHTTP.+?"\s(?<http_code>\d+?)\s.+?\s"(?P<referer>.+?)"\s"(?P<user_agent>.+?)"
Sample data:
10.71.40.57 - admin 23/Apr/2013:16:15:14 -0400 "GET /crx/server/crx.default/jcr%3aroot/etc/map/http.1.json?_dc=1366748119022&node=xnode-339 HTTP/1.1" 200 175 "https://twcc-ci01.lab.webapps.rr.com:4602/crx/de/index.jsp" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0"
10.71.40.57 - admin 23/Apr/2013:16:15:13 -0400 "GET /crx/de/icons/crxde_favicon.ico HTTP/1.1" 200 295606 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0"
127.0.0.1 - admin 23/Apr/2013:16:42:31 -0400 "GET /bin/receive?sling:authRequestLogin=1 HTTP/1.1" 200 32 "-" "Jakarta Commons-HttpClient/3.1"
... View more