I'm trying to create a logs-based metric in GCP for use in an alerting rule (StackDriver, now part of GCP proper). The source are HTTPS (L7) LB logs. After a lot of trial and error, I found out that my metric doesn't work if I use a regex-based filter (note: regex-based label extraction works, after the change described below).
Concretely, if my filter looks like this:
httpRequest.status=403
httpRequest.requestMethod="GET"
httpRequest.requestUrl=~"^.*some/regex/pattern/.*$"
I do see filtered log entries after clicking "Submit Filter", and when I set up regex-based label extractors, I also see sample records/labels, but in the metrics explorer, there never is any data in the time series. However, if I change the filter to do a plain fulltext substring match, i.e.,
httpRequest.status=403
httpRequest.requestMethod="GET"
"some/substring/match"
it does work - I do see time series in the metrics explorer.
Note that this is not a timing issue, I wasn't just impatient enough. I can reliably make it work (or not) by switching between the two filters.
This is an issue for me since I do need a regex match on the request URL; substring match is not sophisticated enough, and additionally, I want to make sure only the request URL gets matched, not the referer. While I'm principally fine with a more coarse filter (even if that blows up the metric), this causes another issue in my alerting rules, as I cannot filter out non-matches -- the extracted labels have no values, and even label !=~ [[:space:]]*
doesn't work to filter out non-matches at the alerting level, as a missing label value apparently is distinct from an empty label value. And FP alerts because of a too coarse log filter are quite the dealbreaker for me.
If this is expected behavior, I'd appreciate a link to the documentation where this limitation is highlighted.
The Logs Viewer does not support regular expression searches and there are no special wildcard characters such as * or ?, in either the basic or advanced query interfaces.