So it turns out that JavaMelody created unique RRD files for each URL and SQL request. Our REST service exposes hundreds of endpoints, and each of them can take thousands of values, like
http://server/get/entity/10
where 10 can be any unique ID in the system. So we had as many RRD files as there were entities being queried.
The documentation at https://github.com/javamelody/javamelody/wiki/UserGuide#6-optional-parameters hints at options called http-transform-pattern
and sql-transform-pattern
.
The parameter http-transform-pattern is a regular expression to
transform descriptions of http requests and to delete dynamic parts
(identifiers of objects for example) in order to be able to aggregate
on these requests
Similarly, the parameter sql-transform-pattern is a
regular expression to transform descriptions of sql requests (not
binded identifiers for a "in" clause for example) in order to be able
to aggregate on these requests.
That is a very vague description of what these options do, and it is not immediately obvious how they help. But others that had the same problems indicated that these settings fixed their issues, so I dug deeper.
I suspect how these options work is by replacing any part of the URL that matches the regular expression with a "$". So setting http-transform-pattern
to \d+
means that the URLs http://server/get/entity/10
and http://server/get/entity/20
both have their digits matched by the regular expression, and are then aggregated into the URL http://server/get/entity/$
. This in turn reduces the number of RRD files, as new ones are no longer created for every id.
In our case we used lookbehinds to group URLs. So we set http-transform-pattern
to something like (?<=/server/get/fuzzballs).*|(?<=/server/get/foobars).*
to aggregate the URLs into the buckets http://server/get/fuzzballs$
and /server/get/foobars$
.