Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey
, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions
. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
- org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom() // the auto-retry feature of the Apache API will retry up to 5 // times on failure, being also allowed to send again requests // that were already sent if necessary (I don't really understand // the purpose of the second parameter below) .setRetryHandler(new StandardHttpRequestRetryHandler(5, true)) // for HTTP 503 'Service unavailable' errors, also retrying up to // 5 times, waiting 500ms between each retry. Guessed is that those // 5 retries are part of the previous "global" 5 retries setting. // The below setting, when used alone, would allow to only enable // retries for HTTP 503, or to get a greater count of retries for // this specific error .setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500)) .build();
, which customizes the Apache API retry behavior
Getting back to the topic :
- A request counter could be created by extending the Apache API retry-related classes quoted before
- Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1: Looks like the Apache API won't permit this to be done. Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.