0

I have a use case where I need to make more than 10,000 external HTTP requests(to one API) in an infinite loop every 10 seconds. This API takes anywhere from 200-800ms to respond.

Goals:

  1. Call an external API more than 10,000 times in 10 seconds
  2. Have a failproof polling system that can run for months at a time without failing

Attempts: I have attempted to use the Async library and limit requests to 300 concurrent calls(higher numbers fail quickly) but after about 300,000 requests I run into errors where I receive a connection refused(I am calling a local Node server on another port). I am running the local server to mimic the real API because our application has not scaled to more than 10,000 users yet and the external API requires 1 unique user per request. The local response server is a very simple Node server that has one route that waits between 200-800ms to respond.

Am I getting this error because I am calling a server running locally on my machine or is it because Node is having an issue handling this many requests? Any insight into the best way to perform this type of polling would be appreciated.

Edit: My question is about how to create a client that can send more than 10,000 requests in a 10 second interval.

Edit2: Client:

//arr contains 10,000 tokens
const makeRequests = arr => {
  setInterval(() => {
    async.eachLimit(arr, 300, (token, cb) => {
      axios.get(`http://localhost:3001/tokens/${token}`)
        .then(res => {
          //do something
          cb();
        })
        .catch(err => {
          //handle error
          cb();
        })
    })
  }, 10000);
}

Dummy Server:

const getRandomArbitrary = (min, max) => {
  return Math.random() * (max - min) + min;
}

app.get('/tokens:token', (req, res) => {
  setTimeout(() => {
    res.send('OK');
  }, getRandomArbitrary(200, 800))
});
9
  • You can try an HTTP load generator? Something like this one?
    – Andrew
    Nov 13, 2019 at 19:02
  • Are you saying that I should use this to see if the issue is my local server? Nov 13, 2019 at 19:10
  • Maybe? To be honest I'm not sure entirely what your question is. To me it looks like you're just trying to run a load test experiment, so I figured I'd throw that suggestion out there.
    – Andrew
    Nov 13, 2019 at 19:13
  • Is this a question about creating a client that can run that many requests to a server that can easily handle everything a client can send at it or a question about creating a server that can handle that many requests? And, if it's about creating a client, it seems like you're already saying that the server involved here has a problem with as many requests as you're trying to do so that would be a target server limitation. I'm very confused about what problem needs to be solved here.
    – jfriend00
    Nov 13, 2019 at 19:14
  • I updated the post, but I am trying to create the client that is sending a minimum of 10,000 requests every 10 seconds to an external API that I do not own. Nov 13, 2019 at 19:17

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.