0

I am intercepting API calls from an iPhone in Charles Proxy. I am able to re-run these API calls through Charles Proxy's 'Repeat' feature on my laptop and they are successful. However, if I export this API request as a cURL and run the cURL it never returns. If I import the cURL into Postman, the request does not succeed. The Postman request never returns and eventually times out. Is there some feature of a Charles Proxy request that is allowing the API call to succeed that is not included within the cURL? Or vice versa?

All API calls are happening from the same IP Address on the same laptop.

I have tried using Postman's convert to code feature to run the request in Python, this also times out. I have also converted the cURL exported from Charles Proxy directly to Python, bypassing Postman completely, this also times out. I would expect any API call that works in Charles Proxy to work anywhere else, such as Postman and Python or as a cURL.

This leads me to believe that the API call I am making in Postman and Python must be different somehow, but I am not sure how it is different. I believe the cURL that is generated by Charles Proxy must be different somehow.

Update: This issue is only occurring on my laptop, regardless of whether I run behind a VPN or not. Trying to replicate this situation on another machine at a different location results in the requests working in Charles Proxy and as a curl/python.

2
  • Are you setting all the request headers when doing a curl?
    – JonSG
    Nov 10 at 15:34
  • I was just using the curl given to me by Charles Proxy. I was not modifying it in anyway. However, I have seen that the 'Connection' and 'Content-Length' headers were not included in the curl. Unfortunately, if I manually add these back in the requests still time out.
    – Aaron
    Nov 10 at 15:58

1 Answer 1

0

It's hard to know without being able to see the specific code, but one possibility is TLS fingerprinting. I've written a detailed introduction to this here (for Node.js, but the concept is the same everywhere).

In essence: by looking at small details of how a client makes the initial TLS connection, before the HTTP request is even received a server can recognize the kind of client, so it can spot curl, Python, browsers or other clients and treat the differently. Many servers use this to try to block automated bots and scraping, by blocking all clients matching certain known fingerprints.

In your case, you would see the behaviour you describe if the server specifically blocks requests from curl, Postman and Python, but doesn't block Charles.

You can confirm this easily by tweaking your curl command to send it via Charles. If the same request works when sent unmodified through Charles, but doesn't work when sent directly, then the difference is almost certainly the TLS fingerprinting (the only other possibility is that Charles is modifying the request itself en route, but that would almost certainly be a Charles bug, and I'm not aware of any such issues).

There are other possibilities of course, particularly if the server is behaving weirdly, or if there's something invalid about your request. Clients can behave in different ways when receiving bad data, where some will accept the response anyway if possible, while others might timeout or fail explicitly, and that would explain what's happening. If you want to know for sure whether the request is different or whether this is due to client handling, you'll need to use a tool like Wireshark to examine the raw traffic directly byte by byte, and see whether there are any differences there (note that for HTTPS traffic, you'll need to use TLS key logging options on the client to be able to decrypt the the traffic in Wireshark)

1
  • I have since discovered that a different machine on a different IP address doesn't have this problem. However, running the requests on my laptop behind a vpn still results in the requests timing out in non-Charles Proxy environments. I will look into the TLS fingerprinting over the weekend. Could TLS fingerprinting also be responsible for requests only failing on my laptop vs others?
    – Aaron
    Nov 10 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.