0

I am trying to obtain an HTTP response asynchronously, but the speed is slow. I am not sure if the asynchronous approach is successful. Removing json::<serde_json::Value>().await can make it faster, but then won't be able to get the returned data Would be grateful for any assistance.

let package_json_string = read_to_string("package.json")
    .expect("read pacakge.json error");
let mut query_key_list = vec![];
let PackageJson {
    dependencies,
    devDependencies,
}: PackageJson = serde_json::from_str(package_json_string.as_str())
    .expect("parse error");
for key in dependencies.keys() {
    query_key_list.push(key)
}
for key in devDependencies.keys() {
    query_key_list.push(key)
}
let client = Client::new();
let mut task = vec![];
let start = Instant::now();
for key in query_key_list {
    let url = format!("{}{}", "https://registry.npmjs.org/", key);
    let client = client.clone();
    task.push(tokio::spawn(async move {
        let result = client.get(url).send().await;
        let res = result.unwrap().json::<serde_json::Value>().await.unwrap();
    }));
}
join_all(task).await;
let duration = start.elapsed();
println!("Time elapsed in expensive_function() is: {:?}", duration);

10
  • 1
    Obligatory "Are you running in --release mode?". Also, are you sure that what's slow is the actual JSON parsing, and not just the receiving, with reqwest (you're using reqwest, right?) throwing away the body if you don't call .json()?
    – Caesar
    Jun 14 at 2:48
  • 2
    On the latter point - I think that means "is there any measurable difference between using .json() and .text()?" (that is, between receiving and parsing vs just receiving).
    – Cerberus
    Jun 14 at 3:46
  • Yes, most of the time is spent on parsing the JSON data. The JSON data returned is several tens of megabytes in size. If we only retrieve result.unwrap().content_length(), it will be faster.
    – Scc
    Jun 14 at 6:18
  • 3
    content_length does not need to retrieve the response data, only the headers. Benchmark again result.unwrap().text() and see if there is still a difference in speed. If not, the JSON parsing is not to blame. Jun 14 at 6:33
  • 1
    Then the problem is not the JSON. Jun 14 at 6:38

0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.