random OOM errors

[Gvwyn]

I did this

I was experimenting with libcurl and I ran into an issue where I keep getting OOM errors like 9/10. Funniest part is that is seems to fix itself sometimes and succeed to print the output and I am so confused.

Code:

#include <curl/curl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

struct memory {
  char *response;
  size_t size;
  size_t capacity;
};

static size_t cb(char *data, size_t size, size_t nmemb, void *clientp) {
  size_t realsize = size * nmemb;
  struct memory *mem = (struct memory *)clientp;

  if (mem->size + realsize + 1 > mem->capacity) {
    mem->capacity = (mem->size + realsize + 1) * 2;
    char *ptr = realloc(mem->response, mem->capacity);
    if (!ptr) {
      return 0;
    }
    mem->response = ptr;
  }

  memcpy(&(mem->response[mem->size]), data, realsize);
  mem->size += realsize;
  mem->response[mem->size] = '\0';

  return realsize;
}

int main(void) {
  curl_global_init(CURL_GLOBAL_WIN32);
  CURL *curl = curl_easy_init();
  CURLcode res;

  struct memory chunk;
  chunk.response = malloc(2048);
  chunk.size = 0;
  chunk.capacity = 2048;

  curl_easy_setopt(curl, CURLOPT_URL, "https://catfact.ninja/fact");
  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, cb);
  curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&chunk);
  curl_easy_setopt(curl, CURLOPT_CAINFO, "cacert-2024-03-11.pem");

  if (curl) {
    for (int x = 0; x < 10; ++x) {
      res = curl_easy_perform(curl);

      if (res != CURLE_OK) {
        printf("%i -> (%lu) whoopsie daisy: %s (%d)\n", x, (unsigned long)chunk.size,
               curl_easy_strerror(res), res);
      } else {
        printf("%i -> (%lu) %s\n", x, (unsigned long)chunk.size,
               chunk.response);
      }

      chunk.size = 0;
      free(chunk.response);
      chunk.response = malloc(2048);
      Sleep(5000);
    }

    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

An example output:

0 -> (353) {"fact":"The world\u2019s rarest coffee, Kopi Luwak, comes from Indonesia where a wildcat known as the luwak lives. The cat eats coffee berries and t
he coffee beans inside pass through the stomach. The beans are harvested from the cat\u2019s dung heaps and then cleaned and roasted. Kopi Luwak sells for about
 $500 for a 450 g (1 lb) bag.","length":319}
1 -> (0) whoopsie daisy: Out of memory (27)
2 -> (0) whoopsie daisy: Out of memory (27)
3 -> (0) whoopsie daisy: Out of memory (27)
4 -> (0) whoopsie daisy: Out of memory (27)
5 -> (0) whoopsie daisy: Out of memory (27)
6 -> (131) {"fact":"A cat named Dusty, aged 1 7, living in Bonham, Texas, USA, gave birth to her 420th kitten on June 23, 1952.","length":107}
7 -> (0) whoopsie daisy: Failure when receiving data from the peer (56)

* HALTS

I expected the following

No stupid "Out of Memory" and etc. errors.

curl/libcurl version

curl 8.8.0 (x86_64-w64-mingw32) libcurl/8.8.0 LibreSSL/3.9.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.62.1 ngtcp2/1.5.0
nghttp3/1.3.0
Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL SSPI threadsafe UnixSockets zstd

operating system

Windows 10

[bagder]

How did you build this?

[Gvwyn]

I used gcc Main.c -o main -lcurl

[bagder]

Which gcc is that and where did the libcurl come from?

(on windows, mixing things is always a cause for problems and this has all the smells of exactly that)

[Gvwyn]

minGW was installed through chocolatey, and I believe I downloaded the 64 bit libcurl binaries from the website (the one named "the curl project"), and just copied everything into mingw's folder

gcc --version:
gcc.exe (x86_64-posix-seh-rev0, Built by MinGW-Builds project) 13.2.0

i'd also like to point out that I think I still have the old curl.exe somewhere in the System32 folder, cause curl -V outputs
curl 7.83.1 (Windows) libcurl/7.83.1 Schannel Release-Date: 2022-05-13
which would kind of? explain it as you said mixing things is not a smart choice

[vszakats]

What matters in this case is the libcurl library linked by gcc Main.c -o main -lcurl. Microsoft doesn't ship libcurl AFAIK. The official binaries, which Chocolatey distributes, do have (or might have?) libcurl.

To tell for certain which libcurl this is, can you add this line to your test code and recompile?

printf("libcurl version: |%s|\n", curl_version());

[Gvwyn]

@vszakats this is what I got
libcurl version: |libcurl/8.8.0 LibreSSL/3.9.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.62.1 ngtcp2/1.5.0 nghttp3/1.3.0|

I also enabled verbose to see if that helps anything, and ran my code 10 times till finish or till it halted or crashed.

This is all the types of errors I've encountered (so far):
It's also worth noting that the first (0) call is ALWAYS successful.

* URL rejected: A memory function failed
* Closing connection
2 -> (0) whoopsie daisy: Out of memory (27)

A pretty common one

* Found bundle for host: 0x*** [can multiplex]
* Re-using existing connection with host catfact.ninja
* Connection #0 to host catfact.ninja left intact
3 -> (0) whoopsie daisy: Out of memory (27)

This one might actually be useful?

* SSL: couldn't create a context: error:14FFF041:SSL routines:(UNKNOWN)SSL_internal:malloc failure
* Closing connection
2 -> (0) whoopsie daisy: Out of memory (27)

* LibreSSL/3.9.2: error:14FFF3E7:SSL routines:(UNKNOWN)SSL_internal:unknown failure occurred
* Closing connection
1 -> (0) whoopsie daisy: SSL connect error (35)

* SSL: couldn't create a context: error:14FFF041:SSL routines:(UNKNOWN)SSL_internal:malloc failure
* Closing connection
3 -> (0) whoopsie daisy: Out of memory (27)

* SSL: couldn't create a context: error:0BFFF041:x509 certificate routines:CRYPTO_internal:malloc failure
* Closing connection
9 -> (0) whoopsie daisy: Out of memory (27)

This one happened multiple times

* process_pending_input: nghttp2_session_mem_recv() returned -901:Out of memory
* Connection #0 to host catfact.ninja left intact
1 -> (0) whoopsie daisy: Failure when receiving data from the peer (56)

[vszakats]

OK, so this is the curl-for-win libcurl.

Could not reproduce this with the C example (used x86_64-w64-mingw32-gcc t.c -I../include -L../lib -lcurl), and an equivalent curl command line also works. Tested under WINE.

[ Gentle request: Could replace "f** y**" with something more cheerful? Thanks. ]

[Gvwyn]

I replaced them with whoopsie daisy 😄

Could not reproduce this with the C example (used x86_64-w64-mingw32-gcc t.c -I../include -L../lib -lcurl), and an equivalent curl command line also works. Tested under WINE.

So it worked just fine on your machine? I guess I could try downloading libcurl again (perhaps trying the 32 bit version?) and see if that does anything differently.

I also tried to spam curl from the command line to see if it ever fails, but no, it's only my code thats bugging out.

But still, thank you for taking your time to test my code.

[vszakats]

Nice, thanks!

Yes, got the binaries from https://curl.se/windows/dl-8.8.0_1/curl-8.8.0_1-win64-mingw.zip. Strictly speaking my build wasn't correct, because GCC uses MSVCRT by default, while the distributed libcurl DLL uses UCRT, so I repeated it with llvm-mingw using <path-to>/llvm-mingw/bin/x86_64-w64-mingw32-clang -I../include -L../lib -lcurl -s -o t-clang.exe test.c. It also worked without issues.

I suggest using llvm-mingw or a GCC configured for UCRT to avoid using two CRTs in one binary.

[Gvwyn]

I tried compiling it with llvm-mingw to see if it changes anything, and damn right it did. The errors are now gone, and the code runs flawlessly, even when I remove the 5 second cooldown between calls.

Honestly this is my first time hearing about CRTs, but I guess learning new things is the whole purpose of trying out and experimenting with stuff...

Thank you!