Error when downloading a large file (>2Gb)

[evgslyusar]

Expected Behavior

Download a large file (> 2 GB)

Actual Behavior

An error occurs: Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647

Steps to Reproduce the Problem

1. The HTTP service returns a file larger than 2 GB.

Notes

When I download a large file (> 2 GB), an error occurs: Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. If to use the HttpCompletionOption.ResponseHeadersRead parameters when calling the HttpClient.SendAsync method, then an error does not occur. But I did not find a way how this can be used in the current version of Ocelot.

Specifications

• Version: 13.5.2
• Platform: Windows

[alekshura]

I've resolved it implementing own IHttpRequester (only for one endpoint that returns big files - "contents")

public class HttpRequester : IHttpRequester
{
    private readonly IHttpClientCache _cacheHandlers;
    private readonly IOcelotLogger _logger;
    private readonly IDelegatingHandlerHandlerFactory _factory;
    private readonly IExceptionToErrorMapper _mapper;
    private readonly IHttpClientFactory _httpClientFactory;

    public HttpRequester(IOcelotLoggerFactory loggerFactory,
        IHttpClientCache cacheHandlers,
        IDelegatingHandlerHandlerFactory factory,
        IExceptionToErrorMapper mapper, IHttpClientFactory httpClientFactory)
    {
        _logger = loggerFactory.CreateLogger<HttpClientHttpRequester>();
        _cacheHandlers = cacheHandlers;
        _factory = factory;
        _mapper = mapper;
        _httpClientFactory = httpClientFactory;
    }

    public async Task<Response<HttpResponseMessage>> GetResponse(HttpContext httpContext)
    {
        if (httpContext.Request.Path.ToString().Contains("contents"))
        {
            var client = _httpClientFactory.CreateClient();
            var request = httpContext.Items.DownstreamRequest();
            var response = await client.SendAsync(request.ToHttpRequestMessage(), HttpCompletionOption.ResponseHeadersRead);
            return new OkResponse<HttpResponseMessage>(response);
        }

        var builder = new HttpClientBuilder(_factory, _cacheHandlers, _logger);
        var downstreamRoute = httpContext.Items.DownstreamRoute();
        var downstreamRequest = httpContext.Items.DownstreamRequest();
        var httpClient = builder.Create(downstreamRoute);

        try
        {
            var response = await httpClient.SendAsync(downstreamRequest.ToHttpRequestMessage(), httpContext.RequestAborted);
            return new OkResponse<HttpResponseMessage>(response);
        }
        catch (Exception exception)
        {
            var error = _mapper.Map(exception);
            return new ErrorResponse<HttpResponseMessage>(error);
        }
        finally
        {
            builder.Save();
        }
    }
}

and adding to DI container before Ocelot:

builder.Services.AddHttpClient();
builder.Services.TryAddSingleton<IHttpRequester, HttpRequester>();
builder.Services.AddOcelot(configuration);

[ggnaegi]

@raman-m var response = await client.SendAsync(request.ToHttpRequestMessage(), HttpCompletionOption.ResponseHeadersRead); for the downloads https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpcompletionoption?view=net-8.0

[raman-m]

@ggnaegi commented on Dec 12

Got it! We will link this issue to the next your PR(s) for Dec'23 release...

[ggnaegi]

@raman-m this point is addressed in #1824

[raman-m]

@alekshura commented on Apr 6, 2023

if (httpContext.Request.Path.ToString().Contains("contents"))

You are cool C# hacker definitely! 😉 🤣
And your solution should work. Congrats! 🥳
How is your life in Warshawa?

[raman-m]

@evgslyusar commented on Jun 8, 2020:

When I download a large file (> 2 GB), an error occurs: Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. If to use the HttpCompletionOption.ResponseHeadersRead parameters when calling the HttpClient.SendAsync method, then an error does not occur. But I did not find a way how this can be used in the current version of Ocelot.

We removed HttpCompletionOption.ResponseHeadersRead life hack in #1853 because we've implemented completely new Http Request processing design by StreamHttpContent vs a new RequestMapper which solve the problem of streaming including large files. Please, see #1724 !

---

And please, re-fork Ocelot repo: it is outdated now having master as a default branch.
So, I've created PR 1 😉

[raman-m]

• Fixed by #827 #1679 Improve performance of Request Mapper #1724