Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: improve fsPromises readFile performance #37608

Closed
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
fs: improve fsPromises readFile performance
Improve the fsPromises readFile performance
by allocating only one buffer, when size is known,
increase the size of the readbuffer chunks,
and dont read more data if size bytes have been read

refs: #37583
  • Loading branch information
Linkgoron committed Mar 5, 2021
commit cfb180594f6d2468b41c3693dc730b0809ba7518
50 changes: 35 additions & 15 deletions lib/internal/fs/promises.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@
// See https://github.com/libuv/libuv/pull/1501.
const kIoMaxLength = 2 ** 31 - 1;

// Note: This is different from kReadFileBufferLength used for non-promisified
// fs.readFile.
const kReadFileMaxChunkSize = 2 ** 14;
const kReadFileBufferLength = 512 * 1024;
const kReadFileUnknownBufferLength = 64 * 1024;
const kWriteFileMaxChunkSize = 2 ** 14;

const {
Expand Down Expand Up @@ -316,25 +315,46 @@ async function readFileHandle(filehandle, options) {
if (size > kIoMaxLength)
throw new ERR_FS_FILE_TOO_LARGE(size);

const chunks = [];
let isFirstChunk = true;
const firstChunkSize = size === 0 ? kReadFileMaxChunkSize : size;
const chunkSize = MathMin(firstChunkSize, kReadFileMaxChunkSize);
let endOfFile = false;
let totalRead = 0;
const noSize = size === 0;
const buffers = [];
const fullBuffer = noSize ? undefined : Buffer.allocUnsafeSlow(size);
do {
if (signal?.aborted) {
throw lazyDOMException('The operation was aborted', 'AbortError');
}
const buf = Buffer.alloc(isFirstChunk ? firstChunkSize : chunkSize);
const { bytesRead, buffer } =
await read(filehandle, buf, 0, buf.length, -1);
endOfFile = bytesRead === 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it looks like it should be enough to change this line to:

    totalRead += bytesRead;
    endOfFile = bytesRead === 0 || totalRead === size;

in order to get the same performance improvement as in this PR (at least that's what I saw on my machine). Another useful optimization here is Buffer.allocUnsafeSlow. So, maybe it's worth considering keeping only these 4 lines of changes, but with the same result.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that some of the changes are not strictly necessary for the improvement, but another feature (IMO) of my changes it that now the logic is essentially the same as the sync readFile.

if (bytesRead > 0)
ArrayPrototypePush(chunks, buffer.slice(0, bytesRead));
isFirstChunk = false;
let buffer;
let offset;
let length;
if (noSize) {
buffer = Buffer.allocUnsafeSlow(kReadFileUnknownBufferLength);
offset = 0;
length = kReadFileUnknownBufferLength;
} else {
buffer = fullBuffer;
offset = totalRead;
length = MathMin(size - totalRead, kReadFileBufferLength);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this is a behavioral change for files that are being appended to while they are being read. I think that should be okay, because ultimately there are no guarantees for the relative timing of the two operations, but it might be something to keep in mind.

Copy link
Member Author

@Linkgoron Linkgoron Mar 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're correct and I should've mentioned it in the PR but forgot. This is aligning the behaviour with the cb version, so I thought that it would be OK.

length = MathMin(kReadFileBufferLength, this.size - this.pos);

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think that’s okay 👍


const bytesRead = (await binding.read(filehandle.fd, buffer, offset,
length, -1, kUsePromises)) || 0;
totalRead += bytesRead;
endOfFile = bytesRead === 0 || totalRead === size;
if (noSize && bytesRead > 0) {
const isBufferFull = bytesRead === kReadFileUnknownBufferLength;
const chunkBuffer = isBufferFull ? buffer : buffer.slice(0, bytesRead);
ArrayPrototypePush(buffers, chunkBuffer);
}
} while (!endOfFile);

const result = chunks.length === 1 ? chunks[0] : Buffer.concat(chunks);
let result;
if (size > 0) {
result = totalRead === size ? fullBuffer : fullBuffer.slice(0, totalRead);
} else {
result = buffers.length === 1 ? buffers[0] : Buffer.concat(buffers,
totalRead);
}

return options.encoding ? result.toString(options.encoding) : result;
}
Expand Down
15 changes: 10 additions & 5 deletions test/parallel/test-fs-promises-file-handle-readFile.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ const {
open,
readFile,
writeFile,
truncate
truncate,
Copy link
Contributor

@RaisinTen RaisinTen Mar 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
truncate,
truncate

nit: Unrelated change

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I'm in favour of not spending mental energy on stylistic changes that are not enforced as part of linting rules.

} = fs.promises;
const path = require('path');
const tmpdir = require('../common/tmpdir');
Expand Down Expand Up @@ -64,6 +64,7 @@ async function doReadAndCancel() {
await assert.rejects(readFile(fileHandle, { signal }), {
name: 'AbortError'
});
await fileHandle.close();
}

// Signal aborted on first tick
Expand All @@ -74,10 +75,11 @@ async function doReadAndCancel() {
fs.writeFileSync(filePathForHandle, buffer);
const controller = new AbortController();
const { signal } = controller;
tick(1, () => controller.abort());
process.nextTick(() => controller.abort());
await assert.rejects(readFile(fileHandle, { signal }), {
name: 'AbortError'
});
}, 'tick-0');
await fileHandle.close();
}

// Signal aborted right before buffer read
Expand All @@ -90,10 +92,12 @@ async function doReadAndCancel() {

const controller = new AbortController();
const { signal } = controller;
tick(2, () => controller.abort());
tick(1, () => controller.abort());
await assert.rejects(fileHandle.readFile({ signal, encoding: 'utf8' }), {
name: 'AbortError'
});
}, 'tick-1');

await fileHandle.close();
}

// Validate file size is within range for reading
Expand All @@ -111,6 +115,7 @@ async function doReadAndCancel() {
name: 'RangeError',
code: 'ERR_FS_FILE_TOO_LARGE'
});
await fileHandle.close();
}
}

Expand Down