[FR] readOpenMemory variant with pointer and size to work with mmap

[FooIbar]

No description provided.

[zhanghai]

It seems to me you can implement it normally with the archive_read_open call - see also https://github.com/bramp/libarchive/blob/master/libarchive/archive_read_open_memory.c .

[FooIbar]

It seems to me you can implement it normally with the archive_read_open call - see also https://github.com/bramp/libarchive/blob/master/libarchive/archive_read_open_memory.c .

ReadCallback requires a ByteBuffer as well.

libarchive-android/library/src/main/java/me/zhanghai/android/libarchive/Archive.java

Line 266 in 1d91b0a

@Nullable SeekCallback<?> callback) throws ArchiveException;

BTW is the ? a typo?

[zhanghai]

ReadCallback requires a ByteBuffer as well.

What about MappedByteBuffer?

BTW is the ? a typo?

Ah yes, I think it's a typo.

[FooIbar]

ReadCallback requires a ByteBuffer as well.

What about MappedByteBuffer?

FileChannel.map has some annoying limitations, like, the size must not exceed Integer.MAX_VALUE (i.e. 2 GB).

[zhanghai]

I can add an Archive.readOpenMemoryUnsafe() that takes a raw pointer and size. Out of curiosity, why do you need to mmap() the file instead of using regular IO (or pread())?

Meanwhile I tried but couldn't easily add an Archive.writeOpenMemoryUnsafe() because it's a bit awkward for how to return the actually used memory size in an async way without the ByteBuffer. So I hope you aren't doing mmap() for writes :P

---

Note to myself: https://bugs.openjdk.org/browse/JDK-6347833 - We could have used MemorySegment if and when our minimum SDK version supported Java FFM API (JEP 454).

[zhanghai]

Actually I think I can add the Archive.writeOpenMemoryUnsafe() because I can & should add a Java API for getting the used out-parameter any time.

[FooIbar]

Thanks!
I need to read multiple entries from the archive simultaneously.
pread() does work to some extent, but mmap() woud be better.

[zhanghai]

I need to read multiple entries from the archive simultaneously.

Out of curiousity, how did you manage to do that? I thought libarchive isn't thread safe by itself:

The library is not thread aware, however. It does no locking or thread management of any kind. If you create a libarchive object and need to access it from multiple threads, you will need to provide your own locking.

https://github.com/libarchive/libarchive/blob/master/README.md

(Closing because the new APIs have been released in v1.1.0 - but please feel free to keep commenting.)

[FooIbar]

I read through the archive to get the entry list first, then use individual archive objects for requested entries.

[zhanghai]

then use individual archive objects for requested entries.

1. Does that mean you are creating multiple archive objects with archive_read_open_memory in order to read in parallel?
2. How are you reusing the entries read earlier in order to read its data? I can imagine one can keep calling archive_read_next_header until the entry name matches with the desired entry, and do this in parallel. But maybe you know some better way to skip directly to the desired entry?

[FooIbar]

1) Yes 2) That's what I did

Here is the commit FooIbar/tachiyomi@35f5f0d

[zhanghai]

I see. I was looking at the other apps of yours which is using mmap() and madvise() with its own build of libarchive.

And curious, will you be upstreaming FooIbar/tachiyomi@35f5f0d to Mihon? (If that's the case I'll be happy that my code is part of Mihon as well - which I'm using right now.)

[FooIbar]

See mihonapp/mihon#949

[zhanghai]

Nice! Subscribed to that PR.

[AntsyLich]

If that's the case I'll be happy that my code is part of Mihon as well - which I'm using right now.

I use your file manager too 🙃

[zhanghai]

I use your file manager too 🙃

Glad you like it 🤠