-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error running rq #1
Comments
Key line is You're running everything correctly, but I think Ubuntu package you're using is missing dependency on "liblz4" or something like that. lz4 dependency was added to libRaptorQ rather recently - only in latest v0.1.7 release - so I'd suggest you file a bug in Ubuntu bugzilla, so that libRaptorQ package maintainer would know about it and fix it. Doesn't have much to do with python code, I think. |
Though to be fair, I didn't really test the thing with v0.1.7, might also happen to be some more generic cffi or libRaptorQ issue, but that seem unlikely. Thanks. |
Hi Yes, I believe liblz4 was already installed as part of the libRaptorQ 0.1.7 install. I have also installed manually just to be sure (apt-get install liblz4-dev).
I have also tried with libRaptorQ 0.1.6, but libRaptorQ only creates:
So it tried to create a softlink from libRaptorQ-0.1.6.so to to libRaptorQ.so, but rq then has problems accessing the library: Tried running ldconfig, but no still no luck. |
Ah, yeah, it's the issue with 0.1.6 that I've raised here some time ago: It's weird that you get similar linking error though, given that 0.1.6 should only be linked against very basic stuff like libc, maybe try Symlinking/moving the lib should totally work in general, if lib itself works, of course. I'll probably try 0.1.7 here later, see if I can reproduce the issue, but seeing how you get similar error for 0.1.6, suspect it might be due to something that I don't have here, unfortunately. |
my library in ubuntu was installed in /usr/local/lib and not /usr/lib So I created a link in /usr/lib to the 0.1.6 lib in /usr/local/lib and rq works now with 0.1.6. I will reinstall 0.1.7 to see if the same fix works. |
If that was the issue and moving stuff is a hassle, you can probably also do it by putting e.g. "local.conf" to |
Still getting same error with 0.1.7, so that is probably not library path issue.
I can rq encode a file with 0.1.6, but with decoding I get:
|
Looking into that, found few loosely-related things, all with 0.1.6 that I have:
Might need to look into python wrapper code, check if I can spot something wrong there, but doubt it can do the straight-up data corruption as in the latter case. Also need to try 0.1.7 for that other lz4 issue and to see if maybe latter thing is also gone there. But seeing the general pattern, I'd advise you against using this module, unless you're totally ok with loosing your data, apparently it's a load of badly-tested crap, sorry. Will also put up a warning in the README. |
Built libRaptorQ here with following PKGBUILD (nothing odd there, same process as in lib's README): https://github.com/mk-fg/archlinux-pkgbuilds/blob/master/libraptorq/PKGBUILD If you'll find what was the cause of that issue, probably worth reporting to libRaptorQ here - https://www.fenrirproject.org/Luker/libRaptorQ/issues - as it might be easy to address in the build system or such. |
Thank you, will do. |
I should probably clarify my message above - while encoding itself works fine, and raises no lz4-related errors, there is still show-stopper issue that anything roughly >100K fails to decode. I've now looked over python code, and couldn't spot anything wrong there (e.g. memory-management gc-related issues sometimes pop-up when passing stuff to C code), so I'd still suggest to avoid using the module. One obvious way to look further into it would be to write C/C++ tool from scratch that'd do same thing, and if it fails in exactly same way, likely that it's a bug in the lib, but I don't know if I'll get around to do that anytime soon. Asked if maybe Luca would be up for the task, as I think such tool would be useful companion to the lib itself (e.g. maybe you won't be needing python wrapper at all then) here: https://www.fenrirproject.org/Luker/libRaptorQ/issues/9 |
Hi @ALL, I'm the developer for libRaptorQ. 1- LZ4 is an optional dependency in the master branch, but not in the 0.1.X releases. the master branch is now working towards v0.2, with new API, performance stuff and so on. I thought the "prealpha" version in the name would have been enough, along with the first lines in the readme... Any pointers on how to avoid future problems like this? 2- Sometimes the encoder gives back 0 blocks, or immediately returns: This probably means that there is something wrong in the encoder initialization. Symbol size must be a multiple of subsymbol_size, max_memory is related to the maximum amount of memory. Funny stuff: what does "related to" mean? Don't know. v0.1 was modelled with RFC6330 in mind, and that parameter is needed. Might be maximum bytes for the internal algorithm? Close, but not really, as the actual amount of memory internally used is not that. But max_mem is used to divide your input in blocks, and choose the block size. You are also limited to 256 blocks, so if you put a really small amount of memory here, RQ won't work. Which is why v0.2 will change APIs, along with performance stuff. the API you are using will be under the RFC630:: C++ namespace, and will be moved to a rfc6330_ prefix in the C functions. So maybe you are using low parameters as max_mem, which would give more than 256 blocks? 3- I see you are using different sizes for subsymbols and symbols. That is only used to interleave the data (again, RFC), but IMHO it's useless. Interleaving makes sense for example in audio streaming so that losing a packet means losing a couple of bytes for multiple samples, basically reducing the quality without hearing the other stuttering. But it also means that the application has to actually de-interleave the data by itself, and be able to work with half-received packets, or wait for RQ to recover everything. Making this a requirement in the RFC for everyone was a really retarded thing to do and has a (low) performance impact. Suggestion: keep subsym_size == sym_size. Again, no such problems in v0.2. 4- I do not know python, so I can't really debug your code, sry. 5- v0.1.X is bufgix only, there are a couple of API problems and inefficiencies, I'd rather just work on v.02 now. But I might think about writing the binary you suggested, should be simple enough. I think one other is needed for streams of data, not just static files. Easy enough. |
Right, my bad, think I've ran grep over git-log and checked lz4-related commit date, then simply checked that vs date on git-tag's, without taking diverging branches into account at all - a mistake.
As mentioned in the issue on the libRaptorQ gitlab, parameters for that particular attached file are fairly fixed and seem to satisfy symbols criterias (ENC_32 subsymbol=8 symbol_size=16)... But max-memory thing indeed makes a world of difference - running e.g. Same for larger files - bumping max-mem value works, doesn't produce symbols that result in corrupted data upon decoding anymore.
Yeah, exactly what happened. Would be cool if lib did some sanity checks there, if at all possible, instead of silently producing stuff that won't be decodable, as that'd clearly indicate where error happens (in this case - script parameters). Have to admit that I didn't actually read RFC, so script parameters have descriptions like this one:
And that's how blame shifts onto poor user who happens to run it ;) I'll look over the RFC and try adding a bunch of sanity checks here in the module, I guess, and try to emphasize that unless one read the RFC, garbage will be produced, as clearly is the case here.
Yeah, will do, thanks for clarification. I actually grabbed current defaults (and used them myself) from the same test_c.c "practical API example", which I guess has them as some kind of "weirdest thing possible" case - i.e. "if that works, everything will" - and not common defaults at all. Again, reading through the RFC would've probably made me pick something saner there.
Actually, you've pretty much fixed it already ;)
Got it, will update the thing for the new API when it comes out, I guess, thanks for the tip.
Assuming you mean something like "read endless stream from stdin, write to stdout", I'd think that it can be same thing as for static files - split input into blocks of pre-defined size encode each one independently (with some fixed redundancy), output in the same order.
I think ideal option for both random person who wants to encode data and me who wants "reference encoder" tool:
That's an ideal though, as it should work for:
For my "test if C API works at all with this file" case (especially now that I'm aware that parameters can cause silent data corruption), simple test.c with hard-coded path to file and parameters (which I can then tweak and re-compile) will work too, but probably not so much for a person who wants to just run it from terminal. |
Made all libRaptorQ encoder parameters like symbol-size and max-memory mandatory in the command-line interface of this module now, and added a bunch of warning to the README as to dangers of these, so hopefully no one will rely on them to "just work" for random use-case. Guess both lz4 and undecodable data / crashes should be addressed by now, so closing the issue here. |
Hi
I have installed libraptorq successfully, but get the following library error when running rq on Ubuntu 16.04 (64bit). Is this a known issue? Or I am doing something wrong?
The text was updated successfully, but these errors were encountered: