Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple support for STEM files #13044

Merged
merged 8 commits into from
May 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert breaking change on soundsource and improve docstrings
  • Loading branch information
acolombier committed Apr 18, 2024
commit 95155b3cdaa5487ee8ac02a1502e25e14ffc782d
6 changes: 3 additions & 3 deletions src/sources/soundsource.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ QString SoundSource::getTypeFromUrl(const QUrl& url) {

//static
QString SoundSource::getTypeFromFile(const QFileInfo& fileInfo) {
const QString fileSuffix = fileInfo.completeSuffix().toLower().trimmed();
const QString fileSuffix = fileInfo.suffix().toLower().trimmed();

if (fileSuffix == QLatin1String("opus")) {
// Bypass the insufficient mime type lookup from content for opus files
Expand All @@ -52,13 +52,13 @@ QString SoundSource::getTypeFromFile(const QFileInfo& fileInfo) {
// https://mixxx.zulipchat.com/#narrow/stream/109171-development/topic/mimetype.20sometimes.20wrong
return fileSuffix;
}
if (fileSuffix == QLatin1String("stem.mp4")) {
if (fileInfo.fileName().toLower().trimmed().endsWith(QLatin1String(".stem.mp4"))) {
// STEM suffix extends the ".mp4" suffix. This is because stem files are
// meant to fall back to MP4 decoder, in case a player doesn't have the
// capability to extract the different stem track. However, we need to
// detect that early as the file natively looks like a traditional MP4
// with multiple tracks (audio and video)
return fileSuffix;
return QLatin1String("stem.mp4");
}

QMimeType mimeType = QMimeDatabase().mimeTypeForFile(
Expand Down
2 changes: 1 addition & 1 deletion src/sources/soundsourcestem.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ namespace mixxx {
class SoundSourceSingleSTEM : public SoundSourceFFmpeg {
public:
// streamIdx is the FFmpeg stream id, which may different than stemIdx + 1
// because STEm may contain other non audio stream
// because STEM may contain other non audio stream
explicit SoundSourceSingleSTEM(const QUrl& url, unsigned int streamIdx);

protected:
Expand Down
18 changes: 9 additions & 9 deletions src/track/steminfoimporter.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,23 +32,23 @@ const uint32_t kStemManifestAtomPath[] = {
/// @param reader The IODevice to search MP4 atom in
/// @param path The list of atom to traverse in the tree, from top to bottom.
/// Must be null terminated
/// @param box_size The size of the box currently under the cursor to focus the
/// @param boxSize The size of the box currently under the cursor to focus the
/// search in
/// @param pathIdx The level of the tree currently search (index of path)
/// @return the size of the box data if found, -1 otherwise
uint32_t seekTillAtom(QIODevice& reader,
acolombier marked this conversation as resolved.
Show resolved Hide resolved
const uint32_t path[],
uint32_t box_size = 0,
uint32_t boxSize = 0,
uint32_t pathIdx = 0) {
if (!path[pathIdx]) {
return box_size;
return boxSize;
}

uint32_t byteRead = 0;
char buffer[kAtomHeaderSize];
while (!reader.atEnd() || (box_size && byteRead >= box_size)) {
while (!reader.atEnd() || (boxSize && byteRead >= boxSize)) {
reader.read(buffer, kAtomHeaderSize);
byteRead += box_size;
byteRead += boxSize;
if (ATOM_TYPE(buffer + 4) == path[pathIdx]) {
return seekTillAtom(reader, path, ATOM_TYPE(buffer) - kAtomHeaderSize, pathIdx + 1);
acolombier marked this conversation as resolved.
Show resolved Hide resolved
}
Expand All @@ -61,8 +61,8 @@ uint32_t seekTillAtom(QIODevice& reader,

// static
bool StemInfoImporter::isStemFile(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we need apply the mime type logic. And probably the checking for easy "stem" properties in addition.
Alternative this may look for these properties only and we can call the function isMp4AStem() or such.
This raises the general question if this Class can be only used with MP4s or if we need to protect every invocation to be called with other files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we need apply the mime type logic

AFAIK, mime type isn't available in the Track object. Reading the file (to detect the MIME using QMimeDatabase::mimeTypeForFile will likely have effect on performance. of every single file being imported, which is likely not negligible when importing a large library.

This raises the general question if this Class can be only used with MP4s

I'm not sure why we would want to support another file type. The STEM atom is part of the NI standard, meaning it needs to be in a mp4 file. If we want to implement other STEM standards, there is no guarantee that there will use a STEM atom, as part of the MPEG standard. My suggestion would be to keep the implantation to what we know and support currently (NI standard, mp4) and cross the bridge to support other standards when we decide to do so.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... will likely have effect on performance. of every single file being imported, which is likely not negligible ...

We currently already support playing mp3s with wav extension for instance. I like to have the same for stems if that is reasonable.

I am not sure if the library scanner does also use mine types, need to check. At least we should treat all the files similar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that that detection based on stem.mp4 extension is overly restrictive. At the extreme any iso media container could be feature tested for moov.udta.stem, but at the least we should be able to detect stems in arbitrary .mp4 containers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, I was simply implementing the specification (which I believe @dszakallas also had access to). Happy to go beyond what it defines.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not expand this PR by general MP4 support. Scope of this PR is the STEMs format and nothing beyond!
Traktor Pro seems to have another detection logic than just checking for .stems.mp4 extension, because it seems to work also with the extension .stems.m4a (or they just check for one of both file extensions).
Performance for file type detection is important in my opinion. What would it cost to probe for moov.udta.stem ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already support mp4 some way, as I was able to drop mp4 tracks into mixxx and play them. And the specification says that the container should be an ISOBMFF with .mp4 as extension. .stem.mp4 is "preferred", not required. So, if we want to follow the spec we should try to detect stems in any mp4.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would it cost to probe for moov.udta.stem ?

That's hard to assess as the atom offset isn't static and require to traverse boxes in the file.

extension. .stem.mp4 is "preferred", not required

That's true but in the meantime, Mixxx aims to be vendor-agonistic IMO, so the flexibility that Tracktor Pro gives to arrange NI shouldn't drive Mixxx implementation. Anyway, last commit adds support for .mp4, but happy to remove depending where the consensus goes.

const QString& aFileName) {
return aFileName.endsWith(kStemFileExtension);
const QString& fileName) {
return fileName.endsWith(kStemFileExtension);
}

QList<StemInfo> StemInfoImporter::importStemInfos(
Expand Down Expand Up @@ -128,8 +128,8 @@ QList<StemInfo> StemInfoImporter::importStemInfos(
}

// Extract DSP information
// TODO(XXX) DSP only contains limit and a compressor, which aren't
// supported by Mixxx yet. parse and implement when supported
// TODO(XXX) DSP only contains a limiter and a compressor effect, which
// Mixxx doesn't have yet. Parse and implement when supported
acolombier marked this conversation as resolved.
Show resolved Hide resolved

file.close();
acolombier marked this conversation as resolved.
Show resolved Hide resolved

Expand Down
3 changes: 1 addition & 2 deletions src/track/steminfoimporter.h
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@

namespace mixxx {

/// Importer class for StemInfo objects that can correct timing offsets when the
/// signal info (channel number, sample rate, bitrate) is known.
/// Importer class for StemInfo objects that contains metadata about the stems of a track.
class StemInfoImporter {
public:
static QList<StemInfo> importStemInfos(
Expand Down
Loading