Allow to use HTML5 audio tag with API Web Audio

[marizuccara]

Introduced the WebAudioMedia backend which allows to load audio as HTML5 audio tag and use it with WebAudio backend.

Setting the WebAudioMedia backend,there is a possibility to load audio of big dimensions, using the API Web Audio features. In this way, filters and other functionalities can be performed as with WebAudio backend without decoding internally audio data, that caused crash of the browser.

Breaking in the external API:

Added the WebAudioMedia backend. The API to use to load an audio or playback it, are the same as with the MediaElement backend.

Breaking changes in the internal API:

Added a file WebAudioMedia that extends WebAudio. Instead of using the AudioBufferSourceNode to push the source to the AudioContext, it was used the MediaElementSourceNode which takes in input the audio tag. Since an HTML5 file is loaded, there were used APIs of MediaElement backend.

Todos/Notes:

It was created another backend to test this functionality and show you my proposal. In fact, I have duplicated the MediaElement code because the resource is handled as HTML5 audio tag. For other operations, there were used WebAudio APIs.

Related Issues and other PRs:

fixes #1763

[coveralls]

Coverage increased (+2.5%) to 80.984% when pulling 4172a53 on marizuccara:master into 5afec8d on katspaugh:master.

[katspaugh]

Thanks for this PR!
What was your incentive to implement this new backend?
My guess would be some people might want to use a media element but control it via Web Audio?

[marizuccara]

Thanks for this PR!
What was your incentive to implement this new backend?
My guess would be some people might want to use a media element but control it via Web Audio?

If I load a big audio file with WebAudio backend, because it internally decodes the AudioBuffer, the browser crashes. Using HTML5 tag, the audio data are not decoded, so in case of a big audio file browser does not crash. Moreover, in this way I can use it with API Web Audio and add filters, like control left and right channels, pannerNode, equalizer, etc

[katspaugh]

Using HTML5 tag, the audio data are not decoded, so in case of a big audio file browser does not crash.

Even with an HTML5 media element, the audio data is still decoded to draw the waveform.

[marizuccara]

Even with an HTML5 media element, the audio data is still decoded to draw the waveform.

But giving peaks, the waveform is drawn with them, no?

[katspaugh]

Yes, correct.

[marizuccara]

Yes, correct.

With WebAudio backend, even if i give peaks to draw the waveform, it internally decodes the audio to transform it to an AudioBuffer and put it to AudioBufferSourceNode. So the waveform is drawn without any problems, but in case of big audio files, the browser crashes because it decodes the data in memory.
If we give an HTML5 tag with peaks, the waveform is drawn without decoding. Moreover, giving the source (HTML tag) to MediaElementSourceNode, it does not decode audio data in memory, so it can be played without any problems (like with MediaElement backend).
So in this way we can add filters and have more controls with audio because we can use API Web Audio, even with big audio files.

[katspaugh]

Cool, gotcha.
I guess this new backend is a good option then. More powerful than a standalone media element, but more efficient than the buffer source.
Let's clean up the PR and merge it. Any idea how to get that coverage back to 79%?

[marizuccara]

Cool, gotcha.
I guess this new backend a good option then. More powerful than a standalone media element, but more efficient than the buffer source.
Let's clean up the PR and merge it. Any idea how to get that coverage back to 79%?

Maybe, we can remove private to functions like createScriptNode, createVolumeNode and Analyser, so we can extend them normally

[aburai]

the support of big files is important, it is good to talk about a solution

[katspaugh]

Wow, that greatly simplified things.
As per my previous comments, could you please restore the empty lines as they were in all the changed files?

[katspaugh]

Looks great! Thanks a lot!

[katspaugh]

We just need to add the new backend type to the docs now. The docs are on another branch, gh-pages, so it should be another PR.

[thijstriemstra]

can you also look at simple unit tests (see existing MediaElement tests)?

[marizuccara]

can you also look at simple unit tests (see existing MediaElement tests)?

Have I run your karma tests or write other tests?

[aburai]

can you also look at simple unit tests (see existing MediaElement tests)?

Have I run your karma tests or write other tests?

before pushing pull requests one should always run the tests with npm run test or yarn test
but in this case we add something new that should have test coverage
an example is found under spec/mediaelement.spec.js
if you need any help, let me know, when i have time to test the new backend, i can have a look at the tests and/or append some tests that make sense

[thijstriemstra]

Added examples and updated CHANGES.md (updating changelog caused conflicts).

I don't see the example?

[marizuccara]

Added examples and updated CHANGES.md (updating changelog caused conflicts).

I don't see the example?

[kavishatalsania]

Hi @marizuccara / @thijstriemstra / @katspaugh ,
Thanks for this great feature.

I used 'MediaElementWebAudio' backend. It does not load whole audio and I am also able to use filters to the mute channel.

One issue I found is when I load the audio file from s3 it is giving error 'MediaElementAudioSource outputs zeroes due to CORS access restrictions'

I am able to resolve this issue by adding line media.crossOrigin = "anonymous" in the load method of wavesurfer at line 1955.

key: "load",
value: function load(url, container, peaks, preload) {
      var media = document.createElement(this.mediaType);
      media.controls = this.params.mediaControls;
      media.autoplay = this.params.autoplay || false;
      media.preload = preload == null ? 'auto' : preload;
      media.src = url;
      media.style.width = '100%';
      media.crossOrigin = "anonymous";
      var prevMedia = container.querySelector(this.mediaType);
      if (prevMedia) {
        container.removeChild(prevMedia);
      }
      container.appendChild(media);
      this._load(media, peaks);
}

Is it right resolution for this problem?

[marizuccara]

Hi! Thanks for the feedback!

You should be able also to create yourself an audio tag.
For example, <audio src="http://domain/audio" crossOrigin="anonymous">,
or by JavaScript,
let audio = document.createElement('audio');
audio.src = "http://domain/audio";
audio.crossOrigin = "anonymous";
Then, pass the audio tag to load method:
wavesurfer.load(audio, peaks).

[kavishatalsania]

Hi @marizuccara
Thanks for your suggestion. It's working.

I got impression from the documentation and code that I could only pass the audio URL in the load method not audio object.

[thijstriemstra]

audio.crossOrigin = "anonymous";

can we add this line to the example?

[thijstriemstra]

merged. thanks @marizuccara!

[katspaugh]

Thanks a lot, @marizuccara! A great contribution!

[marizuccara]

Thanks so much to you, @katspaugh and @thijstriemstra, for this fantastic library and for the support!

[Leaybc]

Looking forward to releasing new version as soon as possible

[kavishatalsania]

Hi @katspaugh / @thijstriemstra

When are you planning to release new version with these changes?

[thijstriemstra]

When are you planning to release new version with these changes?

Documentation first needs to be updated and stuff. See #1782