Skip to content
This repository has been archived by the owner on Apr 19, 2023. It is now read-only.

Call mediaConvert.getJob in MediaConvert status lambda #315

Closed
kylekirkby opened this issue Oct 6, 2021 · 8 comments
Closed

Call mediaConvert.getJob in MediaConvert status lambda #315

kylekirkby opened this issue Oct 6, 2021 · 8 comments

Comments

@kylekirkby
Copy link

Is your feature request related to a problem? Please describe.
The Media Convert status lambda currently returns a MediaConvert job ID. This on its own is not very useful. Even when submitting the job to MediaConvert you don't get returned a job ID to match to the status events. I was hoping to call mediaConvert.getJob with the job ID to get the name of the file that the job relates to (with this I could then update the DynamoDB entry for the video with the conversion progress).

I'm currently getting this error message when calling getJob().

const AWS = require('aws-sdk');
const mediaConvert = new AWS.MediaConvert({
	region: process.env.REGION,
});

console.log('Loading function');

exports.handler = async (event, context) => {
	console.log(event);
	console.log(process);
	// console.log('Received event:', JSON.stringify(event, null, 2));
	const message = JSON.parse(event.Records[0].Sns.Message);
	console.log('From SNS:', message);
	let params = {
		Id: message.id /* required */,
	};
	console.log('Params:', params);
	let jobDetails = await mediaConvert.getJob(params).promise();
	console.log('Job details: ', jobDetails);
	console.log('context ', context);
	return message;
};
  "errorType": "BadRequestException",
    "errorMessage": "You must use the customer-specific endpoint 'https://xxxxxx.mediaconvert.us-east-1.amazonaws.com' for this operation.",
    "code": "BadRequestException",
    "message": "You must use the customer-specific endpoint 'https://xxxxxx.mediaconvert.us-east-1.amazonaws.com' for this operation.",

How can I get the "custom specific" endpoint? I'd put this in by hand but I'd rather it be dynamic if possible due to IaaC principles etc.

Describe the solution you'd like
Some guidance on how to access this endpoint within the MediaConvert status lambda. It would also be good to add this by default as, AFAIK, the MediaConvert response is useless without some way of correlating the status to a specific video.

Describe alternatives you've considered
I've added the region environment variable in hope that this would resolve the issue to no avail.

As I'm writing this issue I've noticed a userMetadata: {}, attribute is returned in the SNS event message. Maybe when creating the MC job I could tag the job with the video ID. I'll update this ticket if I can get that working.

Cheers!

@kylekirkby
Copy link
Author

Also... just noticed this block of code in the input lambda:

	let mcClient = new AWS.MediaConvert();
	if (!AWS.config.mediaconvert) {
		try {
			const endpoints = await mcClient.describeEndpoints().promise();
			AWS.config.mediaconvert = { endpoint: endpoints.Endpoints[0].Url };
			// Override so config applies
			mcClient = new AWS.MediaConvert();
		} catch (e) {
			console.log(e);
			return;
		}
	}

🤦‍♂️

@nathanagez
Copy link
Collaborator

nathanagez commented Oct 6, 2021

Hi @kylekirkby, if I'm not wrong you should have inside your SNS message the job detail, no ?

What does your message.detail object contains ?

const path = require('path')
const { AudioAnalysis } = require('/opt/audio-analysis')

exports.handler = async (event, context) => {
  const message = JSON.parse(event.Records[0].Sns.Message)
  console.log('Status', message.detail.status)
  if (message.detail.status === 'COMPLETE') {
    try {
      const s3uri =
        message.detail.outputGroupDetails[1].outputDetails[0]
          .outputFilePaths[0] || undefined
      if (!s3uri) throw new Error('s3uri is undefined')
      const bucket = path.parse(s3uri).dir.split('/')[2]
      const tag = path.parse(s3uri).dir.split('/')[3]
      const { transcribe } = new AudioAnalysis({
        tag,
        mediaFileUri: s3uri,
        outputBucket: bucket,
        vocabularyFilterName: process.env.VOCABULARY_FILTER_NAME,
        outputKey: `${tag}/transcript/`,
        mediaFormat: 'mp3',
      })
      await transcribe.startJob()
    } catch (error) {
      if (error.name === "ConflictException") {
        console.info(error)
        return;
      }
      console.error(error)
    }
  }
  return message
}

@kylekirkby
Copy link
Author

Arhhh! Thanks @nathanagez! I've just noticed inputDetails in the detail section..

  detail: {
    timestamp: 1633534485958,
    accountId: '120662376178',
    queue: 'arn:aws:mediaconvert:us-east-1:120662376178:queues/Default',
    jobId: 'xxxxxxx,
    status: 'INPUT_INFORMATION',
    userMetadata: { videoId: '953942b4-7cd8-44a4-bfcb-ed88e7c4d31c' },
    inputDetails: [ [Object] ]
  }

@kylekirkby
Copy link
Author

@nathanagez - on another note, I'm not seeing outputGroupDetails in my SNS message?

@kylekirkby
Copy link
Author

@nathanagez - I'm also transcribing my videos but I'm using the input mp4 video file. Are you using the audio-only version that MediaConvert creates?

@nathanagez
Copy link
Collaborator

@nathanagez - on another note, I'm not seeing outputGroupDetails in my SNS message?

I think it depends of your MediaConvert job template, I'm using a custom template where I have output groups 🤔

@kylekirkby
Copy link
Author

@nathanagez - on another note, I'm not seeing outputGroupDetails in my SNS message?

I think it depends of your MediaConvert job template, I'm using a custom template where I have output groups 🤔

Hmmm.. strange! I'm also using a custom template (based on default one) with output groups 🤔. I've added an output group for poster images

      SettingsJson:
        OutputGroups:
          - Name: Apple HLS
            Outputs:
              - ContainerSettings:
                  Container: M3U8
                  M3u8Settings:
                    AudioFramesPerPes: 4
                    PcrControl: PCR_EVERY_PES_PACKET
                    PmtPid: 480
                    PrivateMetadataPid: 503
                    ProgramNumber: 1
                    PatInterval: 0
                    PmtInterval: 0
                    Scte35Source: NONE
                    NielsenId3: NONE
                    TimedMetadata: NONE
                    VideoPid: 481
                    AudioPids:
                      - 482
                      - 483
                      - 484
                      - 485
                      - 486
                      - 487
                      - 488
                      - 489
                      - 490
                      - 491
                      - 492
                VideoDescription:
                  ScalingBehavior: DEFAULT
                  TimecodeInsertion: DISABLED
                  AntiAlias: ENABLED
                  Sharpness: 50
                  CodecSettings:
                    Codec: H_264
                    H264Settings:
                      InterlaceMode: PROGRESSIVE
                      NumberReferenceFrames: 3
                      Syntax: DEFAULT
                      Softness: 0
                      FramerateDenominator: 1
                      GopClosedCadence: 1
                      GopSize: 90
                      Slices: 1
                      GopBReference: DISABLED
                      HrdBufferSize: 4500000
                      MaxBitrate: 3000000
                      SlowPal: DISABLED
                      SpatialAdaptiveQuantization: ENABLED
                      TemporalAdaptiveQuantization: ENABLED
                      FlickerAdaptiveQuantization: DISABLED
                      EntropyEncoding: CABAC
                      FramerateControl: SPECIFIED
                      RateControlMode: QVBR
                      QvbrSettings:
                        QvbrQualityLevel: 9
                        QvbrQualityLevelFineTune: 0
                      CodecProfile: MAIN
                      Telecine: NONE
                      FramerateNumerator: 30
                      MinIInterval: 0
                      AdaptiveQuantization: HIGH
                      CodecLevel: AUTO
                      FieldEncoding: PAFF
                      SceneChangeDetect: ENABLED
                      QualityTuningLevel: SINGLE_PASS
                      FramerateConversionAlgorithm: DUPLICATE_DROP
                      UnregisteredSeiTimecode: DISABLED
                      GopSizeUnits: FRAMES
                      ParControl: INITIALIZE_FROM_SOURCE
                      NumberBFramesBetweenReferenceFrames: 1
                      RepeatPps: DISABLED
                      DynamicSubGop: STATIC
                  AfdSignaling: NONE
                  DropFrameTimecode: ENABLED
                  RespondToAfd: NONE
                  ColorMetadata: INSERT
                AudioDescriptions:
                  - AudioTypeControl: FOLLOW_INPUT
                    AudioSourceName: Audio Selector 1
                    CodecSettings:
                      Codec: AAC
                      AacSettings:
                        AudioDescriptionBroadcasterMix: NORMAL
                        Bitrate: 96000
                        RateControlMode: CBR
                        CodecProfile: LC
                        CodingMode: CODING_MODE_2_0
                        RawFormat: NONE
                        SampleRate: 48000
                        Specification: MPEG4
                    LanguageCodeControl: FOLLOW_INPUT
                OutputSettings:
                  HlsSettings:
                    AudioGroupId: program_audio
                    AudioOnlyContainer: AUTOMATIC
                    IFrameOnlyManifest: EXCLUDE
                    SegmentModifier: $dt$
                NameModifier: _3000
              - ContainerSettings:
                  Container: M3U8
                  M3u8Settings:
                    AudioFramesPerPes: 4
                    PcrControl: PCR_EVERY_PES_PACKET
                    PmtPid: 480
                    PrivateMetadataPid: 503
                    ProgramNumber: 1
                    PatInterval: 0
                    PmtInterval: 0
                    Scte35Source: NONE
                    NielsenId3: NONE
                    TimedMetadata: NONE
                    TimedMetadataPid: 502
                    VideoPid: 481
                    AudioPids:
                      - 482
                      - 483
                      - 484
                      - 485
                      - 486
                      - 487
                      - 488
                      - 489
                      - 490
                      - 491
                      - 492
                VideoDescription:
                  Width: 960
                  ScalingBehavior: DEFAULT
                  Height: 540
                  TimecodeInsertion: DISABLED
                  AntiAlias: ENABLED
                  Sharpness: 50
                  CodecSettings:
                    Codec: H_264
                    H264Settings:
                      InterlaceMode: PROGRESSIVE
                      NumberReferenceFrames: 3
                      Syntax: DEFAULT
                      Softness: 0
                      FramerateDenominator: 1
                      GopClosedCadence: 1
                      GopSize: 90
                      Slices: 1
                      GopBReference: DISABLED
                      HrdBufferSize: 2250000
                      MaxBitrate: 1500000
                      SlowPal: DISABLED
                      SpatialAdaptiveQuantization: ENABLED
                      TemporalAdaptiveQuantization: ENABLED
                      FlickerAdaptiveQuantization: DISABLED
                      EntropyEncoding: CABAC
                      FramerateControl: SPECIFIED
                      RateControlMode: QVBR
                      QvbrSettings:
                        QvbrQualityLevel: 8
                        QvbrQualityLevelFineTune: 0
                      CodecProfile: MAIN
                      Telecine: NONE
                      FramerateNumerator: 30
                      MinIInterval: 0
                      AdaptiveQuantization: HIGH
                      CodecLevel: AUTO
                      FieldEncoding: PAFF
                      SceneChangeDetect: ENABLED
                      QualityTuningLevel: SINGLE_PASS
                      FramerateConversionAlgorithm: DUPLICATE_DROP
                      UnregisteredSeiTimecode: DISABLED
                      GopSizeUnits: FRAMES
                      ParControl: INITIALIZE_FROM_SOURCE
                      NumberBFramesBetweenReferenceFrames: 1
                      RepeatPps: DISABLED
                      DynamicSubGop: STATIC
                  AfdSignaling: NONE
                  DropFrameTimecode: ENABLED
                  RespondToAfd: NONE
                  ColorMetadata: INSERT
                AudioDescriptions:
                  - AudioTypeControl: FOLLOW_INPUT
                    AudioSourceName: Audio Selector 1
                    CodecSettings:
                      Codec: AAC
                      AacSettings:
                        AudioDescriptionBroadcasterMix: NORMAL
                        Bitrate: 96000
                        RateControlMode: CBR
                        CodecProfile: LC
                        CodingMode: CODING_MODE_2_0
                        RawFormat: NONE
                        SampleRate: 48000
                        Specification: MPEG4
                    LanguageCodeControl: FOLLOW_INPUT
                OutputSettings:
                  HlsSettings:
                    AudioGroupId: program_audio
                    AudioOnlyContainer: AUTOMATIC
                    IFrameOnlyManifest: EXCLUDE
                    SegmentModifier: $dt$
                NameModifier: _1500
              - ContainerSettings:
                  Container: M3U8
                  M3u8Settings:
                    AudioFramesPerPes: 4
                    PcrControl: PCR_EVERY_PES_PACKET
                    PmtPid: 480
                    PrivateMetadataPid: 503
                    ProgramNumber: 1
                    PatInterval: 0
                    PmtInterval: 0
                    Scte35Source: NONE
                    NielsenId3: NONE
                    TimedMetadata: NONE
                    TimedMetadataPid: 502
                    VideoPid: 481
                    AudioPids:
                      - 482
                      - 483
                      - 484
                      - 485
                      - 486
                      - 487
                      - 488
                      - 489
                      - 490
                      - 491
                      - 492
                VideoDescription:
                  Width: 768
                  ScalingBehavior: DEFAULT
                  Height: 432
                  TimecodeInsertion: DISABLED
                  AntiAlias: ENABLED
                  Sharpness: 50
                  CodecSettings:
                    Codec: H_264
                    H264Settings:
                      InterlaceMode: PROGRESSIVE
                      NumberReferenceFrames: 3
                      Syntax: DEFAULT
                      Softness: 0
                      FramerateDenominator: 1
                      GopClosedCadence: 1
                      GopSize: 90
                      Slices: 1
                      GopBReference: DISABLED
                      HrdBufferSize: 1250000
                      MaxBitrate: 750000
                      SlowPal: DISABLED
                      SpatialAdaptiveQuantization: ENABLED
                      TemporalAdaptiveQuantization: ENABLED
                      FlickerAdaptiveQuantization: DISABLED
                      EntropyEncoding: CABAC
                      FramerateControl: SPECIFIED
                      RateControlMode: QVBR
                      QvbrSettings:
                        QvbrQualityLevel: 7
                        QvbrQualityLevelFineTune: 0
                      CodecProfile: MAIN
                      Telecine: NONE
                      FramerateNumerator: 30
                      MinIInterval: 0
                      AdaptiveQuantization: HIGH
                      CodecLevel: AUTO
                      FieldEncoding: PAFF
                      SceneChangeDetect: ENABLED
                      QualityTuningLevel: SINGLE_PASS
                      FramerateConversionAlgorithm: DUPLICATE_DROP
                      UnregisteredSeiTimecode: DISABLED
                      GopSizeUnits: FRAMES
                      ParControl: INITIALIZE_FROM_SOURCE
                      NumberBFramesBetweenReferenceFrames: 2
                      RepeatPps: DISABLED
                      DynamicSubGop: STATIC
                  AfdSignaling: NONE
                  DropFrameTimecode: ENABLED
                  RespondToAfd: NONE
                  ColorMetadata: INSERT
                AudioDescriptions:
                  - AudioTypeControl: FOLLOW_INPUT
                    AudioSourceName: Audio Selector 1
                    CodecSettings:
                      Codec: AAC
                      AacSettings:
                        AudioDescriptionBroadcasterMix: NORMAL
                        Bitrate: 96000
                        RateControlMode: CBR
                        CodecProfile: LC
                        CodingMode: CODING_MODE_2_0
                        RawFormat: NONE
                        SampleRate: 48000
                        Specification: MPEG4
                    LanguageCodeControl: FOLLOW_INPUT
                OutputSettings:
                  HlsSettings:
                    AudioGroupId: program_audio
                    AudioOnlyContainer: AUTOMATIC
                    IFrameOnlyManifest: EXCLUDE
                    SegmentModifier: $dt$
                NameModifier: _750
              - ContainerSettings:
                  Container: M3U8
                  M3u8Settings:
                    AudioFramesPerPes: 4
                    PcrControl: PCR_EVERY_PES_PACKET
                    PmtPid: 480
                    PrivateMetadataPid: 503
                    ProgramNumber: 1
                    PatInterval: 0
                    PmtInterval: 0
                    Scte35Source: NONE
                    NielsenId3: NONE
                    TimedMetadata: NONE
                    VideoPid: 481
                    AudioPids:
                      - 482
                      - 483
                      - 484
                      - 485
                      - 486
                      - 487
                      - 488
                      - 489
                      - 490
                      - 491
                      - 492
                VideoDescription:
                  Width: 640
                  ScalingBehavior: DEFAULT
                  Height: 360
                  TimecodeInsertion: DISABLED
                  AntiAlias: ENABLED
                  Sharpness: 50
                  CodecSettings:
                    Codec: H_264
                    H264Settings:
                      InterlaceMode: PROGRESSIVE
                      NumberReferenceFrames: 3
                      Syntax: DEFAULT
                      Softness: 0
                      FramerateDenominator: 1
                      GopClosedCadence: 1
                      GopSize: 90
                      Slices: 1
                      GopBReference: DISABLED
                      HrdBufferSize: 450000
                      MaxBitrate: 325000
                      SlowPal: DISABLED
                      SpatialAdaptiveQuantization: ENABLED
                      TemporalAdaptiveQuantization: ENABLED
                      FlickerAdaptiveQuantization: DISABLED
                      EntropyEncoding: CABAC
                      FramerateControl: SPECIFIED
                      RateControlMode: QVBR
                      QvbrSettings:
                        QvbrQualityLevel: 6
                        QvbrQualityLevelFineTune: 0
                      CodecProfile: MAIN
                      Telecine: NONE
                      FramerateNumerator: 30
                      MinIInterval: 0
                      AdaptiveQuantization: HIGH
                      CodecLevel: AUTO
                      FieldEncoding: PAFF
                      SceneChangeDetect: ENABLED
                      QualityTuningLevel: SINGLE_PASS
                      FramerateConversionAlgorithm: DUPLICATE_DROP
                      UnregisteredSeiTimecode: DISABLED
                      GopSizeUnits: FRAMES
                      ParControl: INITIALIZE_FROM_SOURCE
                      NumberBFramesBetweenReferenceFrames: 2
                      RepeatPps: DISABLED
                      DynamicSubGop: STATIC
                  AfdSignaling: NONE
                  DropFrameTimecode: ENABLED
                  RespondToAfd: NONE
                  ColorMetadata: INSERT
                AudioDescriptions:
                  - AudioTypeControl: FOLLOW_INPUT
                    AudioSourceName: Audio Selector 1
                    CodecSettings:
                      Codec: AAC
                      AacSettings:
                        AudioDescriptionBroadcasterMix: NORMAL
                        Bitrate: 96000
                        RateControlMode: CBR
                        CodecProfile: LC
                        CodingMode: CODING_MODE_2_0
                        RawFormat: NONE
                        SampleRate: 48000
                        Specification: MPEG4
                    LanguageCodeControl: FOLLOW_INPUT
                OutputSettings:
                  HlsSettings:
                    AudioGroupId: program_audio
                    AudioOnlyContainer: AUTOMATIC
                    IFrameOnlyManifest: EXCLUDE
                    SegmentModifier: $dt$
                NameModifier: _325
              - ContainerSettings:
                  Container: M3U8
                  M3u8Settings:
                    AudioFramesPerPes: 4
                    PcrControl: PCR_EVERY_PES_PACKET
                    PmtPid: 480
                    PrivateMetadataPid: 503
                    ProgramNumber: 1
                    PatInterval: 0
                    PmtInterval: 0
                    Scte35Source: NONE
                    NielsenId3: NONE
                    TimedMetadata: NONE
                    TimedMetadataPid: 502
                    VideoPid: 481
                    AudioPids:
                      - 482
                      - 483
                      - 484
                      - 485
                      - 486
                      - 487
                      - 488
                      - 489
                      - 490
                      - 491
                      - 492
                AudioDescriptions:
                  - AudioTypeControl: FOLLOW_INPUT
                    AudioSourceName: Audio Selector 1
                    CodecSettings:
                      Codec: AAC
                      AacSettings:
                        AudioDescriptionBroadcasterMix: NORMAL
                        Bitrate: 96000
                        RateControlMode: CBR
                        CodecProfile: LC
                        CodingMode: CODING_MODE_2_0
                        RawFormat: NONE
                        SampleRate: 48000
                        Specification: MPEG4
                    LanguageCodeControl: FOLLOW_INPUT
                OutputSettings:
                  HlsSettings:
                    AudioGroupId: program_audio
                    AudioOnlyContainer: AUTOMATIC
                    IFrameOnlyManifest: EXCLUDE
                    SegmentModifier: $dt$
                NameModifier: _audio
            OutputGroupSettings:
              Type: HLS_GROUP_SETTINGS
              HlsGroupSettings:
                ManifestDurationFormat: INTEGER
                SegmentLength: 10
                TimedMetadataId3Period: 10
                CaptionLanguageSetting: OMIT
                TimedMetadataId3Frame: PRIV
                CodecSpecification: RFC_4281
                OutputSelection: MANIFESTS_AND_SEGMENTS
                ProgramDateTimePeriod: 600
                MinSegmentLength: 0
                MinFinalSegmentLength: 0
                DirectoryStructure: SINGLE_DIRECTORY
                ProgramDateTime: EXCLUDE
                SegmentControl: SEGMENTED_FILES
                ManifestCompression: NONE
                ClientCache: ENABLED
                StreamInfResolution: INCLUDE
          - Name: File Group
            CustomName: Poster Images
            Outputs:
              - ContainerSettings:
                  Container: RAW
                VideoDescription:
                  CodecSettings:
                    Codec: FRAME_CAPTURE
                    FrameCaptureSettings:
                      FramerateNumerator: 30
                      FramerateDenominator: 150
                      MaxCaptures: 3
                      Quality: 80
                Extension: .jpg
                NameModifier: poster-image
            OutputGroupSettings:
              Type: FILE_GROUP_SETTINGS
        AdAvailOffset: 0

@kylekirkby
Copy link
Author

@nathanagez - outputGroupDetails only exists when status is set to "COMPLETE" 👍🏼

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants