Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

concat.py: problems iterating over input directory #239

Closed
mcampos-quinn opened this issue Feb 16, 2018 · 8 comments
Closed

concat.py: problems iterating over input directory #239

mcampos-quinn opened this issue Feb 16, 2018 · 8 comments

Comments

@mcampos-quinn
Copy link
Contributor

Hey-o

I was playing around with concat.py and had to make some mods to concat.py and ififuncs to get it to play nice. I'm testing it on Ubuntu 16.04 fwiw.

Basically the input arg -i produces a list (due to nargs=+) that contains a single item, the directory path to process. But that path doesn't get iterated over and the rest of the functions try to operate on the directory path rather than the files under that path. For example get_milliseconds returns an integer that is the duration of all the files concatenated together as a single string and the timedelta that is used to get chapter markers is like several days.

Here's a simple set of loops I used to list the files from the input dir:

    # GRAB THE DIRECTORY THAT IS THE INPUT GIVEN IN CL ARGS
    source_dir = os.path.abspath(video_files)
    source_list = []
    # ITERATE OVER THE DIR (ONE LEVEL) AND ADD THE FILE ABSOLUTE PATHS TO A LIST
    for _file in os.listdir(source_dir):
        source_list.append(os.path.join(source_dir,_file))
    source_list.sort()
    video_files = os.listdir(video_files)
    video_files = ififuncs.sanitise_filenames(video_files)
    for source_files in video_files:
        ififuncs.generate_log(
            log_name_source,
            'source_files = %s' % source_files)
    # MAKE CHAPTERS BASED ON THE SOURCE_LIST
    make_chapters(source_list)

I don't know how actively you're using this but I'd be curious to know how you do. If your setup doesn't give you the same error, I would wonder what the difference is. I think we're going to try 'borrowing' your functionality....

This is unrelated but I also have not had the validation check pass in any of my tests... I'll look at the concat/mkvpropedit commands next.

Cheers!
@kieranjol
Copy link
Owner

kieranjol commented Feb 16, 2018 via email

@kieranjol
Copy link
Owner

kieranjol commented Feb 16, 2018 via email

@mcampos-quinn
Copy link
Contributor Author

Yeah! It's the end of Friday before a three day weekend here in California, so... enjoy your weekend!

I left the recursive option alone since that seems to want multiple subdirectories. I'll look again at it next week.

@kieranjol
Copy link
Owner

kieranjol commented Feb 17, 2018 via email

@kieranjol
Copy link
Owner

kieranjol commented Feb 17, 2018 •
edited
Loading

Hey, so I've been able to take a look at this. Replies below:

Hey-o

I was playing around with concat.py and had to make some mods to concat.py and ififuncs to get it to play nice. I'm testing it on Ubuntu 16.04 fwiw.

I'm curious to see what kinds of changes - also I write a lot of code on Ubuntu 16.04, OSX and Windows 7. The scripts usually get run on OSX and Windows though.

Basically the input arg -i produces a list (due to nargs=+) that contains a single item, the directory path to process. But that path doesn't get iterated over and the rest of the functions try to operate on the directory path rather than the files under that path. For example get_milliseconds returns an integer that is the duration of all the files concatenated together as a single string and the timedelta that is used to get chapter markers is like several days.

Ya. So I should make this more clear in the documentation. If you do not use -r, then -i will iterate over several file inputs, for example this worked fine for me : concat.py -i /home/kieran/bishop_berkeley/test_files/bars.mov /home/kieran/bishop_berkeley/test_files/mandel.mov -o /home/kieran110s/sips

So I need to add a catch in the script that when -r is not used, then multiple inputs must be present, and they must all be files.

Here's a simple set of loops I used to list the files from the input dir:

# GRAB THE DIRECTORY THAT IS THE INPUT GIVEN IN CL ARGS
source_dir = os.path.abspath(video_files)
source_list = []
# ITERATE OVER THE DIR (ONE LEVEL) AND ADD THE FILE ABSOLUTE PATHS TO A LIST
for _file in os.listdir(source_dir):
    source_list.append(os.path.join(source_dir,_file))
source_list.sort()
video_files = os.listdir(video_files)
video_files = ififuncs.sanitise_filenames(video_files)
for source_files in video_files:
    ififuncs.generate_log(
        log_name_source,
        'source_files = %s' % source_files)
# MAKE CHAPTERS BASED ON THE SOURCE_LIST
make_chapters(source_list)

I don't know how actively you're using this but I'd be curious to know how you do. If your setup doesn't give you the same error, I would wonder what the difference is. I think we're going to try 'borrowing' your functionality....

We've used it several hundred times in order to make more accessible versions of some XDCAM packages. We still retain the original XDCAMs of course. So yeah,we usually just do
concat.py -i /path/to/xdcam_folder -r -o /path/to/output but sometimes, the filenaming would not process in the correct order or something weird might be happening, so we'd instead do:
concat.py -i path/to/xdcam_video1.mp4 path/to/xdcam_video2.mp4 -o path/to/output but we rarely had to do that.

And yes, please borrow/reuse whatever,and let me know what improvements you make and what issues you find! This script is clearly quite hardcoded to our workflows as it expects a package that was generated by sipcreator.py and it is looking for users,Object Entry identifiers and UUIDs and such, but it should be very easy to remove that kind of stuff.

This is unrelated but I also have not had the validation check pass in any of my tests... I'll look at the concat/mkvpropedit commands next.

Is that the losslessness check? Yeah, that never worked well. I think I might have raised some ffmpeg issues about this that never went anywhere. I was initially doing framemd5 tests,and I could see that the frames in my output concat were indeed identical to the original files.However, some of our xdcam files had some weird timestamps where maybe the very first frame (usually colour bars) would not end up in the outputted file. So I'm not happy about this,but in the grand scheme of things, it's a compromise that I'm willing to make. And I'm logging that this is not a 100% lossless process..

Cheers!

Thank you! Hopefully this clarified things.

@kieranjol
Copy link
Owner

I saw a mistake there, but just in case the edit won't show up in your email:

We've used it several hundred times in order to make more accessible versions of some XDCAM packages. We still retain the original XDCAMs of course. So yeah,we usually just do
concat.py -i /path/to/xdcam_folder -r -o /path/to/output but sometimes, the filenaming would not process in the correct order or something weird might be happening, so we'd instead do:
concat.py -i path/to/xdcam_video1.mp4 path/to/xdcam_video2.mp4 -o path/to/output but we rarely had to do that.

@kieranjol
Copy link
Owner

Btw, when using -r, files don't need to be in subfolders. You could just have a bunch of files in your input folder and the script will process them. Or you can have files in subfolders, the recursive os.walk will find the AV files one way or another.

@mcampos-quinn
Copy link
Contributor Author

Cool thanks! Maybe I ran into some of the hard coded stuff the script expects to find with the recursive option. Anyway thanks for the clarifications and I think we have some use cases for this. I’m working on pulling out some of what would be useful to us!

I haven’t dug deep enough yet to troubleshoot why the losslessness verification failed on my test files, but it might be moot for how we may use your functionality. We’ll see.

Thanks a ton for your replies!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kieranjol @mcampos-quinn