Minor changes for 4.1 in tub conversion script and developer doc #708
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
DocGarbanzo
commented
Dec 22, 2020
DocGarbanzo
commented
- Update conversion script to translate discontinuous data.
- In developer guide add disclaimer for version and correct intra-page links.
* Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links.
Codecov Report
@@ Coverage Diff @@
## dev #708 +/- ##
=======================================
Coverage 28.66% 28.66%
=======================================
Files 61 61
Lines 7683 7683
Branches 805 805
=======================================
Hits 2202 2202
Misses 5361 5361
Partials 120 120
Continue to review full report at Codecov.
|
tikurahul
requested changes
Dec 23, 2020
scripts/convert_to_tub_v2.py
Outdated
|
|
||
| record_paths = legacy_tub.gather_records() | ||
| bar = IncrementalBar('Converting', max=len(record_paths)) | ||
|
|
||
| prev_rec_num = None |
There was a problem hiding this comment.
Nit: Call this previous_index.
scripts/convert_to_tub_v2.py
Outdated
| prev_rec_num += 1 | ||
| while prev_rec_num < rec_num: | ||
| idx = output_tub.manifest.current_index | ||
| output_tub.write_record({}) |
There was a problem hiding this comment.
Define a dict EMPTY which looks something like:
{'__empty__': true}Write that for every deleted record. That way the intention here is very clear when reading though the file.
There was a problem hiding this comment.
Good point. I first thought we might want to support such field in the tub itself, but not sure... I 've updated the script to handle this.
* Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback
* Add empty records concept to tub * Minor updates to conversion script
* Add empty record type into conversion script
tikurahul
approved these changes
Dec 23, 2020
tikurahul
added a commit
that referenced
this pull request
Jan 4, 2021
* Update nano install docs - python3-opencv vs libopencv-python * change slack to discord on readme * Donkeycar 4.x release. (#644) ## Major Improvements - New Datastore. - More ways to pre-process image data for training. - Use the 2.x version of Tensorflow. - Lots of other minor improvements. * Improvements to the car app and handling of KerasPilot parts: (#648) * Improvements to the car app and handling of KerasPilot parts: * Created a simpler webserver or joystick car app by modifying and renaming the basic_web into basic template and switched this on as default. Also renamed the target from 'manage.py' to 'drive.py' because there is only driving and no training included * Simplified the handling of uint8 and float32 numpy image arrays. KerasPilot.run() now expects uint8 data, transforms into float32 and delegates to its children. The corresponding rescaling step is gone from the car app * smaller updates/fixes to environment and config files * added support for tflite pilots in makemovie * Improvements to the car app and handling of KerasPilot parts: * Added docstring to KerasPilot methods * Fixed bad merge for model types in complete config * Add configuration to allow creating sub directories for each tub (like the legacy tub) or by default store all recordings in the data directory directly. (#649) * Update setup.py with progress module for pi (#650) * Update setup.py * Update setup.py Made changes online. * Add testing for training: (#651) * added example tub data as tar.gz in tests with 1000 records * check validation data size in train * created new test_train.py * added support for read-only tub, required to create r/o tub in tmp dir * changed loss in categorical model to equally weight throttle and steering * changed mod on scripts to u+x * Rename `drive.py` to `manage.py` to preserve CLI compat. (#656) * Train still prompts you to move to then new entry point. * Also fixes #655 * Cleanup exit handlers in the new datastore. (#657) * Minor datastore_v2 improvements. (#658) * Add __exit__ handlers again. * Use `os.linesep` to deal with line separators consistently. * Be explicit about newline characters (#660) * Be explicit about newline characters #2 (#661) * Remove the extra flush(). Windows is less forgiving (#662) * Remove redundant code (#663) * Windows Doc Updates (#668) Added more options for Installing Donkey Car on Windows. - Anaconda - Native - Windows Subsystem for Linux (WSL) - experiential These options give more flexibility and could make deployment easier for some individuals. I personally always install donkey car natively onto Python installed on the System. I will be looking more into WSL going forward as it has some interesting benefits. * Integrate osx into travis. (#665) * add in --user flag for MakeMovie so that you can select if you want to draw the line on the video or not. default: true * updated args.draw_user_input * update to include recommendations * added in brake functionality for simulator only fix typo in drivemode return which checked pilot/throttle instead of brake * fix autorope/issue #671 (#672) * Jonathans changes from issue #634 and PR #646 (#676) * Bump version to 4.0.1 (#677) * Fix template docopt, as it's called managed.py and not drive.py * Allow overriding WEB_CONTROL_PORT from env variable * - Add simulator support to basic.py (#682) - Update cfg_basic.py with simulator parameters * Switch to using memory-mapped files when reading. (#691) * This makes reading a 100x faster. * new: mqtt telemetry support (#688) * A configurable training pipeline. (#693) * Implement a Lazy transformable pipeline. * Implement basic batching. However, this will need to be improved further for models with multiple outputs. * Replace the old `Sequence` implementation used with a new `Pipeline`. Test: Ran end to end tests. * The telemetry mqtt test is breaking w/ a timeout on the connection to the server. * change the public mqtt server name to one that's listening * * Overriding inference in TensorRT as pass as it is not needed here. (#698) * Need to implement inference to run on the normalised image (#699) * Fix sequence iterators (#704) * Change training pipeline from tf.Sequence to tf.data (#701) * Improve pipeline use: move from building list of pipelines of single transforms to building a single pipeline with a list of transforms (actually just looping through function to go from TubRecord -> image -> augment -> normalise -> x and TubRecord -> y). Fixed TfmIterators and TfmIterables. * Iterables are the containers and are sized - these are the user objects * Iterators are protocol objects to allow iteration, they have no logic and are local to the Iterables * build/map_pipeline both return sized Iterables * removed all batch logic, this is not required * still commented but left code that uses generator based pipeline as this is simpler code Using new small temporary pipeline generator * this keeps the TubSequence lazy and avoids to roll out the pipeline into a list * added a test to check consistency of the pipeline * remove empty (after moved) augmentation file * removed augmentation from old tub (as it's not needed and we removed the old augmentation) New pipeline changes: * moved augmentation into own class that is used above and can be used a threaded or non-threaded part * moved train functionality out of template and added 'donkey train', train.py just a simple dummy script for backward compatibility * Address code reviews: * Re-base on current dev to use un-altered sequence.py * Add iterator consistency test to pipeline tests * Undo changes in fast_stretch.py * better tf shape manipulation * small code improvements in training.py * remove sleep in augment part * Address code reviews: * Add clearing of tubrecord list and minor renamings * * Add support for multi-dimensional model input and making the x, y interface symmetrical on the model / training interface: (#707) - x/y_transform extract x, y as numpy arrays or floats out of the record - x/y_translate convert the numpy arrays of floats into tf-readable dictionaries used in tf data. * Simplify model interface by implementing output_types() directly in the base class using output_shapes() dictionary. * Adding developer guide for own model development * Updated donkey command documentation * Improve asserts and type hints in keras.py * Added missing __init__.py in parts module. * Add cool ascii text for donkey init and update yml and setup files including mypy * Remove model training test from Travis and change the test to relative convergence. This avoids random fall overs in CI. * Added test of tf.data as used in the training pipeline through re-implementation of data transformation from tub records to tf expected dictionaries, for all currently supported models. * Minor changes for 4.1 in tub conversion script and developer doc (#708) * Minor changes for 4.1 * Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links. * * Update doc with donkey train command. * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Incorporate PR feedback * Add empty records concept to tub * Minor updates to conversion script * Incorporate PR feedback * Add empty record type into conversion script * docs: fix simple typo, unfarmilar -> unfamiliar (#714) There is a small typo in docs/guide/host_pc/setup_windows.md. Should read `unfamiliar` rather than `unfarmilar`. * Added the ability to train PyTorch models (#706) * Added in PyTorch and PyTorch Lightning to train a DC model Successfully able to train a ResNet18-based model using PyTorch Lightning. * Removed hard-coded max number of epochs (used for debugging) * Added an inference transform to ResNet18 to convert PIL -> tensor * Unsqueezed input tensor during inference for batch dimension * Reshaped ResNet output from (1, 2) to (2,) * Added the ability to resume training from a checkpoint * Added helper print message when tensorboard logging is enabled * Updated docopt arguments for train.py. Made checkpoint optional * Changed TorchTubDataset from sub-classing Dataset to IterableDataset This was done in response to #706 (comment) * Renamed load_image_arr to load_image. Updated load_pil_image load_pil_image will now handle converting the image to greyscale (vs. this being done in load_image). * Updated enviroments for Mac and Ubuntu. Set Python=3.7 * Updated installation documentation. Added script to setup Nano Updated the installation instructions for Ubuntu, Mac, and Windows. Clarified a common issue that occurs when running pip install -e .[pc] with ZSH. Also added a script to setup the Jetson Nano and updated the documentation for the Nano (it previously was installing tensorflow 1.x). * Added torch flag to setup.py to install pytorch * Moved pytorch training into base.py and removed from train.py * Moved Jetson Nano python package installation into requirements.txt * Formatted with PEP8 to clean up pytorch code * Updated docs to provide work-around for ZSH pip install -e .[pc] * Removed duplicate dependencies in conda env files * ResNet18 torch model now returns training loss history * Added test file for PyTorch training Still need to make sure this passes Travis CI. * Added lightning_logs to .gitignore * You can now specify the default AI framework to use in config.py This reduces the number of command line arguments you are required to provide. * get_model_by_type for PyTorch now lazy imports ResNet18 * Added help message to torch_train. Got rid of linear model type * Updated pytorch tests and fixed some syntax errors * ResNet18 example input shape updated to be (B, 3, 224, 224) Also now passing output_shape to load_resnet18 to modify how many output classes are used * No longer pinning requirement versions for Jetson Nano * Fixed formatting in setup.py * update configuration so that IMAGE_H and IMAGE_W are passed through to the simulator (#674) Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: sctse999 <jonathan@tsez.net> Co-authored-by: DGarbanzo <47540921+DocGarbanzo@users.noreply.github.com> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: DocGarbanzo <dirk.prange@web.de> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com>
tikurahul
added a commit
that referenced
this pull request
Mar 20, 2021
* - Update tensorflow version to 1.15.0 - Add doc for Jetpack 4.4 - Remove opencv installation * Add dataset and pre-trained model page * fix issue #634 (#646) - remove jquery dependency - add clearTimeout * Allow overriding WEB_CONTROL_PORT from env variable * pin h5py version * fix h5py version for mac * * Update doc with donkey train command. (#709) * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Donkey 4.x is now official (#723) * Update nano install docs - python3-opencv vs libopencv-python * change slack to discord on readme * Donkeycar 4.x release. (#644) ## Major Improvements - New Datastore. - More ways to pre-process image data for training. - Use the 2.x version of Tensorflow. - Lots of other minor improvements. * Improvements to the car app and handling of KerasPilot parts: (#648) * Improvements to the car app and handling of KerasPilot parts: * Created a simpler webserver or joystick car app by modifying and renaming the basic_web into basic template and switched this on as default. Also renamed the target from 'manage.py' to 'drive.py' because there is only driving and no training included * Simplified the handling of uint8 and float32 numpy image arrays. KerasPilot.run() now expects uint8 data, transforms into float32 and delegates to its children. The corresponding rescaling step is gone from the car app * smaller updates/fixes to environment and config files * added support for tflite pilots in makemovie * Improvements to the car app and handling of KerasPilot parts: * Added docstring to KerasPilot methods * Fixed bad merge for model types in complete config * Add configuration to allow creating sub directories for each tub (like the legacy tub) or by default store all recordings in the data directory directly. (#649) * Update setup.py with progress module for pi (#650) * Update setup.py * Update setup.py Made changes online. * Add testing for training: (#651) * added example tub data as tar.gz in tests with 1000 records * check validation data size in train * created new test_train.py * added support for read-only tub, required to create r/o tub in tmp dir * changed loss in categorical model to equally weight throttle and steering * changed mod on scripts to u+x * Rename `drive.py` to `manage.py` to preserve CLI compat. (#656) * Train still prompts you to move to then new entry point. * Also fixes #655 * Cleanup exit handlers in the new datastore. (#657) * Minor datastore_v2 improvements. (#658) * Add __exit__ handlers again. * Use `os.linesep` to deal with line separators consistently. * Be explicit about newline characters (#660) * Be explicit about newline characters #2 (#661) * Remove the extra flush(). Windows is less forgiving (#662) * Remove redundant code (#663) * Windows Doc Updates (#668) Added more options for Installing Donkey Car on Windows. - Anaconda - Native - Windows Subsystem for Linux (WSL) - experiential These options give more flexibility and could make deployment easier for some individuals. I personally always install donkey car natively onto Python installed on the System. I will be looking more into WSL going forward as it has some interesting benefits. * Integrate osx into travis. (#665) * add in --user flag for MakeMovie so that you can select if you want to draw the line on the video or not. default: true * updated args.draw_user_input * update to include recommendations * added in brake functionality for simulator only fix typo in drivemode return which checked pilot/throttle instead of brake * fix autorope/issue #671 (#672) * Jonathans changes from issue #634 and PR #646 (#676) * Bump version to 4.0.1 (#677) * Fix template docopt, as it's called managed.py and not drive.py * Allow overriding WEB_CONTROL_PORT from env variable * - Add simulator support to basic.py (#682) - Update cfg_basic.py with simulator parameters * Switch to using memory-mapped files when reading. (#691) * This makes reading a 100x faster. * new: mqtt telemetry support (#688) * A configurable training pipeline. (#693) * Implement a Lazy transformable pipeline. * Implement basic batching. However, this will need to be improved further for models with multiple outputs. * Replace the old `Sequence` implementation used with a new `Pipeline`. Test: Ran end to end tests. * The telemetry mqtt test is breaking w/ a timeout on the connection to the server. * change the public mqtt server name to one that's listening * * Overriding inference in TensorRT as pass as it is not needed here. (#698) * Need to implement inference to run on the normalised image (#699) * Fix sequence iterators (#704) * Change training pipeline from tf.Sequence to tf.data (#701) * Improve pipeline use: move from building list of pipelines of single transforms to building a single pipeline with a list of transforms (actually just looping through function to go from TubRecord -> image -> augment -> normalise -> x and TubRecord -> y). Fixed TfmIterators and TfmIterables. * Iterables are the containers and are sized - these are the user objects * Iterators are protocol objects to allow iteration, they have no logic and are local to the Iterables * build/map_pipeline both return sized Iterables * removed all batch logic, this is not required * still commented but left code that uses generator based pipeline as this is simpler code Using new small temporary pipeline generator * this keeps the TubSequence lazy and avoids to roll out the pipeline into a list * added a test to check consistency of the pipeline * remove empty (after moved) augmentation file * removed augmentation from old tub (as it's not needed and we removed the old augmentation) New pipeline changes: * moved augmentation into own class that is used above and can be used a threaded or non-threaded part * moved train functionality out of template and added 'donkey train', train.py just a simple dummy script for backward compatibility * Address code reviews: * Re-base on current dev to use un-altered sequence.py * Add iterator consistency test to pipeline tests * Undo changes in fast_stretch.py * better tf shape manipulation * small code improvements in training.py * remove sleep in augment part * Address code reviews: * Add clearing of tubrecord list and minor renamings * * Add support for multi-dimensional model input and making the x, y interface symmetrical on the model / training interface: (#707) - x/y_transform extract x, y as numpy arrays or floats out of the record - x/y_translate convert the numpy arrays of floats into tf-readable dictionaries used in tf data. * Simplify model interface by implementing output_types() directly in the base class using output_shapes() dictionary. * Adding developer guide for own model development * Updated donkey command documentation * Improve asserts and type hints in keras.py * Added missing __init__.py in parts module. * Add cool ascii text for donkey init and update yml and setup files including mypy * Remove model training test from Travis and change the test to relative convergence. This avoids random fall overs in CI. * Added test of tf.data as used in the training pipeline through re-implementation of data transformation from tub records to tf expected dictionaries, for all currently supported models. * Minor changes for 4.1 in tub conversion script and developer doc (#708) * Minor changes for 4.1 * Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links. * * Update doc with donkey train command. * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Incorporate PR feedback * Add empty records concept to tub * Minor updates to conversion script * Incorporate PR feedback * Add empty record type into conversion script * docs: fix simple typo, unfarmilar -> unfamiliar (#714) There is a small typo in docs/guide/host_pc/setup_windows.md. Should read `unfamiliar` rather than `unfarmilar`. * Added the ability to train PyTorch models (#706) * Added in PyTorch and PyTorch Lightning to train a DC model Successfully able to train a ResNet18-based model using PyTorch Lightning. * Removed hard-coded max number of epochs (used for debugging) * Added an inference transform to ResNet18 to convert PIL -> tensor * Unsqueezed input tensor during inference for batch dimension * Reshaped ResNet output from (1, 2) to (2,) * Added the ability to resume training from a checkpoint * Added helper print message when tensorboard logging is enabled * Updated docopt arguments for train.py. Made checkpoint optional * Changed TorchTubDataset from sub-classing Dataset to IterableDataset This was done in response to #706 (comment) * Renamed load_image_arr to load_image. Updated load_pil_image load_pil_image will now handle converting the image to greyscale (vs. this being done in load_image). * Updated enviroments for Mac and Ubuntu. Set Python=3.7 * Updated installation documentation. Added script to setup Nano Updated the installation instructions for Ubuntu, Mac, and Windows. Clarified a common issue that occurs when running pip install -e .[pc] with ZSH. Also added a script to setup the Jetson Nano and updated the documentation for the Nano (it previously was installing tensorflow 1.x). * Added torch flag to setup.py to install pytorch * Moved pytorch training into base.py and removed from train.py * Moved Jetson Nano python package installation into requirements.txt * Formatted with PEP8 to clean up pytorch code * Updated docs to provide work-around for ZSH pip install -e .[pc] * Removed duplicate dependencies in conda env files * ResNet18 torch model now returns training loss history * Added test file for PyTorch training Still need to make sure this passes Travis CI. * Added lightning_logs to .gitignore * You can now specify the default AI framework to use in config.py This reduces the number of command line arguments you are required to provide. * get_model_by_type for PyTorch now lazy imports ResNet18 * Added help message to torch_train. Got rid of linear model type * Updated pytorch tests and fixed some syntax errors * ResNet18 example input shape updated to be (B, 3, 224, 224) Also now passing output_shape to load_resnet18 to modify how many output classes are used * No longer pinning requirement versions for Jetson Nano * Fixed formatting in setup.py * update configuration so that IMAGE_H and IMAGE_W are passed through to the simulator (#674) Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: sctse999 <jonathan@tsez.net> Co-authored-by: DGarbanzo <47540921+DocGarbanzo@users.noreply.github.com> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: DocGarbanzo <dirk.prange@web.de> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com> * Updated path follower with Intel T265 (#729) Properly rebased on Master this time, I hope Co-authored-by: Chris Anderson <gca2@earthlink.net> * Remove '-dev' from version output (#734) * Updated training instructions. (#744) * Make clear how to call "run" vs "run_threaded" Derived from Tawn Kramer's excellent video https://www.youtube.com/watch?v=YZ4ESrtfShs * Discuss update function * Added Tawn's video * More explanation of inputs and outputs * Explain shutdown * typo * typo * Make clear that threaded parts can run faster * Remove part about running faster than main loop. It appears that the main loop block threaded parts from running faster * Replace Travis CI with GitHub Actions (#801) * Add github action files and remove travis files * Suppress train tests in github actions * Improvements in workflow, allow linter failure and don't stop build matrix on single os failure and add missing linter trigger on PRs * Replace Travis CI with GitHub Actions (#801) * Add missing pc.yml * Add "./" before shell script so it will run (#805) * Add "./" before shell script so it will run * Add automatic pushing of docs (#806) * Adding auto push donkey docs for changes to docs in master * Add missing checkout. * Fix heading in faq * Add force and verbose flag to mkdocs * Fix link to roll-your-own in faq.md * Fix minor typos in faq.md * Fixed import for tub writer * Revert unintended changes to code from merge from master. * Added "DRIVE_TRAIN" back to compete template, too (#809) Again, apologies to everyone whose installation has been failing. Blame Travis! * Added DRIVE_TRAIN back (#808) This was causing everyone's installations to fail Co-authored-by: Jonathan Tse <jonathan@tsez.net> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: Rahul Ravikumar <tikurahul@gmail.com> Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com> Co-authored-by: Chris Anderson <analoguedevices@gmail.com> Co-authored-by: Chris Anderson <gca2@earthlink.net>
DocGarbanzo
added a commit
to DocGarbanzo/donkeycar
that referenced
this pull request
Apr 20, 2021
…orope#708) * Minor changes for 4.1 * Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links. * * Update doc with donkey train command. * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Incorporate PR feedback * Add empty records concept to tub * Minor updates to conversion script * Incorporate PR feedback * Add empty record type into conversion script
DocGarbanzo
added a commit
to DocGarbanzo/donkeycar
that referenced
this pull request
Apr 20, 2021
* - Update tensorflow version to 1.15.0 - Add doc for Jetpack 4.4 - Remove opencv installation * Add dataset and pre-trained model page * fix issue autorope#634 (autorope#646) - remove jquery dependency - add clearTimeout * Allow overriding WEB_CONTROL_PORT from env variable * pin h5py version * fix h5py version for mac * * Update doc with donkey train command. (autorope#709) * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Donkey 4.x is now official (autorope#723) * Update nano install docs - python3-opencv vs libopencv-python * change slack to discord on readme * Donkeycar 4.x release. (autorope#644) ## Major Improvements - New Datastore. - More ways to pre-process image data for training. - Use the 2.x version of Tensorflow. - Lots of other minor improvements. * Improvements to the car app and handling of KerasPilot parts: (autorope#648) * Improvements to the car app and handling of KerasPilot parts: * Created a simpler webserver or joystick car app by modifying and renaming the basic_web into basic template and switched this on as default. Also renamed the target from 'manage.py' to 'drive.py' because there is only driving and no training included * Simplified the handling of uint8 and float32 numpy image arrays. KerasPilot.run() now expects uint8 data, transforms into float32 and delegates to its children. The corresponding rescaling step is gone from the car app * smaller updates/fixes to environment and config files * added support for tflite pilots in makemovie * Improvements to the car app and handling of KerasPilot parts: * Added docstring to KerasPilot methods * Fixed bad merge for model types in complete config * Add configuration to allow creating sub directories for each tub (like the legacy tub) or by default store all recordings in the data directory directly. (autorope#649) * Update setup.py with progress module for pi (autorope#650) * Update setup.py * Update setup.py Made changes online. * Add testing for training: (autorope#651) * added example tub data as tar.gz in tests with 1000 records * check validation data size in train * created new test_train.py * added support for read-only tub, required to create r/o tub in tmp dir * changed loss in categorical model to equally weight throttle and steering * changed mod on scripts to u+x * Rename `drive.py` to `manage.py` to preserve CLI compat. (autorope#656) * Train still prompts you to move to then new entry point. * Also fixes autorope#655 * Cleanup exit handlers in the new datastore. (autorope#657) * Minor datastore_v2 improvements. (autorope#658) * Add __exit__ handlers again. * Use `os.linesep` to deal with line separators consistently. * Be explicit about newline characters (autorope#660) * Be explicit about newline characters #2 (autorope#661) * Remove the extra flush(). Windows is less forgiving (autorope#662) * Remove redundant code (autorope#663) * Windows Doc Updates (autorope#668) Added more options for Installing Donkey Car on Windows. - Anaconda - Native - Windows Subsystem for Linux (WSL) - experiential These options give more flexibility and could make deployment easier for some individuals. I personally always install donkey car natively onto Python installed on the System. I will be looking more into WSL going forward as it has some interesting benefits. * Integrate osx into travis. (autorope#665) * add in --user flag for MakeMovie so that you can select if you want to draw the line on the video or not. default: true * updated args.draw_user_input * update to include recommendations * added in brake functionality for simulator only fix typo in drivemode return which checked pilot/throttle instead of brake * fix autorope/issue autorope#671 (autorope#672) * Jonathans changes from issue autorope#634 and PR autorope#646 (autorope#676) * Bump version to 4.0.1 (autorope#677) * Fix template docopt, as it's called managed.py and not drive.py * Allow overriding WEB_CONTROL_PORT from env variable * - Add simulator support to basic.py (autorope#682) - Update cfg_basic.py with simulator parameters * Switch to using memory-mapped files when reading. (autorope#691) * This makes reading a 100x faster. * new: mqtt telemetry support (autorope#688) * A configurable training pipeline. (autorope#693) * Implement a Lazy transformable pipeline. * Implement basic batching. However, this will need to be improved further for models with multiple outputs. * Replace the old `Sequence` implementation used with a new `Pipeline`. Test: Ran end to end tests. * The telemetry mqtt test is breaking w/ a timeout on the connection to the server. * change the public mqtt server name to one that's listening * * Overriding inference in TensorRT as pass as it is not needed here. (autorope#698) * Need to implement inference to run on the normalised image (autorope#699) * Fix sequence iterators (autorope#704) * Change training pipeline from tf.Sequence to tf.data (autorope#701) * Improve pipeline use: move from building list of pipelines of single transforms to building a single pipeline with a list of transforms (actually just looping through function to go from TubRecord -> image -> augment -> normalise -> x and TubRecord -> y). Fixed TfmIterators and TfmIterables. * Iterables are the containers and are sized - these are the user objects * Iterators are protocol objects to allow iteration, they have no logic and are local to the Iterables * build/map_pipeline both return sized Iterables * removed all batch logic, this is not required * still commented but left code that uses generator based pipeline as this is simpler code Using new small temporary pipeline generator * this keeps the TubSequence lazy and avoids to roll out the pipeline into a list * added a test to check consistency of the pipeline * remove empty (after moved) augmentation file * removed augmentation from old tub (as it's not needed and we removed the old augmentation) New pipeline changes: * moved augmentation into own class that is used above and can be used a threaded or non-threaded part * moved train functionality out of template and added 'donkey train', train.py just a simple dummy script for backward compatibility * Address code reviews: * Re-base on current dev to use un-altered sequence.py * Add iterator consistency test to pipeline tests * Undo changes in fast_stretch.py * better tf shape manipulation * small code improvements in training.py * remove sleep in augment part * Address code reviews: * Add clearing of tubrecord list and minor renamings * * Add support for multi-dimensional model input and making the x, y interface symmetrical on the model / training interface: (autorope#707) - x/y_transform extract x, y as numpy arrays or floats out of the record - x/y_translate convert the numpy arrays of floats into tf-readable dictionaries used in tf data. * Simplify model interface by implementing output_types() directly in the base class using output_shapes() dictionary. * Adding developer guide for own model development * Updated donkey command documentation * Improve asserts and type hints in keras.py * Added missing __init__.py in parts module. * Add cool ascii text for donkey init and update yml and setup files including mypy * Remove model training test from Travis and change the test to relative convergence. This avoids random fall overs in CI. * Added test of tf.data as used in the training pipeline through re-implementation of data transformation from tub records to tf expected dictionaries, for all currently supported models. * Minor changes for 4.1 in tub conversion script and developer doc (autorope#708) * Minor changes for 4.1 * Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links. * * Update doc with donkey train command. * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Incorporate PR feedback * Add empty records concept to tub * Minor updates to conversion script * Incorporate PR feedback * Add empty record type into conversion script * docs: fix simple typo, unfarmilar -> unfamiliar (autorope#714) There is a small typo in docs/guide/host_pc/setup_windows.md. Should read `unfamiliar` rather than `unfarmilar`. * Added the ability to train PyTorch models (autorope#706) * Added in PyTorch and PyTorch Lightning to train a DC model Successfully able to train a ResNet18-based model using PyTorch Lightning. * Removed hard-coded max number of epochs (used for debugging) * Added an inference transform to ResNet18 to convert PIL -> tensor * Unsqueezed input tensor during inference for batch dimension * Reshaped ResNet output from (1, 2) to (2,) * Added the ability to resume training from a checkpoint * Added helper print message when tensorboard logging is enabled * Updated docopt arguments for train.py. Made checkpoint optional * Changed TorchTubDataset from sub-classing Dataset to IterableDataset This was done in response to autorope#706 (comment) * Renamed load_image_arr to load_image. Updated load_pil_image load_pil_image will now handle converting the image to greyscale (vs. this being done in load_image). * Updated enviroments for Mac and Ubuntu. Set Python=3.7 * Updated installation documentation. Added script to setup Nano Updated the installation instructions for Ubuntu, Mac, and Windows. Clarified a common issue that occurs when running pip install -e .[pc] with ZSH. Also added a script to setup the Jetson Nano and updated the documentation for the Nano (it previously was installing tensorflow 1.x). * Added torch flag to setup.py to install pytorch * Moved pytorch training into base.py and removed from train.py * Moved Jetson Nano python package installation into requirements.txt * Formatted with PEP8 to clean up pytorch code * Updated docs to provide work-around for ZSH pip install -e .[pc] * Removed duplicate dependencies in conda env files * ResNet18 torch model now returns training loss history * Added test file for PyTorch training Still need to make sure this passes Travis CI. * Added lightning_logs to .gitignore * You can now specify the default AI framework to use in config.py This reduces the number of command line arguments you are required to provide. * get_model_by_type for PyTorch now lazy imports ResNet18 * Added help message to torch_train. Got rid of linear model type * Updated pytorch tests and fixed some syntax errors * ResNet18 example input shape updated to be (B, 3, 224, 224) Also now passing output_shape to load_resnet18 to modify how many output classes are used * No longer pinning requirement versions for Jetson Nano * Fixed formatting in setup.py * update configuration so that IMAGE_H and IMAGE_W are passed through to the simulator (autorope#674) Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: sctse999 <jonathan@tsez.net> Co-authored-by: DGarbanzo <47540921+DocGarbanzo@users.noreply.github.com> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: DocGarbanzo <dirk.prange@web.de> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com> * Updated path follower with Intel T265 (autorope#729) Properly rebased on Master this time, I hope Co-authored-by: Chris Anderson <gca2@earthlink.net> * Remove '-dev' from version output (autorope#734) * Updated training instructions. (autorope#744) * Make clear how to call "run" vs "run_threaded" Derived from Tawn Kramer's excellent video https://www.youtube.com/watch?v=YZ4ESrtfShs * Discuss update function * Added Tawn's video * More explanation of inputs and outputs * Explain shutdown * typo * typo * Make clear that threaded parts can run faster * Remove part about running faster than main loop. It appears that the main loop block threaded parts from running faster * Replace Travis CI with GitHub Actions (autorope#801) * Add github action files and remove travis files * Suppress train tests in github actions * Improvements in workflow, allow linter failure and don't stop build matrix on single os failure and add missing linter trigger on PRs * Replace Travis CI with GitHub Actions (autorope#801) * Add missing pc.yml * Add "./" before shell script so it will run (autorope#805) * Add "./" before shell script so it will run * Add automatic pushing of docs (autorope#806) * Adding auto push donkey docs for changes to docs in master * Add missing checkout. * Fix heading in faq * Add force and verbose flag to mkdocs * Fix link to roll-your-own in faq.md * Fix minor typos in faq.md * Fixed import for tub writer * Revert unintended changes to code from merge from master. * Added "DRIVE_TRAIN" back to compete template, too (autorope#809) Again, apologies to everyone whose installation has been failing. Blame Travis! * Added DRIVE_TRAIN back (autorope#808) This was causing everyone's installations to fail Co-authored-by: Jonathan Tse <jonathan@tsez.net> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: Rahul Ravikumar <tikurahul@gmail.com> Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com> Co-authored-by: Chris Anderson <analoguedevices@gmail.com> Co-authored-by: Chris Anderson <gca2@earthlink.net>
DocGarbanzo
added a commit
that referenced
this pull request
Apr 29, 2021
* - Update tensorflow version to 1.15.0 - Add doc for Jetpack 4.4 - Remove opencv installation * Add dataset and pre-trained model page * fix issue #634 (#646) - remove jquery dependency - add clearTimeout * Allow overriding WEB_CONTROL_PORT from env variable * pin h5py version * fix h5py version for mac * * Update doc with donkey train command. (#709) * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Donkey 4.x is now official (#723) * Update nano install docs - python3-opencv vs libopencv-python * change slack to discord on readme * Donkeycar 4.x release. (#644) ## Major Improvements - New Datastore. - More ways to pre-process image data for training. - Use the 2.x version of Tensorflow. - Lots of other minor improvements. * Improvements to the car app and handling of KerasPilot parts: (#648) * Improvements to the car app and handling of KerasPilot parts: * Created a simpler webserver or joystick car app by modifying and renaming the basic_web into basic template and switched this on as default. Also renamed the target from 'manage.py' to 'drive.py' because there is only driving and no training included * Simplified the handling of uint8 and float32 numpy image arrays. KerasPilot.run() now expects uint8 data, transforms into float32 and delegates to its children. The corresponding rescaling step is gone from the car app * smaller updates/fixes to environment and config files * added support for tflite pilots in makemovie * Improvements to the car app and handling of KerasPilot parts: * Added docstring to KerasPilot methods * Fixed bad merge for model types in complete config * Add configuration to allow creating sub directories for each tub (like the legacy tub) or by default store all recordings in the data directory directly. (#649) * Update setup.py with progress module for pi (#650) * Update setup.py * Update setup.py Made changes online. * Add testing for training: (#651) * added example tub data as tar.gz in tests with 1000 records * check validation data size in train * created new test_train.py * added support for read-only tub, required to create r/o tub in tmp dir * changed loss in categorical model to equally weight throttle and steering * changed mod on scripts to u+x * Rename `drive.py` to `manage.py` to preserve CLI compat. (#656) * Train still prompts you to move to then new entry point. * Also fixes #655 * Cleanup exit handlers in the new datastore. (#657) * Minor datastore_v2 improvements. (#658) * Add __exit__ handlers again. * Use `os.linesep` to deal with line separators consistently. * Be explicit about newline characters (#660) * Be explicit about newline characters #2 (#661) * Remove the extra flush(). Windows is less forgiving (#662) * Remove redundant code (#663) * Windows Doc Updates (#668) Added more options for Installing Donkey Car on Windows. - Anaconda - Native - Windows Subsystem for Linux (WSL) - experiential These options give more flexibility and could make deployment easier for some individuals. I personally always install donkey car natively onto Python installed on the System. I will be looking more into WSL going forward as it has some interesting benefits. * Integrate osx into travis. (#665) * add in --user flag for MakeMovie so that you can select if you want to draw the line on the video or not. default: true * updated args.draw_user_input * update to include recommendations * added in brake functionality for simulator only fix typo in drivemode return which checked pilot/throttle instead of brake * fix autorope/issue #671 (#672) * Jonathans changes from issue #634 and PR #646 (#676) * Bump version to 4.0.1 (#677) * Fix template docopt, as it's called managed.py and not drive.py * Allow overriding WEB_CONTROL_PORT from env variable * - Add simulator support to basic.py (#682) - Update cfg_basic.py with simulator parameters * Switch to using memory-mapped files when reading. (#691) * This makes reading a 100x faster. * new: mqtt telemetry support (#688) * A configurable training pipeline. (#693) * Implement a Lazy transformable pipeline. * Implement basic batching. However, this will need to be improved further for models with multiple outputs. * Replace the old `Sequence` implementation used with a new `Pipeline`. Test: Ran end to end tests. * The telemetry mqtt test is breaking w/ a timeout on the connection to the server. * change the public mqtt server name to one that's listening * * Overriding inference in TensorRT as pass as it is not needed here. (#698) * Need to implement inference to run on the normalised image (#699) * Fix sequence iterators (#704) * Change training pipeline from tf.Sequence to tf.data (#701) * Improve pipeline use: move from building list of pipelines of single transforms to building a single pipeline with a list of transforms (actually just looping through function to go from TubRecord -> image -> augment -> normalise -> x and TubRecord -> y). Fixed TfmIterators and TfmIterables. * Iterables are the containers and are sized - these are the user objects * Iterators are protocol objects to allow iteration, they have no logic and are local to the Iterables * build/map_pipeline both return sized Iterables * removed all batch logic, this is not required * still commented but left code that uses generator based pipeline as this is simpler code Using new small temporary pipeline generator * this keeps the TubSequence lazy and avoids to roll out the pipeline into a list * added a test to check consistency of the pipeline * remove empty (after moved) augmentation file * removed augmentation from old tub (as it's not needed and we removed the old augmentation) New pipeline changes: * moved augmentation into own class that is used above and can be used a threaded or non-threaded part * moved train functionality out of template and added 'donkey train', train.py just a simple dummy script for backward compatibility * Address code reviews: * Re-base on current dev to use un-altered sequence.py * Add iterator consistency test to pipeline tests * Undo changes in fast_stretch.py * better tf shape manipulation * small code improvements in training.py * remove sleep in augment part * Address code reviews: * Add clearing of tubrecord list and minor renamings * * Add support for multi-dimensional model input and making the x, y interface symmetrical on the model / training interface: (#707) - x/y_transform extract x, y as numpy arrays or floats out of the record - x/y_translate convert the numpy arrays of floats into tf-readable dictionaries used in tf data. * Simplify model interface by implementing output_types() directly in the base class using output_shapes() dictionary. * Adding developer guide for own model development * Updated donkey command documentation * Improve asserts and type hints in keras.py * Added missing __init__.py in parts module. * Add cool ascii text for donkey init and update yml and setup files including mypy * Remove model training test from Travis and change the test to relative convergence. This avoids random fall overs in CI. * Added test of tf.data as used in the training pipeline through re-implementation of data transformation from tub records to tf expected dictionaries, for all currently supported models. * Minor changes for 4.1 in tub conversion script and developer doc (#708) * Minor changes for 4.1 * Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links. * * Update doc with donkey train command. * Update doc with developer section for building own models in donkey 4.1 * Integrate changes from PR feedback * Incorporate PR feedback * Add empty records concept to tub * Minor updates to conversion script * Incorporate PR feedback * Add empty record type into conversion script * docs: fix simple typo, unfarmilar -> unfamiliar (#714) There is a small typo in docs/guide/host_pc/setup_windows.md. Should read `unfamiliar` rather than `unfarmilar`. * Added the ability to train PyTorch models (#706) * Added in PyTorch and PyTorch Lightning to train a DC model Successfully able to train a ResNet18-based model using PyTorch Lightning. * Removed hard-coded max number of epochs (used for debugging) * Added an inference transform to ResNet18 to convert PIL -> tensor * Unsqueezed input tensor during inference for batch dimension * Reshaped ResNet output from (1, 2) to (2,) * Added the ability to resume training from a checkpoint * Added helper print message when tensorboard logging is enabled * Updated docopt arguments for train.py. Made checkpoint optional * Changed TorchTubDataset from sub-classing Dataset to IterableDataset This was done in response to #706 (comment) * Renamed load_image_arr to load_image. Updated load_pil_image load_pil_image will now handle converting the image to greyscale (vs. this being done in load_image). * Updated enviroments for Mac and Ubuntu. Set Python=3.7 * Updated installation documentation. Added script to setup Nano Updated the installation instructions for Ubuntu, Mac, and Windows. Clarified a common issue that occurs when running pip install -e .[pc] with ZSH. Also added a script to setup the Jetson Nano and updated the documentation for the Nano (it previously was installing tensorflow 1.x). * Added torch flag to setup.py to install pytorch * Moved pytorch training into base.py and removed from train.py * Moved Jetson Nano python package installation into requirements.txt * Formatted with PEP8 to clean up pytorch code * Updated docs to provide work-around for ZSH pip install -e .[pc] * Removed duplicate dependencies in conda env files * ResNet18 torch model now returns training loss history * Added test file for PyTorch training Still need to make sure this passes Travis CI. * Added lightning_logs to .gitignore * You can now specify the default AI framework to use in config.py This reduces the number of command line arguments you are required to provide. * get_model_by_type for PyTorch now lazy imports ResNet18 * Added help message to torch_train. Got rid of linear model type * Updated pytorch tests and fixed some syntax errors * ResNet18 example input shape updated to be (B, 3, 224, 224) Also now passing output_shape to load_resnet18 to modify how many output classes are used * No longer pinning requirement versions for Jetson Nano * Fixed formatting in setup.py * update configuration so that IMAGE_H and IMAGE_W are passed through to the simulator (#674) Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: sctse999 <jonathan@tsez.net> Co-authored-by: DGarbanzo <47540921+DocGarbanzo@users.noreply.github.com> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: DocGarbanzo <dirk.prange@web.de> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com> * Updated path follower with Intel T265 (#729) Properly rebased on Master this time, I hope Co-authored-by: Chris Anderson <gca2@earthlink.net> * Remove '-dev' from version output (#734) * Updated training instructions. (#744) * Make clear how to call "run" vs "run_threaded" Derived from Tawn Kramer's excellent video https://www.youtube.com/watch?v=YZ4ESrtfShs * Discuss update function * Added Tawn's video * More explanation of inputs and outputs * Explain shutdown * typo * typo * Make clear that threaded parts can run faster * Remove part about running faster than main loop. It appears that the main loop block threaded parts from running faster * Replace Travis CI with GitHub Actions (#801) * Add github action files and remove travis files * Suppress train tests in github actions * Improvements in workflow, allow linter failure and don't stop build matrix on single os failure and add missing linter trigger on PRs * Replace Travis CI with GitHub Actions (#801) * Add missing pc.yml * Add "./" before shell script so it will run (#805) * Add "./" before shell script so it will run * Add automatic pushing of docs (#806) * Adding auto push donkey docs for changes to docs in master * Add missing checkout. * Fix heading in faq * Add force and verbose flag to mkdocs * Fix link to roll-your-own in faq.md * Fix minor typos in faq.md * Fixed import for tub writer * Added "DRIVE_TRAIN" back to compete template, too (#809) Again, apologies to everyone whose installation has been failing. Blame Travis! * Added DRIVE_TRAIN back (#808) This was causing everyone's installations to fail * ubuntu.yml was renamed pc.yml (#819) * Push an important change (for a PC conda file should be pc.yml, not ubuntu.yml) from Dev to Master (#822) * Added manual installation of Tensorflow with --user flag Conda says not to put "--user" in yml scripts so we'll have the users do this manually until we get to the bottom of the TF 2.2 installation errors on Windows * Changed pc.yml to windows.yml as per PR review Avoids ambiguity between computer type and OS type (PCs can run both Windows and Ubuntu) * Take out Tensorflow from automated windows installation (#824) * Take out Tensorflow from automated windows installation because it's causing … …an error with latest conda * Changed pc.yml to windows.yml * Rename pc.yml to windows.yml * Create ubuntu.yml * Update python-package-conda.yml Update workflow after rename of ubuntu conda file. * Fix ubuntu yml file and update ubuntu install docs. * some more changes (#831) * Fix typo (#836) * Add link to parts guide (#841) * Fixed markdown typo (#843) * Fixed markdown typo * Create docs.md * Added link * Update docs.md * Fixed typo. * Removed mkdocs and gh actions referring to docs. * Release 4.2 (#849) * Copied release_4_2 which itself is a copy of dev and removed 2 accidental merges 61f4742, cb50317 * Fixed small bug in UI * Merged from master Co-authored-by: Jonathan Tse <jonathan@tsez.net> Co-authored-by: Tawn Kramer <tawnkramer@gmail.com> Co-authored-by: Rahul Ravikumar <tikurahul@gmail.com> Co-authored-by: Jordan Whited <jordan@jordanwhited.com> Co-authored-by: wallarug <wallarug@coliemore.com.au> Co-authored-by: Craig <cfox570@mac.com> Co-authored-by: Meir Tseitlin <meir.tech@gmail.com> Co-authored-by: Tim Gates <tim.gates@iress.com> Co-authored-by: Eric Wiener <ericwiener3@gmail.com> Co-authored-by: Chris Anderson <analoguedevices@gmail.com> Co-authored-by: Chris Anderson <gca2@earthlink.net> Co-authored-by: Joep B <joepbarmentlo@gmail.com> Co-authored-by: Maxime Ellerbach <maxime@ellerbach.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.