Important
This GitHub repository serves as a legacy archive, marking the culmination of past contributions within the SignON European project. A new repository has been created to build upon this foundation, where ongoing work and advancements can be found. Please visit the successor repository (here) to stay up-to-date with the latest developments.
This web application estimates computer animations from a single-camera input video. It is hosted in this link.
The first view of the application is a selector of the provided input. It supports pre-recorded videos or the video stream from the webcam. It can also load previously created animations for editing purposes.
Loading a video or capturing it live through the webcam redirects the user to the video editing stage. Here the resulting video can be trimmed if desired and/or redo the video by recording it again or uploading a new one. Once the user is fine with the result, the option "Convert Data to 3D Animation" moves the application to the last view.
The last station is a viewer and an editor of the final estimated animation. The application converts the video (2D) into a 3D animation through a diverse set of Machine Learning (ML) techniques. Since the output might not be perfect, the editor allows the user to edit/correct as desired the animation with the help of a timeline and a GUI to move the joints of the skeleton.
Finally, the user can download the animation as a BVH format file to use it in any other applications with different purposes.
It contains (ordered from left to right):
- The SignON logo linked to the oficial project link
- The Project button with some editing options
- The Timeline button with some timeline editing options and shortcuts
- Play/Pause button
- Restart button
The panel on the right is divided into four sections.
- The Skeleton heriarchy shows the different bones and their relations. It provides also a quick way to find and select a specific bone by name
- Animation Clip shows generic information about the animation like the name and duration of the animation. The playback speed can be adjusted through the "speed" slider.
- Gizmo changes some settings of the rotation tool (the ball on the shoulder of the avatar ). The gear button offers some options such as making the yellow joint points biggers/smaller.
- Bone offers the possibility of directly modifying the rotation values by writing them instead of using the in-scene tool
The center of the screen shows the scene with the avatar in a pose, determined by the animation frames and the current time in the timeline.
Clicking on a bone (yellow point) selects that joint. The ball with several lines that appears allows the user to rotates that joint in different axes by just clicking and dragging in the desired axis. The coloured axes are the typical perpendicular axes: red x-axis, green y-axis and blue z-axis. The outer circle rotates in the same plane as the camera is showing.
The buttons in the right side (not in the side panel) shows/hides the skeleton (lines) and the joints points for a clearer view
To move the camera position of the scene either left-click and drag in it (not on a bone) to orbit or right-click and drag to move the camera sideways. Scrolling zooms in and out (up to certain limits).
The timeline located at the bottom of the screen, shows a point for each frame of the animation of the selected bone. Each bone has its own track. Dragging the the timeline moves the current selected time, thus changing the pose in which the avatar is. To modify a frame, a point in the timeline needs to be selected, which will change colour into yellow. When a rotation is performed through the gizmo, the dot will become purple. Right clicking in the timeline allows to insert a new frame or deleting an existing one. The timeline can be zoomed in and out.
This project makes use of the following libraries:
- Three.js - An open-source JavaScript library for creating interactive 3D and 2D graphics in web browsers using WebGL.
- Mediapipe - An open-source library from Google for building cross-platform multimodal applied ML pipelines.
- Tensorflow.js - A JavaScript library for training and deploying machine learning models in the browser and on Node.js.
- LiteGUI - A simple and lightweight GUI library for creating graphical user interfaces for C++ programs.
- w2ui - A lightweight and easy-to-use JavaScript UI library for creating web user interfaces.
We would like to extend our gratitude to the creators and maintainers of these libraries for their invaluable contributions to the open-source community.
- Víctor Ubieto @victorubieto
- Pablo García @PZerua
- Eva Valls @evallsg
- Alex Rodríguez @jxarco
- Jaume Pozo @japopra
This application was developed using funding from the European SignON project (https://signon-project.eu/) and is intended to serve as a tool for creating animations to be used within the project.