YScope Log Viewer

yscope-log-viewer is a tool that can be used to view log files that were compressed using CLP's IR stream format. The viewer can be used to navigate the log file, filter by log level, pretty print the logs and generate direct links to specific log events.

See the features in development section for upcoming features.

Online Demo

• A demo of the log viewer can be found at https://yscope.com/log-viewer
• The demo loads a Hadoop YARN log file from the hive-24hrs log dataset.
  • More info on the dataset and other datasets can be found here.
• To open an IR stream, drag and drop it onto the log viewer or use the open file dialog.

Generating IR Stream Logs

IR stream log files can currently be generated using these libraries:

• Python Logging Library
• Log4j Logging Library
• Golang Logging Library (in development)

Providing Feedback

You can use GitHub issues to report a bug or request a feature.

Join us on Zulip to chat with developers and other community members.

Developing yscope-log-viewer

• Clone the repo to get a copy of the code for development

  git clone https://github.com/yscope/yscope-log-viewer.git
  cd yscope-log-viewer

• Install Node.js using a release or via nvm

• Install the dependencies:

  $ npm install

• Run the development server:

  $ npm start

• The application should now be served at http://localhost:3010.

Distribute

To create a build, run the following command and the build will be placed in the dist folder:

$ npm run build

How does it work?

yscope-log-viewer is written using the ReactJS framework and uses the open source monaco-editor.

To open IR stream files, the viewer spawns workers to do the following:

• Decompress the Zstd-compressed file
• Build an index of log events
• Paginate the indexed logs based on the number of log events per page
• Decode the CLP-encoded log data as needed

Once the worker decompresses, decodes, and extracts logs, the viewer UI can be used to navigate the log. Tasks are passed to the worker as needed and changes are rendered to the UI.

Validation

Currently, there is limited automated testing to verify that changes don't cause bugs. While this is being developed, the following tests can be run manually:

• Verify that the following features work:
  • Changing the number of events per page
  • Navigating to the first/last/next/previous page
  • Loading a log file using the open file dialog and dragging & dropping
  • Copying a link to a log event
  • Change the log level
  • Prettifying logs
  • Using the keyboard shortcuts
• Perform a build and verify that all features are functional

Running tests

We use jest as our testing framework. Tests can be run with:

npm test

Features in Development

• Dashboard to visualize distribution of log types and log levels
• Opening log files which are larger than 2 GB when uncompressed
• Support for plain text, archived log files, and other requested formats
• Search across file, multiple files or within provided time ranges
• Infinite scrolling instead of using pagination
• Log correlation with sync by timestamp across multiple editors
• Automatic conversion of text log files to IR stream format in the browser
• Deployment of components via NPM