- a cloud scale, fault tolerant, distributed computation system
- written in Clojure, for Clojure
- batch and stream processing hybrid
- exposes an information model for the description and construction of distributed workflows
- enabled by hardware advances in the last 10 years
- Competes against Storm, Cascading, Map/Reduce, Dryad, Apache Sqoop, Twitter Crane, etc
- Realtime event stream processing
- Continuous computation
- Extract, transform, load
- Data transformation à la map-reduce
- Data cleaning
- Data ingestion and storage medium transfer
Available on Clojars:
[com.mdrogalis/onyx "0.3.2"]
Feeling impatient? Hit the ground running ASAP with the onyx-starter repo and walkthrough.
- What does Onyx offer?
- Concepts
- Environment
- Hardware
- APIs
- Constraints
- Architecture
- Coordinator and Peer Configuration
- Information Model
- Plugins
- HornetQ Internal Plugin
- Job and Peer Execution Scheduling
- Reliability Guarantees
- Coordinator High Availability
- Logging
- Performance Tuning
- Examples
- Frequently Asked Questions
Official plugins are vetted by Michael Drogalis. Ensure in your project that plugin versions directly correspond to the same Onyx version (e.g. onyx-core-async version 0.3.2 goes with onyx version 0.3.2). Fixes to plugins can be applied using a 4th versioning identifier (e.g. 0.3.2.1).
Generate plugin templates through Leiningen with onyx-plugin.
Check out the Onyx Google Group.
Feel free to use it anywhere. You can find a few different versions here.
A simple lein midje will run the full suite, which takes about 15 minutes on my quad-core MacBook Pro. Expect a long, 5-10 minute pause during the coordinator tests as it's being completely slammed with requests and checked for correctness. The pause is waiting for the Coordinator to catch up and close out cleanly.
Contributions are welcome. Please fork the repository and send a pull request to the master branch.
Onyx uses a similiar branching strategy to Clojure itself. Onyx uses semantic versioning, and each minor version gets its own branch. All work is done on master or feature branches and dropped into a major.minor.x branch when it's time to cut a new release. Pull requests into the master branch are welcome.
Anyone who has a patch accepted may request commit rights. Please do so inside the pull request post-merge.
Onyx is not a battle-tested framework compared to Storm or Cascading. Please keep this in mind when considering taking it to production. That being said, the test suite uses simulation testing to create exceptionally bad data center network partitions, and checks against a number of correctness criteria afterwards. Community effort will help a great deal on this front.
It's worth noting that nearly all inter-cluster communication happens via HornetQ. HornetQ is a battle tested queueing platform from the JBoss stack that has been in production for many years. This does give Onyx a significant leg up on reliability.
At the time of writing this, I do not have any performance benchmarks to publish. Creating a correct, useful benchmark is extremely difficult. I'm working on it - hang tight.
This project is authored by Michael Drogalis, an independent software consultant. Get in touch (mjd3089.at.rit.edu) to work together.
Copyright © 2014 Michael Drogalis
Distributed under the Eclipse Public License, the same as Clojure.