Platform Benchmark

Hardware configuration

  • I7 Intel machine @ 1.6 GHz / core
  • 3.9 Gb of RAM
  • OS: Ubuntu 12.04 Linux x64,
  • Oracle Java 1.7.

Protocol

The setup is to send a series of random messages, of a given size, from one component to another. We choose to send random sequences in order to prevent the caching speed of ActiveMQ, which applies a set of heuristics in case the same message is sent over the network. Moreover, in order to prevent local traffic peaks, we sent 100 messages and presented only the average time. We want to prevent the time delay imposed by different queuing strategies implemented into ZeroMQ or ActiveMQ, therefore we measure the processing time between two consecutive messages (a sprint), with a small delay between the sprints.

For the message throughput, we represented the number of messages that pass between the two components in one second. The table and the figure bellow present this measurement, side-by-side, on a logarithmic scale for all the platforms included in the benchmark.

Source code

The code for the benchmark on each platform is available here.

Results

I. ZeroMQ vs ActiveMQ

101001,00010,000100,0001,000,000
ActiveMQ
4,0474,7925,7142,47929235
ZeroMQ50,0008,33316,6663,22555866

II. AgentSlang (MyBlock) vs SEMAINE vs VHToolkit (VHTMsg-Java)

101001,00010,000100,0001,000,000
Semaine2,6082,6492,7471,62529840
VHToolkit (VHTMsg - Java)1,7322,1432,0906599816
AgentSlang (ASF)1,8721,9771,8661,47235151
AgentSlang + MsgPack (ASF)2,4092,5552,2901,84864993
AgentSlang (Simple)3,0643,2753,1781,83434753
AgentSlang + MsgPack (Simple)5,9544,9114,5843,75076593

benchmark

AgentSlang, by integrating MyBlock, is presented in two versions:

  • Automatic System Feedback (ASF). This mechanism enables AgentSlang to send a feedback message each time an action is successfully executed. This is executed in the case of sending and receiving a message.
  • non-ASF (simple).  SEMAINE and VHToolkit does not provide a similar mechanism, therefore, in order to achieve a fair comparison of the two systems, this feedback has also been disabled.

A second round of experiments is conducted in order to see the influence of the serialization mechanism. In this case two scenarios are tested:

  • Java Default Object Serialization
  • MsgPack serialisation

Conclusion

  1. ZeroMQ is a the best choice for the communication library. Other benchmarks reached similar conclusions.
  2. AgentSlang is faster than SEMAINE and VHToolkit, when having the ASF disabled.
  3. For messages longer than 10,000 characters, the ASF does not increase the sending time.
  4. Since AgentSlang targets a large spectrum of data types, both scenarios can be used in practical situations.
  5. The choice of a platform depends on the application. To achieve the best speed while sending data, AgentSlang simple (non-ASF) is a good choice. To ensure that a message has been successfully processed, AgentSlang ASF is currently the only choice.
  6. MsgPack is obviously a better choice for serialization than the default Java mechanism