Your Comet Server In Scala

I thought it will be cool to display real time stock market streaming ticks in our marketsimplified application.
Small Watchlist

I needed wee bit of flash code that opens an XML socket to our comet server,accepts data and invokes corresponding javascript function in browser for displaying streaming data. I took the flash code from Juggernaut project and compiled it to a swf. Now, we needed a Comet Server. The initial version I wrote in Ruby using EventMachine. It was pretty darn good, you can plugin and stream how many type of data you want, you just need to inherit a class and you were done. But it was slow and was unable to handle influx of stock market ticks. We had to restart the damn comet server everyday.

So I proceeded to port Comet Server in Scala. Before choosing Scala, I played with Erlang and D. My attempts at learning Erlang and using it for our Comet Server were serious. Why I gave up Erlang was mostly because:

  • String handling and near absence of it. Yes sure one can write a recursive descent parser, but for small string manipulation, its an overkill. Our internal data exchange protocol is line oriented and the parers that I wrote used string manipulation.
  • I dunno, if many will agree, but Erlang is hard. Yeah, probably not for simple “hello world” applications, but on whole you need to turn your head quite a bit. I wasn’t sure, if my colleagues will buy into it.
  • Even if you ignore above two critireas, I believe Erlang needs quite different Eco System. Our application is already built around open source technologies, such as – Mysql, MemCache, Rails, Hibernate, Ruby, Python, Nginx, Mongrel and controlled by YAML files. Hunting Erlang libraries for mysql, memcache, yaml, json and making them work seemed like too much work.

Anyways, that was my decision so get over it. I briefly flirted with D. I wasn’t very happy with libraries, their installation procedure and their API. I started with Scala long ago, playing it with now and then. But after my Erlang tryst fizzled out, I decided to look into Scala seriously. There were no decent Network programming libraries for Scala. Sure, I could have used Mina, but I wanted a more Scalasque library, which helps in me translating my EventMachine code to Scala and hence I wrote Eventfax. ( Code in public svn is a bit stale, our corporate svns have latest code, which I will be publish soon )

For example a EchoServer using Eventfax library will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
import scala.actors._
import scala.actors.Actor._
import java.io.IOException
 
import eventfax.core._
import eventfax.protocol._
 
// define a reactor core
class EchoReactor(starterBlock: ReactorCore => Unit) extends ReactorCore(starterBlock) {
  def check_for_actor_messages = {}
}
 
// define a factor class
object EchoServerFactory extends ConnectionFactory {
  def create_connection(): Connection = {
    new EchoServer()
  }
}
 
class EchoServer extends Connection {
  val buftok: LineProtocol = new LineProtocol("[\n]")
  val masterActor: EchoReactor = reactor.asInstanceOf[EchoReactor]
  def receive_data(p_data: FaxData) = {
    buftok.extract(p_data)
    buftok.foreach(str_data => {
      dispatch_request(str_data)
    })
  }
 
  def dispatch_request(str_data: String) = {
    val t_str = str_data.trim()
    if(!t_str.isEmpty)
      send_data(str_data+"\n")
  }
 
  def on_write_error() = {
    try { close_connection }
    catch { case ex: IOException => println("error")}
  }
  def unbind() = {
  }
}
 
object EchoServerRunner {
  def main(args: Array[String]) = {
    new EchoReactor(t_reactor => {
      t_reactor.start_server("localhost",8700,EchoServerFactory)
    })
  }
}

Also, I had to port our protocol parsing libraries to Scala, which proved trivial. After a week of effort, our watchlist is powered by a Comet Server written Scala. I used Scala Specs for testing, buildr for compiling and packaging.

I am yet to get buildr working properly with Scala specs, but I will perhaps look into that later. Getting Memcache,Mysql, YAML or JSON working with Scala was trivial. Although, Json parser of Scala is one cane full of worms. Every new scala release seems to introduce new bugs in it. I had to spend quite sometime in getting around them.

4 thoughts on “Your Comet Server In Scala

  1. raggi

    I recently committed working makefiles and patches to get the JRuby EventMachine reactor up and running, you can find it in http://rubyeventmachine.com/svn/branches/raggi/ if you’re interested to checkout the performance difference for this app. One of the notable things with JRuby is that you can have real threads running alongside the reactor without the same penalty as you have in ruby.

    Well done on the Scala front. I’ve had a look at Erlang in the past and also found it over complex for a lot of work.

    Reply
  2. Anonymous

    Im sorry, but i can’t seem to be able to find an actual “comet” server here (or anything even remotely resembling one)? Just a pretty basic TCP/IP networking library and server. By that definition we should be calling any network program able to listen for incoming connections a comet server (and netcat the ultimate comet swiss army knife).

    All in all a very misleading title and post about Scala comet server which seems to only serve the purpose of promoting your own commercial app (and technology behind it).

    Reply
  3. Hemant Post author

    I am truly sorry to disappoint you. But the moment, I wrote this article I was in talks with my employer to open source the Eventmachine port of Scala and open source the Comet Server.

    Recently, I have switched jobs and my previous employer is no longer interested in open sourcing that stuff and so I am stuck.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *