Category Archives: scala

Constructor memory leaks

Yesterday I fixed a nasty memory leak in our PushServer which is written in Scala. As so happens with all the memory leaks, application was  running smoothily in test mode and just when I deployed in production it bombed within hours of launching the application.  The problem was with following bit of code:

class Quote(dataHash: Map[Int,Any],oldQuote: Quote) extends AmtdStockData {
  val symbol: String = dataHash.getOrElse(0,oldQuote.symbol)
  val bid: Float = dataHash.getOrElse(1,oldQuote.bid)
  val ask: Float = ....
}

Above code defines Quote class, the constructor of class takes a data Map as an argument and old quote for the same symbol. Class populates its field from data Map, but if a certain field is not found in the Map, it uses the field’s value from oldQuote. The Quote class basically works around the problem from data providers which send only updated fields for Level 1 quote.

Probably you have guessed by now, the problem with above code is that memory that is held by old quotes are never freed, because newer quotes always keep a reference to old quote values.

I can probably say, similar logic wouldn’t have created problems with Java or C++, but since constructor arguments are automatically taken as class fields in scala, we have a problem. The ability to pass arguments while defining a class and using it as a constructor is a life saver.It helps in avoiding placeholder field declaration which will be later populated through constructor or another method call as done in Java/C++.

But apparently, in this particular case I needed old ugly way back and hence new code is:

class Quote extends AmtdStockData {
  var symbol: String = _
  var bid: Float = _
  var ask: Float = _
  def populateValues(dataHash: Map[Int,Any],oldQuote: Quote) = {
    // populate above fields
  }
}

Above class design solves the problem at the expense of using “var”.Its not ideal, but I am scouting for ideas if the class can be designed in any other way!

Update: Turns out compiler usually doesn’t emit field for constructor parameter unless parameter is accessed by an instance method/inner class. In above case, getOrElse creates an anonymous inner class and hence compiler emits a field for oldQuote and its never garbage collected.

Your Comet Server In Scala

I thought it will be cool to display real time stock market streaming ticks in our marketsimplified application.
Small Watchlist

I needed wee bit of flash code that opens an XML socket to our comet server,accepts data and invokes corresponding javascript function in browser for displaying streaming data. I took the flash code from Juggernaut project and compiled it to a swf. Now, we needed a Comet Server. The initial version I wrote in Ruby using EventMachine. It was pretty darn good, you can plugin and stream how many type of data you want, you just need to inherit a class and you were done. But it was slow and was unable to handle influx of stock market ticks. We had to restart the damn comet server everyday.

So I proceeded to port Comet Server in Scala. Before choosing Scala, I played with Erlang and D. My attempts at learning Erlang and using it for our Comet Server were serious. Why I gave up Erlang was mostly because:

  • String handling and near absence of it. Yes sure one can write a recursive descent parser, but for small string manipulation, its an overkill. Our internal data exchange protocol is line oriented and the parers that I wrote used string manipulation.
  • I dunno, if many will agree, but Erlang is hard. Yeah, probably not for simple “hello world” applications, but on whole you need to turn your head quite a bit. I wasn’t sure, if my colleagues will buy into it.
  • Even if you ignore above two critireas, I believe Erlang needs quite different Eco System. Our application is already built around open source technologies, such as – Mysql, MemCache, Rails, Hibernate, Ruby, Python, Nginx, Mongrel and controlled by YAML files. Hunting Erlang libraries for mysql, memcache, yaml, json and making them work seemed like too much work.

Anyways, that was my decision so get over it. I briefly flirted with D. I wasn’t very happy with libraries, their installation procedure and their API. I started with Scala long ago, playing it with now and then. But after my Erlang tryst fizzled out, I decided to look into Scala seriously. There were no decent Network programming libraries for Scala. Sure, I could have used Mina, but I wanted a more Scalasque library, which helps in me translating my EventMachine code to Scala and hence I wrote Eventfax. ( Code in public svn is a bit stale, our corporate svns have latest code, which I will be publish soon )

For example a EchoServer using Eventfax library will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
import scala.actors._
import scala.actors.Actor._
import java.io.IOException
 
import eventfax.core._
import eventfax.protocol._
 
// define a reactor core
class EchoReactor(starterBlock: ReactorCore => Unit) extends ReactorCore(starterBlock) {
  def check_for_actor_messages = {}
}
 
// define a factor class
object EchoServerFactory extends ConnectionFactory {
  def create_connection(): Connection = {
    new EchoServer()
  }
}
 
class EchoServer extends Connection {
  val buftok: LineProtocol = new LineProtocol("[\n]")
  val masterActor: EchoReactor = reactor.asInstanceOf[EchoReactor]
  def receive_data(p_data: FaxData) = {
    buftok.extract(p_data)
    buftok.foreach(str_data => {
      dispatch_request(str_data)
    })
  }
 
  def dispatch_request(str_data: String) = {
    val t_str = str_data.trim()
    if(!t_str.isEmpty)
      send_data(str_data+"\n")
  }
 
  def on_write_error() = {
    try { close_connection }
    catch { case ex: IOException => println("error")}
  }
  def unbind() = {
  }
}
 
object EchoServerRunner {
  def main(args: Array[String]) = {
    new EchoReactor(t_reactor => {
      t_reactor.start_server("localhost",8700,EchoServerFactory)
    })
  }
}

Also, I had to port our protocol parsing libraries to Scala, which proved trivial. After a week of effort, our watchlist is powered by a Comet Server written Scala. I used Scala Specs for testing, buildr for compiling and packaging.

I am yet to get buildr working properly with Scala specs, but I will perhaps look into that later. Getting Memcache,Mysql, YAML or JSON working with Scala was trivial. Although, Json parser of Scala is one cane full of worms. Every new scala release seems to introduce new bugs in it. I had to spend quite sometime in getting around them.