A Place for my head

I thought it will be cool to display real time stock market streaming ticks in our marketsimplified application.

I needed wee bit of flash code that opens an XML socket to our comet server,accepts data and invokes corresponding javascript function in browser for displaying streaming data. I took the flash code from Juggernaut project and compiled it to a swf. Now, we needed a Comet Server. The initial version I wrote in Ruby using EventMachine. It was pretty darn good, you can plugin and stream how many type of data you want, you just need to inherit a class and you were done. But it was slow and was unable to handle influx of stock market ticks. We had to restart the damn comet server everyday.

So I proceeded to port Comet Server in Scala. Before choosing Scala, I played with Erlang and D. My attempts at learning Erlang and using it for our Comet Server were serious. Why I gave up Erlang was mostly because:

• String handling and near absence of it. Yes sure one can write a recursive descent parser, but for small string manipulation, its an overkill. Our internal data exchange protocol is line oriented and the parers that I wrote used string manipulation.
• I dunno, if many will agree, but Erlang is hard. Yeah, probably not for simple “hello world” applications, but on whole you need to turn your head quite a bit. I wasn’t sure, if my colleagues will buy into it.
• Even if you ignore above two critireas, I believe Erlang needs quite different Eco System. Our application is already built around open source technologies, such as – Mysql, MemCache, Rails, Hibernate, Ruby, Python, Nginx, Mongrel and controlled by YAML files. Hunting Erlang libraries for mysql, memcache, yaml, json and making them work seemed like too much work.

Anyways, that was my decision so get over it. I briefly flirted with D. I wasn’t very happy with libraries, their installation procedure and their API. I started with Scala long ago, playing it with now and then. But after my Erlang tryst fizzled out, I decided to look into Scala seriously. There were no decent Network programming libraries for Scala. Sure, I could have used Mina, but I wanted a more Scalasque library, which helps in me translating my EventMachine code to Scala and hence I wrote Eventfax. ( Code in public svn is a bit stale, our corporate svns have latest code, which I will be publish soon )

For example a EchoServer using Eventfax library will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

import scala.actors._
import scala.actors.Actor._
import java.io.IOException
 
import eventfax.core._
import eventfax.protocol._
 
// define a reactor core
class EchoReactor(starterBlock: ReactorCore => Unit) extends ReactorCore(starterBlock) {
  def check_for_actor_messages = {}
}
 
// define a factor class
object EchoServerFactory extends ConnectionFactory {
  def create_connection(): Connection = {
    new EchoServer()
  }
}
 
class EchoServer extends Connection {
  val buftok: LineProtocol = new LineProtocol("[\n]")
  val masterActor: EchoReactor = reactor.asInstanceOf[EchoReactor]
  def receive_data(p_data: FaxData) = {
    buftok.extract(p_data)
    buftok.foreach(str_data => {
      dispatch_request(str_data)
    })
  }
 
  def dispatch_request(str_data: String) = {
    val t_str = str_data.trim()
    if(!t_str.isEmpty)
      send_data(str_data+"\n")
  }
 
  def on_write_error() = {
    try { close_connection }
    catch { case ex: IOException => println("error")}
  }
  def unbind() = {
  }
}
 
object EchoServerRunner {
  def main(args: Array[String]) = {
    new EchoReactor(t_reactor => {
      t_reactor.start_server("localhost",8700,EchoServerFactory)
    })
  }
}

Also, I had to port our protocol parsing libraries to Scala, which proved trivial. After a week of effort, our watchlist is powered by a Comet Server written Scala. I used Scala Specs for testing, buildr for compiling and packaging.

I am yet to get buildr working properly with Scala specs, but I will perhaps look into that later. Getting Memcache,Mysql, YAML or JSON working with Scala was trivial. Although, Json parser of Scala is one cane full of worms. Every new scala release seems to introduce new bugs in it. I had to spend quite sometime in getting around them.

Update: Do not forget to get rid of autogenerated backgroundrb etcetera files that were generated during last run of rake backgroundrb:setup.

A plain fork() is evil under Ruby and hence there are some issues with memory usage of BackgrounDRb. I have made some changes to packet library so as it uses fork and exec rather than just fork for better memory usage. However, there were quite bit of changes and BackgrounDRb is affected by them. You can try bleeding edge version as following.Also you need rspec for building packet gem.

clone the packet git repo:

git clone git://github.com/gnufied/packet.git
cd packet;rake gem
cd pkg; sudo gem install --local packet-0.1.6.gem

Go to your vendor/plugins directory of your rails directory and remove or
backup older version of backgroundrb plugin and backup related config
file as well.

from vendor/plugins directory:

git clone git://github.com/gnufied/backgroundrb.git
cd RAILS_ROOT
rake backgroundrb:setup
./script/backgroundrb start

Let me know,how it goes.

Newest version of operating system for human beings is out. Kicking ass and users. Yes users, you heard it right.
I am not very happy with this release. In fact, I regret that I upgraded to this version. Why?

• Today every upcoming Linux distro is bundling Firefox3 beta 5 as default browser so ubuntu is not alone there. But in my opinion this is a bad decision. Firefox3 beta5 is not stable. Most of the extensions aren’t supported yet. You are being hard on users by bundling a software thats still 2 months from final release. My main gripes with Firefox3 beta 5 is,
  • Its a CPU hog. Even with only one tab open, it consumes considerable CPU cycles. Bad for laptop users. Now before you jump the gun, in my tests, I have a simple page open (no fancy gmail with lots of background processing) and I am not interacting with firefox window
  • It hangs and freezes now and then. I have experienced this kinda behavior often, while downloading a page firefox will freeze momentarily and then suddenly it will come back.
  • Extensions, extensions. Many of my favorite extensions are not available for firefox3.
  • Also, I am using fully updated version of Hardy and have disabled Security->Tell Me, thingies. and have deleted urlclassifier3.sqlite without much success.
• I guess all Linux users have seen occasional freezes and they have no other go than to hard reboot the machine. With Hardy and Compiz fusion, I just saw too many of freezes on my notebook. Also, did anyone notice metacity compositioning is buggy? I tried using metacity compositioning in place of compiz and had to press alt-ctrl-backspace couple of times. One sure way is, try enabling compositioning and then disabling it. I experienced X freeze when doing that.
• Many Emacs users i think swap ctrl and caps keys. I am one of them. After upgrading to hardy swapping them through Gnome keyboard settings works, but whenever I press caps ( now its ctrl ), it still turns on caps LED display on my notebook. A bit annoying. This issue was not there in Guts.
• Shipped Xorg is a CPU hog, even if you are not interacting with your machine, Xorg cpu usage stays around 10-30%. I have Nvidia graphics card and I have tried using <code>”UseEvents” “on”</code>. It doesn’t seem to have any effect.
• Oh boy, I don’t even know, why they shipped this half hearted PulseAudio integration? For getting sound back on my laptop, I had to manually choose ALSA as Sound Playback device.

Being a LTS edition, I was expecting something solid, but I can confirm with some certainity that Hardy release is certainly inferior than Gutsy. Thank you Ubuntu for scrweing it up.