Saving space by adding a column

Reik | 26 March, 2010 | 0 comments

We are currently working on a feature which will impact the rake distribution in our Poker platform. As part of the activity, my team had to add some columns in two existing database tables. Unfortunately these tables contain vast amount of data. For example one table contains all the Poker rounds we play. Even though we keep the data only for some days in this table, we almost always have around 300 million entries. This makes it very hard for us to do schema changes without seriously affecting other parts of the system. A good example for scalability issues you might get using a relational database with huge amount of data. More on that later.

Anyway, our DBD team found a good workaround to make the schema changes without downtime. While we were looking for the best possible solution, we found some columns having unrealistic data-types. For instance the column containing the amount of money players had bet in one round, was a BIGINT. This would theoretically have allowed us to store 184 quadrillion US-Dollar, which is a couple of thousand times bigger than the GNP of Sweden in 2009 and veeery unlikely to happen. So we made this an INT column - to allow the high-rollers bid their multi-million dollars - and fixed two other column types on our way. Since the database table is so big, making these data-types smaller will save us 5.6 GB of disk-space, which is pretty cool given that we were actually adding a column. So it can worth it to re-evaluate your old database schemas, especially if they are big.

Some more thoughts on scalability. Even though we have found a solution which avoids downtime, it will still take a long time copying the data around. Personally, I am very interested in the No-SQL movement, which seems to gain more and more momentum in the developer community. Since it can be foreseen, that our amount of data will rather become bigger than smaller, I embrace looking into alternatives scaling better than a relational database. The last few weeks, I tried Hadoop for a bit, which is the basis for HBase or Hive, and I can imagine it being useful in some of our scenarios. Also I know some guys working in our ITOG department are looking into Cassandra, which is a distributed database management system published open source by Facebook - interesting. For those of you interested in No-SQL solutions, I can recommend the Berlin Buzzwords conference, which I am going to attend. It is a 2 days conference in June in Berlin and is organized by Isabel Drost, co-founder and committer at the Apache Mahout project.

First blog post after a little pause

Reik | 12 March, 2010 | 2 comments

First blog post after a little pause. I came back from the US last week, still feeling a bit displaced. Anyway I started in a brand new team called Data Storage and Statistics. Looking forward to new areas and applications after having worked with web development for the last 2 years. This week I had the chance to evaluate Apache Hadoop for a little bit. Hadoop at it's core is a distributed file system and a framework for parallel data processing. So it is definitely interesting for my new team. A lot of companies like Facebook or Yahoo use Hadoop in production. However, the online documentation is really bad or even not existent at all. Also version 0.20 of Hadoop brought a number of big API changes, which makes it even harder to follow because all the samples and tutorials have not been updated. Good thing I had some e-books for Hadoop at hand.

Reading Hadoop in Action right now on my Nook. I bought this e-book reader in New York three weeks ago and I am still very happy about it. Unfortunately PDF documents do only look great when the whole page is displayed at the screen at once, which makes it hard to read because in that case the font will be really small. Fortunately some of the e-books are published in much better formats than PDF. O'Reilly and Manning to some extend publish all their e-books in .epub and .mobi format, which the Nook can display. Reading a book in .epub format looks awesome. Hopefully Apress starts to sell their e-books in other formats than just PDF at some point.

There is a free software called Calibre, which can convert from and to PDF and other formats. However, the results for converting an existing PDF document into .epub or .mobi were not so great. It is basically loosing all formatting of the original e-book which makes reading impossible or at least not a very nice experience.

Top Code Session

Reik | 12 February, 2010 | 0 comments

 This week I participated in Top Code, which is an internal training program for developers in my department at bwin Games. The session took place Monday and Tuesday at the Nordic Light Hotel. I thought this is the Hotel where the ice bar is. It caused a little bit of confusion on Monday morning when I went in there, pretending that the training was supposed to be there and they could not find bwin Games AB on the list. Anyway, I learned that the icebar hotel is called Nordic Sea Hotel and the Nordic Light Hotel is just around the corner.

The Top Code session is presented by two consultants who work here in the Poker client department of bwin Games. There was only little code we looked at. The focus was more on improving the processes before we implement but also how to ensure that our software is of high quality. We got a recap of UML, OOP principles and software patterns. I felt it was very nice to be reminded about patterns that I had read about a long time ago but not used so often in practice. Between the slides, we had a couple of exercises and I thought it was nice to use a common terminology and a notation like UML to attack a problem.

During the second day we talked about code reviews, unit-testing and Design by Contract. Personally I thought the one and a half hour covering Design by Contract was wasted. In general DBC is a great idea. Unfortunately it is not built in into the Java language. You are left with the option to create your own DBC implementation or use one of the existing frameworks. In Top Code however, we were only introduced to DBC but nothing more. No advisory if we really should use DBC in practice or not. No concrete guide how to implement DBC into every day's code. I decided not to use it for now.

Top Code was a great training, especially since it did not focus so much on coding in specific.

ipad misery

Reik | 29 January, 2010 | 0 comments

Yesterdays disclosure of Apple's new toy the iPad, did not come at a very good time for me. In two weeks I will go on a 2 week vacation in New York and Las Vegas. One of the things I had on my personal "must-buy" list, was an eBook reader. I read a lot of books, mostly on paper. However, some computer books are already available as digital early access versions before they are even printed on paper. Recently I bought "Hadoop in Action" and "Mahout in Action" from Manning as early access versions in PDF format. Both books will be released later in 2010. I will definitely buy early access copies of "Taming Text" and Lucene in Action 2nd Edition" too.

Anyway, the eBook reader I was going to buy is the Nook from Barnes & Noble's. The Nook came second in the Top10 gadget of the year 2009 list of the Time magazine. I think it is much nicer than the Kindle from Amazon, as it has a color display, native PDF support which only the expensive Kindle DX has if I am not mistaken - and many more. So I spent the last the last weeks looking forward to reading these great books on a great device. Now Apple announces this ridiculously nice looking iPad, which will not be available until April. This device is like the best eBook reader you can imagine plus a lot more. The price starting at around $500 which is not even twice as much as you pay for the Nook. I am not really sure what to buy anymore.

For those of you interested in early access versions of the Manning Publication books, you can use the following coupons to get a discount. "j2035" is 35% off when ordering for $25 or more. "j2042" same but 42% off and $50, finally "j2047" 47% off and $100. Have fun.

Java lacking blocking Executor

Reik | 22 January, 2010 | 0 comments

Last week I fixed a problem which we found in production in our generic storage application. The application is used, to store vast amounts of data in the Filesystem over two Linux HA nodes. The number of files on each node varies, but it must be about 40 million files we store on each instance. We had a “little” tool, which you can use to roll files. Pretty much in the same way, like webservers do with their log files. When we ran the tool before Christmas. It would run for a while but die after a couple of hours with an OutOfMemoryException. Nasty!

Unfortunately, this was hard to reproduce or even unit-test. Even the worst P2P file-sharer might not have 40 million files at hand. So we started looking at the code. The traversal of the directories, is implemented using the visitor pattern. Starting at the root, we begin with listing all sub-directories. For each sub-directory, we created a Runnable instance and submit it to a fixed size ThreadPoolExecutor. The directory wrapped inside the Runnable, will become the new root directory. The recursion continues, until we finally find files which we can roll. These files, are then added to a bounded Queue, which the main Thread constantly polls and rolls. So far so good.

It turned out, that traversing the directories created so much Runnables for the ThreadPoolExecutors internal work queue, that memory would run out sooner or later. A thing that people (including us) tend to forget if they use ThreadPoolExecutor.newFixedThreadPool(n), is that even though the ThreadPoolExecutor is limited to a fixed size of n-Threads, you can still put an unlimited number of  Runnables into it's work queue. So what you really want instead, is a ThreadPoolExecutor where the work queue is limited in size and that also blocks the submitting Thread, if the queue is full until space becomes available.

Sounds easy. ThreadPoolExecutor has a constructor, where you can specify your own BlockingQueue like a ArrayBlockingQueue. Unfortunately this worked not as expected. The ThreadPoolExecutor will per default not invoke the blocking put method in the ArrayBlockingQueue. Instead it will use the offer method and, if the queue is full, it will not block the invoking thread but throw a RejectedExecutionException instead. I started looking for alternatives. To implement the behavior we wanted, you are basically left with 3 options. Option 1: You can use a Semaphore in conjunction with the invocation of ThreadPoolExecutor.submit, so that the threads need a permit before they can submit Runnables to the thread pool. Option 2: you instantiate your ThreadPoolExecutor specifying a RejectedExecutionHandler. This handler will be invoked as a saturation measure, whenever a Runnable is rejected because the queue is full. You can use CallerRunsPolicy as RejectedExecutionHandler for instance. This will saturate the RejectedExecutionException by running the given Runnable in the submitting Thread instead of running it in the thread pool. This will slow down execution and give the thread pool time to catch up and make space in the work queue. This is the solution I decided to use in out application. Option 3: is to implement your own RejectedExecutionHandler with a custom rejectedExecution method, which can then mimic blocking behavior, ie. by invoking put on the BlockingQueue instead of offer. There is also a concurrency library for Java called Prometheus, which contains a BlockingExecutor class. However, since the last version is 0.1 RC and I don't know anything about the code quality, I decide not to use this library in our Generic Storage application.

If this blog post gives you headache, I can recommend the excellent book Concurrency in practice by Brian Goetz, who suggested Option 1 above in a IBM developer forum.

After Work Snowboarding

Reik | 15 January, 2010 | 0 comments

This week the snowboard season started for me. A friend and bwin college of mine and I grabbed the boards and went to Flottsbro, a small skiing place in Huddinge - very close to Stockholm. Since last Monday, Flottsbro and some other skiing sites in Stockholm, have extended their opening hours until 10pm. The perfect opportunity to go snowboarding after work, which has never been possible when I still worked in Berlin. We decided to ride in Flottsbro every week from now on, even though the Hill is rather small with 3 or 4 slopes. It can get boring after 2 hours but is a good practice anyways.

I have not checked out the other two sites in Stockholm, Hammarby Backen and Väsjöbacken, yet. They have also extended opening hours to 10pm earlier this week. My favorite area for a one-day trip is Romme Alpin. Also of the same is size and distance is Kungsberget but I have never been there so far. For those of you who are interested, I have put up a list of Swedish ski areas including info about size and distance on my personal blog.

Thoughts from a JBoss training

Reik | 18 December, 2009 | 0 comments

Last week I attended a JBoss training at Redpill Linpro in Solna. In was for JBoss 4.0.3, which we still use as application server in my department. We will probably upgrade JBoss to version 4.2.2 next year. It is a very big step to upgrade the application server version, if you have a lot of applications running inside. You need to have excellent knowledge about all applications and verify that they will still work after the upgrade.

The JBoss training at Redpill was quite interesting. It was a two days course and we were 11 people. Usually the training lasts for an entire week, but we decided to cut scope and only talk about JBoss features which we are actually using in our department. The content quality varied from great, normal to useless. On the first day, we spent 2 hours talking about JMX and MBeans. We learned the "old school" way to write a managed bean. However, since 90% of our applications are Spring based, it would be madness to implement MBeans the old fashion way. Instead you would use the Spring JMX annotations. It would have been nicer, if Spring had been covered in the training, because then the content would have been closer to our application design.

Very good on the other hand, were the training parts about Deployers and Classloading. In these two areas, we are having trouble every now and then. Also useful was the part about Interceptors. We use different Server Interceptors for logging and metrics. However, in the hands-on lab we also implemented a Client Interceptor, which we will never use in practice. Client to server communication is almost every time implemented as a loosely coupled service. The de-facto standard in my department at bwin Games, is that if an application offers a service, the service is implemented via the binary protocol Hessian. So in practice, there is no use for a JBoss client Interceptor, since it does not fit into our current architecture.

On a side note, the Java code presented in samples and exercises during the training, did not have a very high standard. Personally I think this is a big problem. If you hand out Java code to participants in a training, most likely the participants will think that the code is reusable. Example 1: in the Client Interceptor example, someone wrote the statement new String(x+"+"+y). This is a very bad practice, because this will create a new String instance, even if the same x and y have been used previously. When you use new String() the JVM will not use the internal String pool. Example 2: use of the Hashtable class. This is a very old class. Basically it is a HashMap with synchronized access. First of all, I believe synchronization was not needed in the code example. The author just picked any map-like Java Collection at random. Secondly, if you really need synchronization, use a ConcurrentHashMap from Java 5 instead - the performance is light-years better. Further details can be found in this article or book. Bottom line is, any code you present to people, should have a high standard even if it is not directly related to the topic you talk about.

Poker Studios Championship 2009

Reik | 11 December, 2009 | 0 comments

Tonight is the big night of the annual Poker championship in the Studios department. We have a couple of smaller tournaments throughout the year but this one is quite important. About 100 people will be playing the championship tonight. Top three finishers will get some nice prices. The winner will win a trip to Helsinki for 2 persons. Second place will get a 2 persons romantic stay at the Nordic Lights Hotel here in Stockholm and the third winner will be given an iPod Nano.

Unfortunately I don't think I have a realistic change to be amongst the top three. A lot of my co-workers have better Poker skills than I do. However, this gives me the opportunity to play very aggressive from the beginning. There is nothing to loose, no need to play tight. Fingers crossed for tonight. If I am out early, at least there will be plenty of time for beer, food, glögg and pepparkaka.

Being a Software Developer - Stockholm vs. Berlin

Reik | 27 November, 2009 | 2 comments

I have worked here in Stockholm for almost 2 years now. Before that, I worked as a Software developer in Berlin, Germany for 8 years. Today I will share some personal perceptions, how working is different into the two countries. Of course my own experiences can be different, depending on the company in which you work.

During most of my German working life, I was employed in a company called Aperto AG in Berlin. We developed for customers like Volkswagen, Siemens, the German Government or the second German television. Also I worked six months for Herlitz, which must be one of the Germany's biggest office equipment manufacturers.

The differences became already apparent before I got the job here. bwin Games invited me to a rather long job interview of 6 hours. I heard that it is common in Sweden to have 2 interviews at different days instead. Since I was flying over from Berlin, they put it into the same day to save travel cost. First I was interviewed by 2 managers and - after lunch - by 3 developers in senior positions. Compared to Germany, the typical job interview will last between 30 Minutes and 1 Hour and you will only be talking to managers or department leaders, but not to developers. I think the aspect that developers meet candidates to see how good they fit in the team, both from a social and technical side, is not very widespread in Germany. If you are invited to a second interview in Germany, it will already be very concrete, maybe even cover contract details. A second interview in Germany means you are almost guaranteed to get the job.

What we also did in Berlin, was a practical work test where candidates had to programmatically solve a task. However, I found the German test harder as you had to do it at our office in Berlin. The bwin Games work test on the other hand, can be done at home. Personally I think testing candidates in the bwin Games office would give us a better picture about the candidates and - over time - boost the quality of our software.

Another thing I had to learn after I got the job was that Swedes are not as hot tempered in discussions. I guess that is both good and bad. Often when you argue, people can get offended and the team spirit may suffer. On the other hand, a hot but productive discussion can lead to better solutions and help to surface underlying problems. For me personally, it is still hard to stay calm when I am in a situation that I am not happy with, i.e. when I code review very bad code by people in senior developer roles or need to do something which I think is useless.

One good side of Sweden is definitely how we deal with overtime. If we had a project in Germany and a milestone (yes we actually used MS Projects back then) or a deadline had to be made, my project manager would come by and order a weekend shift for the whole team. Two or three times it even happened that we worked through three or four weeks in a row, which was very exhausting. Also in Germany employees do overtime every day for no real reason. Often you even felt guilty that you “already” went home after working 8 ½ hours. For me it is no problem to work longer, or come in on the weekend - but there must be a logical reason to do so.

I guess the reason is that there’s much more competition in the German market. 75 million people compared to 7.5 million in Sweden. A lot of my German co-workers thought they could make a good impression on their boss by working overtime every day.

Overall I must say team spirit and companionship is much better here in Sweden. Of course, when you have light you also have shadow. If there’s a lot of work do be done and not enough time, the German solution might be to add overtime, the Swedish way might be to cut scope. Which one do you prefer?

Collective Intelligence

Reik | 20 November, 2009 | 0 comments

I have never been to a book circle before I started here at bwin Games. Since 2008 the Studios department have book circles twice a year. It is totally voluntary. Each employee can choose from a selection of about 10 books. You pick up to 3 books which you would like to read. All the books are somewhat related to Software development, testing, Poker or project management. The group meets every week for a period of 8 to 10 weeks. Bwin Games is sponsoring the lunch, which is very very nice and probably one reason why the book circles are so popular here.Often people in the circle have not read all chapters, so it is okay to be a bit lazy. The idea is to have a lunch together and talk about the book content. Often this becomes a nice discussion and new ideas are being born.

I was avoiding the book circle in 2008. As a part time and distance student, I have some other stuff to read in April and October, because this is the start of the semester. Also I am reading a lot of other books too, so I did not think I would have enough time to read 2 additional chapters for the book circle. Anyway, the colleague who is organizing the book circle meetings, kept asking me which book I would like to read, so I gave in. I signed up for a book about Facebook applications and one book about Collective Intelligence. Since Facebook development seems to be very unpopular amongst my co-workers and no one else signed up for that book, I ended up being in Collective Intelligence in Action from Satnam Alag. Luckily this turned out to be the greatest book I have read in the whole 2009.

Collective Intelligence is one very interesting aspect in Tim O'Reillys Web 2.0 definition. It is a way to discover and harness intelligence created by the users of your application. We had some great talks how we could harness Collective Intelligence in our Poker software. I wrote a small review about this book in my personal blog. After finishing Collective Intelligence in Action I started looking for other books in this area. Here is a small list of other books I have found that might be worth reading if you are interested in Collective Intelligence.

Programming Collective Intelligence: Building Smart Web 2.0 Applications: this book is a bit older than Collective Intelligence in Action. I have ordered this book but I have not started reading it. I was a bit scared off since all the examples are in Python, a programming language which I have never programmed in.

Algorithms of the Intelligent Web: I looked through this book at Java One when it was brand new. It seem to be a very good book but not on the implementational level. The authors write more about the general concepts, the algorithms and not so much how to use this in a real application. Employees in my department can order books through bwin Games multiple times per year, so next time this one will be mine.

Java Data Mining: this is a book about a Java API presented in Collective Intelligence in Action. I started to read the first 300 somewhat pages but then I gave up. I think JDM as the Java Data Mining API has no future. I think the JDM 1 API is not very well designed, you can easily guess that the API is a few years old already.

Building Search Applications - Lucene, LingPipe, Gate: this is another book I have at home but not read yet. It was actually recommended by a colleague in the same book circle. Part of Collective Intelligence is extracting tags from plain text. Building Search Applications explains very concrete examples how to do this using Apache Lucene, LingPipe and Gate.

How we work

Reik | 13 November, 2009 | 0 comments

Today I will blog a bit about how we work here in the bwin Games IT development department. In the Stockholm office it is all about Poker. Here we develop the Poker platform, which our clients use to offer Poker gaming to their users. The poker players are all playing together on an infrastructure, that we call the Network. The department here in Stockholm which develops the Network is called Studios. I don't have the exact numbers but more than 100 employees work at Studios. To my knowledge, it is the biggest department here in the Stockholm office.

In itself, Studios is again split into several smaller sub-departments, for example Gamedev, Platform or Mobile. In Gamedev, we have a lot of developers working with graphical interfaces programmed in Flex, Flash or Java. Platform is the sub-department which I am working in. Here we develop all the infrastructural applications that you need to offer a Poker platform. Currently we have almost 50 different applications in the Platform department. Think of each application as a puzzle piece. The whole Platform can only work properly, when all pieces are put together.

The puzzle pieces which I am responsible for as Technical Lead, are called Rappo, Backoffice and Generic History. Rappo is an application to create daily reports for our clients. A typical report might contain data such as how many players played in a particular tournament and which player finished in which position. Backoffice is a group name for a group of applications written in Java and PHP. Our customers can use Backoffice to access Poker data in almost real time. Here they can search or alter individual players, check their revenue statistic and other things. Generic History finally is a file based generic data store written in Java, where we keep Poker related data over time. The data store must be very responsive and well performing as we have a lot of data floating around. At peak times, Generic History stores and serves 10 Mbyte per second, almost maxing out a 100 Mbit line – and currently we only use the data store for tournament data and nothing else.

On a organizational level, we have different teams working with the different applications. My team is called the Backoffice team and we work with Rappo and of course the Backoffice applications. Other Platform teams are called Twilight Zone, Transactionsystem Team or Swat. Last week I wrote that we use SCRUM to develop our software. A typical Sprint will start with a planning meeting on a Monday morning. The two product owners for Rappo and Backoffice will show up in a conference room with a handful of printouts. Each paper contains a feature request, an improvement or a bug fix. Before the planning meeting with the whole team, I have already selected and estimated the Sprint content together with the two product owners. We have removed all obstacles and answered the open questions – at least most of the time :) In the planning meeting, the team will go through every paper (in SCRUM called a backlog entry) compiling a list of activities with time estimations. When we have collected enough hours to fill the whole 3 weeks, we stop the meeting and put everything up on our Sprint wall.

The typical “in-Sprint day” starts with a morning meeting sometime between 9am and 10am. You talk about progress, problems and pick the activities, which you would like to work on throughout the day and start. The team can choose how to implement a certain requirement, which is great. We can choose the technology, the approach, the design as long as it fits into the overall technical roadmap. A lot of freedom how to develop, I like this.

Chasing to be a better team

Reik | 6 November, 2009 | 0 comments

Premiere. This is my first blog post for bwin Games. It is now almost two years since I moved to Sweden and started to work for bwin Games here in the Stockholm office. Originally, I come from Berlin Germany, where I worked as a Java Developer and System Architect for 8 years. When I decided it was about time to change job, I saw this ad from bwin Games on Monster. I liked Sweden and I liked Java - so this was the perfect position for me. They hired me and I started in January 2008 as Technical Lead in the Platform department. Platform is a sub-section of a bigger department called Studios, I will probably write more about our organization later. Personally I like Snowboarding, Swimming, American Football and Sportsbetting. If you want to know more about me, you can read my personal blogs where I write about Java development, Moving to Sweden and Sportsbetting.

We here at Studios develop using a methodology called SCRUM, which seems to be very popular in agile software development teams all over the world. We have been using SCRUM since I started in 2008 but since mid 2009, we have re-organized our way of working a little bit. For the application I am responsible for, projects and a project team was only set up, when we did new feature or change development. Each project has a name and the team worked together for a couple of Sprints. Now we are organized in, more or less, fixed teams which develop together for the whole year. The content of each Sprint is now being selected from one big backlog, that is being fed continuously by project management. So instead of developing features and changes for a limited time only, the team stays together and the backlog is not only being maintained as long as the project is ongoing.

This new way of working felt a bit strange in the beginning. There was no real “ending” in sight anymore. New items are being added to the backlog every week. I missed (and still miss) this “done” feeling that you get, when projects are over. Not sure but maybe this was one of the reasons that motivation in my team dropped rapidly in the first weeks after switching to the new scheme. After failing in some sprints, where the sprint goal was not reached, and after being called the “bad attitude” team, we are getting better and better now. A couple of weeks ago, we got a new scrum-master because our previous one had a baby. His name is Tomas. It was very fortunate for us. He brought with him some fresh ideas. Sometimes it is great, if someone from outside joins the Scrum team and you have the possibility, to reflect about the everyday-work situation.

One very productive thing, that we did last week on Tomas’ initiative, was a team workshop. The goal was to define a handful of conduct rules, which we wanted to implement into the team work. Tomas handed out an A4 paper to everyone, which contained a long list of nouns. Each team member had to pick 3 words he found important for his job. It was really cool to see the different views of every team member. Some picked economic safety or friendship. I Anyway, the workshop continued by everyone writing down statements about “what makes up a good team” and “what makes up a bad team”. We got a lot of different opinions on the whiteboard. It soon became apparent that we all had pretty much the same opinions which was interesting.

It is clear that such workshops will not turn things around from one day to another. However, I liked the idea just because it gave me a better picture about the other developers in my team.