Neue Wege im IT-Management: Artikel im VentureCapital Magazin 12/2009

Im aktuellen Heft vom VentureCapital Magazin ist mein Artikel über Scrum erschienen (Volltext wegen Copyright nicht frei verfügbar).
Keywords: IT-Management Agile Entwicklungsmethoden Investoren Internet-Startups Softwareentwicklung Pitch Projektsteuerung Scrum Cloud-Computing Open Source Kapitalgeber Time to Market Scrum eliminiert Unproduktivität "The Mythical Man-Month" Frederick Brooks wenige Funktionen Output mit Scrum verbessert Scrum-Prinzipien doppelte Geschwindigkeit Games- und Internetbereich Microsoft, Intel, Yahoo, Accenture, Adobe und GE Healthcare Enfant Terrible Mainstream Jeff Sutherland OpenView Venture Partners Venture Capitals Erfolgschance Investments Qualität Effizienz Due-Diligence Prüfung Impediments Hindernisse Heiner Wolf Gründer CTO virtuelle Welt weblin 3 Mio. User Chief Scientist Open Virtual World Projekt Social Games und Virtual Goods

IT-Projekt der Bundeswehr in der Krise. Wie dumm kann man sein?

Die neue Nachricht: "beim IT-Projekt Herkules der Bundeswehr ufern die Kosten aus".

Eigentlich nicht ganz so neu. "Bereits im Juni hatte ein interner Bundeswehr-Bericht dem Projekt eine verheerende Bilanz ausgestellt".
Aber: Herkules kam in diesem Blog schon vor 5 Jahren vor: Grosse Softwareprojekte in der Krise.

Besonders interessante Fakten:

• IBM, Siemens und der Bund haben für das Projekt ein Joint Venture gegründet
• Seit 5 Jahren ist das Projekt in der Krise, aber Führungskräfte haben auch dieses Jahr Boni bekommen
• Der Bund hat nichts zu sagen, da er nur 49 % der Anteile hält
• Das Projekt "lässt sich inzwischen nicht einmal mehr kalkulieren"
• Ministerium: Grund für steigenden Kosten ist fehlender Wettbewerb

Ist eigentlich schon jemand aufgefallen, dass der Bund 100% zahlt. Bei jedem Drecks-Startup setzen Investoren, die 30 % zahlen, jemand in den Aufsichtsrat. Ab 50 % Finanzierung haben sie gerne die Mehrheit im AR.

Der Auftraggeber hat den Dienstleistern gestattet, eine Abwicklungsfirma zu gründen. Wahrscheinlich hat jemand das dem Bund empfohlen, "um alles zu bündeln". Aber die wichtigste Eigenschaft dieser Firma ist, dass man sie einfach dicht machen kann, falls etwas schiefgeht. Der Bund kann dann nicht mal gegen den Auftragnehmer klagen. Der ist ja dann gelöscht. Siemens/IBM sind fein raus. Eine speziell gegründete GmbH ist DIE Methode, um Verantwortung abzustreifen. Das weiss doch jeder.

"Der fehlende Wettbewerb lässt die Kosten explodieren", hä? Was erwartet man denn, wenn man eine Firma mit der Abwicklung von mehreren Milliarden Euro beauftragt. Wo soll der Wettbewerb denn herkommen?

Wer hat eigentlich diesen Vertrag verhandelt? Das ist ja nicht nur Dummheit, sondern Untreue. Vielleicht sollte man sich mal anschauen wo das Management vorher gearbeitet hat. Ob da nicht jemand aus dem Ministerium rüber gewechselt ist und jetzt gut verdient.

Wie wäre es mit einer parlamentarischen Anfrage: Arbeitet jemand, der auf Seite des Bundes den Auftrag verhandelt hat, jetzt beim Joint Venture oder bei IBM/Siemens?

_happy_moneywashing()

Integration Tests are a Superset of Unit Tests

The agile world is unit test crazy. That's ok. But it is not enough. Integration tests are a much under valued species. I like unit tests. I need unit tests. I do not know how we could ever build software without unit tests. But pure unit tests jump too short. We need more.

I am talking about unit test frameworks like NUnit. You just write the test function, add an attribute and the framework finds the function and adds it to the test list. Your test function tests a real function. Hundereds of those make a complete test set. Great, but not enough. What is missing?

You are supposed to write unit test code, which tests only single functions and methods. The idea is to isolate functionality and test isolated. I know the theory: Interfaces are contracts. There are unit tests for every behaviour. Even failures should be verified. TDD (test driven develpment) write the test as specification. I know all that bloat.

Fact is: these tests are important, but they ignore reality. Most problems result from interdependencies and side effects. In theory, there are no side effects. In reality they are there. Unit testing reduces them. Unit testing gives complex systems a chance to work. Chance is not enough. We must make sure, that systems work, not only functions. That's the part of the integration test.

Integration tests verify complex operations. Example: assumed I have a typical cached business object. An integration test would check the complete process:

• fetch the object from the cache,
• if it's not there, construct it from the DB,
• put it in the cache and
• return it at the same time.
• communication with the DB includes a
• web service interface and 2 layers of
• storage driver and
• SQL access code.

This is a vertical test.

In contrast: unit tests would test everything isolated:

• does the cache access work? with a simulated in-memory cache. Beware of the network inside of unit tests.
• does the database access work? using a fake DB, because relying on a real DB server is uncool, not "isolated" enough
  
• would the webservice return the correct data? using a mock request object and fake data.
• does the webservice format correctly? again: fake data
  
• everything tested with made up configuration data and carefully constructed dependency injection.
  

I am totally pro isolated tests. But they are so un-real, mocked-up, simulated. They need something real on top. They need integration tests. Integration tests assert, that complex systems work despite complexity. We need integration tests. Wenn need lots of them. Unit tests just hope, that everything works together. Dream nicely.

We need integration tests anyway. And integration tests are in the real system. They are live. Unit tests are in a separate project and can not be live. Two separate test sets. Too many for me. There is no reason to split testing into 2 test sets, which are operated very differently. You run the test project from time to time and you are happy about the green bar. I run the real project all the time. Why should I run the test project from time to time, when I can run tests in the real environment with just one click.

Integration tests can be written to test isolated functions. They are a superset of unit tests. My interation tests check isolated functions PLUS vertical functionality PLUS live system health.

Unit test frameworks jump too short. They help, but ignore reality.
Get real. Switch to integration testing.

Update:

Changed the title from "Superior to" to "a Superset of".

_happy_testing()

mod_mono Control Panel Extension

Just added a list of URLs to the mod_mono Control Panel (CP).

The Apache module mod_mono has a small control panel. The CP has some (actually very few) control features. You can see how many requests are currently processed, how many are waiting and you can restart the server process (the mono worker process, not the web server).

It can be enabled by

1. <Location /mono>
2. SetHandler mono-ctrl
3. </Location>

Then go to http://your.host/mono. Unfortunately, it only shows how many requests are in work. I was missing information which requests are processed and waiting. I need the URLs. I want to know which ones take much time and which are blocking the server under high load.

So, I extended the CP to show a simple list of currently processed and waiting URLs.

• First column is the request serial number.
  
• The second colum is the processing time in seconds.
• The third column is the URL with query.

Waiting requests are also listed. This happens only if the max processing limit is exceeded and requests wait (not in the screenshot). The waiting list shows the wait time instead of the processing time.

All data is stored in a shared memory segment.

BTW: mod_mono is programmed in C with not too much structure. No offense guys. I am very thankful for it. Great work. I know, it's open source. I should not complain, but improve it (which I did).

Why did I extend it? 2 reasons:

1. I believe that an operator of a real service needs more information about what is going on than just the number of requests. We used apache server-status heavily. This is the same for mono.
   
2. Flooding the server with dozens of integration tests, which spawn 200 backend requests each, stalled the server. I was afraid, that mon_mono has a problem under high load.
   

Once I knew the URLs, I could clearly see, that mod_mono is fine. It was just a normal deadlock situation where an frontend request tried to make a backend call via HTTP and could not get a free slot. Processing slots are limited by the thread pool. It can be increased, but there must be a hard limit. Allowing unlimited threads would make the server unusable under load, whereas deadlock situations can be avoided.

All slots are occupied by requests which wait for completion of backend requests, which do not find a free slot. Not a mod_mono issue. The situation has been resolved by splitting frontend and backend into separate mono applications. This is the normal configuration of a real multi-tier system, anyway.

The patch:

1. % cd mod_mono-2.4.2/src
2. % wget http://wolfspelz.de/download/mod_mono-2.4.2.patch
   
3. % patch < mod_mono-2.4.2.patch

_happy_patching()

Update: the patch has been integrated into mono 2.6

How to Reactivate Weblin with 2 XML Files

In the forum of the Open Virtual World project someone posted a way to reactivate Weblin. Based on this information and after my own tests I can confirm, that it works.

It is even simpler and works without programming. All you need is a Web space to put some files.

You create a Jabber account on a public Jabber server, (e.g. http://register.jabber.org). Then put some files on your Web space and configure Weblin to use them. You need 2 XML files and an image as avatar. If you want an animated avatar then you need an additional XML and some animated GIFs.

Quick Try:

Enter in the Windows registry ...

[HKEY_CURRENT_USER\Software\zweitgeist\Firebat\2.0.0\Config\Operation]
PlatformLoginURL=http://ydentiti.org/test/Planta/login.xml

Restart Weblin .. and you are Planta. But, we do not want to be Planta, therefore to create your own...

HowTo:

The 2 most important files are login.xml and identity.xml. I have put both as examples on my web server:

http://ydentiti.org/test/Planta/login.xml
http://ydentiti.org/test/Planta/identity.xml

In login.xml is the URL of identity.xml and you have to enter the Jabber account: Jabber passwort ("secret"), Jabber user name ("planta") und Jabber server ("jabber.org").

In identity.xml is a link to the avatar image:

http://ydentiti.org/test/Planta/still.gif

Now you tell Weblin to use login.xml at startup. You set the registry key like this:

[HKEY_CURRENT_USER\Software\zweitgeist\Firebat\2.0.0\Config\Operation]
PlatformLoginURL=http://ydentiti.org/test/Planta/login.xml

Of course, that's the URL of your web space.

Where do you get the software from? Just copy the folder "C:\Program Files\weblin" from a friend. There is no installer required.

For Advanced Users:

Because the PlatformLoginURL is public, anyone can use this avatar (and the Jabber account). The Weblin login screen requests a password. It sends the password, but your web space probably returns the login.xml without checking the password. If you want to protect your login.xml, then you must create a login.php instead of the login.xml. Your login.php can check the HTTP Basic Authentication. You could also protect the login.xml with .htaccess. Attention: .htaccess protects the entire folder. Do not accidentally deny access to the other files. Anyone must be able to fetch your identity.xml and avatar image. You can put login.xml into a separate folder and set PlatformLoginURL accordingly.

For Advanced Users:

identity2.xml has an additional link to an animation file:

http://ydentiti.org/test/Planta/avatar.xml

which links to animated GIFs. This provides your weblin with actions and animations.

Required:

<config xmlns="http://schema.bluehands.de/character-config" version="1.0">
  <sequence group="idle" name="idle1" probability="1000" type="status" in="standard" out="standard">
    <animation src="still.gif" />
  </sequence>
  <sequence group="moveleft" name="moveleft1" probability="1000" type="basic" in="moveleft" out="moveleft">
    <animation dx="-100" src="left.gif" />
  </sequence>
  <sequence group="moveright" name="moveright1" probability="1000" type="basic" in="moveright" out="moveright">
    <animation dx="100" src="right.gif" />
  </sequence>
</config>

For the start copy this file and use your own GIFs. The XML and the GIFs are cached by the client. If you change something, you have to delete the client cache (C:\Documents and Settings\%USERNAME%\Application Data\zweitgeist\cache).

avatar.xml format in short:

• <config> has multiple <sequence>
• <sequence> has one <animation>
• <sequence> has an attribut "group". This is the action, walk, stand, sleep, wave. If there are multiple actions in the same "group", then Weblin searches a <sequence> randomly controlled by the relative "probability".
• <sequence> has an attribut "name". Must be unique, but can be anything.
• <sequence> has an attribut "probability": relative frequency of the <sequence> in a "group"
• <sequence> has an attribut "type" with values "status", "basic", "emote". "emote" appear as actions in the avatar-menu.
• <sequence> attributes "in" and "out" can be ignored. They are for advanced uses with automatic selection of transitions between actions.
• <animation> has only the attribut "src". An animated GIF URL absolute or relative the XML file

_happy_reviving()

Database as a Backend Web Service

The database is always the bottleneck. This is what all the admins of massive services tell in talks about their scaling efforts.

In short:

• Database used to mean SQL
• It is difficult to scale SQL CPU
• It is simple to scale Web-Frontend CPU
• The SQL philosophy puts the burden on Read by enabling very complex SELECTs and JOINs while Write is usually simple with short INSERT. Just the wrong concept in a massive world. We need quick and simple read operations, not complex reporting features.

Therefore many people step back from SQL and use other databases. Read more about the NoSQL movement. You have the choice: CouchDB, MongoDB, Tokyo Tyrant, Voldemort, Cassandra, Ringo, Scalaris, Kai, Dynomite, MemcacheDB, ThruDB, Cassandra, HBase, Hypertable, AWS SimpleDB, or just use Amazon S3 as stupid document store. Also SQL can be 'misused' as quick document/key-value oriented storage. It still has some key benefits.

Basically all you need is a key-value collection store with some indexing, alias document store. Whatever you decide: you are bound to it and this sucks. So, why not decouple the application logic from the database? Decoupling can be done in different ways. Traditionally you had a thin database code layer that tried to abstract from different (SQL) databases. Now, I need more abstraction, because there might well be a non-SQL database in the mix.

I decided to put a web service style frontend-backend separation between application code and database. This makes the DB a web service. In other words: There is HTTP between application and DB which allows for massive scaling. Eventually, my DBs can be scaled using web based load balancing tools. This is great. I can also swap out the DB on a per table basis for another database technology. Also great, because I do not have to decide about the database technology now and this is what this article really is about, right?

So, now I design the DB web service interface. I know what I need from the database interface. This are the requirements:

1. Database items (think: rows) are Key-Value collections
2. Sparse population: not all possible keys (think: column names) exist for all items
3. One quick primary key to access the collection or a subset of key-values per item
4. Results are max one item per request. I will emulate complex searches and multi-item results in the application (disputed by Ingo, see Update 1)
5. Required operations: SET, GET, DELETE on single items
6. Support auto-generated primary keys
7. Only data access operations, no DB management.

This is the interface as code:

1. interface IStorageDriver
2. {
3. // Arguments:
4. // sType: Item type (think: table).
5. // properties: The data. Everything is a string.
6. // names: Column names.
7. // condition: A simple query based on property matching inside the table. No joins. Think: tags or WHERE a=b AND c=d
8. 
   
9. // Add an item and return an auto created ID
10. string Add(string sType, Dictionary<string, string> properties);
11. // returns Created ID
12. 
    
13. // Set item properties, may create an item with a specified ID
14. void Set(string sType, string sId, Dictionary<string, string> properties);
15. 
    
16. // Fetch item properties by ID or condition, may return only selected properties
17. Dictionary<string, string> Get(string sType, string sId, List<string> names);
18. List<Dictionary<string, string>> Get(string sType, Dictionary<string, string> condition, List<string> names);
19. // returns The data. Everything is a string
20. 
    
21. // Delete an item by ID
22. bool Delete(string sType, string sId);
23. // returns True = I did it or False = I did not do it, because not exist, result is the same
24. }

I added the "Add" method to support auto-generated primary keys. Basically, "Set" would be enough, but there are databases or DB schemes which generate IDs on insert, remember?

All this wrapped up into a SRPC interface. Could be SOAP, but I do not want the XML parsing hassle (not so much the overhead). WSDLs suck. Strong typing of web services is good, but can be replaced by integration tests under adult supervision.

On the network this looks like:

Request:

1. POST /srpc HTTP/1.1
2. Content-length: 106
3. 
   
4. Method=Data.Add
5. _Type=TestTable
6. User=Planta
7. Age=3
8. Identity=http://ydentiti.org/test/Planta/identity.xml

Response:

1. HTTP/1.1 200 OK
2. Content-length: 19
3. 
   
4. Status=1
5. _Id=57646

Everything is a string. This is the dark side for SQL people. The application knows each type and asserts type safety with integration tests. On the network all bytes are created equal. They are strings anyway. The real storage drivers on the data web service side will convert to the database types. The application builds cached objects from data sets and maps data to internal types. There are no database types as data model in the application. Business objects are aggregates, not table mappings (LINQ is incredibly great, but not for data on a massive scale).

BUT: I could easily (and backward compatible) add type safety by adding type codes to the protocol, e.g. a subset of XQuery types or like here:

1. User=Planta
2. User/Type=text
3. Age=3
4. Age/Type=int32
5. Identity=http://ydentiti.org/test/Planta/identity.xml
6. Identity/Type=url

The additional HTTP is overhead. But SQL connection setup is bigger and the application is INSERT/UPDATE bound anyway, because memcache will be used massively. Remember the coding rule: the database never notices a browser reload.

Now, I can even use AWS S3, which is the easiest massively scalable stupid database, or Simple DB with my data web service on multiple load balanced EC2 instances. I don't have to change anything in the application. I just implement a simple 4-method storage driver in a single page. For the application it is only 1 line configuration to swap the DB technology.

I can proxy the request easily and do interesting stuff:

• Partitioning. User IDs up to 1.000.000 go to http://zero.domain.tld. The next million goes to go to http://one.domain.tld.
• Replication: All the data may be stored twice for long distance speed reasons. The US-cluster may resolve the web service host name differently than the EU cluster. Data is always fetched from the local data service. But changes are replicated to the other continent using the same protocol. No binary logs across continents.
• Backup: I can duplicate changes as backup into another DB, even into another DB technology. I don't know yet how to backup SimpleDB. But if I need indexing and want to use SimpleDB, then I can put the same data into S3 for backup.
• Eventual persistence:The data service can collect changes in memory and batch-insert them into the real database.

All done with Web technologies and one-pagers of code and the app won't notice.

Update 1:

Supporting result sets (multi-item) as 'Get' response might be worth the effort. I propose to have 2 different 'Get' operations. The first with the primary key and no condition. This will always return at most 1 item. A second 'Get' without pimary key but with condition might return multiple items. (Having both, a primary key and a condition in the 'Get' makes no sense anyway). The multi-item response will use the SRPC Array Response.

On the network:

Request:

1. POST /srpc HTTP/1.1
2. Content-length: ...
3. 
   
4. Method=Data.Get
5. _Type=TestTable
6. _Condition=Age=3\nGender=male
7. _Names=Nickname Identity

Comment: _Condition is a key-value list. This is encoded like an 'embedded' SRPC. A key=value\n format with \n escaping to get it on a single line. _Names is a value list. Tokens of a value lists are separated by a blank (0x20) and blanks inside tokens are escaped by a '\ '. Sounds complicated, but easy to parse and read.

Response:

1. HTTP/1.1 200 OK
2. Content-length: ...
3. 
   
4. Status=1
5. 0:Planta
6. 0:Identity=http://ydentiti.org/test/Planta/identity.xml
7. 1:Wolfspelz
8. 1:Identity=http://wolfspelz.de/identity.xml

I am not yet decided about queries with multiple primary keys. They could be implemented as

1. SRPC Batch with multiple queries in a single HTTP request, or
2. with a specific multi-primary-key syntax, similar to SQL: "WHERE id IN (1,2,3)".

The response would be almost identical, because a SRPC Batch response is very much like SRPC Array Response. Solution 2 adds a bit of complexity to the interface with a new multi-key request field. Solution 1 does not need an interface extension, but puts the burden on the data webservice, which must re-create multi-key semantics from a batch of single-key queries for optimal database access.

Update 2:

I agree with Ingo, that solution 1 (SRPC Batch) makes all operations batchable and has a simple interface at the same time. The trade off, that the webservice must detect multi-key semantics from a batch is probably not too severe. Clients will usually batch ony similar requests together. For the beginning the webservice can just execute multiple database transactions. Later the webservice can improve performance with a bit of code that aggregates the batch into a single multi-key database request.


Update 3:

In order to allow for later addition of type safety and other yet unknown features, I define here, now and forever, that SRPC keys with "/" (forward slash) be treated as meta-data for the corresponding keys without "/". Specifically, that they should not be treated as database (column) names. That's no surprise from the SRPC point of view, but I just wanted to make that clear. I have no idea why someone would use "/" in key names anyway. I find even "_" and "-" disturbing. By the way: ":" (colon) is also forbidden in keys for the benefit of SRPC Batch. In other words: please use letters and numbers, the heck.



Update 4:

I removed the "Database". "Type" is enough for the data service to figure out where to look for the data. "Type" is a string. It can contain "Database/Table".



_happy_decoupling()

Google Toolbar Search History Menu is a Browser

I recently installed a Google toolbar to test Sidewiki. The toolbar replaces the search box by Google's own even if the toolbar is hidden.

Of course, Google has a better search box. It's popup menu has styled text and links. Probably cound even show videos. Guess what, the popup menu is an embedded browser.

The funny thing is, that weblin regards the popup menu as the frontmost browser window and the avatar jumps to the base of the window. As soon as the popup opens while typing, the avatars jump there.

It actually IS the front most browser window and weblin is right. Weblin also correctly calculates the base offset and moves the entire scene to the popup. Great stuff for insiders. Well done by those who worked on the brower positioning code over time.

_happy_popuping()