3. November 2009

mod_mono Control Panel Extension

Just added a list of URLs to the mod_mono Control Panel (CP).

The Apache module mod_mono has a small control panel. The CP has some (actually very few) control features. You can see how many requests are currently processed, how many are waiting and you can restart the server process (the mono worker process, not the web server).

It can be enabled by

  1. <Location /mono>
  2. SetHandler mono-ctrl
  3. </Location>
Then go to http://your.host/mono. Unfortunately, it only shows how many requests are in work. I was missing information which requests are processed and waiting. I need the URLs. I want to know which ones take much time and which are blocking the server under high load.

So, I extended the CP to show a simple list of currently processed and waiting URLs.
  • First column is the request serial number.
  • The second colum is the processing time in seconds.
  • The third column is the URL with query.
Waiting requests are also listed. This happens only if the max processing limit is exceeded and requests wait (not in the screenshot). The waiting list shows the wait time instead of the processing time.

All data is stored in a shared memory segment.

BTW: mod_mono is programmed in C with not too much structure. No offense guys. I am very thankful for it. Great work. I know, it's open source. I should not complain, but improve it (which I did).

Why did I extend it? 2 reasons:
  1. I believe that an operator of a real service needs more information about what is going on than just the number of requests. We used apache server-status heavily. This is the same for mono.
  2. Flooding the server with dozens of integration tests, which spawn 200 backend requests each, stalled the server. I was afraid, that mon_mono has a problem under high load.
Once I knew the URLs, I could clearly see, that mod_mono is fine. It was just a normal deadlock situation where an frontend request tried to make a backend call via HTTP and could not get a free slot. Processing slots are limited by the thread pool. It can be increased, but there must be a hard limit. Allowing unlimited threads would make the server unusable under load, whereas deadlock situations can be avoided.

All slots are occupied by requests which wait for completion of backend requests, which do not find a free slot. Not a mod_mono issue. The situation has been resolved by splitting frontend and backend into separate mono applications. This is the normal configuration of a real multi-tier system, anyway.

The patch:

  1. % cd mod_mono-2.4.2/src
  2. % wget http://wolfspelz.de/download/mod_mono-2.4.2.patch
  3. % patch < mod_mono-2.4.2.patch
_happy_patching()

Update: the patch has been integrated into mono 2.6

1 Kommentar:

Anonym hat gesagt…

A proper solution for this common "http to localhost" problem would be a request slot limit by vhost. Some sites even implemented this as apache module :-)