6. Oktober 2009

Scripting Configuration for C#

Configuration is a scripting task.

Since I am fed up with the bracket mess of mono/.NET web apps, I was thinking about a more convenient configuration system. Most configuration systems lack flexibility, because they just evaluate a static file and set some values. Some allow for includes, but not much more. But advanced applications need more flexibility, in other words: code. If the config file is a script and has access to the application, then you have all the flexibility you need, and type safety.

And what is the most natural config script language for a C# project? C#, of course.

All you need is a script engine like CS-Script. Add the script engine, create a script host, load the config script file on application start and execute it. The best part is, that the configuration file is type safe and can be edited by the usual development system.

Here are my code bits:

Add the CSScriptLibrary.dll to the project References.

Create a class that implements the script file loader: ScriptedConfig.cs

using System;

namespace MyApp
{
public class ScriptedConfig : MarshalByRefObject
{
public string BasePath { get; set; }

public void Include(string sFile)
{
using (CSScriptLibrary.AsmHelper helper =
new CSScriptLibrary.AsmHelper(CSScriptLibrary.CSScript.Load(
BasePath + sFile, null, true)))
{
helper.Invoke("*.Configure", this);
}
}
}
}
A class that implements my specific configuration object derived from ScriptedConfig: MyConfig.cs:
using System;
using System.Collections.Generic; // just for the sample Dictionary

namespace MyApp
{
public class MyConfig : ScriptedConfig
{
// Just some sample values
public string sText1_ = "empty1";
public string sText2_ = "empty2";

// A sample dictionary to hold more values
public Dictionary<string, string> values_ = new Dictionary<string, string>();
public Dictionary<string, string> Values { get { return values_; } }

public string Text2 { get { return sText2_; } }
}
}
The application startup code:

MyConfig myconfig = new MyConfig ();
myconfig.BasePath = AppDomain.CurrentDomain.BaseDirectory; // For my web app
myconfig.Include("ConfigGlobal.cs");
I have a global config file, that is the same for all installations and a local config file that overrides the global setting for the test syste, the stage system or the production system. This is the ConfigGlobal.cs:
using System;
using MyApp;

public class Script
{
public static void Configure(MyConfig c)
{
c.sText1_ = "Global Config 1";
c.sText2_ = "Global Config 2";

c.Include("ConfigLocal.cs");
}
}
The global config file invokes the local config file, which configures the same object. ConfigLocal.cs:
using System;
using MyApp;

public class Script
{
public static void Configure(MyConfig c)
{
c.sText2_ = "Local Config 2";
c.values_["Text3"] = "Local Config 3";
}
}
You access the config values like:

string t1 = myconfig.sText1_;
string t2 = myconfig.Text2;
Actually, in my current project I separate my code from the implementation of MyConfig by a facade pattern using an interface (IConfig) and in addition access the config instance via StructureMap and a global static wrapper (Config).

It rather looks like:

string t1 = Config.Get("Text1", "Default1");
string t3 = Config.Instance().Values["Text3"];
But that's for later.

_happy_configuring()

2. Oktober 2009

Copying Large Files on Windows

Copying large (>50 GB) files seems to be a problem on Windows (XP). Windows Explorer fails after some time with "Insufficient system resources exist to complete the requested service. (error 1450)". I use GoodSync as a folder syncing program. It has the same problem. I even have a commercial license, because I use it for software deployment on WebDAV and other remote file systems. Suprisingly this commercial file copy program also fails after about 50 GB.

GoodSync support told me "i am not sure why you expect that windows can handle 85 gb file". Good joke. This is the 21st century, guys. Why should it not? Apparently, GoodSync uses the Windows function CopyFileEx. Probably the same as Windows Explorer. I don't know what's wrong with CopyFileEx. There is an old MSDN knowledge base article (kb259837), which admits a problem and continues, that it has been fixed in Windows 2000 SP2. Since Win XP is just an updated Win 2k (XP major version 5.1 compared to 5.0, Vista is 6.0), I suppose the problem is also fixed in XP.

I wrote a simple copy program using the POSIX API (fread/fwrite) which uses Win32 ReadFile under the hood instead of CopyFileEx. Just works. It does as many MB/sec as physically possible, and uses only 1.6 MB memory. So, what's the problem?

Considered my half page of code, CopyFileEx really sucks. The funny part is, that while copying large files in Windows Explorer makes the PC unresponsive and is no fun, my small program copies peacefully in the background. I am so good. Or is CopyFileEx just so bad? Probably the latter. There is nothing special with my code. No fancy tricks, just the basic 1st year CS bachelor stuff.

Lets call it "scopy" for simple copy because all things I do are not stupid, but simple:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{
if (argc != 3) { printf("Usage: %s src dst\n", argv[0]); return 1; }

char const *szSrc = argv[1];
char const *szDst = argv[2];

FILE* fSrc = fopen(szSrc, "rb");
if (fSrc == 0) { printf("Error opening %s\n", szSrc); return 1; }

FILE* fDst = fopen(szDst, "wb");
if (fDst == 0) { printf("Error opening %s\n", szDst); return 1; }

size_t nSize = 1024*1024;
unsigned char* pBuf = new unsigned char[nSize];
if (pBuf == 0) { printf("Error allocating %d bytes\n", nSize); return 1; }

int nCnt = 0;
__int64 nTotal = 0;

bool bDone = false;
while (!bDone) {
size_t nRead = fread(pBuf, 1, nSize, fSrc);
if (ferror(fSrc)) {
perror("Read error"); return 1;
} else {
size_t nWritten = fwrite(pBuf, 1, nRead, fDst);
if (ferror(fDst)) {
perror("Write error"); return 1;
}

nCnt++;
nTotal += nWritten;

if (feof(fSrc)) {
bDone = true;
printf("\n%I64d\n", nTotal);
} else {
printf("%d ", nCnt);
}
}
}

delete[] pBuf;
pBuf = 0;

fclose(fSrc);
fclose(fDst);

//char c = getchar();

return 0;
}

_happy_copying()

1. Oktober 2009

Weblin wiederbeleben mit 2 XML Dateien

Im Forum des Open Virtual World Projekts hat jemand einen Weg gepostet mit dem man Weblin wieder betreiben kann obwohl die Server abgeschaltet worden sind. Basierend auf diesen Informationen und nach meinen eigenen Tests kann ich bestätigen, dass es geht.

Es ist sogar noch einfacher und ohne Programmierkenntnisse.

Man muss nur auf einem öffentlichen Jabber Server einen Jabber Account anlegen (z.B. http://register.jabber.org). Dann auf einem Web-Space ein paar Dateien hinlegen und Weblin konfigurieren, dass es diese Dateien benutzt. Man braucht 2 XML Dateien. Dann natürlich ein Bild als Avatar und je nachdem, ob man ein animiertes Avatar will, noch ein XML mit GIF Animationen.

Schnell mal ausprobieren:

In Registry eintragen...

[HKEY_CURRENT_USER\Software\zweitgeist\Firebat\2.0.0\Config\Operation]
PlatformLoginURL=http://www.ydentiti.org/test/Planta/login.xml
Weblin neu starten ... und du bist Planta. Aber wir wollen ja nicht jeder Planta sein, deshalb...

So geht es:

Die 2 wichtigsten Dateien sind login.xml und identity.xml. Als Beispiele habe ich diese 2 Dateien gemacht und auf meinen eigenen Webserver gelegt:
http://www.ydentiti.org/test/Planta/login.xml
http://www.ydentiti.org/test/Planta/identity.xml
In login.xml steht der URL von identity.xml und man muss den JabberAccount, also Jabber Passwort ("secret"), Jabber Username ("planta") und Jabber Server ("jabber.org") eintragen.

In identity.xml ist ein Link auf das Avatarbild:
http://ydentiti.org/test/Planta/still.gif
Jetzt muss man nur noch Weblin beibringen, dass es das login.xml benutzt beim Starten. Dazu einen Registry Key setzen. Beispiel:
[HKEY_CURRENT_USER\Software\zweitgeist\Firebat\2.0.0\Config\Operation]
PlatformLoginURL=http://www.ydentiti.org/test/Planta/login.xml
Natürlich muss hier der richtige URL von deinem Web-Space stehen. Vielleicht programmiert Marmel ja ein Tool dafür :-)

Ach ja, wo bekommt man die Software her? Einfach von eine(r|m) Freund(in|) den Ordner C:\Programme\weblin kopieren. Kein Installer nötig.

Für Fortgeschrittene:

Da der PlatformLoginURL öffentlich ist, kann jeder dieses Avatar (und den Jabber Account) benutzen. Weblin fragt beim Start zwar ein Passwort. Das wird aber nicht verwendet, da bei PlatformLoginURL ein statischer URL eingetragen ist und nicht eine echte Web-Applikation, die das Passwort prüfen würde. Wenn man nicht will, dass andere das gleiche Avatar verwenden, muss man wie in dem Forum Post vorgehen und statt login.xml ein login.php programmieren, dass HTTP Basic Authentication prüft. Alternativ kann man auch das login.xml durch .htaccess schützen. Vorsicht: dabei nicht Zugriff auf die anderen Dateien verbieten. Man kann das login.xml dafür in ein anderes Verzeichnis legen und den PlatformLoginURL entsprechend setzen.

Für Fortgeschrittene:

Man kann den PlatformLoginURL auch in Firebat.sreg eintragen, wenn man mit leerer Registry startet. Registry überschreibt Einstellungen in Firebat.sreg.

Für Fortgeschrittene:

In identity2.xml ist auch ein Link auf eine Animationsdatei:
http://ydentiti.org/test/Planta/avatar.xml
die auf die GIFs mit den Animationen verweist. Dann hat der Weblin auch Aktionen und bewegt sich.

Notwendig ist mindestens:
<config xmlns="http://schema.bluehands.de/character-config" version="1.0">
<sequence group="idle" name="idle1" probability="1000" type="status" in="standard" out="standard">
<animation src="still.gif" />
</sequence>
<sequence group="moveleft" name="moveleft1" probability="1000" type="basic" in="moveleft" out="moveleft">
<animation dx="-100" src="left.gif" />
</sequence>
<sequence group="moveright" name="moveright1" probability="1000" type="basic" in="moveright" out="moveright">
<animation dx="100" src="right.gif" />
</sequence>
</config>
Zum Start am Besten dieses kopieren und nur eigene GIFs verwenden. Das XML und die GIFs werden im Client gecached. Wenn man was ändert und die Änderung sehen will, dann Client Cache löschen (C:\Dokumente und Einstellungen\%USERNAME%\Anwendungsdaten\zweitgeist\cache).

Format in Kürze:
  • <config> hat mehrere <sequence>
  • <sequence> hat ein <animation>
  • <sequence> hat Attribut "group". Das ist die Aktion, laufen, stehen, schlafen, winken. Es kann mehrer mit gleicher "group" geben, dann sucht Weblin eine der <sequence>s zufällig aus gesteuert durch die relative Häufigkeit ("probability").
  • <sequence> hat Attribut "name". Muss nur eindeutig sei, aber beliebig.
  • <sequence> hat Attribut "probability": Relative Häufigkeit der <sequence> innerhalb einer "group"
  • <sequence> hat Attribut "type" mit Werten "status", "basic", "emote". "emote" tauchen als Aktionen im Avatar-Menu auf.
  • <sequence> Attribute "in", "out" kann man mal ignorieren
  • <animation> hat nur das Attribut "src". GIF URL absolut oder relative zu der XML Datei (mit oder ohne http://...)

_happy_reviving()

PS:
Liebe Dunkle Seite der Macht: Das sind alles öffentliche Informationen, bzw welche, die schon vor 2006 bekannt waren. Keine Chance Ärger zu machen. Weblin ist einfach so gut programmiert, dass sowas geht. Vielen Dank auch an das Programmierteam, dass es trotz neuem Accountwizard so geblieben ist.

25. September 2009

Google Sidewiki

Google launched its new Sidewiki feature. Sidewiki opens an annotation sidebar on every Web page. Sidewiki is a Google Toolbar feature. You can leave notes (annotations) on every Web page and see notes of other people. The annotations appear is a sidebar. This is not the browser sidebar, but an extension driven page element. The feature looks quite advanced. It has rating system integrated, crowd driven spam identification, you can choose to see only annotations in your language. While posting an annotation you can send the same text also to your blogs.

Sidewiki is not real virtual presence, as you do not see the people who are on the page at the same time. Sidewiki is also not a new concept. Annotations have been here before, as early as 1999. But Sidewiki is important for virtual presence, because it shows, that a major player is moving closer. A big advantage of annotations compared to virtual presence is that annotations persist. Chats on a Web page are transient. They are gone after you leave. Annotations stay (for some time). You can leave your mark in the world. That's what people want. They want to create something that persists.

Important for virtual presence is, that Google invests in the notion, that Web pages are places. They are not just an anchor for places, like in Lively. The plain Web page identified by it's URL is a place where things happen.

_happy_wikking()

The usual weblin annotation: when a "great manager" told investors, that virtual goods would not work as a business model, he proposed "annotations". What a disaster would that have been to get Google as a direct competitor. Oh right, the decision was correct, because nobody could have known. I heared that before. Being right because of not knowing seems to be a constant.

24. September 2009

TwiX - Jabber Twitter Gateway Version 0.4.8

I was spamming my twitter account accidentally. Actually it was my Jabber/Twitter gateway, which got an XMPP error message from a broken jabber server and tweeted the error, then sent me a confirmation, got another error message and tweeted it again. Infinite loop!

The main problem is, that the error message looks very much like a good message. That's a Jabber "feature". A good message looks like:

<message from="JID" to="JID">
<body>...Started</body>
</message>
If that produces an error, then comes back:
<message from="JID">to="JID"type="error">
<body>...Started</body>
  <error code="500" type="wait">
  <internal-server-error xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" />
  </error>
</message>
...which looks very much like the original message, if you do not check for the type="error" bit and just forward the body to the twitter API.

Fixed.

TwiX has now been updated to version 0.4.8. Get it here.

_happy_looping()

16. September 2009

ejabberd crash solved by raising number of network connections

I was bothered by ejabberd crashes. Suddenly it started to crash quicky. The crash dumps said something like:

eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "heap")
and:
eheap_alloc: Cannot allocate 934157120 bytes of memory (of type "old_heap").
I tried the installer, tried compiling erlang and ejabberd from source. I usually run ejabber in a virtual server. I tried on the native host. Nothing helped. We had a similar problem before with a memory bug related to mnesia tables where a server with 5000 clients reaches 3 GB in 5 days and crashes. That case was solved by upgrading the erlang runtime from R12B-3 to R12B-5.

But in this case it needed only 5 minutes and 1000 users to suck up the entire system memory. This was different. The number of about 1000 client connections was the constant of all crashes. @zeank suspected that the problem is the limit of open file descriptors, 1024 by default. The open file descriptor limit means also the max number of TCP (client) connections. Setting this to a higer value, e.g. 32.000 solves the problem.

In practice: insert into ejabberdctl
after:
ERL_MAX_PORTS=32000
the line:
ulimit -n $ERL_MAX_PORTS
_happy_ejabbering()

2. September 2009

Twitter Microtagging = Personal Tags

It is time for structure in Twitter. I will now start microtagging my tweets so that followers can filter and read only what they are interested in.

Hashtags are currently used for several purposes:

  1. Global tagging: is done in the spirit of tag-clouds to support global search for topics. example: "Official hashtag for Barcamp Bangkok 3 is #barcampbkk3".
  2. Comment tagging: is a very recent development. People started to add single word comments which support or augment the original tweet in form of hashtags. Example: "Got my iPhone today #happy". Nobody would search for #happy. Its just a funny (and completely valid) mis-use of tagging.
  3. Data tagging: to use Twitter as a data channel for later evaluation. Example: "My current #weight 72.2 kg" with the convention that after #weight follow a number and a unit. Example: "I am now at Los Angeles Airport #geoloc 33.94 -118.40" with the convention that after #geoloc follow longitude and latitude. Rarely used though and probably yet underrated.
  4. Follower phishing: is done by adding as many hashtags as possible to the tweet in the hope to attract attention of followers, specifically auto-followers which are triggered by certain keywords. Example: I once was followed within seconds by @timmendorf when I twittered #helgoland (both are beach holiday resorts).
Hashtags are the community driven Twitter way to give a litte bit of structure. But this is not enough. What we need is a way to filter relevant information from streams. I am tweeting different parts of my life. I tweet about my work, about various projects, about my lunch, about the kids, about fun stuff I find. But, there are followers who are not interested in my kids. My familiy is not interested in technical tweets. Parts of it is always spam for some of my followers. They have no way to filter only the interesting tweets. There should be a way for me to tag my tweets with my own personal tags, so that followers or their client software can select what they read. This is Microtagging: short personal tags.

I will now start to add short tags to my tweets which indicate a kind of "sub-channel" inside my tweet stream. You (or your software) will be able to ignore #dev tweets if you are not interested in my development work. You may ignore #kids tweets if you do not want to know when we are having ice cream in the local zoo. Microtags are personal tags. Everyone can define her own. They are not globally defined. Microtags have only meaning inside a single Twitter account. There might be standards later. Until then read my microtags at microtags.org to learn what they mean.

I will start with:
#life What I do, eat, where I walk and other spam
#me About me, but not the trivial #life stuff
#chat Chats via twitter, which you'd probably ignore if your r not @'d
#fun Fun stuff I find and want to share
#pol Politics (I am trying to avoid this)
#fam Family matters and events
#me About me, but not the mundane #life stuff
#blog New blog posts announced on twitter
#web The Web, Internet including social media, social networks and Twitter
#vw About virtual worlds
#soc Society, including rants about how bad/good the world is
#sci Science, if any
#kids Not yet
#dev Software development and tools
#tec Technical stuff except software development
#ovw That's the project I am currently working on most of the time
#vg My series of Virtual Goods Articles (see http://blog.wolfspelz.de/2009/10/virtual-goods-ticker.html)

A twitter client could have multiple input fields, so that I do not have to choose the microtag for each tweet. A twitter client could automatically fetch the microtags of a new follower from microtags.org and show you a list to select from. My list is at http://microtags.org/tags/?http://twitter.com/wolfspelz. Your twitter client could also fetch my microtags from time to time and offer new choices if they change.

I'd rather make these tags metadata for the tweet and not as part of the text. But since there is not metadata in microblogging, there will now be microtags in the text. Clients can strip them anyway.

_happy_microtagging()

See also: http://www.microsyntax.org but I can't wait anymore.