Following on from showing how easy it was to create web services in F# with Service Stack, it’s now a good time to step it up a bit and do something useful to showcase some F#’s async goodness!

The final solution is available as a stand-alone HttpListener or an ASP.NET host which I just happen to have deployed earlier on a Linux server at http://www.servicestack.net/ftweetstack/. Here’s a couple example urls to see the stats of all my friends and followers:

http://www.servicestack.net/ftweetstack/followers/demisbellot
http://www.servicestack.net/ftweetstack/friends/demisbellot

Edit Screenshots added:

As I’m now getting rate limited by twitter, the live services have stopped working so I’ve included screenshots of the web service responses, reminiscent of happier times

[format=html]

[format=csv]

[format=json]

Back to the code editor…

The best way to showoff some Async code is to do some IO, and the most interesting IO people do these days is against 3rd Party APIs, and the API nearly everyone is familiar with is Twitter’s, so let’s write some code that calls Twitter’s API!

Stuck with the old

We’ll first start with something simple – a basic HTTP request. Unfortunately F# doesn’t come with hot new networking libs to make life easy for us. Nope we’re stuck with the same Async APIs .NET was born with – and .NETs Asynchronous Programming Model (APM) of using the BeginXXX/EndXXX pattern could very well be the worst async model in use today. But what F# doesn’t fix with new cake, it fixes with sweeteners providing us some nice sugar coating we can use over the top to make these APIs more palatable.

Async.FromBeginEnd(,) takes the BeginXXX,EndXXX async method pair and returns an Asynchronous computation that wraps the Async task for use in F#’s built-in Asynchronous workflows.

Doing something new with it

Taking advantage of F#’s ability to extend existing types, we can add this helper method to the familiar WebRequest type with:

	type System.Net.WebRequest with
	    member x.GetResponseAsync() =
	    	Async.FromBeginEnd(x.BeginGetResponse, x.EndGetResponse)

With this in-place we can build a high level asyncHttp API that downloads the text contents of a http resource asynchronously.

	let asyncHttp (url:string) =
	    async {
	        printfn "downloading: %s" url
	        let req = System.Net.WebRequest.Create(url)
	        let! rsp = req.GetResponseAsync()
	        printfn "processing response..."
	        use stream = rsp.GetResponseStream()
	        use reader = new System.IO.StreamReader(stream)
	        return reader.ReadToEnd() }

What is this async { … } ?

What’s different with this function is it doesn’t actually return a string as the implementation might suggest, it actually returns an Async which is basically says “I wrap an asynchronous computation that returns a string”. Async workflows in F# are surrounded by an async { … } block, inside this block you will occaisionally see a let! statement. The let! statement above which happens to be against the async helper we created earlier is used when calling Async APIs which performs the async task without blocking the current execution, when the task completes it resumes, and executes the remaining logic in the async block.

In code terms this looks like:

	let fetchMe = asyncHttp "http://api.twitter.com/1/users/lookup.json?screen_name=demisbellot"
	printfn "doing other stuff... la, di, da..."
	let json = fetchMe |> Async.RunSynchronously

Which when run in fsi returns:

	doing other stuff... la, di, da...
	downloading: http://api.twitter.com/1/users/lookup.json?screen_name=demisbellot
	processing response...
	val fetchMe : Async<string>
	val json : string =
	  "[{"id_str":"17575623","profile_link_color":"43594A","follower"+[1330 chars]

As illustrated invoking the asyncHttp method doesn’t execute it immediately, instead it returns an async workflow that can be called upon later, in this way it works like the deferred exeuction of yield iterators in C# where the iterator itself is only executed when it’s enumerated, i.e. someone calls .ToList() or in this case Async.RunSynchronously which runs the async task and awaits the results.

Great, we now know how to make an async HTTP request so let’s create a service out of it. To help with error handling I’ve added a wrapper over the raw asyncHttp call to detect and log errors of any fail whales caught and return an empty JSON array muting Exceptions. I’ve also added a join method to join any lists we need to and a short jsonTo alias to make using ServiceStack’s fast Json Serializer easier on the eyes

	let asyncJson url =
	    async {
	        try
	            return! asyncHttp url
	        with
	            | ex -> printfn "Error downloading: %s => %s" url ex.Message; return "[]" }

	let joinWith delim seq =
	    let sb = new System.Text.StringBuilder()
	    seq |> Seq.iter (fun x -> sb.Append((if sb.Length > 0 then delim else "") + x.ToString()) |> ignore)
	    sb.ToString()

	let join seq = joinWith "," seq

	let jsonTo<'a> json = ServiceStack.Text.JsonSerializer.DeserializeFromString<'a>(json)

We also need a model to dehydrate twitter’s response. I’m only interested in Users statistics for this example which I can easily map into an F# Record Type which makes definiing POCO’s effortless.

And as I’m always striving for DRY, readable code, I’ll wrap the async calls into more readable methods. Immediately you can start to see the benefits of F# async workflows as we’re able to compose higher level async APIs together without infecting the calling code, a common problem with many Async APIs.

    type UserStat = {
        mutable id: uint64;
        mutable screen_name: string;
        mutable name: string;
        mutable friends_count: int;
        mutable followers_count: int;
        mutable listed_count: int;
        mutable statuses_count: int }

    let asyncUsers screenNames = asyncJson("http://api.twitter.com/1/users/lookup.json?screen_name=" + (screenNames |> join))

    let usersByNames screenNames = asyncUsers screenNames |> Async.RunSynchronously |> jsonTo<UserStat[]> |> Seq.toList

With the above helpers in place all we need to do is to create the service, which is easy as creating the UserStats Request DTO so we know what to expect then and the actual implementation which just passes the list of names into our above usersByNames function, returning the results as-is:

    type UserStats = { mutable ScreenNames: string; }
    type UserStatsService() =
        interface IService<UserStats> with
            member this.Execute (req:UserStats) = req.ScreenNames.Split(',') |> usersByNames :> Object

After registering the appropriate Route in our AppHost we can now call our service passing in multiple twitter user names:

http://localhost:8080/users/demisbellot,servicestack,dsyme

More Async

Although we’ve created a web service that calls Twitter’s REST API asynchronously, we’re synchronously waiting for it straight after so it’s not doing us much good. The benefits of Async begin to show itself when you’re performing multiple IO calls, which is what will be needed in order to gather the stats of your friends or followers using twitter’s APIs.

Twitter only provides an API to return your followers or friends as a list of user ids, to get the user info, another API call is needed to lookup.json which at most only accepts 100 user ids at a time. Knowing this, the quickest way to fetch the user info for your friends of followers is to send the requests for userinfo asynchronously, in parallel.

The actual API calls given they’re just JSON urls are the easiest to do, just 1 line each. I’ll further wrap them in higher level followerIds and friendIds functions which deserializes the json ids in the response into a list of longs:

    let asyncFollowerIds screenName = asyncJson("http://api.twitter.com/1/followers/ids.json?screen_name=" + screenName)
    let asyncFriendIds screenName = asyncJson("http://api.twitter.com/1/friends/ids.json?screen_name=" + screenName)
    let asyncUserIds userIds = asyncJson("http://api.twitter.com/1/users/lookup.json?user_id=" + (userIds |> join))

    let followerIds screenName = asyncFollowerIds screenName |> Async.RunSynchronously |> jsonTo<uint64[]> |> Array.toList
    let friendIds screenName = asyncFriendIds screenName   |> Async.RunSynchronously |> jsonTo<uint64[]> |> Array.toList

The first piece of functionality required is batching an unbounded list of ids into manageable 100-size chunks. My first attempt at this was short and Linq-y:

	let batchesOf size (sequence: _ seq) : _ list seq =
	    seq {
	        let s = ref sequence
	        while not (!s |> Seq.isEmpty)  do
	            yield !s |> Seq.truncate size |> List.ofSeq
	            s := System.Linq.Enumerable.Skip(!s, size)
	    }

But was later discovered to be fairly slow, given the repetitive call to Skip. Since I’m particularly sensitive to perf in my library functions, I opted to go with a more verbose version, but one that only enumerates the sequence once:

	let batchesOf size (s: seq<'v>) =
	    seq {
	        let en = s.GetEnumerator()
	        let more = ref true
	        while !more do
	        let group = [
	            let i = ref 0
	            while !i < size && en.MoveNext () do
	                yield en.Current
	                i := !i + 1 ]
	        if List.isEmpty group then
	            more := false
	        else
	            yield group }

As third party IO calls are amongst the most expensive things you can do, having a cache is a good idea to reduce unnecessary IO calls. We’ll need a threadsafe collection here since we’ll be reading and writting to it at runtime, and .NET 4′s Generic ConcurrentDictionary handles the task nicely and made easily available to F#:

    let userCache = System.Collections.Concurrent.ConcurrentDictionary<uint64,UserStat>()

With the rest of what we need in place we can focus on the most complicated piece to asynchronously download in parallel the the user info for an unlimited list of user ids in batches of 100. In addition, the function should make use of the userCache only fetching the missing entries before merging the new and cached UserStat’s. Surprisingly this entire behaviour can be achieved in just a few lines of F# code:

    let usersByIds userIds =
		let cachedIds, missingIds = userIds |> List.partition userCache.ContainsKey
		missingIds
		|> batchesOf 100
		|> Seq.map asyncUserIds
		|> Async.Parallel |> Async.RunSynchronously
		|> Seq.map jsonTo<UserStat[]>
		|> Seq.collect (fun xs -> xs |> Seq.map
			(fun x -> userCache.TryAdd(x.id, x) |> ignore; x)) |> Seq.toList
		|> List.append (cachedIds |> List.map (fun x -> userCache.TryGetValue x |> snd))

As there’s a bit going on here I’ll explain what’s happening for each line:

#2: Split the entire list of user ids into the ids we have cached and the ones that are missing.
#3: Group the ids in batches of 100
#4: Create separate async API calls for each of those batches
#5: Call each of the API calls in Parallel and wait till they’re all completed
#6: Deserialize the response into lists of UserStat[] types
#7: Merge the UserStat’s together adding each entry into the cache
#8: Append the resulting entries with the previously cached UserStats

Neat! the hard part’s over now it’s just a matter of exposing them in web services and then we’re done:

    let followers screenName = followerIds screenName |> usersByIds
    let friends screenName = friendIds screenName |> Seq.toList |> usersByIds

    type FollowerStats = { mutable ScreenName: string; }
    type FollowerStatsService() =
        interface IService<FollowerStats> with
            member this.Execute (req:FollowerStats) = followers req.ScreenName :> Object

    type FriendsStats = { mutable ScreenName: string; }
    type FriendsStatsService() =
        interface IService<FriendsStats> with
            member this.Execute (req:FriendsStats) = friends req.ScreenName :> Object

After adding the routes for each service we can now see the stats for all my friends and followers at these two friendly urls:

http://localhost:8080/followers/demisbellot
http://localhost:8080/friends/demisbellot

We can test the above urls live using the deployed web service on servicestack.net:

http://www.servicestack.net/ftweetstack/followers/demisbellot
http://www.servicestack.net/ftweetstack/friends/demisbellot

Because we’re using ServiceStack, viewing these urls in a web browser (or any REST client with Accept: text/html) returns the data in a human readable HTML Report Format. Naturally, the usual XML and JSON formats are available and likewise the CSV format which will load the response data into your preferred spreadsheet program or database. All the different formats are available by appending the format to the url e.g: ?format=csv or by specifying it in the Accept HTTP Header.

http://www.servicestack.net/ftweetstack/followers/demisbellot?format=csv
http://www.servicestack.net/ftweetstack/friends/demisbellot?format=json

Should you feel like angering the REST gods, you can commit heresy by accessing twitters REST-ful API over the SOAP endpoints and WSDLS at /soap11 and /soap12 urls

Download in ASP.NET or HttpListener modes

The complete source code for the above is available in different hosting flavours which can be compiled and run on all Windows/OSX/Linux platforms:

• ASP.NET Host [gist]
• Stand alone HttpListener [gist]

Check the previous post for more information on creating ServiceStack Web Services with F#.

Following on from my previous post where I took a first look at F# through my C# tinted eyes, I looked into some of the advantages of what this versatile language brings to a C# devs table. One of the major benefits I mentioned in passing but failed to explain in detail was it’s exceptionally good async story, which in many respects provides arguably the best async programming development experience to date on any platform.

F# and .NET Web Services

If F# only had to interface with other code as its inputs and outputs, the async world of F# is pristine and beautiful. Such is rarely the case so we’re stuck with having to interface with existing frameworks to provide these additional user facing services. Unfortunately the Web Services story for .NET isn’t pretty, where if for whatever reason you’re forced to use Microsoft-only technologies, you’re left with either having to deal with the WCF problem or extend MVC’s framework and roll your own REST-ful framework-in-a-framework yourself – which although less capable, I’ve witnessed to be a viable alternative providing a better iterative experience than WCF in the wild. Another web framework worth considering along similar lines, but developed in true Open Source spirit is NancyFX.

However if like me you like Web Services and still want to use a framework optimized for the task but consider WCF to be an anti-pattern promoting, over architected abstraction I’m happy to highlight a more elegant option in Service Stack which does the same job as WCF in allowing you to expose XML, JSON and SOAP services (as well as CSV, JSV & HTML formats) but does so cleanly, out-of-the-box, without any configuration required! It’s typed, DTO-first development model gets out of your way providing a friction-free dev experience that encourages the development of clean, best-practice web services – easily consumable from any client.

I should note the latest Web Service offering coming out of Redmond: WCF Web API is not be confused with WCF (even if the naming tries) does a much better job of ‘exposing HTTP and REST’ rather than hiding it like the original WCF framework upon which it is built. This may work better for you, although my preference for Open source solutions favouring light, DRY, typed, performance-focused API’s with the ability to run cross platform keeps me using and continue to recommend and maintain Service Stack for the foreseeable future.

Hello, F# World!

To get started we just need the Service Stack dlls that can be downloaded from GitHub. In the unzipped folder you can spark up your favourite text editor and write your first Hello, World web service which looks like [gist]:

open System
open ServiceStack.ServiceHost
open ServiceStack.WebHost.Endpoints

type Hello = { mutable Name: string; }
type HelloResponse = { mutable Result: string; }
type HelloService() =
    interface IService<Hello> with
        member this.Execute (req:Hello) = { Result = "Hello, " + req.Name } :> Object

//Define the Web Services AppHost
type AppHost =
    inherit AppHostHttpListenerBase
    new() = { inherit AppHostHttpListenerBase("Hello F# Services", typeof<HelloService>.Assembly) }
    override this.Configure container =
        base.Routes
            .Add<Hello>("/hello")
            .Add<Hello>("/hello/{Name}") |> ignore

//Run it!
[<EntryPoint>]
let main args =
    let host = if args.Length = 0 then "http://*:8080/" else args.[0]
    printfn "listening on %s ..." host
    let appHost = new AppHost()
    appHost.Init()
    appHost.Start host
    Console.ReadLine() |> ignore
    0

The rest of the code except for the 5 lines of code needed to create the actual service need only be written once which is just used to setup the AppHost and run it, in this case as a Console Application.

Great, once you copy all that in, and hit save using the filename Hello.fs – you can compile with the 1-liner:

fsharpc -r:ServiceStack.Common.dll -r:ServiceStack.Interfaces.dll -r:ServiceStack.Text.dll -r:ServiceStack.dll Hello.fs

Which cuts you a nice Hello.exe that you can double-click to run on Windows, otherwise run on OSX or Linux with:

sudo mono Hello.exe

And Voila! your stand alone F# self-hosting Web Service is ready to serve you at: http://localhost:8080/hello/FSharp!

Convert into an ASP.NET Web Service

To convert this into an ASP.NET application we just need to do some minor tweaks add the appropriate files to deal with the nuances of having an ASP.NET application. For this we need to venture back to a time before VS.NET templates existed that did all this for us.
Note: All files including HelloAsp.fs, Global.asax and Web.Config files used in the walkthrough below are available on this [gist].

First lets take a copy of the file and call it HelloAsp.fs

1. As we want to create a .NET assembly we wrap the entire script in a namespace adding the declaration on top and indenting the rest of the file

namespace HelloFSharp

2. Change the AppHost from a Http Listener to an ASP.NET AppHost by removing the word HttpListener from AppHostHttpListenerBase e.g:

    type AppHost =
        inherit AppHostBase
        new() = { inherit AppHostBase("Hello F# Services", typeof<HelloService>.Assembly) }
        override this.Configure container =
            base.Routes
                .Add<Hello>("/hello")
                .Add<Hello>("/hello/{Name}") |> ignore

3. Replace starting the AppHost from the command line with initialising it from HttpApplication Application_Start() event, i.e:

    type Global =
        inherit System.Web.HttpApplication
        new() = { }
        member x.Application_Start() =
            let appHost = new AppHost()
            appHost.Init()

4. Create the Global.asax stub to point it to the Global type, i.e:

echo "<%@ Application Inherits=\"HelloFSharp.Global\" %>" >> Global.asax

5. Compile HelloAsp.fs into a library since its now an ASP.NET app instead of a stand-alone executable:

fsharpc -r:ServiceStack.Common.dll -r:ServiceStack.Interfaces.dll -r:ServiceStack.Text.dll -r:ServiceStack.dll --target:library HelloAsp.fs

6. make a bin directory and copy all the dlls into it:

mkdir bin && cp *.dll bin

7. Copy the desired Web.config mapping from Hello World Tutorial

<?xml version="1.0"?>
<configuration>
<system.web>
  <httpHandlers>
    <add path="*" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*"/>
  </httpHandlers>
</system.web>
<!-- Required for IIS 7.0 -->
<system.webServer>
  <handlers>
    <add path="*" name="ServiceStack.Factory" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*" preCondition="integratedMode" resourceType="Unspecified" allowPathInfo="true" />
  </handlers>
</system.webServer>
</configuration>

8. Run the .NET 4.0 version of xsp in the current directory

xsp4

If all went well you should see the following output:

Listening on address: 0.0.0.0
Root directory: /Users/mythz/src/fsharp/fsharpstack
Listening on port: 8080 (non-secure)
Hit Return to stop the server.

and you’ll be able to hit the service again using the same url at http://localhost:8080/hello/FSharp!

You now have a fully-fledged ServiceStack ASP.NET application that can be hosted in any ASP.NET compatible web server e.g:

• IIS
• Apache + mod_mono
• Nginx + MonoFastCGI
• XSP
• WebDevServer.exe

Now that you know how to create functional F# web services, you can read more about the goodies you get for free when using Service Stack, complete with live links to all those juicy XML, JSON, SOAP, HTML, CSV and JSV Formats check out: http://www.servicestack.net/ServiceStack.Hello/

Next – Async Goodness!

With the basics out of the way tomorrow we’ll tackle a more complex service taking advantage of F#’s async and parallel good ness to provide nice high-level aggregate features off twitters public API.

Stay Tuned!

Out of all its features I believe what promotes the least code bloat is the quality of F#’s community (see: The Python Paradox) and its preference to simple, elegant composable solutions.  This is contrast to C#’s abstraction fetish it copied from Java and its relentless need to cater for the lowest common Drag n’ Drop developer and impress them with 5 minute demos. Although it’s a subject for another post, this rarely leaves us with quality frameworks or APIs.

Functional vs Imperative

public int Square(int num)
{
	return num * num;
}

public int SumOfSquares(IEnumerable<int> list)
{
	var sum = 0;
	foreach (var num in list)
	{
		sum += Square(num);
	}
	return sum;
}

var nums  = Enumerable.Range(1,100);
Console.WriteLine("Sum of squares of 1-100: {0}", SumOfSquares(nums));

let squares x = x * x
let sumOfSquares nums =
	nums
	|> Seq.map squares
	|> Seq.sum

printfn "Sum of squares of 1-100: %d" (sumOfSquares [1..100])

let sumOfSquares nums = nums |> Seq.sumBy squares

let taxOf salary taxRates =
	((0m,0)::taxRates, taxRates)
		||> Seq.zip
		 |> Seq.map(fun ((_, prevBand),(rate, band)) -> (prevBand, rate, band))
		 |> Seq.sumBy(fun (prevBand, rate, band) ->
			match salary with
				| x when x < prevBand -> 0m
				| x when x > band -> decimal(band - prevBand) * rate
				| x -> decimal(x - prevBand) * rate
		    )

let israelTaxRates = [
	0.10m, 5070;
	0.14m, 8660;
	0.23m, 14070;
	0.30m, 21240;
	0.33m, 40230;
	0.45m, System.Int32.MaxValue]

let taxOfIsrael salary = israelTaxRates |> taxOf salary

//Usage: taxOfIsrael 5800

Wow what a week! it’s been an eventful one to remember: there are Rebellions overthrowing dictatorships in Libya, Earthquakes in Washington with tremors being felt in New York and hurricane Irene swirling around keen on paying us a visit anytime now. All this just setting the stage for the sad news I hoped not to hear for many more years with Mr. Apple (aka Steve Jobs) bowing out on top, relinquishing command as king of the tech world.

Meanwhile all this is happening I’ve undergone some radical lifestyle changes as well with just having just moved to the city of New York after a 2 month hiatus from programming, travelling the wrong way round the world from London. I’m now currently settling into my new apartment in Brooklyn whilst just having finished the first day at my new job after having joined the talented team at: http://stackexchange.com!

My new, New York life is largely thanks to the opportunity I received from the eloquent Joel Spolsky. Which was pretty surreal in itself , given I’ve been following and enjoying his entertaining articles on his world famous blog since the dawn of my computing career >10 years ago!

My first call to action at StackExchange will be helping “Stand Up Matt” @clipperhouse and “Mad Typist” @JasonPunyon (my first impressions – sorry guys! ) enhance the new StackOverflow Careers 2.0 site – Something I’ll be looking forward to as It’ll have the potential to help and reach a very large audience. This is in stark contrast to the standard enterprise developer positions I’ve found myself in over the years – developing custom backend solutions that are usually only destined to support a handful of people.

Day 1

With having survived Day 1 (and not having done much), the biggest change I’ve noticed so far is the view. Where back in London I had a quiet desk next to a peaceful, lifeless cemetery where on a winters day I’m greeted with a ghostly calming atmosphere:

In New York that image is a distant memory where sitting next to a window on the 26th floor, neighbouring skyscrapers hit you in the face:

Although this changes very quickly with the bit of rain today, you still get a ghostly picture but instead of snow tipped tombstones – it’s the city of New Jersey!

That’s about it for my first day, I’m told my second will be even better with some in-house Team StackOverflow beer festival on Friday to close off the week!

Since I’m pretty happy with the Dark Visual Studio Theme I’ve been rolling for a long time with I thought I’d share it with the wider .NET world so others can bask in the glow of an eye friendly dark theme when they’re doing their midnight code runs!

I’ve had it for so long that I’m no longer sure where it was originally based, but looking at Scott Hanslemans VS.NET theme gallery it looks like it’s a mix of Rob Conery’s TextMate and John Lam’s Vibrant Ink.
Over the years where I’ve felt the contrast could be improved, I’ve spiced it up with my own unique blend of colours from the pastel famiglia palette.

And since this theme isn’t complete without a name I’m calling it….

Dark Pastel

Here’s a code screenshot:

Some Web.config love:

A splash of HTML:

And the all important download.

.

Enjoy!

Obviously the title of this post doesn’t make sense to be all in same article, yet I’ve still managed to carry an underlying theme interlinking the above topics : -). Effectively it’s fairy forest through the trees stuff where I’m hoping to encourage the pursuit of end user value over and above all else.

Disadvantages of Custom Mediatypes

Although I’m going to focus on Custom Media Types in this instance (since it reflects my most recent public discussion), the essence of my argument can be applied to many other areas, protocols, libraries, patterns, etc being advocated in software development today.

The main problem I have with Custom Media Types, is the same I have with any other technologies asking me to devote more effort to produce to what I ultimately conceive as a less valueable output. Where the only time when it proves to be beneficial over raw canonical data forms is when the consuming client has built-in support for the custom media format saving the integration point since the client already understands how to process the response.

Although this post may contain a critique on certain facets of REST, I’m by no means advocating against it as overall it’s a far superior more elegant approach to designing services than the SOAP and WS-* stack (which I’m happy to see now starting to slowly fade out of existence). In this light I instead consider myself more a devils advocate providing critique on parts which provide minimal value yet require significant more effort to implement. So consider this a public service announcement as due to the taboo nature of REST you’re unlikely to find many posts like this which is partly my motivation for writing it.

There are times when custom mediatypes are useful however what’s not being discussed in these cases (and what I continue to have issue with) is the amount of airtime given to what I consider a small % of applicability and real-world use. This has a tendency to result in a significant body of knowledge flowing around new developers information streams and if they didn’t know any better would think that this is the right way to go about developing web services. What we’re going to end up with are solutions looking for problems and a swath of invented custom formats – more then there deserves to be. Given the unhealthy amount of coverage given to REST frameworks used to flex different parts of REST theory, I fully expect solutions developed just because developers want to obtain full-compliance REST reputation badges implementing non-trivial CONNEG + Custom Media Types multi-hop solutions when far simpler, more direct solutions can elegantly solve the same problem space. Of course I fully expect people to argue that a superior solution can be achieved even with full-compliance of REST HATEOS restrictions, and I simply want to re-iterate for developers to consider, to understand and weigh all options, and that Getting Things Done in the minimum LOC and effort possible is also a worthy goal to strive for, as when we get older, time becomes more precious and being able to hone this skill will ultimately pay dividends later : -)

Web Services should be optimized for Data Formats

In my 10+ years as a backend systems developer, I can count on one hand the number of times when creating custom media types has proven beneficial or was a customer requirement. The majority of my time is spent developing systems with a primary purpose of servicing clients, thick, thin or otherwise. In my web services world view, not all media types are created equal and its the popular data formats XML, JSON, SOAP, CSV, etc  that are ultimately exposed by web services and consumed by its clients. It’s these formats I believe Web Service Frameworks should be optimized for as they provide the most utility, re-usability and value.

As expected my real-world experience is also the primary influence on http://www.servicestack.net where the above data formats are built-in and are configured to work with all your services automatically. Your C# logic remains DRY and is fused with the expected HTTP result providing a free logic block to be able to return any C# POCO or throw any C# Exception. At the same time making all services available on all above data formats. These are the traits I’ve found to be the most productive and ultimately provide a friction-free development experience.

The other notable mediatypes which are actually more prominent than web services data formats are the media types understood by web browsers. By this I mean HTML/CSS/JavaScript /img/vids/etc. The development of web sites have long been the focus for most development platforms where websites have now become the most important deployment target for most new applications. These web formats are first class formats where entire web frameworks are built around and optimized for, which you should be using instead of Web Service Frameworks.

Custom formats are still important, just not of primary importance

Despite my criticisms, I still think the production of custom formats is important when it’s required, and ServiceStack actually has very good support for it with one of the DRY-est API’s you’re likely to see in accomplishing this task. I just don’t personally give prominence to this functionality since it’s not where I believe web service developers should be speding their time in implementing solutions with. Although for the interested this is how ServiceStack implements the VCard Format. Where its non-invasive layered API makes it trivial to support any custom media format that is well supported since its the same integrated pipeline the newer built-in HTML and CSV formats are using.

Note: at the lowest HTTP API level in every web framework (e.g. IHttpHandler, HttpServletRequest, etc) this has always been possible, so all web service frameworks are showcasing its high-level API to achieve this end. Some frameworks make this functionality available using APIs mapping directly to REST concepts, ServiceStack however prefers to make it as DRY, typed and expressive as possible.

Data formats the preferred choice in SOA systems

A key objective in SOA systems and web services today is to servicify and free your data making it as universally available and re-usable as possible. Given these objectives it is common to see SOA systems today make use of HTTP for their transport protocol and utilizing either XML or JSON for their data formats. This is largely because the data formats contain the essence of your data where should they need to at a later date, your clients can use this canonical form of your data and create the custom media types themselves. The same is not true the other way around where if the only format your service provided were custom media formats (e.g. the VCard format) it not only takes much more effort to extract the raw data, but the specific media type will only contain the subset of your contact data as supported by the format. So if your developers only have an option to deliver 1 format for your resource, you’re best to make it a raw data format since the development of other formats can still be achieved by your clients long after the developers have moved on.

In summary you should only be accessing the specific VCard format (or other specific mediatype) only if your client already has built-in support for it. Since it requires more effort on the client and server to implement and access otherwise.

It is therefore important to remain vigilant and keep the 80/20 Rule in mind (or in the case of custom media formats the 95/5 rule : -) and treat custom media types as complimentary formats that you should only add in addition to your data formats. When development of your SOA services have finished you want to ensure it’s data is as accessible and re-usable as possible in the formats that provide the most utility to your current and future consumers of your services, i.e. the data formats.

The guiding value light

With the current demise of SOAP, REST is making it’s way back into the forefront of IT minds.
The value of REST has always been available in HTTP/1.1 since its inception but between the marketing and committees pushing their own agendas, the Enterprise development world was largely oblivious to its possibilities as they were taught if your data didn’t come in a XML decorated envelope it’s not a real web service.

Its come-back can largely be attributed to developers and companies ignoring the blind advice of big iron Company’s and instead discovered that they are better able to provide value by embracing the inherent properties in HTTP rather than the friction-encumbered dev experience of building SOAP web services.

Displacing big iron-backed monolithic standards is hard, it takes courageous companies willing to discard current enterprise teachings who are able to re-think the essence of what they’re trying to achieve.

Focusing on max value and min effort

It’s therefore important we don’t repeat the same mistakes and don’t lose sight of what the value proposition on the technologies we ultimately choose to adopt, can provide. I’m constantly seeing blind embracing of software process, approach or technology where at the most extreme levels its treated akin to a religion, instead of what it should be: productive tools serving a means to an end.

HATEOS

REST at its extreme conformance imposes some hefty HATEOS restrictions which coupled with most Hyper media driven applications fixation on XML, does not leave you with a very productive window to develop in. If you see real-world value in conforming to these restrictions then by all means adopt them, I personally don’t, and since collecting stamps of approvals from the REST community is not my #1 priority, I routinely discard they exist.

JSON vs XML

XML is another one of those funny technologies handed down from standard committees as the holy grail data format.  Unfortunately this only holds true when you don’t have to develop with it, as it’s a poor programattic fit, making it harder to work with than JSON, that’s both more verbose and slower as well. A good reference that explores these differences in more detail is Douglas Crockford’s entertaining talk on Heresy & Heretical Open Source.

XML as a interoperable structured data format has its place, at the same time I’m finding cases where it would be the ideal data format harder to find. I no longer store it in databases, make use of it for efficient messaging or let it go near my mobile or web apps.

Choosing the right technology

The way I defend myself from adopting the wrong technology is to take a cynical view against all promises, hype and marketing materials. I usually form a picture of the easiest way to develop something and use that as a benchmark to compare other solutions against. The onus then becomes on other frameworks to prove their value by either reducing the amount of effort or code required or in someway enhancing re-usability, performance or promote better maintainability by reducing coupling. If it does not provide any realizable real-world value I don’t spend too much time researching it and continue what I was doing before this Hot Tech Tip landed on my lap.

Basically don’t take any advice at face value, understand the benefits and amount of effort to achieve the desired outcome of each approach and choose the one that offers the maximum end user utility given the least effort to implement. If you don’t understand the full value that your architecture/approach/framework is providing, you won’t be able to maximize its benefits and will soon find you’ve inadvertently subscribed to a solution looking for a problem.

So if you’re spending a lot of time fighting with your framework just to try to get it to prescribe to some artificial standard or religion it might be time to explore other approaches/technologies. When evaluating technologies think about the concept of Last principles where you first look at the ultimate end-user result you want to deliver, then look backwards at the tools, technoligies and frameworks that will best meet this end with the least effort and maximum efficiency.

Heavy weight libraries and frameworks

I personally believe that the Java/.NET worlds are plagued with leaky, heavy abstractions that introduce more friction an inertia that they’re trying to solve. The pain in configuring these monstrosities can be attributed to:

“So much complexity in software comes from trying to make one thing do two things.”

– Ryan Singer

Whilst the un-intended, unknown or magic behaviour I frequently see at runtime whilst developing in said frameworks I’m attributing to its poor, leaky abstractions.

Prefer thin abstractions promoting DRY code

In recent years I’ve been more active in proactively shunning heavy frameworks and look to libraries allowing me to write less code and configuration. Configuration is actually a form of inferior code which happens to be more awkward to define object graphs with than your general purpose programming language. What’s left is a fragile, un-typed mess that you can usually only determine correctness at runtime.

I have many reasons to prefer shorter code-bases which follow my general mantra of “You never have to maintain any line of code you didn’t write”. However here a couple of my favourite quotes I’ve collected on the subject that put this more eloquently than I:

“I happen to hold a hard-won minority opinion about code bases. In particular I believe, quite staunchly I might add, that the worst thing that can happen to a code base is size.”

– Steve Yegge

“A programmer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”

– Antoine St Exupery

With that said, my general preference is for light-weight development libraries and frameworks with thin abstractions allowing me to capture my logic’s intent as DRY as possible. Usually this means these libraries have a low artificial construct / concept count whilst remaining expressive at its core.

The rise of Ruby and Literate Programming

Despite its poor runtime performance and no sight of any corporate backing, Ruby has been making deep inroads into developers mindshare simply because it allows developers to be more productive. Although the language itself is quite expressive, I’m attributing it’s large productivity gains on the communities focus on Simplicity, Convention, DRY-ness and it’s ultimate goal of its ability to compress logic into pseudo code that runs.
In this goal, it seems either by accident or in a way that was cleverly conceived by the languages author it appears that much of Rubys libraries follow a deep symmetry to Donald Knuth’s Literate programming which enables programmers to develop programs in the order demanded by the logic and flow of their thoughts.

Code as literature

“Meaning belongs to a text or to a program to the extent that it acts upon intelligence in a particular way.”

We say that a textual work is “literary” if it contains a high density of meaning.

The above quotes are from @jashkenas enlightening talk at Ruby’s SunnyConf 2010 exploring the subject.

Literate programming is an intriguing topic that not only looks to enhance developer productivity but at the same time increases developer enjoyment and satisfaction. I can definitely see its appeal as it seems like a breadth of fresh air especially if you spend your time fighting heavy frameworks in your day job. Having said that Ruby’s lack of good performance characteristics which I consider important keeps me from using the language in any capacity.

JavaScript’s star new language – CoffeeScript

Luckily CoffeeScript is a hot new language that looks to embrace this vision and since its built on JavaScript, can take advantage of the arms race going on amongst browser vendors in recent years trying to make JavaScript as fast as possible.

Unlike other JavaScript frameworks tackling this domain, it’s not interpreted at runtime and simply compiles 1:1 with efficient JavaScript (in some cases more efficient than a developer would write).

This gives it some unique qualities where it is able to run in a browser without any plugins and thanks to node.js the same CoffeeScript code can run very efficiently on servers as well.

We’re now starting to see this powerfull and expressive combination powering new servers like Ruby’s new Pow Rack server for OSX.
Its recent inclusion in the default Rails install is causing the biggest rift I recall in recent times for a single line of config change.

So while everyone may not be in love with it yet, they at least have an opinion on it

What’s yours?

The rise and fall of Internet riches

Within the last 15 years on the rise of the Internet we have seen many a phenomena gracing technology circles promising great riches to early adopters and investors. It first started out with a gold rush land grab for domain names where in this era a 3-letter domain name like sex.com was able to earn millions a year just by providing a holding page hosting adult-related advertising banners and was eventually sold for 14M.  Then came the dotcom boom where anything Internet related were the most sought after prize on the stock market where traffic and the size of your user base was the hard currency of the day and was rewarded with over-inflated valuations never mind that many had no visible path to profitability. The preferred sport of many startups at the time was who had the highest burn-rate which were fueled by expensive Aeron chairs and Sparc servers.

The Bigger They Come, The Harder They Fall

But just as quick as the riches came to dotcom companies, they vanished again almost entirely within one weekend (which this author remembers quite well as an eager investor on the friday before black monday). Eventually the market corrected itself and only businesses with viable business models were the focus of cautious investors whose fingers were still smarting from the bursting of their dotcom investments.

In time faith was restored in the online advertising market spurred largely by Google’s foray into the area of targeted advertising. Confidence was restored in Internet companies and a smaller bubble is starting to emerge that many casual observers are labelling Web 2.0.

Cometh the App Store, Cometh the riches again

Out of the all the events to show up on the technology radar recently, none has been as exciting and show as much promise as the introduction of the App Store by Apple. With the introduction of their revolutionary smart phone, Apple for the first time put the power of a desktop operating system in the palms of technophiles (with above-average discretionary incomes). Although its potential wasn’t realized until Apple put the first version of the iPhone SDK in developers hands and opened the much publicized App Store on July 10, 2008. Within months of its introduction stories or riches started to be told of hobbyist programmers earning $600,000 in just 1-month enough to quit their full-time job to pursue a lucrative career in developing mobile apps for the App store. Soon enough a new economy was created and in pretty short order a growing number of developers made it to the App store millionaire club. This brings us to where we are today where as of September 1, 2010 there were 250,000 third-party applications with over 6.5 billion total downloads generating its developers over USD $1 billion in revenue.

The reports of App Stores death are greatly exaggerated

With now over 250,000 applications available for sale and many of them free many people have chimed in with their opinions some proclaiming the app bubble is bursting and the path to easy riches is over while others maintain the App store maintains a fruitful future. Unfortunately the ubiquity of the Internet has made opinions cheap and plentiful where it looks like anyone with a keyboard can pluck 2 numbers out of the air to show a downward trend.

An App Store Developers story…

As a developer on the App Store I wanted to weigh in on the state of the App store with some empirical evidence of my own as it evolves. My story dates back 18 months ago when it took me 3 weeks to learn enough Objective-C to submit my first app. Within a couple of weeks, inspired by the success of iFart apps I put aside my morals and pursuit of software excellence and ended up submitting my own cookie cutter apps. Although my own fart apps didn’t do nearly as well as notable others my translator app earned a noticeable side income where it managed to pay off its development effort many times over. Unfortunately around this time I joined my first startup and between the all nighters and 12 hour work days my free time was sapped and my once App making machine days were over. That is until my recent resignation which saw the return of my free time and I was finally able to open xcode again after an 18 month hiatus. After tying up some loose ends on my open source project I got to work on a new App.

How much does an App built today earn?

So that Non-App developers can get an idea of what they can expect when developing an App today, I’m going to be completely open and transparent about the development and the financial results of my latest App that has just been approved today (14th Sept UK). I will keep this blog updated with live financial results as I get them.

Coming up with an idea for an App

Developing a successful app in this age can be a tricky balance to get right where the subject and appeal of your app is every much as important as its features and quality. Because of this, I’m not a true believer of investing a lot of time in developing an App before I see some ROI. So I’m always on the lookout for small, quick wins with short development cycles. It just so happen that at the time I was seeking inspiration for my new app, I also spent a large portion of my free time playing Words with friends – an enjoyable, network-enabled SCRABBLE® clone. Although many would see my poor scrabbling skills the result of a long absence from participating in any academic curriculum, I saw it as an opportunity to build an App to improve my scrabble game! (read: cheat) After a quick sanity check to make sure there weren’t a hundred free apps already available on the App store that did the same thing: Word Mate was born – a collection of word finding utilities that help you play SCRABBLE® like a computer

Development efforts and total cost

Word Mate took me roughly a week full-time to develop which included downloading all the latest SDK’s and tools, a bit of reading designed to re-ignite my love with the Objective-C language and its Cocoa inspired frameworks as well as creating all the App and website graphics. As the website is freely hosted on Google App Engine and the only 3rd party icons used were free, no costs were incurred into making this app other than the $99/year iOS Developer Program fee and my own time.

App Store submission process

Although the language and tools haven’t improved much, the app submission and code-signing process has improved significantly. When it was first released, code-signing your app so it can be submitted to the App store was considered a black-art which thanks to my careful literal following of every word in an online tutorial caused me to be stuck with the embarrassing title of ‘iPhone Developer: Demis Bellot’ for the name of my developer profile. Anyway I submitted my app on the 1st September and it was only approved at midnight on the 13th September. Where the QA staff that Mr Jobs says approves most apps within 7 days are? I don’t know – I’ve never had an any of my apps approved within 2 weeks.

Regardless today will be my first full day on the App Store market. Like all my apps, Word Mate will be free for the first week. I like to do this as it gives a chance for my friends and anyone else who wants one a chance to download my app for free. Making an App for free and then charging for it will also give you an idea of the Free vs Premium download ratio. From the experience of my first app the difference was about 6-9% where for every 100 free apps that were downloaded (when it was free) only 6-9 were paid for (when I started charging USD $0.99 for it). It will be interesting to see if that trends continues with Word Mate.

Live Financial Results

I’m going to maintain a live list of results on the table below, so you can check back on this post to see how it develops. iTunes Connect (where we login to get our sales reports) doesn’t publish daily results until 12pm GMT the next day. You can follow @demisbellot on twitter to find out as soon as the latest results have been added.

*NOTE: Word Mate will be free until 20 Sept 2010, so if you think you might find this app useful now would be a good time to get it for free

Follow the conversation

A discussion thread has started in hacker news which you can get involved in at:

• http://news.ycombinator.com/item?id=1694752

SCRABBLE® is a registered trademark of Hasbro, Inc. in the United States and Canada, and Mattel, inc. elsewhere. I’m not affiliated with either company.

Update 23/09/210

It’s just come to my attention that I may have stuffed up the SEO of my app in the App Store as Word Mate can’t be found with the words ‘Word Finder’ or ‘Word Cheat’  2 of the most popular searches for finding like apps. Unfortunately Apple wont let you change it after you’ve added it so I’m going to have to submit a new app. Unfortunately SEO is just as important as having a quality app as if users can’t find it, they can’t buy it.

These Redis vs RavenDB benchmarks have been made into a bar chart for a better visualization.

Seeing that Redis v2.0 has been just been released and Oren Eini (aka @ayende) has just checked in performance optimization improvements that show a 2x speed improvement for raw writes in RavenDB, I thought it was a good time to do a benchmark pitting these 2 popular NoSQL data stores against each other.

Benchmarks Take 2 – Measuring write performance

For the best chance of an Apples to Apples comparison I just copied the RavenDB’s benchmarks solution project and modified it slightly only to slot in the equivalent Redis operations. The modified solution is available here. Redis was also configured to run in its ‘most safest mode’ where it keeps an append only transaction log with the fsync option so the operation does not complete until the transaction log entry is written to disk. This is so we can get Redis to closely match RavenDB’s default behaviour. Enabling this behaviour in Redis is simply a matter of uncommenting the lines below in redis.conf:

appendonly yes
appendfsync always

To use this new configuration simply run ‘redis-server.exe /path/to/redis.conf’ on the command line.
Other changes I made for these new set of benchmarks was to remove batching from the Redis benchmark since its an accidental complexity not required or useful for the Redis Client.

Here are the benchmarks with these new changes in place:

Which for this scenario show that:

Redis is 11.75x faster than RavenDB

Note: The benchmarks here are of Redis running on a Windows Server through the Linux API emulation layer - Cygwin. Expect better results when running Redis on Unix servers where it is actively developed and optimized for. It is understood that the Cygwin version of redis-server is 4-10x slower than the native Linux version so expect results to be much better in production.

I attribute the large discrepancy between Redis and RavenDB due to the fact that Redis doesn’t use batches so only pays the ‘fsync penalty’ once instead of once per batch.

The ‘appendfsync always‘ mode is not an optimal configuration for a single process since Redis has to block to wait for the transaction log entry to be written to disk, a more sane configuration would be ‘appendfsync everysec‘ which writes to the transaction log asynchronously. Running the same benchmark using the default configuration yields the following results:

Which is a 39% improvement over the previous benchmarks where now:

Redis is 16.9x faster than RavenDB

Which unless I hear otherwise? should make this the fastest NoSQL solution available for .NET or MONO clients.

Measuring raw write performance using Redis is a little unfair since it has a batchful operation MSET specifically optimized for this task. But that is just good practice, whenever you cross a process boundary you should be batching requests to minimize the number of your calls minimizing latency and maximizing performance.

Even though performance is important, its not the only metric when deciding which NoSQL database to use. If you have a lot of querying and reporting requirements that you don’t know up front then a document database like RavenDB, MongoDB or CouchDB is a better choice. Likewise if you have minimal querying requirements and performance is important than you would be better suited to using Redis – either way having a healthy array of vibrant choices available benefits everybody.

Notes about these benchmarks

Since these benchmarks just writes entities in large batches to a local Redis or RavenDB instance using a single client, I don’t consider this to be indicative of a *real-world* test rather a measure is raw write performance, i.e. How fast each client can persist 5,163 entities in their respective datastore.

A better *real-world* test would be one that accesses the server over the network using multiple concurrent clients that were benchmarking typical usage of a real-world application rather than just raw writes as done here.

So why is Redis so fast?

Based on the comments below there appears to be some confusion as to what Redis is and how it works. Redis is a high-performance data structures server written in C that operates predominantly in-memory and routinely persists to disk and maintains an Append-only transaction log file for data integrity – both of which are configurable.

For redundancy each instance has built-in support for replication so you can turn any redis instance into a slave of another, which can also be trivially configured at runtime. It also features its own Virtual Machine implementation so if your dataset exceeds your available memory, un-frequented values are swapped out to disk whilst the hot values remain in memory.

Like other high-performance network servers e.g. Nginx (the worlds fastest HTTP server), Node.js (a popular, very efficient web framework for JavaScript), Memcached, etc it achieves maximum efficiency by having each Redis instance run in a single process where all IO is asynchronous and no time is wasted context-switching between threads. To learn more about this architecture, check out Douglas Crockford (of JavaScript and JSON fame) imformative speech comparing event-loops vs threading for simulating concurrency.

It achieves concurrency by being really fast and achieves integrity by having all operations atomic. You are not just limited to the available transactions either as you can compose any combination of Redis commands together and process them atomically in a single transaction.

Effectively if you wanted to create the fastest NoSQL data store possible you would design it just like Redis and Memcached. Big kudos to @antirez for his continued relentless pursuit of optimizations resulting in Redis’s stellar performance.

The Redis Client,  JSON and the Redis Admin UI

Behind the scenes the Redis Client automatically stores the entities as JSON string values in Redis. Thanks to the ubiquitous nature of JSON I was easily able to develop a Redis Admin UI which provides a quick way to navigate and introspect your data in Redis. The Redis Admin UI runs on both .NET and Linux using Mono – A live demo is available here.

Download Benchmarks

The benchmarks (minus the dependencies) are available in ServiceStack’s svn repo.

I also have a complete download with including all dependencies available here:
http://servicestack.googlecode.com/files/NoSqlPerformance.zip (18MB)

Gaining in Popularity

Redis is sponsored by VMWare and has a vibrant pro-community behind it and been gaining a lot of popularity lately. Already with a library for every popular language in active use today, it is gaining momentum outside its Linux roots with twitter now starting to make use of it as well as popular .NET shops like the StackOverflow team taking advantage of it.

Unlike RavenDB and MongoDB which are document-orientated data stores, Redis is a ‘data structures’ server which although lacks some of the native querying functionalities found in Document DBs, encourage you to leverage its comp-sci data structures to maintain your own custom indexes to satisfy all your querying needs.

Try Redis in .NET

If these results have piqued your interest in Redis I invite you to try it out. If you don’t have a linux server handy, you can still get started by trying one of the windows server builds.

Included with ServiceStack is a feature-rich C# client which provides a familiar and easy to use C# API which like the rest of Service Stack runs on .NET and Linux with Mono.

Useful resources for using the C# .NET Client

I also have some useful documentation to help you get started:
- Designing a NoSQL Database using Redis
+ A refactored example showing how to use Redis behind a repository pattern
- Painless data migrations with schema-less NoSQL datastores and Redis
- How to create custom atomic operations in Redis
- Publish/Subscribe messaging pattern in Redis
- Achieving High Performance, Distributed Locking with Redis

Yep the title of this post is correct! With a little hacking to get around MonoTouch Limitations I’ve managed to shoe-horn Service Stack’s HttpListener and host it inside of an iPhone with the help of MonoTouch. Basically, in not so many words I made this happen:

Basically the above is the result of hosting a web server on your iPhone making it possible to view your iPhone content with any web browser. In a addition to a static HTTP server there is also an Embedded version of Service Stack turning your iPhone into a Web Services server. This happens to be a pretty convenient way to get at your data. As a basic example I created a simple Web Service returning all the contacts in your iPhone Address book:

namespace EmbeddedServiceStack
{
	public class ContactsService : IService<Contacts>
	{
		//Singleton defined in AppHost, Injected by Funq IOC
		public ABAddressBook AddressBook { get; set; }

		public object Execute (Contacts request)
		{
			var response = new ContactsResponse();

			foreach(var person in AddressBook.GetPeople())
			{
				var emails = person.GetEmails().GetValues();
				response.Contacts.Add(new Contact {
					FirstName = person.FirstName,
					LastName = person.LastName,
					Email = emails.Length > 0 ? emails[0] : null,
				});
			}

			return response;
		}
	}
}

Which once you start the AppHost’s HttpListener lets you view your data using REST-ful  HTTP requests like so:

Some of you might be thinking WTF dude why the hell would you want to do this to an iPhone?? Actually I did for a while too! However in a recent thread on the mailing list Miguel De Icaza actually identified some useful use cases where this might be a good idea.

For one reason you can sprinkle a little JavaScript in your HTML page to display the list of Contacts below:

Which granted doesn’t look like much however this task is infinitely less tedious with JavaScript.

Having thought about for a little more I can think of a few more potential use-cases:

• Giving you more flexibility to access and export your iPhone’s data without needing to use iTunes
• Develop PhoneGap-like iPhone apps without PhoneGap
• Use a full-featured desktop browser to view your iPhone data rather than it’s constrained mobile interface
• Access your iPhone from over the Internet (allowing async callback duplex requests)
• Provide easy 2-way communication between iPhone Apps

Other potential use-cases heard on the tweet vine:

• Mobile device management (Landesk like) and Software testing (@itdnext)

Developer Notes

How the Web Server works?

Basically any url with a Web Service prefix is interpreted as a Web Service and delegated to Service Stack to process.

http://localhost:8080/Xml/
http://localhost:8080/Json/
http://localhost:8080/Jsv/

All other urls are treated like a standard HTTP GET request where it looks for a matching file in the Projects www/ resource directory and simply copies it to the Response Output Stream. Requests that ends with a ‘/’ (i.e. a directory path)  are treated like ‘/default.html’

Note: In order to have your static files packaged and deployed with your application you need to set the ‘Build Action’ to ‘Content’ for each file.
Also if you don’t see any contacts you need to use the Contact App in the iPhone simulator and add some in yourself.

MonoTouch Quirks

Since I found the JsonDataContractSerializer in Mono to be unreliable I’ve had to write and included my own Json Serializer whose heavily reliance on generics requires some extra attention when running in an No JIT environment like MonoTouch. Basically we have to tell the MonoTouch compiler what concrete generic classes to generate – which we do by Registering all DTO’s like so:

private void RegisterDtoTypes_RequiredWhenTheresNoJit()
{
	Register.Type<Hello>();
	Register.Type<HelloResponse>();
	Register.Type<Contacts>();
	Register.Type<ContactsResponse>();
	Register.Type<Contact>();
}

Download the Example project

http://servicestack.googlecode.com/files/MonoTouch-EmbeddedServiceStack.zip

Since I only have the trial version of MonoTouch I have only been able to verify that this works in the iPhone Simulator. So if anyone with the full version of MonoTouch gets this running on an iOS device – can you let me know. Feel free to file any issues you run into at the projects website: http://code.google.com/p/servicestack/issues/list

Happy Hacking!

Further Reading

A tutorial showing you how to create and call web services from MonoTouch is available here:
http://www.servicestack.net/monotouch/remote-info/

Confident that I’ve optimized ServiceStack’s JSON web services performance enough with the adoption of my latest efforts developing .NET’s fastest JSON Serializer, I’m now turning my attention towards creating apps that take advantage of it.

I’m a firm believer that performance is one of, if not the most important feature in developing an App that most users will love and use on a regular basis.  It’s the common trait amongst all the apps and websites I regularly use and is why I’m continually seeking software components and/or techniques that can help make my software run faster; or whenever there is no alternative to develop them myself. Although having said this I’m not a complete perf maniac and find that its important to strike a balance between productivity, utility and performance – which is what has effectively kept me tied to C# language for all my server development.

Redis, Sweet Redis

One of the exciting movements that have occurred in recent times is the introduction of NoSQL suite of data persistence solutions. There are numerous impressive NoSQL solutions out there but the one that I have been most interested in is Redis which from the projects website:

is an advanced key-value store. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists, sets, and ordered sets. All this data types can be manipulated with atomic operations to push/pop elements, add/remove elements, perform server side union, intersection, difference between sets, and so forth.

I found this fascinating since it provides an extremely fast data-store (that gets routinely persisted) supporting rich data-structures that can be safely accessed by multiple app servers concurrently since all operations are atomic. Sweet just what I always wanted – although to make it productive I developed a C# Redis Client that apart from supporting Redis’s entire feature-set also provides a high-level typed API that can persist any .NET POCO Type which gets persisted as JSON in Redis.

The Redis Admin UI

One of the disadvantages that comes with making use of a shiny new tech is that there is sometimes not a lot of tooling available for it. Despite its vibrant community this is also true for Redis where although it sports a rich command-line interface (Unix software is good like this) the GUI admin tools are somewhat lacking. Not to worry, I actually needed a project to work on to learn about Google’s closure-library anyway so this ended up being a pretty good fit.

Screenshots

Before we get into more detail its probably a good idea to showcase some of screenshots of where its currently at:
Note: You can also try it out live: http://www.servicestack.net/RedisAdminUI/AjaxClient/

Admin tab showing redis instance info

Aggregate view of complex types

View single complex type

Redis Web Services

In order to be able to access Redis from a web page some JSON web services are in order. I could’ve just implemented the services required by the Admin UI although I wanted to flex some ServiceStack muscle so decided to create web services for all of Redis’s operations which on final count totalled near 100 web services that I ended up knocking out over a single weekend. One of the benefits of using ServiceStack to develop your web services is that you get SOAP, XML, JSON and JSV endpoints for free. So after spending the next couple of days creating unit tests to provide 100% coverage, the back-end was complete – thus giving Redis CouchDB-like powers by allowing it to be accessed from any HTTP client.

Those interested in the Redis Web Services component can check out a live preview – with the complete list of available web services are available here:

http://www.servicestack.net/RedisAdminUI/Public/Metadata

And some examples on how to call them:

Ajax UI

Restrictions and Assumptions

In order to provide a useful Generic UI I’ve had to make a few assumptions on conventions used. Coincidentally these also happen to be the same conventions that the ServiceStack’s C# Redis Client uses when storing data .

1. Keys parts should be separated with a ‘:’
2. Keys within the same group are expected to be of the same type
3. Complex types are stored as JSON

There are likely to be others I’ve subconsciously used so I’ll make an effort to keep this list of assumptions live.

Download and installation

Like the rest of ServiceStack the Redis Admin UI is Open source released under the liberal new BSD licence.

In keeping with tradition with most of my software, the Redis Admin UI works cross-platform on Windows with .NET and Linux and OSX using Mono (Live demo is hosted on CentOS/Nginx).
I’ve had an attempt at some basic installation instructions that are included in the downloaded and viewable online.

The latest version is hosted on Service Stacks code project site at the following url:

https://github.com/downloads/mythz/ServiceStack.RedisWebServices/RedisAdminUI.zip

The Admin UI is highly customizable and very hackable since its written all in Java Script, so if you are interested in customizing the UI for your own purposes I invite you get started by downloading the development version from svn trunk.