When you need to get some creative work done after a week with too much work and too little sleep, a change of scenery helps. So I've been trying some completely mobile working.

My new phone (HTC Touch Diamond) works quite well for reading feeds and writing notes and blog posts, with a level of portability you don't have with a larger device.

An onscreen keyboard may not seem ideal for writing longer posts, but any problems there are made up for by being able to start writing without any waiting to get set up. Downloading a better keyboard is a must though. The one that comes with the diamond is a big improvement on the windows mobile default, but TouchPal does better for writing longer posts in full.

Most of the writing I do with OneNote mobile, which is a good text editor with easy synchronisation and the ability to add photos. I don't actually post from my phone, after syncing I post from OneNote on my tablet.

I'd like to be able to post directly but while I have found programs that edit text well and programs that post to a blog well, I haven't found anything that does both well. I don't trust web forms not to delete all my work unexpectedly.

The past week has been fairly intense here, so I haven't had time to blog the latest developments on the startup. As well as having some day job deadlines to deal with, a Friday deadline for a demo resulted in a couple of days working 8am to 5am.

We got a lot done though - finding a name, designing the website and logo, describing the service clearly for the first time, hiring another developer and even some improvements to the code.

Finding a name is one of the hardest tasks for a new startup, and we went through quite a few options before finding something available that works. Even something as obscure as uwigi is taken, though that is probably a good thing given that it is such a horrible name. Fortunately we managed to find a name which is both relevant to the service and correctly spelled - TwoNeeds.

Although I am a very good developer, I am completely hopeless at making a site look nice. The rest of the team is even worse, so we ended up outsourcing to someone in the states. We found the designer through 99designs, though not using the crowdsourcing model - As long as you can find someone competent, a trusted relationship with one person is much more productive.

For the demo itself our main problem was that the service doesn't lend itself to discoverability when there is only one user. We ended up going with a script/screenshot based presentation, implemented in javascript with elements of the real system to make it look less like powerpoint.

And now back to work - I have some rather thorny UI flow issues to sort out.

A web form I am currently working on has a couple of simple requirements:

• The user should be notified as soon as an incorrect value is entered.
• The values entered need to be validated using a complicated set of rules in a back end system, available via a web service.

The first requirement is a perfect fit for a CustomValidator with ClientValidationFunction and the AJAX control toolkit ValidatorCalloutExtender, which is what we are using for simpler things like checking that a number field does not contain letters.

The second requirement doesn't fit in so well - since a database connection is involved it is impossible to do client side, and a server side validation function won't check the value until the form is submitted.

Initially I thought that I could simply call a web service from within the client side validation function. Unfortunately that isn't as simple as it seems, because the ASP.NET AJAX ScriptManager can only call web services asynchronously, while validation methods are synchronous and must set args.IsValid before exiting.

My next idea was to make the web service call synchronous. I found a few useful posts on this, in particular SJAX Call, but most references I found were simply "don't do it". Since adding a custom XMLHttpExecutor is way more complexity than I was looking for, especially for something with potential delay issues for the user, so I decided to try a different approach.

Since making the web service synchronous is both an undesirable solution and too much work, I have gone with making the validation asynchronous with a combination of caching and a custom ValidationResult object.

First, the server side component:

    [WebMethod]

public ValidationResult TestWebServiceValidation(string customValidatorId, string param1, string param2)

{

ValidationResult vr = new ValidationResult();

vr.CustomValidatorId = customValidatorId;

vr.Parameters = param1 + ";" + param2;

vr.IsValid = false;

vr.Result = "This is a test result";

return vr;

}

The Validation result entity is designed to be easily added to a cache, with the key being based on the validator id and the parameters (which will generally be the values of the controls being validated).

Result is an object that can be anything, as long as your client side code knows what to expect.

CustomValidatorId is passed in as a parameter and included in the returned object because otherwise it would be impossible to get the validator that triggered the call without writing a seperate callback method for every validator on the page.

On the client side, we start with the cache:

var ValidationResultCache =
				new Array();
				

function GetValidationResultFromCache(customValidatorId, parameters)
				{
				

for
				(i =
				0; i < ValidationResultCache.length; i++)
				{
				

				var vr = ValidationResultCache[i];
				

				if
				(vr.CustomValidatorId == customValidatorId && vr.Parameters == parameters)
				{
				

				return vr;
				

				}
				

				}
				

return
				null;
				

}
				

This is about as simple as caching gets, so it has potential issues of course, but it at least performs the basic function - returning a ValidationResult without calling the web service if it has previously been called to validate the same values.

Next is the validation function. It checks the cache for a validation result and calls the web service only if an appropriate result in the cache. Note that this will break rather badly if the cache key calculated here is different from the one generated by the web service method.

When the web service needs to be called, IsValid is set to false. This is to ensure that the form cannot be submitted while the web service call is still being processed. An alternative option is to set IsValid to true, but make sure an equivalent server side validator is in place.

Note that there are some things in this function that are part of the larger validtion framework. In particular:

• this function is called with args.IsValid = ValidateDuplicateInvoiceNumber...
• SetError updates the message on both the custom validator and the callout extender
• Message 8 is "{0} is a duplicate invoice number"
• Message 12 is "{0} is being validated - please wait"

function ValidateDuplicateInvoiceNumber(sender, currentRules, value, fieldDescription)
				{
				

				if
				(currentRules[8]
				!=
				null
				&& value !=
				""
				&& value !=
				null)
				{
				

				var result = GetValidationResultFromCache(sender.id, value+";param2")
				

				if
				(result ==
				null)
				{
				

        ValidationTest2.UIService.TestWebServiceValidation(sender.id, value,
				"param2", TestWebServiceSuccess,TestWebServiceFail);
				

        SetError(sender,
				12, fieldDescription);
				

				return
				false;
				

				}
				

				else
				{
				

    if
				(!result.IsValid)
				{
				

            SetError(sender,
				8, fieldDescription);
				

				return
				false;
				

            }
				

          }
				

        }
				

				return
				true;
				

				}
				

The callbacks passed as the last two parameters on the web service call are standard, and can be used for multiple validation web services:

function TestWebServiceSuccess(result, userContext, methodName)
				{
				

//alert("success - "+result);
				

ValidationResultCache[ValidationResultCache.length]
				= result;
				

var cv = document.getElementById(result.CustomValidatorId);
				

ValidatorValidate(cv,
				null,
				null)
				

}
				

function TestWebServiceFail(error, userContext, methodName)
				{
				

alert("Validation web service error - "+error);
				

}
				

When the web service call finishes running, the ValidationResult is added to the cache. The validator that originally called the web service is retrieved using the id in the ValidationResult, and ValidatorValidate is called so that the original validation method (in this case ValidateDuplicateInvoiceNumber) will run again.

When run the second time, the result from the cache is used, and controls can be updated as happens with a standard client side validator.

When creating a new online service, two aspects of the development are particularly important:

• Developing something quickly - if you don't release something you're going to have trouble making any money
• Scalability and good design - if a million users show up overnight, do you have to buy more servers and fix the problem straight away or spend six months redesigning the system from scratch?

In many ways these two requirements are opposites - a system that does one well will not do well on the other. Choose scalable and you never get around to launching. Choose quick development and you get all twitter's problems.

Fortunately it is possible to produce a reasonable compromise between these attributes. The key is flexibility - develop quickly with the easiest toolset, but make sure the system can change easily enough to add scalability when it is needed.

That is why we are starting development using a standard .net web application - SQL Server Express, LINQ, WCF and ASP.NET AJAX.

For much of what we want to do this is far from the ideal platform. The various cloud computing platforms offer massive scalability out of the box, and for the web search portion of the system this architecture is about as far from what google uses as you can get.

What makes up for that is the ease of development - It's relatively easy to find developers who know .net, and once you install visual studio and download the source code it only takes a single keypress to get everything running.

Having chosen the platform based on ease of development, we need to work out how scalability can be built in. There are two parts to this.

First is the completely scalable design, which is what you would build if you had an unlimited development budget and the requirement to support a billion users. You don't have to go into a lot of detail with this, but you need to have some idea of how you would handle the scenario where using a more powerful server is no longer an option. In particular you should look for things that will cause trouble later that might be possible to remove from the design early on.

The other part is the interim solution that lets you grow the service quickly without requiring a complete redesign. This is the part that .NET does really well. Deploying a web application to a new server is trivial, and at the database level (usually the bottleneck when you get to large numbers of users) you can go from a single instance of SQL Express running on a development box to a high performance cluster without needing to change any code.

In planning this part of scaling, you leave out the harder problem of splitting up the database, but there are still design decisions to be made so that the rest of the system can be split across servers easily. Examples in our system include:

• Strong separation of application layers - Using as many distinct modules as reasonably possible means that the application is easily broken up across servers later. For ease of development we are taking advantage of the fact that a WCF service can also be referenced as a regular class.
• As much as possible, operations are stateless - since operations only affect the database, the application server should be easy to duplicate.

Any operations that may take a long time with larger datasets (mostly in the search part of the system) are designed to be asynchronous. While they actually run synchronously within the one process to start with as that is much easier to develop, they have minimal connection to the rest of the code so they can be moved to a separate server if necessary.

Working on my own project, without the usual constraints of detailed requirements and specifications, I get a lot of freedom in the methodology I use.

This means that I can focus efforts a little differently - As long as the available development time is being used productively I don't really care what tasks are being done or how long each one takes.

This approach leads to last minute design and minimal specifications, which isn't as bad as it sounds. With plenty of other stuff to do, spending less time writing documentation suits me just fine, and flexibility of design is usually a good thing.

The one thing you need to be careful with when running a project this way is making sure that your developers can handle minimal specifications. I have in the past worked with developers who thought that the correct response to an unclear requirement was to stare at the screen for three weeks until someone noticed. Fortunately this issue is reasonably easy to avoid. The best way is to only hire perfect developers, but failing that you can do reasonably well just by making sure you know what your developers are doing.

While I don't advocate micromanagement - giving developers the impression they are not trusted is a really bad idea - you should be reading the code that gets checked in. Don't expect the code to be perfect, but look for places where there is a mismatch between what the developer thought you asked for and what you thought you asked for.

Being a couple of weeks into the project, I'm getting a good feel for what works. Initially we tried a more conventional spec, but with very little detail. That works ok to start with, but building the right thing beyond the most basic level requires either more detailed documentation, which would take too much time, or knowledge of the complete system, which you can't expect from someone new to the project.

From there we have been trying a more test driven approach, which has been working quite well. Initially we tried the minimalist approach, creating the test methods but only populating them with comments. That didn't work all that well, since the comments left too much flexibility in how to implement the tests when the developer doesn't know anything about the UI the tests are simulating. It also means needing a thorough review of the test code rather than just running the tests and looking over the code briefly, which is more time we don't have.

The tests written as specs need to be quite detailed. Writing the detailed tests does take more time, but it's not too hard to write them once you get a good template set up. We've been making heavy use of Visual Studio's generate method stub feature, though unfortunately that doesn't work for extension methods, which we are also using a lot of.

Once the tests are set up, getting the right code written is easy, and writing the tests is also quite helpful for working out details of the design that would be overlooked with a written spec. If the tests aren't ready we fall back to the minimal written spec so no development time is wasted, though we do seem to be running out of places that will work.

A couple of days ago I saw a post on Venture Hacks, Free Ideas - Just Add Execution, linking to Mike Speiser's blog. Mike's position is that ideas aren't valuable and execution is everything.

There are a lot of non-valuable ideas out there, and the ideas in Mike's presentation are good examples. However, that doesn't mean that there are not also valuable ideas.

Making an idea valuable is mostly a matter of detail. The idea needs to be thought through enough that execution becomes trivial. If hiring a good team of developers would produce a successful business, the idea is valuable.

Mike's ideas are the sort where hiring developers won't get you anywhere, because there is a nearly impossible step 1 to get past before you can get to the benefits of the idea.

• Community algorithm development on sites with established datasets has a lot of potential, but step one is to have a site with a better algorithm. The better algorithm doesn't exist on day one, and because existing sites are good enough for most users, you won't get enough users to make algorithm improvements worthwhile. It's worth noting that the reason for Google's initial success was not so much the improved algorithm but the uncluttered search page - the better algorithm isn't immediately obvious, but anyone can see a fast loading page and come back for that. Then of course there's the whole evil users thing to deal with.
• Personalizr has different problems - but it's still the same issue of day one value and user adoption. You start out with no value (because no data has been added to the system) and probably bad publicity for privacy issues.
• Support.com is somewhat closer to a valuable idea, although it would take a lot of investment to get the dataset up to a useful standard. I generally don't think much of services that can only make money from advertising, though I do like the idea of targeting ads at people who are annoyed with a particular product.
• The other idea linked in the Venture Hacks post is the shadow market - It's got potential, but there's no way of proving the idea is any better than existing funds management methods until well after launch. A large amount of money is needed to start, which nobody will provide - those who usually invest in managed funds won't give their money to someone with no track record, and VCs, who don't mind the risk, won't be interested because they expect higher returns than you will get even if it goes well.

Fortunately, non valuable ideas can be turned into valuable ideas - basically by adding more ideas. Maybe you come up with a search algorithm that is better for finding support information and you can launch support.com right away without having to put together the data manually. Maybe you find a way to start a managed fund without needing anyone to trust you with lots of money. Maybe you design a service people will pay for that complements the free service that was your original idea.

Once you have that valuable idea execution is the easy part. It may not quite be trivial, but there is at least plenty of advice available on how not to screw it up.

A couple of weeks I posted about hiring a development team in Thailand. Having completed that process, it's now time to write about the results.

Actually getting the job ad posted took a bit more doing than expected - doing business online in Thailand tends to involve phone calls and paper forms more than you might expect. While the "Pay by credit card" bit was easy enough, it ended up being very helpful having someone who can speak the language to figure out that the reason nothing happened after paying was that they wanted to be sent a copy of the company registration certificate.

Having sorted that out, which took several days, we started getting resumes. The quality was about what you would expect anywhere - quite a lot that may or may not be any good and a small number that stand out.

Naturally some of those stand out for the wrong reasons - The one with no experience and qualifications in theatre design, or the one who sent his resume as a 2MB rar file copied to half a dozen other companies.

Most of the rest responded by ignoring the request to apply by email and using the apply button on the job site, which produces a very standard Thai CV. Those are pretty good if you want to hire based on weight or religion, but next to useless for actually determining ability to do the job. I think the standard Thai recruitment process is based on doing a large number of in person interviews rather than early filtering. From what little I could tell from the available information, most of the applications received would probably be worth hiring if we were adding to an established team. However, when starting from scratch you need someone you are absolutely sure can get the job done.

There was actually only one applicant whose technical ability I was really sure about, as he included a link to his technical blog. Unfortunately when we set up an interview via IM, he turned out to speak almost no English.

After that we moved on to interviewing the one remaining applicant I was almost sure about - and ended up making an offer to start the next day.

We actually ended up hiring the least experienced candidate - he finished studying just a couple of weeks ago. After years working with corporate development teams I know just how incompetent people experienced and qualified on paper can be. It is far more important to find someone who really enjoys coding and can learn new stuff - and we seem to have succeeded in that.

While we had been thinking about hiring more developers, we'll probably hold off on that for a while. A few days into the work we're finding that the biggest problem is producing specs fast enough, and having more people writing code isn't going to fix that. Besides, sometimes it seems like hiring a larger team is mostly aimed at making sure there is at least one good developer involved who can get all the work done.

There are a lot of challenges in getting someone else to write your code, both in communicating what you want up front and in making sure that what gets written is what you wanted. This is hard enough when you have a clear requirement like duplicating an existing system, and we are trying to produce something that hasn't been done before. Working with someone 4500 miles away who we haven't worked with before isn't going to make things any easier either.

All this means that it is especially important to get the development environment right from the start. The key requirement is being able to see what progress everyone is making easily, which means a good source control and continuous integration server.

Fortunately, that has become a lot easier since last time I tried to set up something similar. Last time I was using SourceSafe, which isn't the most intuitive product to get set up to run over the web.

First, the simple part - client side setup.

• Visual Studio 2008
• TortoiseSVN
• VisualSVN
• TestDriven.Net

     

This is easy enough, just a matter of running the installers, though of course with a Visual Studio install it still takes most of the day to set up. At least using VS2008 improves things a bit as it is too new to have the excessive number of service packs and extensions you have to install to get VS2005 working.

The server takes a little more work, though not too much. For the most part I was following Rob Conery's excellent post Source Control and Continuous Integration On The Cheap.

Since I already had a windows server available, I just had to connect with remote desktop and run the installers.

• VisualSVN Server
• TeamCity

     

Visual SVN gives you a simple desktop app to create a repository and add users. TeamCity lets you do everything after the install through a webapp.

Adding the project configuration to TeamCity was easy enough, though it did take me a couple of attempts to work out where everything was supposed to go. Where I started running into problems was with getting nunit running. Nunit turns out to be quite sensitive to the version used, which I didn't seem to be able to get the same on both the server and client. The solution I ended up with was including nunit.framework in a solution items folder rather than expecting it to be installed on the build server.

After getting the unit tests working I added ncover integration as well (Integration of NCover into Team City for Tech Head Brothers), but I wouldn't recommend trying that. Getting it working took about 4 hours (longer than setting up everything else). Not only was the end result not really worth it, I ended up turning it off because it was interfering with building locally as soon as I added a failing test. It would probably have worked if I set up a separate build script to run on the server, but I didn't want to have to deal with multiple configurations.

Aside from that, the setup has been quite successful - while actual remote development won't start until tomorrow, I've been using it for a couple of weeks and it's running a lot better than the SourceSafe setup I've been using on another project.

While randomly looking through the incoming searches on my blog, I came across a post from Bamboo Solutions - How to install Windows SharePoint Services 3.0 SP1 on Vista. Hacking the installer to allow SharePoint development without building a VM is something I've had sitting on my list of stuff to try out for about six months, so it's good to see someone has put the solution out there.

Removing the VM requirement is a fairly important part of making SharePoint development easier. Visual Studio is really painful to use if you have to go through remote desktop, and even running locally with Virtual PC has its little issues like not being able to bring up intellisense with alt-right.

Being able to install SharePoint locally is certainly a step in the right direction. I've had all sorts of issues with things like virtual machines not being approved to connect to the network and therefore being unable to use source control or backup properly. Of course, in that sort of environment Vista probably isn't SOE either, but it could make things easier.

However, it's not a perfect solution, mainly in how easy it is to completely screw up a SharePoint installation. Despite the issues with performance and running visual studio, a virtual environment does have a big advantage in being easy to restore to a previous state.

When I'm next working on SharePoint development tools (which may be a while with all the other projects I'm working on at the moment), I'll probably go with a hybrid approach - SharePoint running on a VM and Visual Studio running natively. I have done this before in a limited way, setting up a web service that deploys an uploaded WSP file and calling it from a piece of the WSPBuilder add-in that hasn't made it into the official codebase as yet. The main piece of work left to make that system really workable is setting up the remote debugger.

More recently I have been thinking of another approach that may work better - setting up a continuous integration server alongside SharePoint. It shouldn't be too hard to make WSPBuilder run as part of the build script, and may actually make a shared development server workable.

I've seen a few posts lately covering whether or not you need to know C to be a good programmer, most recently Do you really need to know C? I think so from Darren Stokes.

To a certain extent it seems like people who know C think everyone should know it, and people who don't know C think it's not important. I think the truth is a little more complicated - Knowing C is something likely to show up in good programmers, but isn't in itself that important.

Assume we are in the context of projects that don't use C directly - few would argue that you can work on a C project without knowing C. Think about what knowing C actually means - that you have been programming long enough to have experience from before Java and .net became the dominant mainstream languages and you have continued to learn new things in moving to newer systems.

It's that continual learning, and to a lesser extent the years of experience that make you a good developer. There is something to be said for understanding the memory management stuff that C programming forces you to look at, but that's not really tied to the language either. Again, knowing C is just an indicator of something else - that you are capable of understanding those concepts.

Requiring knowledge of C when hiring is probably a fairly effective way of filtering out the people who would be incapable of being developers at all in C, and therefore incapable of being really good developers in more modern languages. However, you're also filtering out the good developers who just never needed to use it.

This is probably not too serious a problem right now, but will be more of an issue in the future, as fewer people have a reason to learn C. I personally haven't written C (or assembly) code in about 10 years. I know C because I have been coding longer than that, but if I was starting out now I wouldn't bother learning C - there are better ways to learn the concepts now.