Tom Clarkson

Adding Test Failures and TODO Comments to RedMine

Tom Clarkson — Thu, 28 Jan 2010 14:27:00 GMT

Over the last few days I've been trying out Redmine as a replacement for Basecamp, which I have been using for the past year or so. Redmine has a new (and more or less undocumented) REST API for working with issues, which opens up some interesting possibilities for connecting to other systems.

Something I've been wanting to try for a while is better handling of test related tasks and todo comments, which is where the code below comes in - everything ends up in the issue tracker and can be assigned/prioritised as for regular issues.

I have TeamCity set up to build the project and produce an xml test report whenever code changes in subversion. When the test report changes, the sync app gets all tasks of the appropriate type from Redmine (I set up two new trackers, "Test" and "Todo"), compares them to what is found in the code and test report, then updates Redmine as needed.

Note that this is proof of concept, works-on-my-machine code - It shouldn't be too hard to make it do what you want, but there are no guarantees.


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Text.RegularExpressions;
using System.Net;
using System.Xml;

namespace OrangeGuava.TaskSync
{
    enum TaskType
    {
        Test,
        Todo
    }

    class Task
    {
        public int? Id { get; set; }
        public TaskType Type { get; set; }
        public string Filename { get; set; }
        public string Result { get; set; }
        public int Line { get; set; }
        public string Text { get; set; }
        public int Status { get; set; }
    }

    class Program
    {

        static int TRACKER_TODO = 5;
        static int TRACKER_TEST = 4;
        static string ServerUrl = "http://redmineURL";
        static int ProjectId = 1;
        static string basepath = //@"C:\code\twoneeds\";
        @"C:\TeamCity\buildAgent\work\597b948abdcc6ce0\";
        static string login = "TaskSync";
        static string password = "[password]";


        static void UpdateTask(Task t)
        {

            var request = (HttpWebRequest)HttpWebRequest.Create(
                string.Format(
                "{0}/issues/{1}.xml", ServerUrl, t.Id));

            string cre = String.Format("{0}:{1}", login, password);
            byte[] bytes = Encoding.ASCII.GetBytes(cre);
            string base64 = Convert.ToBase64String(bytes);
            request.Headers.Add("Authorization", "Basic " + base64);
            request.ContentType = "application/xml";

            request.Method = "PUT";

            var reqsw = new StreamWriter(request.GetRequestStream());

            WriteTask(reqsw, t);

            reqsw.Close();

            var response = request.GetResponse();
            response.Close();


        }

        static void WriteTask(StreamWriter reqsw, Task t)
        {
            XmlDocument doc = new XmlDocument();
            doc.AppendChild(doc.CreateElement("issue"));
            doc.DocumentElement.SetAttribute("project_id", ProjectId.ToString());
            doc.DocumentElement.SetAttribute("tracker_id", (t.Type == TaskType.Test ? TRACKER_TEST : TRACKER_TODO).ToString());
            doc.DocumentElement.SetAttribute("status_id", t.Status.ToString());

            if (t.Type == TaskType.Todo)
            {
                var fn = t.Filename;
                if (fn.Contains('/')) fn = fn.Substring(fn.LastIndexOf('/'));

                var subj = doc.CreateElement("subject");
                doc.DocumentElement.AppendChild(subj);
                subj.InnerText = fn + ":" + t.Line + " - " + t.Text;

                var desc = doc.CreateElement("description");
                doc.DocumentElement.AppendChild(desc);
                desc.InnerText = t.Filename + "\n" + t.Line + "\n" + t.Text;
            }
            else
            {

                var subj = doc.CreateElement("subject");
                doc.DocumentElement.AppendChild(subj);
                subj.InnerText = t.Result + ": " + t.Filename;

                var desc = doc.CreateElement("description");
                doc.DocumentElement.AppendChild(desc);
                desc.InnerText = t.Filename + "\n" + t.Result + "\n" + t.Text;
            }

            reqsw.Write(doc.DocumentElement.OuterXml);

        }

        static void CreateTask(Task t) {

            var request = (HttpWebRequest)HttpWebRequest.Create(
                string.Format(
                "{0}/issues.xml", ServerUrl));

            string cre = String.Format("{0}:{1}", login, password);
            byte[] bytes = Encoding.ASCII.GetBytes(cre);
            string base64 = Convert.ToBase64String(bytes);
            request.Headers.Add("Authorization", "Basic " + base64);
            request.ContentType= "application/xml";

            request.Method = "POST";
            
            var reqsw = new StreamWriter(request.GetRequestStream());

            WriteTask(reqsw, t);

            reqsw.Close();
            
            var response = request.GetResponse();
            response.Close();

        }

        static List GetTasksFromIssueTracker(TaskType type, int page)
        {
            var result = new List();

            var request = (HttpWebRequest)HttpWebRequest.Create(
                string.Format(
                "{0}/issues.xml?project_id={1}&tracker_id={2}&status_id=*&page={3}", 
                ServerUrl, 
                ProjectId,
                type == TaskType.Test ? TRACKER_TEST : TRACKER_TODO,
                page
                ));
            //request.Credentials = new NetworkCredential("tqc01", "asdf12");
            string cre = String.Format("{0}:{1}", login, password);
            byte[] bytes = Encoding.ASCII.GetBytes(cre);
            string base64 = Convert.ToBase64String(bytes);
            request.Headers.Add("Authorization", "Basic " + base64);

            var response = (HttpWebResponse)request.GetResponse();

            StreamReader reader = new StreamReader(response.GetResponseStream());
            string xml = reader.ReadToEnd();
            response.Close();

            XmlDocument doc = new XmlDocument();
            doc.LoadXml(xml);

            var total = int.Parse(doc.DocumentElement.GetAttribute("count"));
            var pages = (int)Math.Ceiling(total / 25.0);

            var issueNodes = doc.DocumentElement.SelectNodes("//issue");

            foreach (XmlElement issueNode in issueNodes)
            {
                var task = new Task();
                task.Type = type;
                task.Id = int.Parse(issueNode.SelectSingleNode("./id").InnerText);
                task.Status = int.Parse(((XmlElement)issueNode.SelectSingleNode("./status")).GetAttribute("id"));
                //task.Text = issueNode.SelectSingleNode("./subject").InnerText;
                var description = issueNode.SelectSingleNode("./description").InnerText;
                var ds = description.Split('\n');
                if (type == TaskType.Todo)
                {
                    task.Filename = ds[0];
                    task.Line = int.Parse(ds[1]);
                    task.Text = (ds[2]);
                }

                if (type == TaskType.Test)
                {
                    task.Filename = ds[0].Trim();
                    task.Result = (ds[1]).Trim();
                    task.Text = description.Substring(description.IndexOf('\n', description.IndexOf('\n')+1) + 1);
                }
                result.Add(task);
            }


            if (page < pages)
            {
                result.AddRange(GetTasksFromIssueTracker(type, page + 1));
            }
            return result;
        }

        static List GetTasksFromCode()
        {
            var result = new List();
            

            var files = Directory.GetFiles(basepath, "*.cs", SearchOption.AllDirectories);

            foreach (var fn in files)
            {
                //Console.WriteLine(fn);
                var lines = File.ReadAllLines(fn);
                for (int i = 0; i < lines.Length; i++)
                {
                    var l = lines[i];
                    var m = Regex.Match(l, "//.*TODO:(.*)$", RegexOptions.IgnoreCase);
                    if (m.Success)
                    {
                        result.Add(new Task()
                        {
                            Type = TaskType.Todo,
                            Filename = fn.Substring(basepath.Length),
                            Line = i,
                            Text = m.Groups[1].Value.Trim(),
                            Status = 1
                        });
                    }
                }
            }
            return result;
        }

        static void SyncTests(string path)
        {
            var oldtasks = GetTasksFromIssueTracker(TaskType.Test, 1);
            var newtasks = GetTasksFromTestRecord(path);


            Console.WriteLine("Server: " + oldtasks.Count);
            Console.WriteLine("Code: " + newtasks.Count);


            // updates - either no change, status change, or no longer there

            foreach (var ts in oldtasks.ToArray())
            {
                bool found = false;
                foreach (var tt in newtasks)
                {
                    if (tt.Filename == ts.Filename)
                    {
                        // test matches
                        found = true;
                        if (tt.Result == ts.Result && Regex.Replace(tt.Text, "\\W", "") == Regex.Replace(ts.Text, "\\W", ""))
                        {
                            // no change
                            oldtasks.Remove(ts);
                            newtasks.Remove(tt);
                        }
                        else
                        {
                            // update required
                            ts.Result = tt.Result;
                            ts.Text = tt.Text;
                            if (ts.Status == 5) ts.Status = 1;
                            newtasks.Remove(tt);
                        }
                        break;

                    }

                    if (!found)
                    {
                        // test passed or removed
                        ts.Status = 5;
                    }
                }
            }

            Console.WriteLine("New: " + newtasks.Count);
            Console.WriteLine("Updated: " + oldtasks.Count);


            // save updates
            foreach (var t in oldtasks)
            {
                Console.WriteLine("Updating " + t.Filename);
                UpdateTask(t);
            }

            // create new tasks
            foreach (var t in newtasks)
            {
                Console.WriteLine("Adding " + t.Filename);
                CreateTask(t);
            }


        }

        private static List GetTasksFromTestRecord(string path)
        {
            var result = new List();
            XmlDocument doc = new XmlDocument();
            doc.Load(path);
            var testnodes = doc.SelectNodes("//test-case");

            foreach (XmlElement testnode in testnodes)
            {
                if (testnode.GetAttribute("executed") == "False")
                {
                    // ignored
                    result.Add(new Task()
                    {
                        Type = TaskType.Test,
                        Filename = testnode.GetAttribute("name"),
                        Result = "Ignored",
                        Status = 1,
                        Text = testnode.SelectSingleNode("./reason").InnerText                        
                    });
                }
                else if (testnode.GetAttribute("success") == "False")
                {
                    // failed
                    result.Add(new Task()
                    {
                        Type = TaskType.Test,
                        Filename = testnode.GetAttribute("name"),
                        Result = "Failed",
                        Status = 1,
                        Text = testnode.SelectSingleNode("./failure").InnerText
                    });
                }
            }
            return result;
        }


        static void SyncToDo()
        {

            var oldtasks = GetTasksFromIssueTracker(TaskType.Todo, 1);
            var newtasks = GetTasksFromCode();
       

            Console.WriteLine("Server: " + oldtasks.Count);
            Console.WriteLine("Code: " + newtasks.Count);

            // ignore exact matches

            foreach (var ts in oldtasks.ToArray())
            {
                foreach (var tc in newtasks)
                {
                    if (ts.Filename == tc.Filename && ts.Line == tc.Line && ts.Text == tc.Text)
                    {
                        // exact match
                        newtasks.Remove(tc);
                        oldtasks.Remove(ts);
                        break;
                    }
                }
            }

            var updates = new List();

            // find moved tasks
            foreach (var ts in oldtasks.ToArray())
            {
                foreach (var tc in newtasks)
                {
                    if (ts.Filename == tc.Filename && ts.Line != tc.Line && ts.Text == tc.Text)
                    {
                        // line changed
                        newtasks.Remove(tc);
                        oldtasks.Remove(ts);
                        ts.Line = tc.Line;
                        updates.Add(ts);
                        break;
                    }
                }
            }

            // find edited tasks
            foreach (var ts in oldtasks.ToArray())
            {
                foreach (var tc in newtasks)
                {
                    if (ts.Filename == tc.Filename && ts.Line == tc.Line && ts.Text != tc.Text)
                    {
                        newtasks.Remove(tc);
                        oldtasks.Remove(ts);
                        ts.Text = tc.Text;
                        updates.Add(ts);
                        break;
                    }
                }
                //                Console.WriteLine(t.Text);
                //   CreateTask(t);                               
            }

            Console.WriteLine("Completed: " + oldtasks.Count);
            Console.WriteLine("New: " + newtasks.Count);
            Console.WriteLine("Updated: " + updates.Count);


            // save updates
            foreach (var t in updates)
            {
                Console.WriteLine("Updating " + t.Text);
                UpdateTask(t);
            }

            // mark old tasks complete
            foreach (var t in oldtasks)
            {
                if (t.Status == 5) continue;
                Console.WriteLine("Closing " + t.Text);
                t.Status = 5;
                UpdateTask(t);
            }


            // create new tasks
            foreach (var t in newtasks)
            {
                Console.WriteLine("Adding " + t.Text);
                CreateTask(t);
            }

        }


        static DateTime lastchange = DateTime.MinValue;

        static void Main(string[] args)
        {

            FileSystemWatcher fsw = new FileSystemWatcher(basepath);
            fsw.Changed += new FileSystemEventHandler(fsw_Changed);
            fsw.Renamed += new RenamedEventHandler(fsw_Changed);
            fsw.Created += new FileSystemEventHandler(fsw_Changed);
            fsw.EnableRaisingEvents = true;

            Console.WriteLine("Waiting for change on "+basepath);
            Console.ReadLine();
        }

        
        static void fsw_Changed(object sender, FileSystemEventArgs e)
        {
            Console.WriteLine(e.FullPath);
            if (!e.FullPath.ToLower().EndsWith("orangeguava.onlineservice.unittests.dll-results.xml")) return;
            if (lastchange > DateTime.Now.AddSeconds(-30)) return;
            lastchange = DateTime.Now;
            
            Console.WriteLine("Changed");
            SyncTests(e.FullPath);
            SyncToDo();
            Console.WriteLine("Sync complete - waiting for change");
        }
    }
}

Gradual Upgrade, Large Databases and Timeouts

Tom Clarkson — Fri, 05 Jun 2009 18:32:00 GMT

My project this week has been migrating a system from SharePoint 2003 to SharePoint 2007 - like many things in SharePoint, this is something that is either very easy or very hard.

The first challenge was getting prescan to work. Prescan kept reporting that SharePoint 2003 SP2 was not installed, even though it was. Turns out that as well as having SP2 installed you need to upgrade/extend all the web applications with that version. It also doesn't like the presence of non-SharePoint web applications on the system.

The next issue was an exception in ParseInt32 when checking the system version. It turned out to be simple enough to fix, though I have no idea why the SystemVersion table contained "6.0.2.6568*" instead of "6.0.2.6568".

With prescan finally running without errors, it was time for the actual migration. The original plan was to just attach the database to a new 2007 server, but that was not an option once we discovered that the database was 30GB larger than the hard drive on the new server - everything would have to be done locally.

After installing 2007 on the 2003 server and selecting gradual upgrade, my first attempt at migration was to attach the 2003 database to 2007 and do database migration with an in place upgrade. This appeared to work quite nicely - the process completed in about 20 minutes - until I tried to access the site. All the data was copied, but nothing had a url and the site collections list was empty.

With in place upgrade ruled out, it was back to doing gradual upgrade of individual sites. All the references I could find on doing this with large sites (the biggest single site being 30GB) just said to use database migration instead.

Gradual migration works by copying everything from the 2003 database into a temp database, performing an in place upgrade on that, then copying everything into the 2007 database. On small sites it's great - All but 5 of the site collections were under 2GB and migrated without problems. On larger sites it uses huge amounts of disk space (maybe 5 times the database size), takes hours to run, and may timeout on some operations.

The first timeout came when migrating a couple of 2GB site collections. This one was in DropFullTextSearch. Reflector led me to sp_fulltext_database and references to needing to reinstall windows. Fortunately restarting the search service seemed to be enough - instead of timing out after 30 minutes, the next run completed in 3 seconds.

Next was the last site, the 30GB one. The timeout came in copying data to the temp database. I found a technet post with instructions to pregrow the temp database transaction log - it probably improved performance, but not enough to stop the next run timing out as well. When copying the Docs table, the upgrade process detects that it is full of 50MB blobs and extends the timeout to a couple of days. When copying the DocVersions table it doesn't do this, and leaves the timeout at the default 30 minutes. Although it's possible a service pack fixes this, I opted for the quick solution - removing all document versions.

To do this I renamed the DocVersions table in the 2003 database with sp_rename and recreated it - I could have deleted it, but this way it can be restored quickly if necessary.

Without the versions, migration completed successfully and as an added advantage the database was down to 24GB, which should make things easier to get onto the new server.

Principles of SharePoint Development

Tom Clarkson — Mon, 04 May 2009 14:11:00 GMT

I have been working with SharePoint 2007 since the first beta came out. The tools and methodologies available for development have improved a lot since then, but I still see a lot of organisations that haven't quite got a handle on how to use SharePoint as an effective development platform.

In working with various clients over the past few years, I have come up with a set of principles that can ensure a SharePoint system both supports initial requirements and continues to work well after a year of use and customisation. Most of these will be most relevant in the planning stages of a project, although I have on several occasions been called in to implement these principles after a system has become unstable.

Treat SharePoint components like any other code - Source control is often omitted from SharePoint projects, and manual deployment is far too common.
Maintain the environment in a supported state - Debugging is very difficult when you can't tell what is deployed.
Control access appropriately - What users and developers are allowed to do has a big impact on code quality.
Make good development easy - Security policies that prevent development environments connecting to source control are not a good thing.
Develop with future changes in mind - Most functionality can be implemented in more than one way, and usually one way is much easier to change later. Don't build a system that will be impossible to upgrade.
Avoid duplicate effort - You can use SharePoint's collaboration functionality to improve communication between development teams and facilitate development of reusable components.
Do basic technical design before confirming requirements - The SharePoint platform makes some requirements very easy and others very hard. Don't agree to build the hard ones before you know how hard they will be.

I will be going into more detail on each of these points in a series of posts over the next couple of weeks.

SharePoint Discussion with jQuery

Tom Clarkson — Mon, 04 May 2009 12:59:00 GMT

Recently I have been doing some work for a large bank with some rather restrictive policies on their SharePoint environment - no custom code can be installed. SharePoint Designer used to be an option, but with some changes to VPN settings I'm limited to IE6 on a painfully slow java remote desktop that changes the quotes in my code to umlauts.

The sites I was building make extensive use of discussion lists, but the out of box web parts weren't really adequate - the threaded view in particular doesn't like appearing on custom pages.

The requirement was for a web part that could display recent posts with threaded comments on the site home page - similar to the blog template, but a bit more flexible. I've done similar things in the past, building a custom web part to display comments with a news article, but that wasn't an option in this environment.

What I came up with was some javascript that could be pasted into a content editor web part to render the required view with jQuery. I kept the rendering and web service access fairly well seperated, rendering from a simple custom data structure - partly to simplify the code and partly so I could debug the javascript independent of any SharePoint issues.

I used Darren Johnstone's JSAPI for web service access - a very useful piece of code, and much easier than the manual approach I used last time I had to use sharepoint web services from javascript.

The included javascript files (jquery, jsapi and optionally the code below if you want reuse instead of quick copy/paste editing) can be placed in any document library with appropriate permissions.

First, the shared functions - one to retrieve the list data and put it into a structure more easily manipulated in javascript, and another to render the html.

function GetTopLevelPosts(weburl, listname, listurl, postlist, src) {
    var lists = new SPAPI_Lists(weburl);
    var res = lists.getListItems(
               listname,
                "",
                "",
                '',
                10,
'TRUE',

                 null);
    //alert(res);
    var rows = res.responseXML.getElementsByTagName('z:row');
    for (var i = 0; i < rows.length; i++) {
        var row = rows[i];

        var fn = row.getAttribute("ows_FileRef");
        var ih = fn.indexOf("#") + 1;
        fn = fn.substr(ih, fn.length - ih);

        var t = row.getAttribute("ows_Body");


        var np = {
            Title: row.getAttribute("ows_Title"),
            Threading: row.getAttribute("ows_Threading"),
            Date: row.getAttribute("ows_Created"),
            PostedBy: row.getAttribute("ows_PersonViewMinimal"),
            Text: t,
            ReplyLink: listurl + "/NewForm.aspx?RootFolder=/" + fn +

"&ContentTypeID=0x0107&DiscussionParentID=" + row.getAttribute("ows_ID") + "&Source=" + src,
            MoreLink: listurl + "/Threaded.aspx?RootFolder=/" + fn + "",

            Replies: []
        };

        var att = row.getAttribute('ows_Attachments');

        if (att != '0') {
            var at2 = att.split(';#');
            np.AttachmentUrl = at2[1];

            var atfnl = np.AttachmentUrl.split('/');
            np.AttachmentName = atfnl[atfnl.length - 1];
        }
        postlist[postlist.length] = np;


        GetChildPosts(lists, listname, listurl, fn, np.Replies, src);

    }

}





function GetChildPosts(lists, listname, listurl, fn, postlist, src) {

    var unthreaded = new Array();

    var res = lists.getListItems(
               listname,
                "",
                "0",
                '',
                100,
                '/' + fn + 'TRUE',
                 null);
    //alert(res);




    var rows = res.responseXML.getElementsByTagName('z:row');


    for (var i = 0; i < rows.length; i++) {
        var row = rows[i];

        var t = row.getAttribute("ows_Body");
        var ii = t.indexOf("= 0) t = t.substr(0, ii);


        var np = {
            Title: row.getAttribute("ows_Title"),
            Threading: row.getAttribute("ows_Threading"),
            Date: row.getAttribute("ows_Created"),
            PostedBy: row.getAttribute("ows_PersonViewMinimal"),
            Text: t,
            ReplyLink: listurl + "/NewForm.aspx?RootFolder=/" + fn +

"&ContentTypeID=0x0107&DiscussionParentID=" + row.getAttribute("ows_ID") + '&Source=' + src,

            Replies: []
        };

        var att = row.getAttribute('ows_Attachments');

        if (att != '0') {
            var at2 = att.split(';#');
            np.AttachmentUrl = at2[1];
            var atfnl = np.AttachmentUrl.split('/');
            np.AttachmentName = atfnl[atfnl.length - 1];
        }
        unthreaded[unthreaded.length] = np;

    }



    for (var i = 0; i < unthreaded.length; i++) {

        for (var j = i - 1; j >= -1; j--) {
            if (j < 0) {
                postlist[postlist.length] = unthreaded[i];
            }
            else {
                if (unthreaded[i].Threading.indexOf(unthreaded[j].Threading) == 0) {
                    unthreaded[j].Replies[unthreaded[j].Replies.length] = unthreaded[i];
                    break;
                }
            }


        }


    }


}

function RenderPosts(parentdiv, postlist) {

    for (var i = 0; i < postlist.length; i++) {
        var post = postlist[i];
        //console.log(post.Text);
        var postdiv = $(document.createElement("div"));
        var posttextdiv = $(document.createElement("div"));
        var postcommentdiv = $(document.createElement("div"));
        var postheaderdiv = $(document.createElement("div"));
        var postfooterdiv = $(document.createElement("div"));

        parentdiv.append(postdiv);
        postdiv.append(postheaderdiv);
        postdiv.append(posttextdiv);
        postdiv.append(postfooterdiv);
        postdiv.append(postcommentdiv);

        postdiv.addClass("post");
        postheaderdiv.addClass("postheader");
        posttextdiv.addClass("posttext");
        postfooterdiv.addClass("postfooter");
        postcommentdiv.addClass("postcomment");


        postheaderdiv.html(post.PostedBy + " - " + post.Date + " Reply");
        posttextdiv.html(post.Text);
        postfooterdiv.html("");

        RenderPosts(postcommentdiv, post.Replies);

    }


}


function RenderTopLevelPosts(parentdiv, postlist) {
    for (var i = 0; i < postlist.length; i++) {
        var post = postlist[i];
        //console.log(post.Text);
        var postdiv = $(document.createElement("div"));
        var posttextdiv = $(document.createElement("div"));
        var postcommentdiv = $(document.createElement("div"));
        var postheaderdiv = $(document.createElement("div"));
        var postfooterdiv = $(document.createElement("div"));


        parentdiv.append(postdiv);
        postdiv.append(postheaderdiv);
        postdiv.append(posttextdiv);
        postdiv.append(postfooterdiv);
        postdiv.append(postcommentdiv);
        postdiv.append("");
        postdiv.addClass("toppost");
        postheaderdiv.addClass("toppostheader");
        posttextdiv.addClass("topposttext");
        postfooterdiv.addClass("toppostfooter");
        postcommentdiv.addClass("toppostcomment");


        postheaderdiv.html("" + post.Title + "
" + post.PostedBy + " - " + post.Date + "");
        posttextdiv.html(post.Text);
        var alnk = "";
        if (post.AttachmentUrl != null) {
            alnk = "Attachment: " + post.AttachmentName + " - ";
        }

        postfooterdiv.html(alnk + post.Replies.length + " comments - Reply");

        RenderPosts(postcommentdiv, post.Replies);


    }


}

To use the code just add the html elements and call to the above methods to a content editor web part.

The css I used is below - this version doesn't look that great, but is easily customised.

.toppost
            {
                border: 1px solid black;
                font-family: Verdana;
                font-size: 8pt;
                margin-bottom: 20px;
                width: 600px;
            }
            .toppostheader
            {
                margin-bottom: 10px;
            }
            .toppostheader .title
            {
                font-weight: bold;
                font-size: 10pt;
            }
            .toppostheader .byline
            {
                font-size: 8pt;
                color: Gray;
            }
            .post
            {
                margin-top: 10px;
                margin-left: 30px;
            }

Reviving a Project

Tom Clarkson — Tue, 24 Mar 2009 20:14:00 GMT

Over the last couple of weeks I have been working on getting TwoNeeds back on track. After six months or so of active development we had a rather complex but solidly designed and scalable back end and UI for most of the important features. However, with lots of changes to the feature set the pieces didn't really fit together in any coherent way. It all worked technically and we could keep making progress, but it would never become a good product that way.

When that happens, the only real solution is to throw out the code and start over. While it may seem like a lot of wasted effort, building the first version was necessary to get a proper understanding of how the app should be built. It is immediately clear what pages are needed, and its possible to do some stuff with jquery that wouldn't have fit into the initial prototype.

I've been doing most of the work myself so far - partly because the rest of the team is busy with other projects, and partly because much of the detail gets designed as I build it - writing full specs would probably take just as long as the initial build. I should be able to delegate more as the work progresses - it's quite easy to have someone make specific changes to a component, but describing an overall vision is pretty hard.

I'm also noticing that a lot of the work is on the less interesting stuff. In order to build the interesting features, we have to build a lot of stuff that has been done before in a way that supports the new stuff that will be added later.

Google API - A Moving Target

Tom Clarkson — Tue, 24 Mar 2009 17:40:00 GMT

A while ago I said that developing for Google Apps was like SharePoint in the lack of documentation and hacks required to work around the issues. I have since discovered something more unique to Google - sudden changes to the behaviour of API calls with neither the old or new behaviour being documented. Quick changes can be good (like the fix for the 404 errors on working with folders I'm hoping will appear in the next week or so - the best workaround I have for that so far is changing the error message to reflect that the bug is in the api rather than my code), but sometimes they just stop my code from working altogether.

The latest issue I've run into is with creating documents. Because Google Docs does not support creating new documents offline, GDNote creates a folder full of blank documents that can be moved and renamed while offline. Yesterday this worked, but today the call gets the error 400 Bad Request - Document content required.

We have very good error logging set up, so I should be able to fix it quickly and not too many users will be affected, but these unexpected issues are still pretty frustrating.

Dealing with Google API Bugs

Tom Clarkson — Wed, 18 Mar 2009 12:56:00 GMT

I just deployed an updated version of GDNote. The bugs fixed are getting fairly small now - the latest one was triggered only if you reload a section while the web service call to create it is running. However, some users are running into issues we can't fix - bugs in the Google APIs.

The most recent case involves a user getting errors doing anything that creates a folder. Everything works fine with my own heavy usage and all of the test accounts, but for one particular user it doesn't work. Fortunately GDNote logs all errors in as much detail as possible without causing security issues, so I can see the request that failed. It's exactly the same api call that works for everyone else, but for this particular user Google returns 404 not found.

A bit of searching reveals that this is a known Google bug that has shown up in the last week or so, with a fix due later this month. Best I can do is wait for the fix and tell the affected user to create the folders manually. For similar issues in the past I've been able to create workarounds in the code, for example checking if a call really failed or just returned an error code for no apparent reason. What I can't do is find a way to prevent these problems from occurring for real users in the future. Normally when an issue comes up and you fix it you would add a test that reproduces the original error, preferably before a real user has problems. But with these api problems the issue only occurs for some accounts, and setting up a test account with the same configuration doesn't trigger the error.

Is it actually possible to do anything better than "Works on My Machine" when working with cloud APIs?

Online Notebooks

Tom Clarkson — Thu, 05 Mar 2009 11:15:00 GMT

The last couple of weeks have been pretty busy, as is to be expected when you have six or so active projects. Among other things, I got GDNote to the point where I could remove the placeholder page and make it publicly available. It's still a beta, but from a technical perspective it's done (the online/offline part, mobile clients are a bit more complicated).

However, from a non-technical perspective there are still some challenges - namely, getting users. I haven't really done anything to market it yet due to taking time to catch up on other projects, but now that I have time to think about it more I realise that finding the right approach may be difficult.

The main problem we have is describing the product in the same way that people looking for it would describe it. So far the best we have is "online notebook", which unfortunately means different things to different people.

GDNote is a fairly structured product, organised into notebooks, sections and pages in much the same way as OneNote. Smaller projects get a section in the projects notebook, while larger projects have their own notebook with multiple sections. Bringing in content from the web is possiblie in theory, but not something I use - most of my usage is writing down thoughts that have come from nowhere in particular. This is the structure that works for me, so it's what I think of as an online notebook.

However, the same term can be used to describe quite different products - there are apps based on clipping web content (Google Reader does most of what I need in that space), there are quite a few that do short notes designed for searching later, and there are those like evernote that are relatively similar in concept but have enough differences to not fit with the way I expect a notebook to work.

GDNote is different from all of these, but the difference is hard to describe in a searchable way for users who have never heard of the product. Being based on Google Docs so that everything is stored in your existing account is a selling point, but nobody is going to be looking for that - it only becomes an advantage after someone makes it to the GDNote website. The way notes are organised is something that people will be looking for but is not easy to clearly describe for a search engine.

Short of trying every online notebook available there doesn't seem to be an easy way to find the one that fits how you think - Obviously something that is going to require a bit more thought.

SharePoint Contracting Market Changes

Tom Clarkson — Thu, 12 Feb 2009 12:26:00 GMT

I've just got back to Sydney after two months away. I've talked to a couple of places about new SharePoint contracts - I'm not looking for that sort of thing actively, but I'm open to the right offer.

What I notice has changed since last year is that going through the standard channels everybody assumes I must be desperate for work, and therefore willing to drop my rate significantly. I'm not of course - I've been planning my exit from full time contracting for a while, so I'm set up with both other clients and some of my own projects. If I was actually in urgent need of a job I'd have one within a few hours, same as the last couple of times.

I can see some unintended consequences coming out of this - in particular, if you only make offers matched to people with no choice but to accept, that's what you'll get - the ones without the skills to do better elsewhere. That's the sort of thing that leads to consultants like myself being brought in later to fix things, so it's all good.

Working in the Cloud More Literally than Usual

Tom Clarkson — Mon, 09 Feb 2009 09:42:00 GMT

I'm writing this somewhere over Indonesia on a flight from Bangkok to Sydney. I find that I am always particularly productive on long flights - I guess it's something to do with knowing I'm sitting here for nine hours whatever I do.

So far I've replied to all my emails (really liking offline gmail), written a reference for a former colleague and written some text for the Fixed Price SharePoint website (I got the rest of the copy done on the Sydney to Bangkok flight and haven't gotten around to working on it since).

In the past I've gotten a lot of code written on planes - mostly when that was the only thing my job required. I find it works best for stuff that has been getting delayed for a while - the stuff you would otherwise have to force yourself to sit down and do.

In terms of tools, I find that the Mac I'm using now is much better than the Windows machines I've used in the past - mainly due to how quickly it starts up. A seat with a power outlet is definitely worth paying a bit extra for - I have in the past used spare batteries to keep my computer running the whole flight, but it only works when the batteries are new, and not having to worry about running out of power makes a big difference.