Refactor vs rewrite (again)

I don't know why I have to blog about this.  It depresses me and reflects poorly on our entire industry.  It is 2014!  Don't we know better?!?

Apparently not.  I have found myself *several* times in the past month, having to argue against a big rewrite of many thousands of lines of code.  I am amazed and appalled that anyone still thinks this way, after *so* many articles have been written for *so* many years.  Two of my favorites are Joel in the year 2000, and this more recent article that references one of my favorite cartoons (thanks to Lance for introducing me to the cartoon).

One thing that is different from when Joel wrote that blog post 14 years ago is that refactoring software is so much easier now.  It is now *incredibly* easy, in addition to being much safer.

I don’t want to go through the same arguments again, because so many others have done it for me.  Unlike some of the people I have to convince, I am not stupid enough to think that I am the first person faced with this decision, or arrogant enough to dismiss what those hundreds of other people have said.  However, I will offer one small piece of advice.  Often times the refactoring can happen *while* people are designing what the rewritten version is going to look like.  At that point the code will be easy to change and it will be a much simpler process to add the new features.

Azure DocumentDB First Look

Step 1: Run the PowerShell Scripts

Step 2: Use the new Azure portal at portal.azure.com

Step 3: Run NuGet: Install-Package Microsoft.Azure.Documents.Client -Pre

Step 4: This is the Flashcard class

 
public class FlashCard
{
    [JsonProperty(PropertyName = "id")]
    public string ID { get; set; }
    [JsonProperty(PropertyName = "question")]
    public string Question { get; set; }
    [JsonProperty(PropertyName = "answer")]
    public string Answer { get; set; }
}

Step 5: This is the console application:

 
internal class Program
{
    private static void Main(string[] args)
    {
      CreateSaveAndGet().Wait();
      //CleanUp().Wait();
    }

public static async Task CreateSaveAndGet()
{
 //connect database
 var client = GetClient();
 //find or create a database
 var database = await GetDatabase(client);

 //create a collection
 var collection = await GetCollection(client, database);

 Console.WriteLine("Save new flashcards...");
 var flashcard1 = await client.CreateDocumentAsync(collection.SelfLink, new FlashCard()
 {
  ID = "1",
  Question = "When did Azure DocumentDB release in preview?",
  Answer = "August 21st, 2014",
 });

 var flashcard2 = await client.CreateDocumentAsync(collection.SelfLink, new FlashCard()
 {
  ID = "2",
  Question = "What is Azure DocumentDBs twitter handle?",
  Answer = "@DocumentDB",
 });

 var flashcards = await Task.Run(() => client.CreateDocumentQuery(collection.DocumentsLink)
  .AsEnumerable()
  .ToList());

 Console.WriteLine("Iterating through flashcards...");
 foreach (var flashcard in flashcards)
 {
  Console.WriteLine("QUESTION " + flashcard.ID.ToString() + ": " + flashcard.Question);
  Console.WriteLine("Answer: " + flashcard.Answer);
  Console.WriteLine("Press a key");
  Console.ReadKey();
 }
 Console.ReadKey();

 var flashcardGet = await Task.Run(() =>
  client.CreateDocumentQuery(collection.DocumentsLink)
   .Where(d => d.ID == "1")
   .AsEnumerable()
   .FirstOrDefault());

 Console.WriteLine(flashcardGet.Question);
 Console.ReadKey();

 var doc = client.CreateDocumentQuery(collection.DocumentsLink)
  .Where(d => d.Id == flashcardGet.ID)
  .AsEnumerable().FirstOrDefault();

 await client.DeleteDocumentAsync(doc.SelfLink);

 flashcardGet = await Task.Run(() =>
  client.CreateDocumentQuery(collection.DocumentsLink)
   .Where(d => d.ID == "1")
   .AsEnumerable()
   .FirstOrDefault());

 Console.WriteLine(flashcardGet.Question);
 Console.ReadKey();
    }

    private static async Task CleanUp()
    {
 //connect database
 var client = GetClient();

 //find or create a database
 var database = await GetDatabase(client);

 //create a collection
 var collection = await GetCollection(client, database);
 await client.DeleteDocumentCollectionAsync(collection.SelfLink);
    }

    private static DocumentClient GetClient()
    {
 string endpoint = ConfigurationManager.AppSettings["EndPoint"];
 string authKey = ConfigurationManager.AppSettings["AuthKey"];

 Uri endpointUri = new Uri(endpoint);
        var client = new DocumentClient(endpointUri, authKey);
 return client;
    }

    private static async Task GetDatabase(DocumentClient client)
    {
 Database database;
 var databaseName = "flashcards";
 var databases = client.CreateDatabaseQuery()
  .Where(db => db.Id == databaseName).ToArray();

 if (databases.Any())
 {
  database = databases.First();
 }
 else
 {
  database = new Database { Id = databaseName };
  database = await client.CreateDatabaseAsync(database);
 }
 return database;
    }
    private static async Task GetCollection(DocumentClient client, Database database)
    {
 var collectionName = "flashcards";
 DocumentCollection collection;

 var collections = client.CreateDocumentCollectionQuery(database.SelfLink)
  .Where(col => col.Id == collectionName).ToArray();

 if (collections.Any())
 {
  collection = collections.First();
 }
 else
 {
  collection = await client.CreateDocumentCollectionAsync(database.SelfLink,
   new DocumentCollection { Id = collectionName });
 }
 return collection;
    }
}

Video of Learn JavaScript Properly Part 1 - SQL Pass Book Readers

Simplifying Andoid Concepts

When it comes to Android there are fundamental concepts that a programmer *must* understand in order to author code for the platform or to understand code found online. You need to understand these concepts even if you are writing the application in Xamarin or some other abstraction layer.

From the Android developer web pages (http://developer.android.com/) you can find out that by default, every app runs in its own Linux process. Components are the essential building blocks of an Android app. Each component is a different point through which the system can enter your app.

There are four types of components:

Activity	Screen with a user interface
Service	Performs long running operations in the background
Content Provider	Manages a shared set of app data
Broadcast Receiver	Responds to system-wide broadcast announcements

Broadcast receiver makes perfect sense, and anyone coming from a Windows world thinks of services as performing background tasks already. If you consider data as content then of course a content provider provides data to the rest of the application. So the only one that has a slightly bizarre name is an Activity. If you think of an activity as a single focused thing that a *user* can do, then it makes sense.

Three of them (Activity, Broadcast Receiver, and Service) are all activated by an asynchronous message called an intent. Intents are confusing, but they are essentially an abstract description of an operation to be performed. Try using the mnemonic, "It is my intent to perform this operation." Another way to think of an intent is a description of an operation written down on a piece of paper (more on this later).

In Android an intent can be either explicit or implicit. In an explicit intent the fully qualified class name of an actual component that needs to run is on the piece of paper. Android makes sure that the component runs.

An implicit intent is a little more vague. Once again it is just a string, but it is global, and you don't want to collide with someone else's string. The implicit intents are generally reverse domain qualified (e.g. "com.mydomain.myapp.myintent") so that there are no naming conflicts.

When apps are *installed* Android takes all of the implicit intents specified in the manifest and adds them to its global table. If we go back to the piece of paper analogy, an implicit intent is a single-sheet newsletter. Applications that include that implicit intent in the filters section of the manifest are subscribing for the newsletter, and then when an app publishes the newsletter, Android makes sure that it gets to all of the subscribers.

There is an addtional type of intent called a *pending* intent. Things get a little more complicated with a *pending* intent. However it is easy to understand if we think of an intent (or piece of paper) that has been placed inside an envelope with some handling instructions, similar to inter-office mail or do not open until your birthday. The action written on the piece of paper is performed on behalf of the sender.

One set of handling instructions is the flags parameter. All of the static methods for creating a PendingIntent, like getActivity, getBroadcast, or getService, accept a flags parameter. Flags can be combined together and some only make sense in the context of the other flags. Here are the flags on PendingIntent and how they fit in with the envelope analogy.

FLAG_CANCEL_CURRENT	This is an update, throw away the old piece of paper you were working on and replace it with this new one
FLAG_NO_CREATE	(NOTE: only relevant in combination of the CURRENT flags). If you didn't already have an envelope, ignore this one also
FLAG_ONE_SHOT	Burn after reading
FLAG_UPDATE_CURRENT	Don't throw the first piece of paper away, instead make these revisions to the piece of paper

Here is a few snippets of code (C# code written using Xamarin Studio) that show how you would use an intent in practice. In this example we are creating a “Geofencing Intent”. A GeoFence is simply a location based circle that a user can enter or leave. When either of those events happen your application can be notified. Think of the example where the user wants to be reminded to “take the trash out when I get home”. The “when I get home” piece indicates a location based area. We can create a GeoFence for this area and be notified when the user enters this area.

Here is an example of of creating an IntentService using Xamarin C# code

using Android.App;
using Android.Content;
using Android.Support.V4.App;
using Android.Gms.Location;
using Android.Locations;
 
namespace TIG.Todo.AndroidApp
{
 [Service]
 [IntentFilter(new[] { "com.sdtig.Todo.WITHIN_PROXIMITY" })]
 public class GeofenceIntentService : IntentService
 {
  protected override void OnHandleIntent (Intent intent)
  {
   //Handle Intent
  }
 }
}

Below is a snippet that shows an example of sending a PendingIntent out (see line 43)

 [Service]
 [IntentFilter(new[] { "com.sdtig.Todo.START_LOCATION", "com.sdtig.Todo.SET_GEOFENCE" })]
 public class GeofencingHelper : Service, Android.Locations.ILocationListener
 {
  public override Android.OS.IBinder OnBind (Intent intent)
  {
   return null;
  }
 
  public override void OnStart (Intent intent, int startId)
  {
   if (intent.Action == "com.sdtig.Todo.START_LOCATION")
   {
     //Elided
   }
   if (intent.Action == "com.sdtig.Todo.SET_GEOFENCE")
   {
    SetFence();
   }
  }
   
 
  public void SetFence()
  {
   bool isWithinRadius = false;
   foreach (var fence in fences)
   {
    float[] results = new float[1];
    Location.DistanceBetween(fence.Latitude, fence.Longitude, currentLatitude, currentLongitude, results);
    float distanceInMeters = results[0];
    if (distanceInMeters < radiusInMeters)
    {
     isWithinRadius = true;
     break;
    }
   }
 
   if (!isWithinRadius)
   {
    var intent = new Intent(CustomActions.TODO_WITHIN_PROXIMITY);
    PendingIntent pendingIntent = PendingIntent.GetService(this, 0, intent, 
     PendingIntentFlags.UpdateCurrent);
    _locationManager.AddProximityAlert(currentLatitude, currentLongitude, 
     radiusInMeters, -1, pendingIntent);
   }
  }
 }

And finally here is a snippet showing the MainActivy.cs file where we start an Intent service to watching for system intent's related to Location Monitoring

using System.IO;
using Android.App;
using Android.Content;
using Android.Widget;
using Android.OS;
using TIG.Todo.Common;
using Android.Content.PM;
using TIG.Todo.Common.SQLite;
 
namespace TIG.Todo.AndroidApp
{
 [Activity (Label = "TIG.Todo.Android", MainLauncher = true, ConfigurationChanges = ConfigChanges.Orientation | ConfigChanges.ScreenSize)]
 public class MainActivity : Activity
 {
  private TaskManager taskManager;
 
  protected override void OnCreate (Bundle bundle)
  {
   base.OnCreate (bundle);
 
   // Set our view from the "main" layout resource
   SetContentView (Resource.Layout.Main);
 
   //Intent intent = new Intent (this, typeof(GeofencingHelper));
   StartService(new Intent("com.sdtig.Todo.START_LOCATION"));
  }
 }
}

Hopefully the analogy helped to simplify some of the more abstract Android concepts.

Web UI Testing Part 4: Extension methods in Page Object Model

When I started my next project I switched from WatiN to Selenium, and I incorporated the PageObjectModel.  I had recently watched John Somnez's video's Pluralsight videos around this topic (http://simpleprogrammer.com/2013/09/28/creating-automated-testing-framework-selenium/) , so a lot of his ideas were shining through.  There was a Pages class which had static properties to all of the Page objects.

Here are some of the highlights of that solution.  We created some additional extension methods for any web element to be able to perform some common functions.  Because Selenium's FindElement normally only looks under an element, and we needed a way of looking above an element, we modified this hack using XPath parent axis. Another really useful function is the ability to extract table information.
   
    public static class WebElementExtensions
    {
        public static IWebElement GetParent(this IWebElement element)
        {
            return element.FindElement(By.XPath("parent::*"));
        }

       public static IWebElement FindParentByClassName(this IWebElement element, string className)
        {
            if (element == null)
            {
                return null;
            }

            var classValue = element.GetAttribute("class");
            if (classValue.Contains(className))
            {
                return element;
            }

            return FindParentByClassName(element.GetParent(), className);
        }

        public static List<string[]> ToTable(this IWebElement element)
        {
            var rows = new List<string[]>();
            foreach (var tr in element.FindElements(By.TagName("tr")))
            {
                var thOrTds = tr.FindElements(By.TagName("th")).Union(tr.FindElements(By.TagName("td")));
                rows.Add(thOrTds.Select(c => c.Text).ToArray());
            }

            return rows;
        }

In addition to the normal page object model there are often times menus, or toolbars, that cross pages.  The original way we did this was just to use the Base classes, but we soon started needing the base classes for things like steps in a wizard.  So instead we moved those to extensions as well, based off the BasePage.  So when we created a new page that used an exiting menu partial we could use the extension methods to call those the methods easily without any modifications.  We found the easiest way to do this was based off empty interfaces, because extension methods don't really support attributes and we needed someway of describing which extension methods were legal on which objects.

public interface IHaveAdminMenu
{
}

public static class AdminMenuExtensions
{
    public static void AdminMenuClickItems(this IHaveAdminMenu adminMenu)
    {
        var basePage = (BasePage) adminMenu;
        basePage.Driver.FindElement(By.Id("itemsLink")).Click();
    }
}

Web UI Testing Part 3: The Page Object Model

Whether you end up WatiN or Selenium for automating the browser actually doesn't matter that much.  Whichever mechanism you use should be hidden behind a Page Object Model.  This actually took me a while to discover because it wasn't really in your face on the WatiN and Selenium forums.  In fact even once I knew about the pattern I didn't feel the need for it at first.  It was similar to having a domain controller for a couple of computers.  However, as the sites I was writing and testing got more complicated, I needed a way of organizing the methods to manipulate the pages into a logical grouping.  It makes sense to make an object model that encapsulates the ID, classes, tags, etc. inside a page so that they can be reused easily.  Let's look at a simple example in WatiN, prior to putting in the Page Object Model.

[Given(@"I am on an item details page")]

public void GivenIAmOnAnItemDetailsPage()

{

browser = new IE("http://localhost:12345/items/details/1?test=true");

}

[When(@"I update the item information")]

public void WhenIUpdateTheItemInformation()

{

browser.TextField(Find.ByName("Name")).TypeTextQuickly("New item name");

browser.TextField(Find.ByName("Description")).TypeTextQuickly("This is the new item description");

var fileUpload = browser.FileUpload(Find.ByName("pictureFile"));

string codebase = new Uri(GetType().Assembly.CodeBase).AbsolutePath;

string baseDir = Path.GetDirectoryName(codebase);

string path = Path.Combine(baseDir, @"..\..\DM.png");

fileUpload.Set(Path.GetFullPath(path));

The ?test=true in the first method is interesting, but the subject of another blog post.  Instead Notice the Find.ByName("Name") in the second method.  Now what if there is another method where I need to check the name to see what is there.  And yet another where I need to both check it *and* update it.  So I would have three places and four lines where that Find.ByName("Name") would be used.

What happens when I change the element to have a different name?  Every test where I have used Find.ByName("Name") breaks.  I have to go through and find them all and update them. 

Let's look at the same two methods, but this time with a PageObject model.

[Given(@"I am on an item details page")]

public void GivenIAmOnAnItemDetailsPage()

{

browser = new IE(Pages.ItemDetails.Url);

}

[When(@"I update the item information")]

public void WhenIUpdateTheItemInformation()

{

Pages.ItemDetails.SetName("New item name");

Pages.ItemDetails.SetDetails("This is the new item description");

Pages.ItemDetails.SetPictureFile("DM.png");

A couple of interesting things happened.  The first is that the test is a lot more readable.  The second is that I now have a central place to change when something from the page changes.  I fix one line, and now all of the tests are running again.

So to recap, Page Object Models are great when either the pages are volatile or the same pages are being used for lots of different tests.

Xamarin Forms changes the game

Although Xamarin has been around for a while now, there were cross platform mobile projects where I did not recommend its use.  These were generally projects that had a large number of screens.  In such cases the process of creating multiple versions of every screen could make the project too difficult and time consuming to write.  In these cases I might recommend going with PhoneGap if the shop has web development experience.  Now that Xamarin Forms has released (at the end of May), all of the UI can go into a common layer, written in a single paradigm.  This allows the vast majority of the assets to be reused across platforms.  Of course it still doesn't mean that *all* of the code will go into a common layer, just that *most* of it will.

One a personal note it is interesting that the Xamarin Forms release happened while we were in the middle of the mobile development track for the San Diego TIG.  It is changing the industry, and it also changed our track.  We removed the PhoneGap meeting from the end of the track after a brief discussion of the technology.

[Update: September One thing I have noticed since I started using Xamarin regularly is the flakiness of the product.  When you pay this much for a product you expect a higher level of quality.  The start debugging, then restart debugging, then stop the simulator and restart debugging again is getting old]

SQL Tip: THe Importance of Changing Schema

SQL Tip: How to share passwords over email

SQL Tip: Change coloring on status bar of SSMS

New SQL Tip: SET XACT_ABORT ON

I have a new SQL Tip about SET XACT_ABORT.  If you're unfamiliar with this setting, you should definitely spend three minutes and watch this tip:



CREATE TABLE t1(
col1 INT PRIMARY KEY)

GO

INSERT INTO t1
VALUES
(1)
,(2)
,(3)
,(4)-- Assuming SET IMPLICIT_TRANSACTIONS is OFF

INSERT INTO t1
VALUES(1)

INSERT INTO t1
VALUES(99)
-- Which rows get inserted?
SELECT * FROM t1-- Which rows get inserted for this explicit transaction?
BEGIN TRANSACTION;
   INSERT INTO t1
   VALUES
   (1)

   INSERT INTO t1
   VALUES
   (100)
COMMIT TRANSACTION;
GO-- Which rows get inserted?  Did 100 get inserted?
SELECT * FROM t1-- With SET XACT_ABORT?
SET XACT_ABORT ON;
BEGIN TRANSACTION;
   INSERT INTO t1
   VALUES
   (1)

   INSERT INTO t1
   VALUES
   (200)
COMMIT TRANSACTION;
GO
DROP TABLE t1



Web UI Testing Part 2: Front-door testing tools

Because of the two problems I mentioned with back-door web testing (changes to layout and no Javascript testing), I was looking to pursue front-door web testing toward the end of 2012. 

My first thought was that whatever framework I chose should have a test recorder so that writing the tests would be much easier than having to code up every little click and wait.  The problem with this philosophy is that most of these test recorders generate code.  It turns out that generating code in a maintainable way is hard, and all code should be maintainable, even test code.  So although recorders can be useful in learning how to write tests I eventually scrapped that path, and started looking at using a nice API to drive the browser.

I looked at two different frameworks in .NET for accomplishing this: WatiN and Selenium.  Both had great feature sets and either one would have been suitable.  At the time, Selenium's documentation was way too fragmented.  There were multiple versions: Selenium 1.0, Selenium RC, Selenium 2.0 , etc.  Because I was new I wasn't sure which one to use (e.g. was 2.0 stable?).  When I did a search I ended up on a blog posts using outdated methods, or the blog posts didn't indicate which version of the API was being used.  I found WatiN's documentation to be much clearer on the .NET side.  I went with that, although both tools offer an API which is easily to understand.  I later ended up regretting my choice.  WatiN only supported IE, and although the bulk of the bugs were in IE, I found a few bugs on the other browsers that I wished I had a way of automating.  WatiN later started supporting Firefox after I finished the project.

[Update: Selenium has been on version 2.0 for a while, and the older documentation is becoming less relevant in search engines, so I would probably go with Selenium today]

New SQL Tip: The Mouse Middle Button

Proper Dashboard Design

Made a new SQL tip on proper dashboard design:

Web UI Testing Part 1: Front-door and back-door testing

Llewellyn Falco and I had a conversation many years ago (June 2010?) about the best way to test Web UI.  During that conversation we referred to the two classifications/mechanisms of web testing as front-door and back-door web testing.  That is how I still think of the two types many years later, although I recognize that not many people in the industry use those terms. 

In front-door web testing you are using the browser to drive the test, which more closely tests what the user sees, but offers limited ability to manipulate or control the data and other dependencies.  The other drawback of this type of testing is if the test modifies data, there needs to be some way to get back to a clean slate after the test finishes.

In back-door web testing you call the controller or presenter directly (assumes you are using MVC pattern, or have done a good job separating the Greedy view into a presenter).  The advantage of this pattern is that you can control the dependencies and data context under which the test runs more easily by using in memory repositories, mocks, and things of that nature.  The main issue with this type of testing is that these controller methods return some sort of model and view name, making it difficult to test what the user sees.  Because of this, you can have complete test coverage over the controllers but still have bugs in the view.

In January of 2011 ASP.NET MVC 3 was released which allowed different view engines to be used to render the views into HTML that would be sent back to the client.  Because the View engines were easily pluggable and the Razor Engine was packaged separately (https://www.nuget.org/packages/RazorEngine/) this allowed back door testing to call the engine to produce HTML.  This allowed back-door web testing to get closer to what the user was seeing and eventually resulted in Llewellyn augmenting Approval tests with a mechanism for approving HTML.

http://blog.approvaltests.com/2012/05/testing-views-in-aspmvc.html

However, there are still problems with this approach.  Two of the biggest problems are

1. changes to the layout template break all tests
2. inability to test JavaScript manipulations of the page.

Project Management is a Prioritized List

I just finished forming a new company called Crafting Bytes with Brad Cunningham and Ike Ellis.  We wanted to start taking on bigger projects using the techniques that have made us so successful as consultants.  Over time we have noticed that we do much better with the projects when we take control of the project management as well as the development rather than simply augmenting the development staff.  What is different about our project management style than the project management style of other companies that we work with?

One major difference is that we don't use scrum.   In a sprint developers spend a lot of time estimating the work.  Estimating can be important when the estimate is used to determine whether or not the work should take place at all.  However, in most cases companies were using the estimate to inform management when the product was supposed to ship, so that they could relay that information to the customer.  It would be a better idea to relay the information to the customer *after* the work has been completed.  It is much more accurate that way.  The other reason managers were requiring estimates was to figure out how much work should be completed this sprint.  So in other words managers were requiring estimates for the sole purpose of the project management methodology they were using.

The thing is estimates take a lot of time, and they are rarely accurate.  Our thought was let's forget scrum and just go with a simple Kanban board (from the Lean school of thinking).  By doing this we can save ourselves countless hours of trying to figure out how much time things are going to take, and spend more time simply doing them.

OK, so that saves a couple days every sprint, but then what is the purpose of having a project manager at all?  I admit that the project managers of many companies are totally unnecessary.  You know the type, they spend most of their time polling individual people "are you done yet".  They could easily be replaced by voice recognition software that recognizes the word "yes".  This isn't really project management, it is instead project reaction.  Project *management* would be managing the work of the project, prior to it starting.  In short a project managers job is to figure out which work is the most important and which work is so unimportant it doesn't need to be done at all.  Great project managers remove all unnecessary tasks, that is all tasks that don't lead to working software, and prioritize which features are the most important for the business and the user.  In short they control the prioritized list.

Thinking of project management as simply controlling the list of things that need to get done and prioritizing the most important simplifies the job of the project manager and helps the team achieve minimum "time to value" - a vastly underrated metric.