NullifyNetwork

Work and life

Sun, 14 Apr 2013 22:02:57

Not sure anyone is still reading this, but I've not really been blogging while contracting for semi-obvious reasons; I'll be leaving this post here at the top just to confirm I am still alive and will post some interesting stuff about both Workflow Foundation and 3D engine development in the future.

All but a few of my servers are now upgraded to Server 2012 or FreeBSD and I'm ditching the remaining Linux ones, overall I've been finding Server 2012 a solid platform and while the initial version had power management bugs and various other quirks updates have really improved it. It has made things like Active Directory extremely lightweight too.

I've also been making a 3d engine in my spare time (though only putting in a concerted effort over holidays!) so should have plenty to say about geometry/meshes, collision detection, mouse movement, shaders and physics. I also toyed with flow-field pathfinding and am amazed how effective it can be with so little code so hopefully that'll be the subject of a few blog posts on its own.

TFS 2012 has been an awesome platform for doing builds and keeping track of what I was working on given quite lengthy gaps too.

Visual Studio 2012 Theme Editor

Sun, 09 Sep 2012 22:32:54

You can download the theme editor from here. I have started to get used to the colours to be honest, though they are still substantially worse than the VS2010 scheme. Commence the addition of colour!

IPv6

Wed, 06 Jun 2012 08:41:46

I'm still contracting so haven't posted much at all but I am still alive! This is just a quick note because it's nice to see people like Microsoft have taken world IPv6 launch day seriously and enabled their main websites permanently with AAAA records.

Unfortunatey major UK ISP's haven't yet enabled it for their customers (JANET excluded), but the smaller ones (and international carriers) are shining through and able to provide service.

I took the IPv6 Enabled logo off my site; however I'm proud to say it has been enabled here for the last eight years or so.

Silverlight 5 Released

Sat, 10 Dec 2011 12:40:30

Microsoft have just released version 5 of Silverlight, the tools for VS2010, the developer runtime and the full runtime!

Hopefully this isn't the end of Silverlight as the media are currently speculating, I'm hopeful that there'll be an announcement of a Silverlight 6 but either way Silverlight 5 has a ten year support cycle and is the underlying technology behind Windows Phone 7 and Windows 8 (albeit with different naming, etc.).

I've been very impressed with the capabilities of WPF/Silverlight and hope that it gets the resources it deserves.

Entity Framework 4 and Replication Guids (RowGuid)

Fri, 30 Sep 2011 21:32:18

This has been delaying my release of an upgraded version of this blog for months now, and has also meant that I have chosen to use Linq To SQL on several large projects when working (Linq To SQL fully supports this scenario despite being the 2nd class citizen in entity mapping at Microsoft), Paul Patterson explains it quite well here so I can be lazy and just link to his article:-

http://www.paulspatterson.com/technology/dot-net-development/entity-framework-4-and-sql-uniqueidentifier-bug/

I wasn't willing to employ his fix though, it was frustrating and would get overwritten whenever the schema was updated however I have just found that there is a hotfix for this now available from Microsoft so that you can correctly configure the EDMX in entity framework to support the rowguid correctly:-

http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=37957

This isn't perfect as you still need to manually set the StoreGeneratedPattern option on the rowguid columns to Identity, but it is certainly better than the total failure that was the case before! Hopefully they will consider better support for replication, guid's and server-generated values in future versions.

I guess this means I can finally switch to Entity Framework and get my new site version released... :)

World IPv6 Day

Wed, 08 Jun 2011 00:01:41

Haven't had time to blog around work (figured I should at least post once a year though), but I'm pleased to see large companies suddenly enabling their IPv6 implementations today, it has been a rough ride. I've seen a gradual climb in IPv6 end users (and web and mail servers) for some time now and it's very pleasing to see it gradually approaching critical mass.

With my network fully IPv6 enabled for some time now thanks to Hurricane Electric (and more recently Goscomb Technologies too) I am looking forward to the time when there's no more NAT being deployed, or at least not in the same fashion.

Edit: I'm glad to see Microsoft as one of the first companies to get their sites advertising their AAAA records today, curtesy of Akamai. The rapid climb towards 100% amongst the participating companies is excellent to see, partcularly as the agreed start time is midnight UTC rather than british summer time when this post is being made (visible at http://www.worldipv6day.org/participants-dashboard/index.html and http://v6day.ripe.net/cgi-bin/index.cgi)

Still contracting...

Sun, 21 Nov 2010 18:41:15

Just a quick note to say that I've been contracting for a while now and have avoided posting anything/not had much to post about but that I'm still around.

I did go to the Microsoft PDC's virtual event in Reading, UK so should really have been evangelizing about how great it is that there'll be private Azure clouds now but I've not really had a proper opportunity to investigate that yet, but perhaps the next post will be related to that.

Yubikey decryption/authentication class

Sat, 31 Jul 2010 16:58:27

So I have encountered a situation where I need to use two factor authentication and recently bought a couple of Yubico YubiKey's to investigate them as a potential solution where I can't use smart cards (mostly with other users who can't install the appropriate drivers or on locked down systems/internet cafe's).

They are fairly cheap (particularly in bulk) and provide similar functionality to an RSA Securid but without the need to type the password (they act as a keyboard and type onto the computer, but can't be modified by the computer in question). They also have two 'slots' for different tokens, you can touch the button once for one and touch and hold for a second one time key.

Whilst there's plenty of code examples around for them (the company are extremely open), I haven't found a simple, logical C# library/class I could use to do authentication locally (they supply ones for remote auth on their servers) - there were always multiple files and loads of code in peoples projects.

My intention is to write a Windows SubAuthentication DLL (in C++ but I learn in C#) to use with these keys which also still checks your normal password - so when remote desktop'ing, using something that authenticates to Windows or using the Windows Radius server (NPS) to authenticate VPN's you can choose to either supply your normal password, or if you're on a system you don't trust you can use your username/low security pin code and Yubikey.

So here's my take on the decoding of a Yubikey's input - note that I'm not interested in making a software token though so the class doesn't generate values for the random value (if you wanted to do this you just need to ask RNGCryptoProvider for a few bytes of random though) and there's also no incrementing of the sessions, in-session use counters or timestamps included - however I did include the code to encrypt it appropriately if you do supply all the values.

To use this class you just need to call it like so - where privateKey is a byte[] array with the private key in and otpCode is a string like "fkfthfffktbnreffrghcldrffeclcgkt" output from a touch of the Yubikey:-

YubiKey otp = YubiKey.FromModHex(privateKey, otpCode);
if (otp.UidMatch(checkIdentity))
{
	Console.WriteLine("Decoded the YubiKey's OTP code:-");
	Console.WriteLine("\tSession:\t" + otp.SessionCounter.ToString());
	Console.WriteLine("\tCount in session:\t" + otp.SessionUseCount.ToString());
	Console.WriteLine("\tTime (high):\t" + otp.TimestampHighPart.ToString());
	Console.WriteLine("\tTime (low):\t" + otp.TimestampLowPart.ToString());
	Console.WriteLine("\tTime (low):\t" + otp.Uid.ToString());
}
else
{
	Console.WriteLine("Failed to identify the keys correct OTP");
}

And you can decode the private key/identity as they're displayed in the YubiKey personalisation tool with the following handy (but not super efficient) hex to byte algorithms:-

public static byte[] StringHexDecode(string hexData)
{
	List<byte> data = new List<byte>();
	foreach (string b in hexData.Split(' '))
	{
		data.Add(Convert.ToByte(b, 16));
	}
	return data.ToArray();
}
 
public static string StringHexEncode(byte[] hexData)
{
	return BitConverter.ToString(hexData).Replace('-', ' ');
}

And finally here's the actual Yubikey class including the algorithm to decrypt the data that the USB dongles actually output in 'ModHex' which is an odd keyboard language independent format but seems to work quite well:-

/// <summary>
/// A YubiKey OTP is symmetric two-factor auth key, this class allows decoding and validating them
/// </summary>
public class YubiKey
{
	public YubiKey()
	{
	}
 
	private const int CRC_OK_RESIDUE = 0xf0b8;

	/// <summary>
	/// Unique (secret) ID.
	/// </summary>
	public byte[] Uid = new byte[6];
	/// <summary>
	/// Session counter (incremented by 1 at each startup).  High bit
	/// indicates whether caps-lock triggered the token.
	/// </summary>
	public UInt16 SessionCounter;
	/// <summary>
	/// Timestamp incremented by approx 8Hz (low part).
	/// </summary>
	public UInt16 TimestampLowPart;
	/// <summary>
	/// Timestamp (high part).
	/// </summary>
	public byte TimestampHighPart;
	/// <summary>
	/// Number of times used within session + activation flags.
	/// </summary>
	public byte SessionUseCount;
	/// <summary>
	/// Pseudo-random value.
	/// </summary>
	public UInt16 RandomValue;
	/// <summary>
	/// CRC16 value of all fields.
	/// </summary>
	public UInt16 CRC;
 
	/// <summary>
	/// Does the included UID match the one we expected?
	/// </summary>
	/// <param name="uid"></param>
	/// <returns></returns>
	public bool UidMatch(byte[] uid)
	{
		for (int i = 0; i < uid.Length; i++)
		{
			if (Uid[i] != uid[i])
			{
				return false;
			}
		}
		return true;
	}
 
 #region Post Decryption Conversion
 
	private static int calculateCrc(byte[] b)
	{
		int crc = 0xffff;
 
		for (int i = 0; i < b.Length; i += 1)
		{
			crc ^= b[i] & 0xFF;
			for (int j = 0; j < 8; j++)
			{
				int n = crc & 1;
				crc >>= 1;
				if (n != 0)
				{
					crc ^= 0x8408;
				}
			}
		}
		return crc;
	}
 
	internal static YubiKey OtpFromRawByteArray(byte[] input)
	{
		if (input.Length < 16)
		{
			throw new YubiKeyException("Invalid OTP data - the amount supplied was
insufficient for a six byte identity.");
		}
 
		if (calculateCrc(input) != CRC_OK_RESIDUE)
		{
			throw new YubiKeyException("CRC was invalid on that OTP");
		}
 
		YubiKey u = new YubiKey();
		u.Uid = input.Take(6).ToArray();
		u.SessionCounter = BitConverter.ToUInt16(input, 6);
		u.TimestampLowPart = BitConverter.ToUInt16(input, 8);
		u.TimestampHighPart = input[10];
		u.SessionUseCount = input[11];
		u.RandomValue = BitConverter.ToUInt16(input, 12);
		u.CRC = BitConverter.ToUInt16(input, 14);
		return u;
	}
 
	internal static byte[] RawByteArrayFromOtp(YubiKey input)
	{
		List<byte> data = new List<byte>();
		data.AddRange(input.Uid);
		data.AddRange(BitConverter.GetBytes(input.SessionCounter));
		data.AddRange(BitConverter.GetBytes(input.TimestampLowPart));
		data.Add(input.TimestampHighPart);
		data.Add(input.SessionUseCount);
		data.AddRange(BitConverter.GetBytes(input.RandomValue));
		data.AddRange(BitConverter.GetBytes(input.CRC));
		return data.ToArray();
	}
 #endregion
 
 #region Cryptographic wrapper
	internal byte[] AESEncrypt(byte[] data, byte[] key, byte[] iv)
	{
		Aes enc = Aes.Create();
		enc.Key = key;
		enc.IV = iv;
		enc.Padding = PaddingMode.None;
		using (ICryptoTransform transform = enc.CreateEncryptor())
		{
			byte[] output = transform.TransformFinalBlock(data, 0, data.Length);
			return output;
		}
	}
 
	internal static byte[] AESDecrypt(byte[] data, byte[] key, byte[] iv)
	{
		Aes aesImplementation = Aes.Create();
		aesImplementation.Key = key;
		aesImplementation.IV = iv;
		aesImplementation.Padding = PaddingMode.None;
 
		using (ICryptoTransform transform = aesImplementation.CreateDecryptor())
		{
			byte[] output = transform.TransformFinalBlock(data, 0, data.Length);
 
			return output;
		}
	}
 #endregion
 
 #region ModHex Support
 
	private const string alphabet = "cbdefghijklnrtuv";
 
	internal static string ModHexEncode(byte[] data)
	{
		StringBuilder result = new StringBuilder();
 
		for (int i = 0; i < data.Length; i++)
		{
			result.Append(alphabet[(data[i] >> 4) & 0xf]);
			result.Append(alphabet[data[i] & 0xf]);
		}
 
		return result.ToString();
	}
 
	internal static byte[] ModHexDecode(String s)
	{
		List<byte> baos = new List<byte>();
		int len = s.Length;
 
		bool toggle = false;
		int keep = 0;
 
		for (int i = 0; i < len; i++)
		{
			char ch = s[i];
			int n = alphabet.IndexOf(ch.ToString().ToLower());
			if (n == -1)
			{
				throw new YubiKeyException(s + " is not properly encoded");
			}
 
			toggle = !toggle;
 
			if (toggle)
			{
				keep = n;
			}
			else
			{
				baos.Add((byte)((keep << 4) | n));
			}
		}
		return baos.ToArray();
	}
 
 #endregion
 
 #region Factory helpers
	/// <summary>
	/// Create a Yubikey object from the mod hex and the private key
	/// </summary>
	/// <param name="privateKey">The private key to decrypt with</param>
	/// <param name="modHex">The modhex content</param>
	/// <returns>A yubikey object</returns>
	public static YubiKey FromModHex(byte[] privateKey, string modHex)
	{
		//no IV in use
		byte[] inputOTP = ModHexDecode(modHex);
		byte[] decrypted = AESDecrypt(inputOTP, privateKey, new byte[16]);
		return OtpFromRawByteArray(decrypted);
	}
 
	/// <summary>
	/// Convert this yubikey instance into a mod-hex OTP string
	/// </summary>
	/// <param name="privateKey">The private key to encrypt with</param>
	/// <returns>The modhext content</returns>
	public string ToModHex(byte[] privateKey)
	{
		byte[] clearotp = RawByteArrayFromOtp(this);
		byte[] encrypted = AESEncrypt(clearotp, privateKey, new byte[16]);
		return ModHexEncode(encrypted);
	}
 #endregion
 
	public class YubiKeyException : Exception
	{
		public YubiKeyException(string message) : base(message)
		{
 
		}
	}
}

Once decrypted it is a simple matter to verify it is a key with a particular code (the simple act of decryption should do that but there's a UID included inside too) and that the Session counter and SessionUseCount have been incremented appropriately to prevent re-use of existing keys.

Note that if you are trying to decode a key which has a prefix added you are interested in the last 32 characters of the output - the first few characters by default on a key are a static identifier so that it's possible to identify which encryption key to use to decrypt the token; I've turned this off and favour the user having to enter their own username/password prior to using the key.

Sorry for the lack of posts!

Sat, 05 Jun 2010 11:27:44

Just a quick note that posting will resume at some point as I'm going to start some more hobby projects, due to some issues with Zorg Solutions though I took a contract and I try to avoid posting about anything work related on my blog.

I've not been doing much proper software development out of work, just helping the odd person with things and making sure I am up-to-speed with things like Windows Azure, Visual Studio 2010 and the new features of C++0x (I ended up doing quite a bit of C++ at my previous job at one point, and found I can do stuff in it with just a bit more time spent working out what library to use than C#, so it seems sensible to keep up to speed with it!).

Luhn check digit algorithm

Wed, 31 Mar 2010 10:41:03

Here's one from my distant past but which may be useful to me at some point today. I can actually see plenty of quicker ways to achieve this but it's also well tested for generating/checking credit card check digits:-

/// <summary>
    /// Creates/appends Luhn/Mod10 check digits to a specified numeric string
    /// Goes belly up with a text string.
    /// </summary>
    public class LuhnCheck
    {
        /// <summary>
        /// Initialise the LuhnCheck class with a number
        /// </summary>
        /// <param name="numberToAddDigitTo">The number to do work on</param>
        public LuhnCheck(string numberToAddDigitTo)
        {
            _numberToAddDigitTo = numberToAddDigitTo;
            setupok = true;
        }
        /// <summary>
        /// Initialise the LuhnCheck class without a number to start with
        /// </summary>
        public LuhnCheck()
        {
            _numberToAddDigitTo = "";
        }

        private string _numberToAddDigitTo;
        /// <summary>
        /// The number to work on
        /// </summary>
        public string NumberToAddDigitTo
        {
            get
            {
                return _numberToAddDigitTo;
            }
            set
            {
                _numberToAddDigitTo = value;
                setupok = true;
            }
        }
        private bool setupok = false;

        /// <summary>
        /// Add the check digit to the string and return the whole string
        /// </summary>
        /// <returns>The whole string with added check digit</returns>
        public string AddCheckDigit()
        {
            if (setupok)
            {
                NumberToAddDigitTo += Convert.ToString(CreateCheckDigit());
                return NumberToAddDigitTo;
            }
            else
            {
                throw new ArgumentException("Please specify the number to add the digit to.");
            }
        }
        /// <summary>
        /// Creates a check digit without using it
        /// </summary>
        /// <returns>The check digit</returns>
        public int CreateCheckDigit()
        {
            if (setupok)
            {
                return CreateCheckDigit(NumberToAddDigitTo);
            }
            else
            {
                throw new ArgumentException("Please specify the number to calculate the digit for.");
            }
        }
        /// <summary>
        /// Creates the check digit for a specified numeric string - it does not add it, only calculates it
        /// </summary>
        /// <param name="numberToAddDigitTo">The number to work the check digit out for</param>
        /// <returns>The check digit</returns>
        public int CreateCheckDigit(string numberToAddDigitTo)
        {
            string validChars = "0123456789";
            foreach (char c in numberToAddDigitTo)
            {
                if (!(validChars.IndexOf(c) >= 0))
                    throw new FormatException("The value passed in was not a valid number");
            }

            string tocheckdigit = numberToAddDigitTo + "0"; //We add zero to use it to calculate
            //the number from the correct point - since we'll be alternating between multiplying it by one or two
            int strlen = tocheckdigit.Length;
            int ncheck = 0;
            int nweight = 1; //this will alternate between one and two
            int intermediary = 0;
            string tosplit = "";

            for (int i = 1; i <= strlen; i++) //loop through the card number string
            {
                intermediary = nweight * Convert.ToInt32(Convert.ToString(tocheckdigit[strlen - i])); //make intermediary
//be the multiple of the variance and the card number
                if (intermediary > 9) //if it's bigger than ten...
                {
                    tosplit = Convert.ToString(intermediary);
                    //Split the two digits
                    int firstdigit = Convert.ToInt32(Convert.ToString(tosplit[0]));
                    int seconddigit = Convert.ToInt32(Convert.ToString(tosplit[1]));
                    //Then add them together
                    intermediary = firstdigit + seconddigit;
                }
                //Add the intermediary result to the rolling total
                ncheck += intermediary;
                //Change the variance as per the Luhn formula
                if (nweight == 2)
                {
                    nweight = 1;
                }
                else
                {
                    nweight = 2;
                }
            }
            //ncheck now has the total
            int remainder = 0;
            //Work out the remainder
            Math.DivRem(ncheck, 10, out remainder);
            //return ten minus the remainder (if it's zero then it divides perfectly by ten so is valid,
            //but since we're CREATING a check digit we want to know how far off the result is and how much we
	    //need to correct it using the check digit)
            if (10 - remainder == 10)
            {
                return 0;
            }
            else
            {
                return 10 - remainder;
            }

        }
        /// <summary>
        /// Whether the number stored in the object is a valid MOD 10/Luhn check digited number
        /// </summary>
        public bool IsValid
        {
            get
            {
                if (CreateCheckDigit() == 0)
                {
                    return true;
                }
                else
                {
                    return false;
                }
            }
        }
    }

Command Line Arguments in .NET are Broken

Wed, 24 Mar 2010 13:34:52

Attempting to pass paths with a space or any other command line argument that requires being enclosed in quotes was causing me some mild frustration, it looks like the .NET implementation that fills args is incorrectly mangling the data that is passed into it. A major bug that shouldn't have gotten through QA exists somewhere...

If you pass a path into a program that is foreach-ing over the args collection through the debug properties in visual studio (and presumably through anywhere else!) you go from:-

program.exe /path "c:\my documents\" /dostuff 1

Turn into two rather than four arguments:-

args[0] is /path
args[1] is c:\my documents" /dostuff 1

Which is downright strange, as you'd expect to get:-

args[0] is /path
args[1] is c:\my documents\
args[2] is /dostuff
args[3] is 1

What on earth are they doing to cause such a mess to command line arguments... Anyway, whilst I'm not sure when this started but thankfully the Environment.CommandLine is left intact and undamaged, so you can do something like this to fix the problem:-

private static string[] FixedExtractCmdLineElements()
        {
            List<string> elements = new List<string>();

            string[] firstParts = Environment.CommandLine.Split(' ');

            //reparse it
            StringBuilder temporaryPart = new StringBuilder("");
            bool inside = false;
            foreach (string part in firstParts)
            {
                //are we inside a quoted part, or did we just find a quote?
                if (!inside && !part.Contains("\""))
                {
                    elements.Add(part);
                }
                else
                {
                    if (inside)
                    {
                        //we are still inside a quote...
                        temporaryPart.Append(" " + part);

                        if (part.Contains("\""))
                        {
                            //then we are also at the end!
                            elements.Add(temporaryPart.Replace("\"", "").ToString()); //add the part minus its quotes to the array
                            //all done!
                            inside = false;
                        }
                    }
                    else
                    {
                        //else we just found a quote!
                        inside = true;
                        temporaryPart = new StringBuilder(part);
                    }
                }
            }

            return elements.ToArray();
        }

Quick AES encryption methods in .NET

Fri, 19 Mar 2010 15:05:33

This is mostly so I don't have to ever type this in again, but it's most useful when you're also using an RSACryptoProvider to encrypt and send the key to the other end where you're decrypting whatever you are sending, but I thought these might be of use to someone... There's really nothing complicated to these at all.

Note that the iv (initialisation vector) should be some random data that you pass with the encrypted data, but it does not need to be protected. It's used to prevent two duplicate bits of input data outputting the exact same (and therefore being obvious), and to prevent someone making you encrypt something special in a way that you can have your private key exposed.

public static string AESEncrypt(string input, byte[] key, byte[] iv)
{
    Aes enc = Aes.Create();
    enc.Key = key;
    enc.IV = iv;
    using (ICryptoTransform transform = enc.CreateEncryptor())
    {
        byte[] data = Encoding.Unicode.GetBytes(input);
        byte[] output = transform.TransformFinalBlock(data, 0, data.Length);

        return Convert.ToBase64String(output);
    }
}

private static string AESDecrypt(string input, byte[] key, byte[] iv)
{
    Aes aesImplementation = Aes.Create();
    aesImplementation.Key = key;
    aesImplementation.IV = iv;

    using (ICryptoTransform transform = aesImplementation.CreateDecryptor())
    {
        byte[] data = Convert.FromBase64String(input);
        byte[] output = transform.TransformFinalBlock(data, 0, data.Length);

        return Encoding.Unicode.GetString(output);
    }
}

WCF serialisation and deserialisation without sending the object over WCF...

Fri, 19 Mar 2010 15:03:37

I needed to send some data over another medium to WCF and wanted to retain my current objects and classes, but also to retain the serialisation format for ease of direct input at the other end of this communications 'tube'. I was actually sending these messages over XMPP after encrypting them, which is what the next post will be about... It's worth noting that serialization using WCF is marginally faster than the more capable XmlSerializer but at the same time this has its own quirks and tends to produce more verbose XML.

So here's methods to serialise and deserialise DataContract marked up objects to XML strings:-

        public static string ContractObjectToXml<T>(T obj)
        {
            DataContractSerializer dataContractSerializer = new DataContractSerializer(obj.GetType());

            String text;
            using (MemoryStream memoryStream = new MemoryStream())
            {
                dataContractSerializer.WriteObject(memoryStream, obj);
                text = Encoding.UTF8.GetString(memoryStream.ToArray());
            }
            return text;
        }

        public static T XmlToContractObject<T>(string data)
        {
            DataContractSerializer dataContractSerializer = new DataContractSerializer(typeof(T));

            byte[] binary = Encoding.UTF8.GetBytes(data);
            using (MemoryStream memoryStream = new MemoryStream(binary))
            {
                object o = dataContractSerializer.ReadObject(memoryStream);
                return (T)o;
            }
        }

(Google thinks I should spell Serialise incorrectly as Serialize so I'm very conflicted about which I should use in this article as anyone searching would probably look for the American spelling...!)

SQL Compact Edition with multiple threads

Wed, 03 Mar 2010 09:26:03

I was experiencing problems with SQL compact edition (complete .NET framework freezes) and found that not only is it not thread safe (fairly obvious) but it's doesn't like to even have multiple instances of itself on the same thread - it deadlocks a lot if it does. I also found that if it runs out of instances then it unloads from memory and has to be reloaded again afterwards, so avoiding that can result in a many tens of thousands of percent performance improvement.

So here's a small class I wrote to track the connection per thread based on someone elses on the Internet (they were just creating one per thread, in this one it is creating one per thread and maintaining the lifespan of that connection to close it when the thread it was for has been closed). I'll probably add per-connection-string pooling but figured that the important thing is the concept of the solution to the COM deadlocks and total freezing of the framework in this.

/// <summary>
/// Manages open connections on a per-thread basis
/// </summary>
public abstract class SqlCeConnectionPool
{
    private static Dictionary<int, IDbConnection> threadConnectionMap = new Dictionary<int, IDbConnection>();

    private static Dictionary<int, Thread> threadMap = new Dictionary<int, Thread>();

    /// <summary>
    /// The connection map
    /// </summary>
    public static Dictionary<int, IDbConnection> ThreadConnectionMap
    {
        get { return SqlCeConnectionPool.threadConnectionMap; }
    }

    /// <summary>
    /// Gets the connection string.
    /// </summary>
    /// <value>The connection string.</value>
    public static string ConnectionString
    {
        get;
        set;
    }

    /// <summary>
    /// Gets a connection for this thread, maintains one open one of each.
    /// </summary>
    /// <remarks>Don't do this with anything but SQL compact edition or you'll run out of connections - compact edition is not
    /// connection pooling friendly and unloads itself too often otherwise so that is why this class exists</remarks> 
        /// <returns>An open connection</returns>
    public static IDbConnection Connection
    {
        get
        {
            lock (threadConnectionMap)
            {
                //do some quick maintenance on existing connections (closing those that have no thread)
                List<int> removeItems = new List<int>();
                foreach (var kvp in threadConnectionMap)
                {
                    if (threadMap.ContainsKey(kvp.Key))
                    {
                        if (!threadMap[kvp.Key].IsAlive)
                        {
                            //close the connection
                            kvp.Value.Close();
                            removeItems.Add(kvp.Key);
                        }
                    }
                    else
                    {
                        kvp.Value.Close();
                        removeItems.Add(kvp.Key);
                    }
                }
                foreach (int i in removeItems)
                {
                    threadMap.Remove(i);
                    threadConnectionMap.Remove(i);
                }

                //now issue the appropriate connection for our current thread
                int threadId = Thread.CurrentThread.ManagedThreadId;

                IDbConnection connection = null;
                if (threadConnectionMap.ContainsKey(threadId))
                {
                    connection = threadConnectionMap[threadId];
                }
                else
                {
                    connection = new SqlCeConnection(ConnectionString);
                    connection.Open();
                    threadConnectionMap.Add(threadId, connection);
                    threadMap.Add(threadId, Thread.CurrentThread);
                }

                return connection;
            }
        }
    }
}

Zorganiser almost ready for beta test...

Tue, 02 Mar 2010 17:04:35

My integration/automation system 'Zorganiser' is almost ready for a closed beta test (albeit with a limited set of features but enough to be useful!). It'll let you control automated tasks on any number of computers, so you can do things like tell 50 webservers to install a particular application, run a particular Windows Workflow or to just shift data around between your installed agents - for example to take data from one business application and deliver it to another in a slightly tweaked format regardless of where the two applications are. You can also configure one event to trigger another - so the workflow could poll something and then trigger an event across all your workstations or servers.

I already have a few people who are interested in testing it, but if you're also interested please e-mail me at (redacted)!

Dedicated server hosting - almost a year with 1and1...

Tue, 02 Mar 2010 16:58:54

Due to a variety of my circumstances changing and the way the exchange rate was fluctuating I changed from having all-US servers to using rented dedicated servers from 1and1 internet - billed in UK pounds. Right now it's particularly appropriate with both the Euro and the Pound hugely devalued against the US dollar.

I didn't expect much - the plan I picked was half literally half the price I was paying before but I got a higher specification server, unlimited bandwidth with no concern about overages anymore or anything of the sort and a significant improvement on the latency to my Texas based servers (which is understandable given the speed of light, but the network is also less congested and I can regularly get 85Mbps and less than 20ms to the UK when testing it).

So I've been with them almost a year now and the availability has far exceeded that of my previous datacenter - I've had about 2 minutes downtime the entire time, so I thought I would mention it as it's relevant to a project I'm working on at the moment...

Fortunately that does mean I've not managed to see if their support is any good, but the service certainly is.

Be warned that their network setup is not really conducive to VMWare or virtualisation in general - they use 1 ip sized subnets and gateways that you can always access even though they aren't in your subnet through some TCP/IP stack tweaks (I think it's "All subnets are local" combined with the switches having some custom configuration, but I've not yet figured it out and it's the first time I've seen something similar done). Their network does have a nice Cisco switch based firewall for every custom because of the strange setup though, and it lets you have public IP's from different subnets so it's got its advantages (no renumbering!).

File dialog filter string format

Fri, 19 Feb 2010 13:00:09

Just a very quick reminder to myself that the file dialogs filter string (OpenFileDialog.Filter and SaveFileDialog.Filter) format it is "Description|*.ext;AnotherDescription|*.txt" etc.

I always seem to get this around the wrong way for some reason.

MethodAccessException: System.Runtime.CompilerServices.StrongBox`1..ctor(System.__Canon)

Thu, 18 Feb 2010 10:00:50

I'm trying to use a Linq datacontext from a partial trust environment (well, after Assert'ing my way back up) and I'm getting the spectacularly verbose error:-

System.Runtime.CompilerServices.StrongBox`1..ctor(System.__Canon)

Which looks like it's lacking an actual human-readable error but at least it's distinct!

In my case rather than asserting the individually needed parts I had to assert full-trust from my intermediary class (the one that is signed by a strongname to allow it to always be full-trust in my custom AppDomain - see my article on creating a restricted AppDomain - see one of the comments in the code example) between the untrusted plugins (things inside the appdomain with the restrictions applied can call into this and anything the intermediary class does is unrestricted but even things it calls get the inherited permissions from the caller) and the Linq data context user:-

PermissionSet set = new PermissionSet(PermissionState.Unrestricted); //fulltrust especially for Linq :(
//removed my carefully crafted assert set for Linq/SQLCE in exchange for the above being unrestricted:-
//set.AddPermission(new FileIOPermission(PermissionState.Unrestricted));
//set.AddPermission(new AspNetHostingPermission(AspNetHostingPermissionLevel.Unrestricted));
set.Assert();

//Make the call to whatever uses the linq datacontext

Which was a little annoying, but worth knowing. (The assertion will revert when the stack pops from the method this is in, however do be aware not to allow anything under this method to allow the plugin to execute a delegate or something).

I did find a possible workaround of moving a member variable I was accessing into a local variable in the Linq query, but this didn't work in my case where asserting this did.

CSV to Datatable parser

Tue, 09 Feb 2010 10:41:44

I'm sure there's an easier way to do this, but I've been maintaining my own class to do the job for a while now and keep needing to find it in my code library every time I use it - I figure it might be useful to others:-

using System;
using System.Collections.Generic;
using System.Text;
using System.Data;

namespace CSVParser
{
    /// <summary>
    /// Simon's stock CSV parser class
    /// </summary>
    public class CSVParser
    {
        /// <summary>
        /// Takes the CSV files contents and creates a data table from it
        /// </summary>
        /// <param name="csvFileContents">The entire contents of a CSV file split by line (rather than the filename)</param>
        /// <param name="validateLayout">Validate the file to check conformance of the layout with expected standards</param>
        /// <param name="topRowIsHeader">The top row is the header row, take it as the column names rather than data</param>
        /// <returns>A datatable</returns>
        public static DataTable DataTableFromCSV(string[] csvFileContents, bool validateLayout, bool topRowIsHeader)
        {
            DataTable outputDataTable = new DataTable();
            List<string[]> csvFileRows = new List<string[]>();
            List<string> columns = new List<string>();

            #region Pre-parse the file
            int columnCount = 0;

            bool gotHeaders = false;
            int rowNumber = 0;

            foreach (string line in csvFileContents)
            {
                string[] parts = ExtractCSVElements(line);

                //initial set of header names but only if the top row is header option is set
                if (!gotHeaders && topRowIsHeader)
                {
                    columns.AddRange(parts);
                    columnCount = parts.Length;
                    gotHeaders = true;
                }
                else
                {
                    if (parts.Length > 0)
                    {
                        csvFileRows.Add(parts);
                    }
                }

                if (parts.Length > columnCount)
                {
                    //if set to validate the layout and that the first row contains the headers then we know any extra columns are wrong
                    if (validateLayout && gotHeaders)
                    {
                        throw new Exception("Row has extra data columns: " + rowNumber.ToString());
                    }

                    //new column detected mid-data-set!
                    for (int i = columnCount; i < parts.Length; i++)
                    {
                        columns.Add("Column " + i.ToString());
                    }

                    columnCount = parts.Length;
                }

                //we always ignore zero length rows as the last line can be empty
                if (parts.Length < columnCount && parts.Length != 0)
                {
                    if (validateLayout)
                    {
                        throw new Exception("Row has missing data columns: " + rowNumber.ToString());
                    }
                }


                rowNumber++;
            }

            #endregion

            #region Build the data tables layout and data

            //columns
            foreach (string column in columns)
            {
                outputDataTable.Columns.Add(column);
            }

            //rows
            foreach (string[] row in csvFileRows)
            {
                outputDataTable.Rows.Add(row);
            }
            #endregion

            return outputDataTable;
        }

        /// <summary>
        /// Extract the elements of a line from a CSV file with support for quotes
        /// </summary>
        /// <param name="line">The data to parse</param>
        private static string[] ExtractCSVElements(string line)
        {
            List<string> elements = new List<string>();

            //do the initial split, based on commas
            string[] firstParts = line.Split(',');

            //reparse it
            StringBuilder temporaryPart = new StringBuilder("");
            bool inside = false;
            foreach (string part in firstParts)
            {
                //are we inside a quoted part, or did we just find a quote?
                if (!inside //we're not inside
                && (!part.Contains("\"") //and we don't contain a quote
                 || ( //or we're handling a single quote enclosed element
                        part.StartsWith("\"") //we start with a quote
                        && part.EndsWith("\"") //and we end with a quote)
                    )
                )
                {
                    elements.Add(part);
                }
                else
                {
                    if (inside)
                    {
                        //we are still inside a quote...
                        temporaryPart.Append("," + part);

                        if (part.Contains("\""))
                        {
                            //then we are also at the end!
                            elements.Add(temporaryPart.Replace("\"", "").ToString()); //add the part minus its quotes to the array
                            //all done!
                            inside = false;
                        }
                    }
                    else
                    {
                        //else we just found a quote!
                        inside = true;
                        temporaryPart = new StringBuilder(part);
                    }
                }
            }

            return elements.ToArray();
        }
    }
}

Come work for me in Basingstoke!

Fri, 29 Jan 2010 11:05:18

Edit: Due to some investment problems I can no longer employ anyone, sorry :(

My new company - Zorg Solutions Ltd is now looking for at least one software developer to fill out a team, preferably you'll be an enthusiastic multi-skilled person but we're flexible and will distribute work based on what you can do.

Do you have a background as a C# software developer that has done either Winforms or ASP.NET in the past? You will need to have at least a couple of years of experience developing with the .NET framework but I will be willing to take hobby or open source work as experience if it's provable that you did it. This could be a great opportunity to jump into the software development world for someone - there's no requirement for a degree or other certification. Any experience of Windows Azure, Silverlight or WPF would be an advantage.

Or do you have any experience (hobby or professionally) in electronics, firmware, C/C++? We have contracts for some hardware development and the opportunity to work with and program the hardware we'll build - probably Atmel microprocessors or PIC microcontrollers, but even some Linux on ARM9 based chips.

If you answered yes to either of these questions and would be willing to commute to Basingstoke then I am interested in hearing from you! Any experience of the following would make you stand out but is not essential:-

SQL experience (preferably SQL server's T-SQL dialect)
Windows Workflow Foundation
Biztalk
Silverlight
Mono

Salary will be negotiable (but as an idea we would be willing to pay either side of £35k) and based on experience. Working hours and benefits will be fully negotiable - we can be very flexible and understand if you have personal commitments. Training and an MSDN subscription will also be provided!

If you apply (just send me your CV) or contact me now to discuss in more detail then you will be ahead of the conventional advertisements for these roles - we'll be putting them out conditional on responses to this posting and a few other enquiries - so you have nothing to lose! No recruiters yet please!

Edit: Due to some investment problems I can no longer employ anyone, sorry :(