NullifyNetwork

The blog and home page of Simon Soanes
Skip to content
[ Log On ]

Archive - Historical Articles

You are viewing records from 08/06/2009 21:03:08 to 07/31/2010 16:58:27. I'll be adding support for selecting a date range in future.

So I have encountered a situation where I need to use two factor authentication and recently bought a couple of Yubico YubiKey's to investigate them as a potential solution where I can't use smart cards (mostly with other users who can't install the appropriate drivers or on locked down systems/internet cafe's).

They are fairly cheap (particularly in bulk) and provide similar functionality to an RSA Securid but without the need to type the password (they act as a keyboard and type onto the computer, but can't be modified by the computer in question).  They also have two 'slots' for different tokens, you can touch the button once for one and touch and hold for a second one time key.

Whilst there's plenty of code examples around for them (the company are extremely open), I haven't found a simple, logical C# library/class I could use to do authentication locally (they supply ones for remote auth on their servers) - there were always multiple files and loads of code in peoples projects.

My intention is to write a Windows SubAuthentication DLL (in C++ but I learn in C#) to use with these keys which also still checks your normal password - so when remote desktop'ing, using something that authenticates to Windows or using the Windows Radius server (NPS) to authenticate VPN's you can choose to either supply your normal password, or if you're on a system you don't trust you can use your username/low security pin code and Yubikey.

So here's my take on the decoding of a Yubikey's input - note that I'm not interested in making a software token though so the class doesn't generate values for the random value (if you wanted to do this you just need to ask RNGCryptoProvider for a few bytes of random though) and there's also no incrementing of the sessions, in-session use counters or timestamps included - however I did include the code to encrypt it appropriately if you do supply all the values.

To use this class you just need to call it like so - where privateKey is a byte[] array with the private key in and otpCode is a string like "fkfthfffktbnreffrghcldrffeclcgkt" output from a touch of the Yubikey:-

YubiKey otp = YubiKey.FromModHex(privateKey, otpCode);
if (otp.UidMatch(checkIdentity))
{
Console.WriteLine("Decoded the YubiKey's OTP code:-");
Console.WriteLine("\tSession:\t" + otp.SessionCounter.ToString());
Console.WriteLine("\tCount in session:\t" + otp.SessionUseCount.ToString());
Console.WriteLine("\tTime (high):\t" + otp.TimestampHighPart.ToString());
Console.WriteLine("\tTime (low):\t" + otp.TimestampLowPart.ToString());
Console.WriteLine("\tTime (low):\t" + otp.Uid.ToString());
}
else
{
Console.WriteLine("Failed to identify the keys correct OTP");
}

 

And you can decode the private key/identity as they're displayed in the YubiKey personalisation tool with the following handy (but not super efficient) hex to byte algorithms:-

 

public static byte[] StringHexDecode(string hexData)
{
List<byte> data = new List<byte>();
foreach (string b in hexData.Split(' '))
{
data.Add(Convert.ToByte(b, 16));
}
return data.ToArray();
}

public static string StringHexEncode(byte[] hexData)
{
return BitConverter.ToString(hexData).Replace('-'' ');
}

And finally here's the actual Yubikey class including the algorithm to decrypt the data that the USB dongles actually output in 'ModHex' which is an odd keyboard language independent format but seems to work quite well:-

/// <summary>
/// A YubiKey OTP is symmetric two-factor auth key, this class allows decoding and validating them
/// </summary> public class YubiKey {
public YubiKey()
{
}

private const int CRC_OK_RESIDUE = 0xf0b8; /// <summary>
/// Unique (secret) ID.
/// </summary> public byte[] Uid = new byte[6]; /// <summary>
/// Session counter (incremented by 1 at each startup).  High bit
/// indicates whether caps-lock triggered the token.
/// </summary>
public UInt16 SessionCounter; /// <summary>
/// Timestamp incremented by approx 8Hz (low part).
/// </summary>
public UInt16 TimestampLowPart; /// <summary>
/// Timestamp (high part).
/// </summary>
public byte TimestampHighPart;
/// <summary>
/// Number of times used within session + activation flags.
/// </summary> public byte SessionUseCount; /// <summary>
/// Pseudo-random value.
/// </summary>
public UInt16 RandomValue; /// <summary>
/// CRC16 value of all fields.
/// </summary>
public UInt16 CRC;

/// <summary>
/// Does the included UID match the one we expected?
/// </summary>
/// <param name="uid"></param>
/// <returns></returns>
public bool UidMatch(byte[] uid)
{
for (int i = 0; i < uid.Length; i++)
{
if (Uid[i] != uid[i])
{
return false;
}
}
return true;
}

#region Post Decryption Conversion

private static int calculateCrc(byte[] b)
{
int crc = 0xffff;

for (int i = 0; i < b.Length; i += 1)
{
crc ^= b[i] & 0xFF;
for (int j = 0; j < 8; j++)
{
int n = crc & 1;
crc >>= 1;
if (n != 0)
{
crc ^= 0x8408;
}
}
}
return crc;
}

internal static YubiKey OtpFromRawByteArray(byte[] input)
{
if (input.Length < 16)
{
throw new YubiKeyException("Invalid OTP data - the amount supplied was
insufficient for a six byte identity."
);
}

if (calculateCrc(input) != CRC_OK_RESIDUE)
{
throw new YubiKeyException("CRC was invalid on that OTP");
}

YubiKey u = new YubiKey();
u.Uid = input.Take(6).ToArray();
u.SessionCounter = BitConverter.ToUInt16(input, 6);
u.TimestampLowPart = BitConverter.ToUInt16(input, 8);
u.TimestampHighPart = input[10];
u.SessionUseCount = input[11];
u.RandomValue = BitConverter.ToUInt16(input, 12);
u.CRC = BitConverter.ToUInt16(input, 14);
return u;
}

internal static byte[] RawByteArrayFromOtp(YubiKey input)
{
List<byte> data = new List<byte>();
data.AddRange(input.Uid);
data.AddRange(BitConverter.GetBytes(input.SessionCounter));
data.AddRange(BitConverter.GetBytes(input.TimestampLowPart));
data.Add(input.TimestampHighPart);
data.Add(input.SessionUseCount);
data.AddRange(BitConverter.GetBytes(input.RandomValue));
data.AddRange(BitConverter.GetBytes(input.CRC));
return data.ToArray();
}
#endregion

#region Cryptographic wrapper
internal byte[] AESEncrypt(byte[] data, byte[] key, byte[] iv)
{
Aes enc = Aes.Create();
enc.Key = key;
enc.IV = iv;
enc.Padding = PaddingMode.None;
using (ICryptoTransform transform = enc.CreateEncryptor())
{
byte[] output = transform.TransformFinalBlock(data, 0, data.Length);
return output;
}
}

internal static byte[] AESDecrypt(byte[] data, byte[] key, byte[] iv)
{
Aes aesImplementation = Aes.Create();
aesImplementation.Key = key;
aesImplementation.IV = iv;
aesImplementation.Padding = PaddingMode.None;

using (ICryptoTransform transform = aesImplementation.CreateDecryptor())
{
byte[] output = transform.TransformFinalBlock(data, 0, data.Length);

return output;
}
}
#endregion

#region ModHex Support

private const string alphabet = "cbdefghijklnrtuv";

internal static string ModHexEncode(byte[] data)
{
StringBuilder result = new StringBuilder();

for (int i = 0; i < data.Length; i++)
{
result.Append(alphabet[(data[i] >> 4) & 0xf]);
result.Append(alphabet[data[i] & 0xf]);
}

return result.ToString();
}

internal static byte[] ModHexDecode(String s)
{
List<byte> baos = new List<byte>();
int len = s.Length;

bool toggle = false;
int keep = 0;

for (int i = 0; i < len; i++)
{
char ch = s[i];
int n = alphabet.IndexOf(ch.ToString().ToLower());
if (n == -1)
{
throw new YubiKeyException(s + " is not properly encoded");
}

toggle = !toggle;

if (toggle)
{
keep = n;
}
else
{
baos.Add((byte)((keep << 4) | n));
}
}
return baos.ToArray();
}

#endregion

#region Factory helpers
/// <summary>
/// Create a Yubikey object from the mod hex and the private key
/// </summary>
/// <param name="privateKey">The private key to decrypt with</param>
/// <param name="modHex">The modhex content</param>
/// <returns>A yubikey object</returns>
public static YubiKey FromModHex(byte[] privateKey, string modHex)
{
//no IV in use
byte[] inputOTP = ModHexDecode(modHex);
byte[] decrypted = AESDecrypt(inputOTP, privateKey, new byte[16]);
return OtpFromRawByteArray(decrypted);
}

/// <summary>
/// Convert this yubikey instance into a mod-hex OTP string
/// </summary>
/// <param name="privateKey">The private key to encrypt with</param>
/// <returns>The modhext content</returns>
public string ToModHex(byte[] privateKey)
{
byte[] clearotp = RawByteArrayFromOtp(this);
byte[] encrypted = AESEncrypt(clearotp, privateKey, new byte[16]);
return ModHexEncode(encrypted);
}
#endregion

public class YubiKeyException : Exception
{
public YubiKeyException(string message) : base(message)
{

}
} }

Once decrypted it is a simple matter to verify it is a key with a particular code (the simple act of decryption should do that but there's a UID included inside too) and that the Session counter and SessionUseCount have been incremented appropriately to prevent re-use of existing keys.

Note that if you are trying to decode a key which has a prefix added you are interested in the last 32 characters of the output - the first few characters by default on a key are a static identifier so that it's possible to identify which encryption key to use to decrypt the token; I've turned this off and favour the user having to enter their own username/password prior to using the key.

Permalink 

Just a quick note that posting will resume at some point as I'm going to start some more hobby projects, due to some issues with Zorg Solutions though I took a contract and I try to avoid posting about anything work related on my blog.

I've not been doing much proper software development out of work, just helping the odd person with things and making sure I am up-to-speed with things like Windows Azure, Visual Studio 2010 and the new features of C++0x (I ended up doing quite a bit of C++ at my previous job at one point, and found I can do stuff in it with just a bit more time spent working out what library to use than C#, so it seems sensible to keep up to speed with it!).

Permalink 

Here's one from my distant past but which may be useful to me at some point today. I can actually see plenty of quicker ways to achieve this but it's also well tested for generating/checking credit card check digits:-

/// <summary>
    /// Creates/appends Luhn/Mod10 check digits to a specified numeric string
    /// Goes belly up with a text string.
    /// </summary>
    public class LuhnCheck
    {
        /// <summary>
        /// Initialise the LuhnCheck class with a number
        /// </summary>
        /// <param name="numberToAddDigitTo">The number to do work on</param>
        public LuhnCheck(string numberToAddDigitTo)
        {
            _numberToAddDigitTo = numberToAddDigitTo;
            setupok = true;
        }
        /// <summary>
        /// Initialise the LuhnCheck class without a number to start with
        /// </summary>
        public LuhnCheck()
        {
            _numberToAddDigitTo = "";
        }

        private string _numberToAddDigitTo;
        /// <summary>
        /// The number to work on
        /// </summary>
        public string NumberToAddDigitTo
        {
            get
            {
                return _numberToAddDigitTo;
            }
            set
            {
                _numberToAddDigitTo = value;
                setupok = true;
            }
        }
        private bool setupok = false;

        /// <summary>
        /// Add the check digit to the string and return the whole string
        /// </summary>
        /// <returns>The whole string with added check digit</returns>
        public string AddCheckDigit()
        {
            if (setupok)
            {
                NumberToAddDigitTo += Convert.ToString(CreateCheckDigit());
                return NumberToAddDigitTo;
            }
            else
            {
                throw new ArgumentException("Please specify the number to add the digit to.");
            }
        }
        /// <summary>
        /// Creates a check digit without using it
        /// </summary>
        /// <returns>The check digit</returns>
        public int CreateCheckDigit()
        {
            if (setupok)
            {
                return CreateCheckDigit(NumberToAddDigitTo);
            }
            else
            {
                throw new ArgumentException("Please specify the number to calculate the digit for.");
            }
        }
        /// <summary>
        /// Creates the check digit for a specified numeric string - it does not add it, only calculates it
        /// </summary>
        /// <param name="numberToAddDigitTo">The number to work the check digit out for</param>
        /// <returns>The check digit</returns>
        public int CreateCheckDigit(string numberToAddDigitTo)
        {
            string validChars = "0123456789";
            foreach (char c in numberToAddDigitTo)
            {
                if (!(validChars.IndexOf(c) >= 0))
                    throw new FormatException("The value passed in was not a valid number");
            }

            string tocheckdigit = numberToAddDigitTo + "0"; //We add zero to use it to calculate
            //the number from the correct point - since we'll be alternating between multiplying it by one or two
            int strlen = tocheckdigit.Length;
            int ncheck = 0;
            int nweight = 1; //this will alternate between one and two
            int intermediary = 0;
            string tosplit = "";

            for (int i = 1; i <= strlen; i++) //loop through the card number string
            {
                intermediary = nweight * Convert.ToInt32(Convert.ToString(tocheckdigit[strlen - i])); //make intermediary
//be the multiple of the variance and the card number
                if (intermediary > 9) //if it's bigger than ten...
                {
                    tosplit = Convert.ToString(intermediary);
                    //Split the two digits
                    int firstdigit = Convert.ToInt32(Convert.ToString(tosplit[0]));
                    int seconddigit = Convert.ToInt32(Convert.ToString(tosplit[1]));
                    //Then add them together
                    intermediary = firstdigit + seconddigit;
                }
                //Add the intermediary result to the rolling total
                ncheck += intermediary;
                //Change the variance as per the Luhn formula
                if (nweight == 2)
                {
                    nweight = 1;
                }
                else
                {
                    nweight = 2;
                }
            }
            //ncheck now has the total
            int remainder = 0;
            //Work out the remainder
            Math.DivRem(ncheck, 10, out remainder);
            //return ten minus the remainder (if it's zero then it divides perfectly by ten so is valid,
            //but since we're CREATING a check digit we want to know how far off the result is and how much we
	    //need to correct it using the check digit)
            if (10 - remainder == 10)
            {
                return 0;
            }
            else
            {
                return 10 - remainder;
            }

        }
        /// <summary>
        /// Whether the number stored in the object is a valid MOD 10/Luhn check digited number
        /// </summary>
        public bool IsValid
        {
            get
            {
                if (CreateCheckDigit() == 0)
                {
                    return true;
                }
                else
                {
                    return false;
                }
            }
        }
    }
Permalink 

Attempting to pass paths with a space or any other command line argument that requires being enclosed in quotes was causing me some mild frustration, it looks like the .NET implementation that fills args is incorrectly mangling the data that is passed into it.  A major bug that shouldn't have gotten through QA exists somewhere...

If you pass a path into a program that is foreach-ing over the args collection through the debug properties in visual studio (and presumably through anywhere else!) you go from:-

program.exe /path "c:\my documents\" /dostuff 1

Turn into two rather than four arguments:-

args[0] is /path
args[1] is c:\my documents" /dostuff 1

Which is downright strange, as you'd expect to get:-

args[0] is /path
args[1] is c:\my documents\
args[2] is /dostuff
args[3] is 1

What on earth are they doing to cause such a mess to command line arguments... Anyway, whilst I'm not sure when this started but thankfully the Environment.CommandLine is left intact and undamaged, so you can do something like this to fix the problem:-

private static string[] FixedExtractCmdLineElements()
        {
            List<string> elements = new List<string>();

            string[] firstParts = Environment.CommandLine.Split(' ');

            //reparse it
            StringBuilder temporaryPart = new StringBuilder("");
            bool inside = false;
            foreach (string part in firstParts)
            {
                //are we inside a quoted part, or did we just find a quote?
                if (!inside && !part.Contains("\""))
                {
                    elements.Add(part);
                }
                else
                {
                    if (inside)
                    {
                        //we are still inside a quote...
                        temporaryPart.Append(" " + part);

                        if (part.Contains("\""))
                        {
                            //then we are also at the end!
                            elements.Add(temporaryPart.Replace("\"", "").ToString()); //add the part minus its quotes to the array
                            //all done!
                            inside = false;
                        }
                    }
                    else
                    {
                        //else we just found a quote!
                        inside = true;
                        temporaryPart = new StringBuilder(part);
                    }
                }
            }

            return elements.ToArray();
        }
Permalink  2 Comments 

This is mostly so I don't have to ever type this in again, but it's most useful when you're also using an RSACryptoProvider to encrypt and send the key to the other end where you're decrypting whatever you are sending, but I thought these might be of use to someone...  There's really nothing complicated to these at all.

Note that the iv (initialisation vector) should be some random data that you pass with the encrypted data, but it does not need to be protected.  It's used to prevent two duplicate bits of input data outputting the exact same (and therefore being obvious), and to prevent someone making you encrypt something special in a way that you can have your private key exposed.

public static string AESEncrypt(string input, byte[] key, byte[] iv)
{
    Aes enc = Aes.Create();
    enc.Key = key;
    enc.IV = iv;
    using (ICryptoTransform transform = enc.CreateEncryptor())
    {
        byte[] data = Encoding.Unicode.GetBytes(input);
        byte[] output = transform.TransformFinalBlock(data, 0, data.Length);

        return Convert.ToBase64String(output);
    }
}

private static string AESDecrypt(string input, byte[] key, byte[] iv)
{
    Aes aesImplementation = Aes.Create();
    aesImplementation.Key = key;
    aesImplementation.IV = iv;

    using (ICryptoTransform transform = aesImplementation.CreateDecryptor())
    {
        byte[] data = Convert.FromBase64String(input);
        byte[] output = transform.TransformFinalBlock(data, 0, data.Length);

        return Encoding.Unicode.GetString(output);
    }
}
Permalink 

I needed to send some data over another medium to WCF and wanted to retain my current objects and classes, but also to retain the serialisation format for ease of direct input at the other end of this communications 'tube'.  I was actually sending these messages over XMPP after encrypting them, which is what the next post will be about...  It's worth noting that serialization using WCF is marginally faster than the more capable XmlSerializer but at the same time this has its own quirks and tends to produce more verbose XML.

So here's methods to serialise and deserialise DataContract marked up objects to XML strings:-

        public static string ContractObjectToXml<T>(T obj)
        {
            DataContractSerializer dataContractSerializer = new DataContractSerializer(obj.GetType());

            String text;
            using (MemoryStream memoryStream = new MemoryStream())
            {
                dataContractSerializer.WriteObject(memoryStream, obj);
                text = Encoding.UTF8.GetString(memoryStream.ToArray());
            }
            return text;
        }

        public static T XmlToContractObject<T>(string data)
        {
            DataContractSerializer dataContractSerializer = new DataContractSerializer(typeof(T));

            byte[] binary = Encoding.UTF8.GetBytes(data);
            using (MemoryStream memoryStream = new MemoryStream(binary))
            {
                object o = dataContractSerializer.ReadObject(memoryStream);
                return (T)o;
            }
        }
 
(Google thinks I should spell Serialise incorrectly as Serialize so I'm very conflicted about which I should use in this article as anyone searching would probably look for the American spelling...!)
Permalink 

I was experiencing problems with SQL compact edition (complete .NET framework freezes) and found that not only is it not thread safe (fairly obvious) but it's doesn't like to even have multiple instances of itself on the same thread - it deadlocks a lot if it does.  I also found that if it runs out of instances then it unloads from memory and has to be reloaded again afterwards, so avoiding that can result in a many tens of thousands of percent performance improvement.

So here's a small class I wrote to track the connection per thread based on someone elses on the Internet (they were just creating one per thread, in this one it is creating one per thread and maintaining the lifespan of that connection to close it when the thread it was for has been closed).  I'll probably add per-connection-string pooling but figured that the important thing is the concept of the solution to the COM deadlocks and total freezing of the framework in this.

/// <summary>
/// Manages open connections on a per-thread basis
/// </summary>
public abstract class SqlCeConnectionPool
{
    private static Dictionary<int, IDbConnection> threadConnectionMap = new Dictionary<int, IDbConnection>();

    private static Dictionary<int, Thread> threadMap = new Dictionary<int, Thread>();

    /// <summary>
    /// The connection map
    /// </summary>
    public static Dictionary<int, IDbConnection> ThreadConnectionMap
    {
        get { return SqlCeConnectionPool.threadConnectionMap; }
    }

    /// <summary>
    /// Gets the connection string.
    /// </summary>
    /// <value>The connection string.</value>
    public static string ConnectionString
    {
        get;
        set;
    }

    /// <summary>
    /// Gets a connection for this thread, maintains one open one of each.
    /// </summary>
    /// <remarks>Don't do this with anything but SQL compact edition or you'll run out of connections - compact edition is not
    /// connection pooling friendly and unloads itself too often otherwise so that is why this class exists</remarks> 
        /// <returns>An open connection</returns>
    public static IDbConnection Connection
    {
        get
        {
            lock (threadConnectionMap)
            {
                //do some quick maintenance on existing connections (closing those that have no thread)
                List<int> removeItems = new List<int>();
                foreach (var kvp in threadConnectionMap)
                {
                    if (threadMap.ContainsKey(kvp.Key))
                    {
                        if (!threadMap[kvp.Key].IsAlive)
                        {
                            //close the connection
                            kvp.Value.Close();
                            removeItems.Add(kvp.Key);
                        }
                    }
                    else
                    {
                        kvp.Value.Close();
                        removeItems.Add(kvp.Key);
                    }
                }
                foreach (int i in removeItems)
                {
                    threadMap.Remove(i);
                    threadConnectionMap.Remove(i);
                }

                //now issue the appropriate connection for our current thread
                int threadId = Thread.CurrentThread.ManagedThreadId;

                IDbConnection connection = null;
                if (threadConnectionMap.ContainsKey(threadId))
                {
                    connection = threadConnectionMap[threadId];
                }
                else
                {
                    connection = new SqlCeConnection(ConnectionString);
                    connection.Open();
                    threadConnectionMap.Add(threadId, connection);
                    threadMap.Add(threadId, Thread.CurrentThread);
                }

                return connection;
            }
        }
    }
}
Permalink 

My integration/automation system 'Zorganiser' is almost ready for a closed beta test (albeit with a limited set of features but enough to be useful!).  It'll let you control automated tasks on any number of computers, so you can do things like tell 50 webservers to install a particular application, run a particular Windows Workflow or to just shift data around between your installed agents - for example to take data from one business application and deliver it to another in a slightly tweaked format regardless of where the two applications are.  You can also configure one event to trigger another - so the workflow could poll something and then trigger an event across all your workstations or servers.

I already have a few people who are interested in testing it, but if you're also interested please e-mail me at (redacted)!

Permalink 

Due to a variety of my circumstances changing and the way the exchange rate was fluctuating I changed from having all-US servers to using rented dedicated servers from 1and1 internet - billed in UK pounds. Right now it's particularly appropriate with both the Euro and the Pound hugely devalued against the US dollar.

I didn't expect much - the plan I picked was half literally half the price I was paying before but I got a higher specification server, unlimited bandwidth with no concern about overages anymore or anything of the sort and a significant improvement on the latency to my Texas based servers (which is understandable given the speed of light, but the network is also less congested and I can regularly get 85Mbps and less than 20ms to the UK when testing it).

So I've been with them almost a year now and the availability has far exceeded that of my previous datacenter - I've had about 2 minutes downtime the entire time, so I thought I would mention it as it's relevant to a project I'm working on at the moment...

Fortunately that does mean I've not managed to see if their support is any good, but the service certainly is.

Be warned that their network setup is not really conducive to VMWare or virtualisation in general - they use 1 ip sized subnets and gateways that you can always access even though they aren't in your subnet through some TCP/IP stack tweaks (I think it's "All subnets are local" combined with the switches having some custom configuration, but I've not yet figured it out and it's the first time I've seen something similar done).  Their network does have a nice Cisco switch based firewall for every custom because of the strange setup though, and it lets you have public IP's from different subnets so it's got its advantages (no renumbering!).

Permalink 

Just a very quick reminder to myself that the file dialogs filter string (OpenFileDialog.Filter and SaveFileDialog.Filter) format it is "Description|*.ext;AnotherDescription|*.txt" etc.

I always seem to get this around the wrong way for some reason.

Permalink 

I'm trying to use a Linq datacontext from a partial trust environment (well, after Assert'ing my way back up) and I'm getting the spectacularly verbose error:-

System.Runtime.CompilerServices.StrongBox`1..ctor(System.__Canon)

Which looks like it's lacking an actual human-readable error but at least it's distinct!

In my case rather than asserting the individually needed parts I had to assert full-trust from my intermediary class (the one that is signed by a strongname to allow it to always be full-trust in my custom AppDomain - see my article on creating a restricted AppDomain - see one of the comments in the code example) between the untrusted plugins (things inside the appdomain with the restrictions applied can call into this and anything the intermediary class does is unrestricted but even things it calls get the inherited permissions from the caller) and the Linq data context user:-

PermissionSet set = new PermissionSet(PermissionState.Unrestricted); //fulltrust especially for Linq :(
//removed my carefully crafted assert set for Linq/SQLCE in exchange for the above being unrestricted:-
//set.AddPermission(new FileIOPermission(PermissionState.Unrestricted));
//set.AddPermission(new AspNetHostingPermission(AspNetHostingPermissionLevel.Unrestricted));
set.Assert();

//Make the call to whatever uses the linq datacontext

Which was a little annoying, but worth knowing.  (The assertion will revert when the stack pops from the method this is in, however do be aware not to allow anything under this method to allow the plugin to execute a delegate or something).
 
I did find a possible workaround of moving a member variable I was accessing into a local variable in the Linq query, but this didn't work in my case where asserting this did.
 
Permalink 

I'm sure there's an easier way to do this, but I've been maintaining my own class to do the job for a while now and keep needing to find it in my code library every time I use it - I figure it might be useful to others:- 

using System;
using System.Collections.Generic;
using System.Text;
using System.Data;

namespace CSVParser
{
    /// <summary>
    /// Simon's stock CSV parser class
    /// </summary>
    public class CSVParser
    {
        /// <summary>
        /// Takes the CSV files contents and creates a data table from it
        /// </summary>
        /// <param name="csvFileContents">The entire contents of a CSV file split by line (rather than the filename)</param>
        /// <param name="validateLayout">Validate the file to check conformance of the layout with expected standards</param>
        /// <param name="topRowIsHeader">The top row is the header row, take it as the column names rather than data</param>
        /// <returns>A datatable</returns>
        public static DataTable DataTableFromCSV(string[] csvFileContents, bool validateLayout, bool topRowIsHeader)
        {
            DataTable outputDataTable = new DataTable();
            List<string[]> csvFileRows = new List<string[]>();
            List<string> columns = new List<string>();

            #region Pre-parse the file
            int columnCount = 0;

            bool gotHeaders = false;
            int rowNumber = 0;

            foreach (string line in csvFileContents)
            {
                string[] parts = ExtractCSVElements(line);

                //initial set of header names but only if the top row is header option is set
                if (!gotHeaders && topRowIsHeader)
                {
                    columns.AddRange(parts);
                    columnCount = parts.Length;
                    gotHeaders = true;
                }
                else
                {
                    if (parts.Length > 0)
                    {
                        csvFileRows.Add(parts);
                    }
                }

                if (parts.Length > columnCount)
                {
                    //if set to validate the layout and that the first row contains the headers then we know any extra columns are wrong
                    if (validateLayout && gotHeaders)
                    {
                        throw new Exception("Row has extra data columns: " + rowNumber.ToString());
                    }

                    //new column detected mid-data-set!
                    for (int i = columnCount; i < parts.Length; i++)
                    {
                        columns.Add("Column " + i.ToString());
                    }

                    columnCount = parts.Length;
                }

                //we always ignore zero length rows as the last line can be empty
                if (parts.Length < columnCount && parts.Length != 0)
                {
                    if (validateLayout)
                    {
                        throw new Exception("Row has missing data columns: " + rowNumber.ToString());
                    }
                }


                rowNumber++;
            }

            #endregion

            #region Build the data tables layout and data

            //columns
            foreach (string column in columns)
            {
                outputDataTable.Columns.Add(column);
            }

            //rows
            foreach (string[] row in csvFileRows)
            {
                outputDataTable.Rows.Add(row);
            }
            #endregion

            return outputDataTable;
        }

        /// <summary>
        /// Extract the elements of a line from a CSV file with support for quotes
        /// </summary>
        /// <param name="line">The data to parse</param>
        private static string[] ExtractCSVElements(string line)
        {
            List<string> elements = new List<string>();

            //do the initial split, based on commas
            string[] firstParts = line.Split(',');

            //reparse it
            StringBuilder temporaryPart = new StringBuilder("");
            bool inside = false;
            foreach (string part in firstParts)
            {
                //are we inside a quoted part, or did we just find a quote?
                if (!inside //we're not inside
                && (!part.Contains("\"") //and we don't contain a quote
                 || ( //or we're handling a single quote enclosed element
                        part.StartsWith("\"") //we start with a quote
                        && part.EndsWith("\"") //and we end with a quote)
                    )
                )
                {
                    elements.Add(part);
                }
                else
                {
                    if (inside)
                    {
                        //we are still inside a quote...
                        temporaryPart.Append("," + part);

                        if (part.Contains("\""))
                        {
                            //then we are also at the end!
                            elements.Add(temporaryPart.Replace("\"", "").ToString()); //add the part minus its quotes to the array
                            //all done!
                            inside = false;
                        }
                    }
                    else
                    {
                        //else we just found a quote!
                        inside = true;
                        temporaryPart = new StringBuilder(part);
                    }
                }
            }

            return elements.ToArray();
        }
    }
}

 

Permalink 

Edit: Due to some investment problems I can no longer employ anyone, sorry :(

My new company - Zorg Solutions Ltd is now looking for at least one software developer to fill out a team, preferably you'll be an enthusiastic multi-skilled person but we're flexible and will distribute work based on what you can do.

Do you have a background as a C# software developer that has done either Winforms or ASP.NET in the past?  You will need to have at least a couple of years of experience developing with the .NET framework but I will be willing to take hobby or open source work as experience if it's provable that you did it.  This could be a great opportunity to jump into the software development world for someone - there's no requirement for a degree or other certification.  Any experience of Windows Azure, Silverlight or WPF would be an advantage.

Or do you have any experience (hobby or professionally) in electronics, firmware, C/C++?  We have contracts for some hardware development and the opportunity to work with and program the hardware we'll build - probably Atmel microprocessors or PIC microcontrollers, but even some Linux on ARM9 based chips.

If you answered yes to either of these questions and would be willing to commute to Basingstoke then I am interested in hearing from you!  Any experience of the following would make you stand out but is not essential:-

  • SQL experience (preferably SQL server's T-SQL dialect)
  • Windows Workflow Foundation
  • Biztalk
  • Silverlight
  • Mono

Salary will be negotiable (but as an idea we would be willing to pay either side of £35k) and based on experience.  Working hours and benefits will be fully negotiable - we can be very flexible and understand if you have personal commitments.  Training and an MSDN subscription will also be provided!

If you apply (just send me your CV) or contact me now to discuss in more detail then you will be ahead of the conventional advertisements for these roles - we'll be putting them out conditional on responses to this posting and a few other enquiries - so you have nothing to lose!  No recruiters yet please!

Edit: Due to some investment problems I can no longer employ anyone, sorry :(

Permalink 

I finally set up my new companies website (thanks to the snow here totally blocking the ability to get to the office!) - it's lacking content at the moment but the design is in place and I think it looks excellent and embodies the company quite well.

http://www.zorg.co.uk/ 

What do you think?  Anything I should change?

I went through the images on fotolia for quite a while before settling on that one for the front page.  We're using SIP phones for the telephony and I am hoping to have my automation system at the alpha stage within a few months so I'll be looking for people to test it before we go public with a beta (Google style!).  I already have a prototype Workflow designer but at the moment there's not enough to use in production yet.

I am hoping that this product will help in a lot of situations, everything from automated deployment of complex systems, testing your software without needing to sit there and do it manually to automating business processes (or virtual machine operations like creation, start-up and shut down!).

All as a simple and easy on-line service you can optionally subscribe to (there will be a free offering to get to try it!).

Permalink 

Today is my last day at my current employer, a company called EasyTrace which has gone from a few people in a cow-shed just outside Basingstoke selling a DOS product to a market leader selling modern software I can feel proud to have written the vast majority of.  Over a year ago now they were also bought by RM PLC and the RM people I have worked with have been enthusiastic and fun to interact with.

I have been there for over five and a half years and would like to wish all the present and ex-employees, suppliers and RM people who have helped out over the years the very best for the future.

Everyone has been amazing and put in a lot of long hours and effort to make a happy client base of schools, colleges, hospitals and big companies.  It has been a wild ride, and it has been a pleasure working with everyone.

I will be providing some help to the company after the relocation to the new offices in Abingdon this Christmas but I am mostly taking this opportunity to set up a new business and have some more relaxing fun for a while exploring an idea I have.  It also hopefully means I'll have the opportunity to get to some conferences and write more technical blog posts - I'm fully aware I haven't made any decent ones since May as I have been spending a lot of time on work projects even out of hours recently.

So, here's to the future!

Permalink 

This is an excellent usability study comparing the major webmail providers sign-up forms out there, it almost certainly has an applicability to all software development to a degree:-

http://www.cxpartners.co.uk/thoughts/web_forms_design_guidelines_an_eyetracking_study.htm 

So take a look.

It covers things like making name fields appear as one entry so they require less thought to enter, and making mobile phone fields not take an age to decide what bit goes in which box.  There were also some good basic layout hints and talk of when to use groupings and when not to!

Permalink 

With development getting more and more complex and thousands of lines of code being required to do things that used to take a couple of hundred, it's nice to see a simpler approach to things.  Sometimes the complexity has great benefit (like reuse and flexibility), but when learning that isn't necessarily first on your mind...

I just noticed a blog post on the Microsoft Small Basic blog and have since had a quick try... It looks like an awesome educational tool - easy to use and I certainly remember playing with logo and basic when I was younger - this provides both of those combined (along with decent graphics and no overhead).

So the blog is here:-

http://blogs.msdn.com/smallbasic/

And the program itself (which is basically a tiny IDE) is at http://smallbasic.com/ - hopefully it will let a new generation of children find programming as fun as it was for me when I was young.

Edit: Yes this is no match for C# which is a wonderful language and has a vast framework beside it, but this IS able to make a logo turtle move around in three lines on screen...

Permalink  3 Comments 

For those that don't know me, I work in an education sector linked business - which means that when the schools/colleges and universities go back we always get a vast increase in the amount of calls and number of issues that people raise.

So I've been trying to make up for the development time that was lost doing support by working later here and there.  Thankfully September is now over so I can return to having a personal life and possibly doing more research/blogging!

I have several projects I am investigating right now and hopefully one of them is getting closer to being something I can talk about and that will be able to have a bit of an ecosystem sit on top of it.  For all those people that were disappointed that Oslo wasn't a platform, this might make up for it, depending on what you were expecting :)

Permalink 

I am looking to merge both a P2P WCF service (for LAN support and accessible hops) and a publish-subscribe server to coordinate everything, this is just a post to keep track of the URL's related to doing this as I find them :)

WCF can't use its PeerChannel to connect to both hosts on the internet and local network at the same time, so there'll need to be two PeerChannel's and a Publish-Subscriber service bus connection to work out where everything else is.

Publish-Subscriber: http://msdn.microsoft.com/en-us/library/ms752254.aspx
PeerResolvers: http://msdn.microsoft.com/en-us/library/aa702771.aspx
Writing a custom peer resolver (new one needs to handle non-local hosts by requesting from the publish subscribe service): http://msdn.microsoft.com/en-us/library/ms751466.aspx
Sample WCF P2P chat app: http://msdn.microsoft.com/en-us/library/ms751502.aspx

(This post will be updated with more links as I find them)

Update: What I didn't realise when I started looking into this is that the PeerResolver service is not a magic 'find everything on your network' concept, but rather just a WCF service you register and unregister at, so it's basically already the public-subscriber list I was looking at.  That leaves the question of if the PeerResolver service does that, how on earth is it supposed to find things on the local network?  Well that would be the PNRP service, which happens to not be compatible with loads of OS' and isn't installed by default.  Great one there guys!

So instead I'll just make my own UDP transport for the peer resolver, and if anything I'll get to learn how to make a new transport :).

Permalink 

So this is a little different to my normal posts that are all 100% technical, but via the Omnima site (company who sell some embedded components) there's a link to a review of one of the products they sell which is used for monitoring stuff, which links to a specialist site on aquariums.

Well, it has some amazing photos of reefs on it which I thought were worth linking to.

Permalink 

Downloading Windows 7 slowly from MSDN since around 6.30PM here and it has been getting gradually slower.

So please wait before downloading it ;)

Edit: And Server 2008 R2 is out too as of the 14th August 2009!

Permalink