Mikey Cooper: The Nerdy Stuff

Firefox 4 Beta static cursor

2010-12-21T16:17:00.001-08:00

After upgrading to Firefox 4 Beta, I found that I was clicking on things multiple times trying to get them to work. For some reason, the default in FF4 is to not have a busy cursor when you click something. Your cursor remains the default arrow with no indication anything is happening. You can restore the FF3-standard (and really Windows-standard) behavior of having a "Busy" hourglass cursor with the following steps:

1) Open a new tab.
2) Browse to the URL "about:config".
3) Click through the warning about voiding your warranty.
4) In the "Filter" box at the top, type "cursor".
5) Double-click on the item "ui.use_activity_cursor". It should turn bold and the value should change from "false" to "true".
6) Close the "about:config" tab.

That's it, no restart required. The hourglass cursor should now be back when you click on a link and the page is loading.

Socket connection aborted when calling a WCF service method.

2010-08-01T13:55:00.001-07:00

We are currently converting our project from .NET Remoting to WCF as the preferred method of remote service calls. One issue we ran into is that some methods worked perfectly while others bombed with a CommunicationException with very little useful information in it. There seemed to be no rhyme or reason as to why some methods worked, but others failed. The exception we always got back on the consumer after trying to make the service call was:

The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.2350000'.

The various levels of nested inner exceptions (the innermost being "An existing connection was forcibly closed by the remote host") were no more helpful. Further debugging showed that the call did make it from the consumer to the service and that the service did successfully execute the method in under a second and return the results, so we definitely weren't running into a socket timeout. After turning on the trace writers for WCF and digging through the event output, we saw issues where some of the types we were returning were unrecognized, despite being decorated with [DataContract] attributes:

Type '(type name)' with data contract name '(contract name)' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.

In our scenario, the culprit ended up being abstract classes our objects inherited. When an object is passed over the wire using WCF, it is instantiated and rehydrated on the other side. If the type is being passed as an abstract, the consuming end does not know what concrete type it should reconstitute the abstract type as. To work around this, you can decorate the abstract class with the [KnownType] attribute, specifying what known concrete types it could be. Why WCF doesn't automatically infer this based on both types being [DataContract]'d, I'm not sure.

For example, if concrete type Person inherits abstract type NamedItemBase and you attempt to call WCF service method "List<NamedItemBase> GetAllPeople()", you would need to decorate your Person class with [DataContract] and your NamedItemBase with [DataContract] and [KnownType(typeof(Person))]

Deserialization considerations when renaming objects, namespaces, or assemblies

2010-07-11T11:53:00.000-07:00

In one of my current projects, we are considering refactoring some legacy code into new assemblies and namespaces. The product is an enterprise backup solution and the objects we're moving are serialized to disk during a backup. To restore, the objects are deserialized from disk and read in for processing.

No types like that around here

The issue we ran into almost immediately was supporting legacy backups. We wanted customers to still be able to use their existing backups to restore, even if they upgraded to the new product. However, moving objects into new namespaces or assemblies breaks deserialization.

Take, as an example, an Objects assembly which contains a Square class in the Bluey.Objects namespace. When a Square is serialized to disk, both the assembly FullName:
  Objects, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
and the object's type name (including namespace):
  Bluey.Objects.Square
are saved to the byte stream.

If you later decide to rename the Objects assembly to Shapes, any objects that were previously serialized will fail to deserialize because the Objects assembly no longer exists and cannot be loaded. Likewise, if you were to leave the assembly name alone and just rename the Bluey.Objects namespace to Bluey.Shapes, the object would still fail to deserialize because Bluey.Objects.Square no longer exists.

Enter SerializationBinder

This problem can be solved by injecting custom SerializationBinder logic into the formatter used to deserialize the objects. The SerializationBinder is responsible for taking a type name and assembly name and binding it to the corresponding type. By injecting our own binder, we can override the default behavior and handle special cases. Here is a very simplistic example binder we can use for the Objects->Shapes rename scenario above:

  public class ObjectsToShapesBinder : SerializationBinder {
    public override Type BindToType(string assemblyName, string typeName) {
      if (typeName == "Bluey.Objects.Square") {
        typeName = "Bluey.Shapes.Square";
      }
      return Type.GetType(typeName);
    }
  }

To use this custom binder, we simply attach it to the formatter we're going to use for deserialization:

  var binFormatter = new BinaryFormatter();
  binFormatter.Binder = new ObjectsToShapesBinder();

At this point, any deserialization done with binFormatter will use our binder to determine the representative Types to create. Because the assembly is likely to be already loaded in memory at runtime, we do not need to worry about the assembly name parameter in our custom BindToType method. However, if that were not the case, we could use the assembly name to determine the correct assembly to load before trying to Type.GetType.

The use of the custom binder applies to any object in the object graph, not just the root object being deserialized. So deserializing a Hashtable containing a Square will work just as well as deserializing a lone Square.

PluralSight's On-Demand training library

2010-07-01T10:13:00.000-07:00

The next phase of the project I'm on will likely move us from .NET Remoting to WCF services. I wanted to get a quick introduction to WCF beyond the basic user-group sessions I found online. I eventually noticed that that as a BizSpark start-up, I had a code for a free month of access to Pluralsight's on-demand training library among my listed MSDN benefits.

I signed up and started on their very in-depth 15-hour course on the fundamentals of WCF. So far, these courses are fantastic and if the rest of their library is of the same caliber, this seems like a goldmine of useful information. In the first two hours alone, most of the questions I had on what we'd need to do to drop WCF in to replace .NET Remoting have been answered. The questions that have come up while watching (can I keep my existing domain objects?) or spots where I thought the code was sloppy (why is the channel being closed before the async call completes?) were addressed and were part of the curriculum.

The only downside is the cost is too high for me ($500/yr to stream the content, $1000/yr if you want to download). If you can get your company to spring for a subscription though, Pluralsight's much cheaper (and much more valuable and less inconvenient, in my opinion) than a 5-day classroom training session.

SPSite.Exists leaks SPSites

2009-11-13T12:04:00.001-08:00

The Sharepoint Object Model is a painful, leak-prone API to work with. Roger Lamb's MSDN blog article is a great reference for deciding when you need to dispose of objects and when doing so will break things, but it's necessity is pretty powerful evidence of how clunky disposal is implemented in this managed code API.

In dev testing, QA found an issue where SPWeb objects were being leaked left and right. It turned out some disposable objects (like SPLimitedWebPartManager) themselves contained disposable objects you had to account for (LimitedWPM's Web).

Today, I ran into another case with a very minimal set of code I had changed. A quick look in Reflector revealed that as of SP2, even Microsoft can't always remember when to dispose of objects. The static method SPSite.Exists instantiates a new SPSite object that needs to be disposed of. However, there is no disposal code in the method. There's no way to work around this issue other than not calling SPSite.Exists. Luckily, no other code in the object model seems to call SPSite.Exists.

Don't use generic domain names

2009-08-23T07:38:00.000-07:00

Ever since I started on my newest client's project over a year ago, I've had problems with their VPN. When connected, I could no longer authenticate to remote Sharepoint or SQL servers on my local network (rending my VMs useless), name resolution would be fine for awhile and then stop resolving (making it a race to sync up source code with their server before I could no longer see it), group policies failed to update... anything that could go wrong seemed to on a very regular basis.

When visiting the client's site last year, I found that I could not log into my laptop with my domain's credentials while plugged into their network. It became a game of turn off the wireless, log in to my laptop, turn wireless back on. We eventually figured out that my domain name (the nice and generic DEV/dev.local) was conflicting with their DEV domain. While plugged into their network, my laptop would suddenly send login requests to their DEV domain controllers. The same thing happened when connected through VPN, causing SQL and Sharepoint to believe a hijack attempt was underway.

Over a year later, I finally got around to renaming the domain to a less generic BLUEYDEV/blueydev.local and suddenly everything works. I can finally work connected to VPN and source control rather than trying to sync everything up as quick as possible when connecting.

The moral of the story: When setting up your domain, never give them generic names that others might use, like DEV or TEST. This is especially important for laptops that you may be connecting to customers' networks.

Windows 7 Homegroup issue finding the homegroup

2009-05-25T05:01:00.001-07:00

Windows 7 Homegroup does not play nice with bridged adapters. If you are bridging your wired NIC to a wireless NIC (in our case, for the Xbox to talk to our wireless network), you can start a Homegroup and others will be able to see it but they will encounter errors trying to join it. Likewise, if someone else has created the Homegroup, you will see it but not be able to join it from the bridged PC. Taking both adapters off of the bridge and then deleting the Network Bridge adapter resolved the issue for me.

That said, Homegroup is SIMPLE and it WORKS. If we want to watch a show on our office computers but the file is downstairs on the TV pc, we can just browse to the file in our Homegroup and watch it. The people who say it's just as easy to set up SMB shares haven't gone through the painful process of walking Mom & Pop through sharing each folder/setting permissions/creating user accounts/opening firewall ports.

Windows 7 and Win2k8 power management issues

2009-04-13T04:59:00.000-07:00

More recent builds of Windows 7 have resolved the sleep issue on Lappy. We've also thrown 7 on the Media PC downstairs. The eye candy looks very crisp on a nice large TV. Bundled codecs meant we did not need to download K-Lite to watch any of our ripped AVI files. It also detected our wireless mouse's media keys out-of-the-box and is able to wake up from sleep when when push one, something we were missing before. Media Center can FINALLY use the timeline to move to a specific spot in AVI files that don't contain an index track. This always stopped us from using Media Center over Windows Explorer/WMP. The feature changes from Vista are small and subtle, but coupled with the performance enhancements, there's enough to warrant an upgrade from Vista.

With Lappy and Media PC sleep issues resolved, I noticed my main Win2k8 development box was no longer sleeping, heating up the office considerably at night. It turns out installing the Hyper-V role disables power management completely, the two are absolutely incompatible. While trying to switch back over to Virtual PC, I found performance of be horrible. The install of Vista under VPC was stuck in the same spot for an hour and a half. Even with all of the Hyper-V services stopped, your parent partition is still virtualized under the hypervisor. After removing the Hyper-V role and rebooting, VPC flew. Win2k8 was sound asleep and the office nice and cool when I walked in this morning.

Windows 7 Wake From Sleep issue and workaround

2009-01-20T14:20:00.000-08:00

Playing with Windows 7 the last few weeks, I've found that my laptop refused to wake from sleep. The display alternated between not even powering back on and powering on but remaining black with a non-responsive mouse cursor. The only way to recover was to hard-power Lappy off and back on. This seems especially prevalent on Intel 945 chipset displays.

The only workaround I've found for now is to turn off sleeping completely and instead use hibernate. I've had no problems powering back on from hibernate so far, it only takes about 20 seconds to do with Windows 7 on my system, and doesn't drain your battery like sleep does.

The updated Intel 945 driver released today through Windows Update did not fix Wake From Sleep yet. Hopefully it's an issue that will be addressed before RTM.

Zoom lever on Microsoft Wireless Comfort Keyboard 4000 scrolls in Firefox

2008-12-13T09:09:00.000-08:00

I got a new Microsoft Wireless Comfort Keyboard 4000 today and found that the little zoom lever to the left of the qwerty keys caused Firefox to scroll (and backwards, even). There are plenty of articles out there on switching the lever to scroll instead of zoom, but I just wanted to fix the Zoom lever so that it zooms in Firefox. To do so, open your commands.xml file (usually located in C:\Program Files\Microsoft IntelliType Pro) and search for "MozillaUIWindowClass".

You should see two lines for for C319 and C320 immediately following the MozillaUIWindowClass line. Replace those two lines with:
<C319 Type="5" KeySeq="ctrl add" />
<C320 Type="5" KeySeq="ctrl subtract" />

The reference to "1.5" in the AppName property of can be safely ignore. I tested this fix with Firefox 3.0.4 and IntelliType Pro 6.2.

Restart your itype.exe process or just log out and back in/restart your PC. The Zoom lever should work in Firefox now.

Trying out ANTS Profiler 4

2008-09-07T17:38:00.000-07:00

I got an email that ANTS Profile 4 was released, so I downloaded my upgrade and poked around with it this weekend. I haven't given a good look at the new features list, but the performance improvements are very apparent when profiling. There is still some degradation of your app performance when profiling, but nothing at all like it used to be. I no longer feel like I'm suffering through horrible performance for the sake of finding my issue.

The other big-money change is nice calltree/methodgraph with the shiny "Call Graph" that brings you right to the line of code in the Line-Level Timings view. I love interactive + pretty! Native support to export to PDF and PNG too. This is either a new feature or something that's just been made much more discoverable, because I would have killed to have this in 3.x when debugging memory issues.

It's very much worth doling out dollars for.

Google Chrome

2008-09-03T06:12:00.001-07:00

Like every other nerd out there, I tried out Google Chrome last night.

The first issue I got bludgeoned with is how many ads are on the internet. Not having Adblock Plus in Chrome (or some comparable ad blocking plug-in) means browsing the web SUCKS.

The second issue is how Chrome sucks at password management. It imported my saved password list from FF3 easily enough and I can see them in Chrome's password list now, but it does not auto-populate most of the websites I visit (like blogger, for example), so I have to go lookup the username and password for each site each time I visit. Since I make up random passwords for every website and then promptly forget them after registering, this is an epic fail of Chrome for me.

Beyond that, performance was noticeably fast compared to FF3. Refreshing the pages side-by-side in both browsers, FF3 is still zippy but Chrome is definitely faster for the pages I visit at least. It seems pretty light on memory, but I never noticed any issues with FF3 in that department. The "new tab" page with thumbnails, recent bookmarks, etc. is nice.

I LOVE that Chrome actually listens to me when I choose "always open files of this type" after downloading something, where as Firefox just presents me with the "Opening file" dialog with the "Do this automatically" check box already checked for me, but will still not open it until I click "OK".

Were it not for the punch-the-monkey banners and the password manager failures, I'd probably give Chrome a shot as my default browser for a few weeks.

Hyper-V and widescreen

2008-08-22T04:47:00.000-07:00

The multimedia performance issues in my previous post about Hyper-V performance turned out to be a display driver issue. With my deal-breaking "must be able to watch Burn Notice" requirement now met, I poked around with Hyper-V some more last night. The Hyper-V Manager interface is rather sexy, absolutely a winner compared to VMware Server's web-based atrocity and more than one snapshot is a bonus. However I still don't see much benefit to using it over VMware Workstation for a development workstation, other than the fact that it's free.

Both perform well, both have good snapshot tree management, both support VLANs ("Team networks" in VMware). Biggest frustration now with Hyper-V (other than not being able to connect to wireless NIC) is the lack of support for widescreen resolutions on the console sessions. With all the window decorations, taskbar, status bar, toolbar, etc. taking up so much space, not having widescreen means I'm stuck at 800x600 for all my Hyper-V VMs. 1024x768 if I don't mind turning off the toolbar and hiding some of the statusbar behind the windows taskbar. VMware Workstation has very nice resizing functionality that allows you to just expand the window to whatever width you want and the desktop will resize to that resolution.

This may seem like a feature that's only a bother for IT people trying to use Hyper-V as a workstation solution, but I imagine it has to be a bit of a pain in the ass for administrators running Hyper-V Manager from their workstation with widescreen monitors.

Hyper-V is definitely NOT for workstations

2008-07-27T14:00:00.001-07:00

EDIT: The multimedia performance problems described below turned out to be a display driver issue with nvidia having some kind of conflict with Hyper-V. Downgrading to Windows XP drivers cured this for me.

Also, the very minimal performance testing I did previously (start-up time of a Windows XP VM in Hyper-V) was a bit apples-to-oranges as I installed a fresh copy of WinXP under Hyper-V. Since I already reinstalled Hyper-V to test out the display drivers fix, I decided to do a better comparison. This time I converted my VMware VM to a Hyper-V VM (minimal services set to start and all) and found startup time on par with VMware workstation, maybe a few seconds faster.

My only roadblock to using Hyper-V now is the lack of wireless NIC support.

I’m at a point in my current project where I need to test out automated installs on different OSes at various states (totally clean, .net framework installed, our software already installed, etc). Having just moved to Win2k8 on my workstation, I decided to give Microsoft’s new paravirtualization product, Hyper-V, a shot.

I’m a pretty big VMware fan. I use VMware Workstation on a daily basis and also used their free VMware Server product before they switched over to the clunky web-only interface in v2. Microsoft’s Virtual PC and Virtual Server offerings never compared with even the free VMware Server, as far as features and performance were concerned. It was even worse when you take into account Workstation’s killer snapshot management. Hyper-V (essentially $28 when bundled with Win2k8) was supposed to bring everything much more in parity with VMware’s enterprise offerings. I was excited to try it out.

That didn’t last long though, as I soon discovered that Hyper-V doesn’t support binding to wireless NICs. There are some painful workarounds involving binding your wireless NIC to the virtual NIC, or using ICS on your wireless NIC to share it out to the virtual NIC. Both are kludgy and neither were very stable for me.

The real killer though was how much host performance suffered. When the Hyper-V role is installed on Win2k8, the hypervisor stack is installed underneath the host OS stack and the host essentially becomes a virtualized environment, just like any other VM. This is great for the server environments that Hyper-V is intended for, but for a workstation where you want interactivity, it means pretty noticeable performance degradation. Videos that would normally load up and play instantly would hang for 5-10 seconds before playing, and even then, the audio and video would stutter for the first few seconds. Performing other activities involving reasonable disk I/O (such as copying files) would make the audio and video stutter more. Scrolling applications, which are smooth without Hyper-V, are jerky when virtualized (reminded me a lot of scrolling on a remote desktop session).

Granted, Hyper-V isn’t designed for workstation use, just like Win2k8 server, but for IT pros/Developers who have switched toWin2k8 for their workstation regardless, stick with VMware. It provides more features and better host performance. Guest performance seems remarkably better as well (a WinXP VM that takes 13 seconds avg to boot to a desktop under VMware took 24 seconds avg under Hyper-V). I’m not really sure I see the benefit Hyper-V provides, at this point.

Error 1079 and 1006 after an AD domain rename

2008-07-22T17:24:00.001-07:00

After having some DNS forwarding issues with my internal dev domain and my external domain name, I decided to rename it from dev.bluey.com to dev.local.

Renaming an AD domain isn’t as big a deal as most people make it out to be. The rendom.exe command takes care of most everything for you. I did all the usual rendom /list, rendom /upload, rendom /prepare, and finally rendom /execute to actually kick off the domain rename.

I noticed after rebooting the DC, it hung for a very long time at “Applying Computer Settings…”. I had the same issue with my lappy after joining it to the new domain, it would hang at “Applying Computer Settings…” during boot. I kept seeing 1079 and 1006 errors in the event log. gpupdate.exe showed that my group policies were refusing to apply. dcdiag.exe spewed back more of the same. Out of desperation I tried reloading the gpo schema with dcgpofix.exe but still had the same issue.

In the end, I realized I forgot to actually FINISH the domain rename. A quick rendom /end, gpfixup, and rendom /clean and everything worked like magic again. So in case you’re dumb like me and forget to complete the rename, there ya go…

SharedView

2008-07-03T06:24:00.001-07:00

Been cranking on a new project. A coworker asked if I had SharedView when we were doing some knowledge transfer. I'd never heard of it but downloaded it and finally got a chance to try it out last night and it is AWESOME. If you regularly get calls from relatives that start off "I'm on my computer and I'm trying to...", you NEED this utility. Where "Remote Assistance" falls flat on its face, SharedView shines.

We used it last night to track down some code issues. The session host is able to share a single application or their entire desktop to all other participants (up to 15) in their session. Initially the session host controls the mouse and keyboard, and other participants get a color-coded read-only mouse cursor. When read-only participants click on things, it draws a circle around their cursor to draw attention to it. The session host can easily hand over control to another participant (and just as easily take it back). There's "handout" functionality if you need to exchange files and it has minimal chat functionality, but we mostly used IM to talk. I was hosting the session, but the other participant said performance was fast when viewing, and a little bit slower when controlling but nothing major.

Most importantly, setting up a session is pretty painless. It automatically generates a random password for your session and populates it into a link you can IM/email to other users to join in. That's about all there is to it. Like most everything Microsoft now-a-days, it does authentication based on a Live account, so you'll need one to have one even to just to watch a session.

Internet Service Bus

2008-06-05T18:07:00.000-07:00

While TechEd is right in my backyard this year, being self-employed means no sugar-daddy to foot the bill for me to go. Instead, I've been doing the poor-man's TechEd, where I nag my friends who could afford to go for daily updates and I watch the web sessions online. One piece that caught my fancy was the Internet Service Bus (basically ESB meets the intercloud).

Mesh is a cool tech preview to show off some of the cloud capabilities, but it really only scratches the surface of what you can do. ISB gives you a good glimpse into what's to come further down the road. On the surface it doesn't seem much more than a centralized .net services advertising broker, allowing you to point your remoting/web services to Microsoft's cloud to figure out how to connect to your services.

It seems to allow a bit more though. While a lot of it is over my head at this point, one obvious benefit is the seamless service brokering. Behind my firewall/NAT'd router, I'd typically have to advertise my regularly-changing dynamic IP and open up one or more ports to forward to my local box. With the ServiceBroker stuff in the Biztalk SDK, when your service starts up, you simply have it register with ISB with your pre-created application/user account (this can be a username/password, a windows live account, a cardspace card, etc). Then users are able to access your service via a biztalk URL without you having to make any router/firewall changes. I have a SimpleShare example service running on my local box, which can be accessed as an RSS feed by anyone (with a free biztalk account) externally at:
http://connect.biztalk.net/services/mikeycooper/SimpleShareService/files/
The service allows anyone to download any files I throw into a test directory on my computer as long as the service is running.

I haven't figured out how to get to Identity information yet to see WHO is requesting data from my service (the IncomingMessageHeaders.From is always null, for some reason) and there's not much documentation yet. The sample applications don't really give a great sense of what kind of cool stuff you can really achieve with such minimal effort, but its pretty impressive to dig around in once it all clicks.

Mesh'd!

2008-05-27T05:15:00.000-07:00

I've been playing around with the Live Mesh technical preview for a few weeks now. Once I found this way to
disable UAC prompting without disabling UAC, I was able to install and use Mesh. So far there's not much more to Live Mesh than file sharing and remote control, which makes it a bit of a cross between Adesso's now-defunct Tubes and GoToMyPc.

On the file sharing front, it's been nothing but awesome. Mesh is like the old-school Windows 95 "Briefcase" for keeping files in sync among multiple PCs, except it actually works. The interface is monumentally easier and more intuitive than Tubes, which required a degree in particle physics just to configure it to share a simple file. Mesh integrates right into Explorer and the Windows UI. Mesh folders appear appear wherever you want them, with a distinctive blue folder (reminds me of the Crystal theme on KDE). I have a "Shared" folder that is automatically sync'd with my laptop and the website. I've started saving my hours logs in there so I don't lose track of hours worked when I'm at a remote location. Mesh works offline and syncs back up when you've got internet access again. Conflict resolution is a manual, prompted process that's easy to understand. If I'm at a PC that's not a device on in my Mesh, I can still easily access my files through the web interface. It's all very simple.

It also supports multiple users accessing folders, so a friend and I both have his folder and can share files with each other just by copying them into their. We've used it to exchange code projects, DLLs, study guides, etc. Each folder has a little news pane associated with it that lets you see what's happened to it recently or post notes.

Very nice in that regard. The remote desktop component is very nice as well, letting you connect to any device in your Mesh that is running the client. It worked without any extra configuration on my router (doesn't appear to use UPnP as far as the router reports, so it must all be server-based). Unfortunately, it's dreadfully slow compared to a plain old Remote Desktop session. Hopefully performance improvements will be in the next version. Again, support to remote into your machine is offered via the web interface if you're on a non-meshed PC.

I'd love to be able to eventually have my Documents/Music/Pictures folders just be mesh folders that are available on any PC I add to my Mesh rather than having to copy files over manually. I think that might be the direction Microsoft heads with this technology if it proves to be successful and they can find a way to profit from it. The infrastructure costs to mass-market support this thing beyond a tech preview would be huge, I'd think.

Pivot queries

2008-05-24T08:08:00.000-07:00

I stumbled across the PIVOT keyword while poking through the dynamic management tables in SQL trying to track down some sproc recompiles. Pivots are great for denormalizing data for easy reporting. OLTP schema is great for quick insert/update/deletes but piecing data together into something useful can be a pain.

A simple budget table that tracks a budget item type and a budget amount (rather than having a separate amount column for each type of item) is a good example of where this is useful...

DECLARE @BudgetItems TABLE
(CustId int, BudgetItemType varchar(20), BudgetAmount money)

--Customer 1 is living the life
INSERT @BudgetItems (CustId, BudgetItemType, BudgetAmount) VALUES (1, 'Mortgage', 1000)
INSERT @BudgetItems (CustId, BudgetItemType, BudgetAmount) VALUES (1, 'Salary', 6000)
INSERT @BudgetItems (CustId, BudgetItemType, BudgetAmount) VALUES (1, 'Food', 140)

--Customer 2 has no income
INSERT @BudgetItems (CustId, BudgetItemType, BudgetAmount) VALUES (2, 'Salary', 400)
INSERT @BudgetItems (CustId, BudgetItemType, BudgetAmount) VALUES (2, 'Food', 340)

The table ends up looking like:

SELECT * FROM @BudgetItems

CustId BudgetItemType BudgetAmount
------ -------------- ------------
1      Mortgage       1000.00
1      Salary         6000.00
1      Food           140.00
2      Salary         400.00
2      Food           340.00

Again, very easy to ins/upd/del, but difficult to get a single client's budget into a row for something like binding a report to it as a datasource. Pivoting the table is an easy (if somewhat syntax-clumsy) solution to this:

SELECT pivotTable.*
FROM (SELECT CustId, BudgetItemType, BudgetAmount FROM @BudgetItems) AS source
PIVOT ( SUM(source.BudgetAmount) FOR BudgetItemType IN (Mortgage, Salary, Food) ) AS pivotTable

CustId Salary  Mortgage Food
------ ------- -------- ------
1      6000.00 1000.00  140.00
2      400.00  NULL     340.00

DTS to SSIS migration

2008-05-18T16:21:00.000-07:00

I've been working the past few weekends on migrating DTS packages off our dying, 8-year old SQL2000 server over to new SSIS packages. The performance is markedly improved, often over 50% faster with SSIS for a single datapump task. The DTS-to-SSIS migration wizard is also working remarkably better than I expected, even for some of our more complex packages.

Other DTSes we've put together that regularly email out Excel spreadsheets to business users have been migrated to the internal SSRS portal with predefined email subscriptions. Users seem really excited about being able to just go to a website and get their data.

Not having to constantly deal with the day-to-day CM/DBA stuff is leaving me lots of time to explore this cool stuff we've been sitting on since the migration from SQL2000 to SQL2005. I wish I'd gotten a chance to play with it sooner.

More ANTS Profiler

2008-05-03T09:07:00.001-07:00

My pleas were answered (thanks!) and I now have a Pro version of ANTS Profiler. I have mostly focused on memory usage over the last week trying to track down issues for our client. When the app started up, there was a long delay before we got UI-responsiveness. I figured this was just overhead from ANTS profiling it. I happened to run it without ANTS today and got the same result though, so something was blocking the UI thread for ~15 seconds during start-up.

When I was trying out ANTS last week, my sales rep at Red Gate talked up the performance profiling aspects quite a bit, so I decided to try tracking down my 15 seconds in ANTS. Unlike the memory profiling, which required some pre-sales support and hand-holding (though a token effort on my part to read the manual would probably have helped ;), the performance profiler was very straight-forward. Just start profiling in performance fast-mode and take a snapshot when slow things occur. After the 15-second lag passed and the UI started up again, I took a snapshot and saw this immediately:

This call is made to determine any timezone difference between the SQL Server and the client our app is running on. It's called once for each server registered in the app, and for some reason this call to the single server I have registered took 14.47 seconds. Digging in further showed that GetServerTime() called RemoteTime() which was making native calls to NetRemoteTOD to get the time from the operating system, parsing out the timezone data returned, adding it to our current time to get the difference, etc. OS time retrieval works great if you have rights to the server itself, but I don't so I lock up for 15 seconds before getting an "Access denied" return code. Changing GetServerTime() to just use our already-established SQL connection to call GETDATE() reduced the method call from 14.47s to 0.03s:

So that took about 10 minutes to find and fix, shaved about 14.4 seconds off each server call during start-up, and probably saved my client 3 hours of billable time had we tried to fix this by manually stepping through everything.

The downside, of course, being that it shaved off 3 hours of billable time. ;)

Quick Launch keyboard shortcuts

2008-05-02T17:41:00.000-07:00

I was trying to use a keyboard shortcut I set up for Windows Key + ` and accidentally fat-fingered the 1 instead. To my bewilderment Visual Studio launched. I hit it again and got another instance of VS. Windows+2 launched VMware and Windows+3 launched SQL Management Studio. After bumbling around for a bit, I realized that it was launching things in the order they were in my Quick Launch toolbar. Trying to get a friend excited about it revealed that it's a new feature in Vista, whoops!

Product Pimping (and pleading)

2008-04-30T17:08:00.000-07:00

A while back, I got the opportunity to test out Hyperbac, a backup solution for SQL Server. I love pimping Hyperbac because it is dirt-cheap and has seamless integration with SQL 2005. Most third-party backup products required using extended stored procs to do backups, which killed my ability to use SQL 2005's vastly improved maintenance plans. Hyperbac works with the native commands as it simply intercepts the write operations at the OS level based on the (completely customizable) file extensions you're writing to. So just using a basic BACKUP DATABASE TO DISK command and specifying a filetype of .HBC would automatically compress the file. Hyperbac also supported backing up to standard .ZIP. So much nicer than the tools it replaces. Not sure why they're not a bigger name in the backup/recovery market yet.

I started my new consulting gig. It's cool getting to just focus on one project and just grind away at it. One of the issues our client wants fixed is memory bloat. The app just consumes gobs of memory, even when not seeming to do anything. I downloaded a copy of ANTS Profiler and once I found the call stack window, I was able to dig into what was going on in about two hours, whereas I had been stumbling around for a good day or so with no results prior. It's pretty reasonably priced ($495/license for Pro), but unfortunately I'm still in start-up mode where "food and shelter" takes precedence. ~~If any rich people that stumble across this want to buy a copy for me before my trial runs out, I'm about $492 shy:~~

~~Your kindness will not be easily forgotten.~~ Red Gate rocks!

It got better

2008-04-16T18:25:00.000-07:00

We've been running automated deployments for about 4 weeks now and they've greatly reduced the amount of time we spend pushing code out for testing. Powershell turned out to be a bit more of a ~~pain-in-the-ass~~learning curve than I expected. Discovering Scott Guthrie's blog post about app_offline.htm simplified another issue of ours, maintenance pages during the release.

Our Xml Updater utility is done. Our TFSDeployer scripts are a bit more modular now. We decided to just deploy everything rather than cherry-picking changes (it only takes about 4 minutes to deploy anyhow). The only thing we can't do yet is automatically apply database changes and deploy to web servers with file sharing disabled.

Incorporating the app_offline.htm functionality to take the site down before deploying, we ended up with a basic code block of:


function Deploy-Directory
{
  param([string] $appSourcePath, [string] $appDestinationPath)

  # create our destination if it's not already there
  new-item -force -path $appDestinationPath -itemtype "directory" | out-null

  # create app_offline.htm to take destination offline for maintenance
  Offline-App $appDestinationPath

  # delete everything from destination, except the offline page.
  get-childitem $appDestinationPath -exclude "app_offline.htm" | remove-item -force –recurse

  # copy the published website to our destination
  get-childitem $appSourcePath | copy-item -force -recurse -destination $appDestinationPath

  # remove app_offline.htm
  Online-App $appDestinationPath

  # create a build indicator
  $buildFile = $appDestinationPath + "\CurrentBuild.txt"
  $TfsDeployerBuildData.BuildNumber + " deployed on " + $(get-date).ToString() > $buildFile
}

# Creates an app_offline.htm file to let everyone know the websites's down.
function Offline-App
{
  param([string] $appDestinationPath)
 
  $offlineFile = $appDestinationPath + "\app_offline.htm"
 
  $output = "<HTML>" +
  "<HEAD><TITLE>App down for maintenance</TITLE></HEAD>" +
  "<BODY>This website is temporarily offline while Build " + $TfsDeployerBuildData.BuildNumber + " is being deployed. " +
  "Please try again in a few minutes. " + 
  "<!-- " +
  " Adding additional hidden content so that IE Friendly Errors don't prevent" +
  " this message from displaying (note: it will show a friendly 404" +
  " error if the content isn't of a certain size)." +
  "-->" +
  "</BODY>" +
  "</HTML>"

  $output > $offlineFile
}

Codecamp and Linq

2008-03-23T09:12:00.000-07:00

Went to the Orlando Code Camp yesterday and was pleasantly surprised by how enjoyable and informative the sessions were. It's always a little humbling to see all the technologies you're not familiar with and realize you haven't even scratched the surface of being an "expert". ;)

The most interesting (and most crowded) seminars were definitely the Linq ones. I tried getting my toes wet with it for an my Xml updater, but just couldn't wrap my head around it and went back to plain old DOM. After seeing LINQ to XML and LINQ to SQL sessions, it seemed a lot cooler, but I still just didn't "get" why it was so much of a benefit. After being clued in to LinqPad, it began to click for me. LinqPad is one of the more ingenious learning tools I've used, especially for its simplicity.

Once I noticed that Linq works not just for Sql or Xml, but for basic collections, arrays, etc, I finally started to see how much more readable and maintainable it can make code. It can eliminate a lot of the nested foreach/if/foreach/if blocks you see scattered around business logic. Code like:

foreach (Order o in currentBatch.Orders)
  if (o.ShipDate < '1/1/2008')
    foreach (OrderLineItem oli in o.LineItems)
      if (oli.Category == "Cookies")
      {
        AddFreeMilkPromotion(o);
        break; //only 1 free milk per order
      }

Becomes:

var query = from o in currentBatch.Orders
  where o.ShipDate < '1/1/2008'
  where o.LineItems.Any(i => i.Category == "Cookies")
  select o;

foreach (Order o in query)
  AddFreeMilkPromotion(o);

While it doesn't save much as far as number of lines, the bottom example just seems so much cleaner and more concise to me.

XML Namespaces

2008-03-18T17:07:00.000-07:00

Martin Honnen is my hero for solving my XML namespace issue in my config file updater utility. We're getting close to closing the automation loop now. Developers can check their changes into their dev branches and we automatically kick off an integration build for one of those branches. Once the developers merge down into trunk, an integration build will automatically kick off there too. TFS 2008's CI stuff works painlessly here, even if it's not that full-featured.

Once the build succeeds, QA can decide they want to deploy it to their test environment by changing the build quality to "Ready for Initial Testing", at which point TFS Deployer does its magic. I'm now done with the little utility we can call after a deployment to update the web.config-like files to have test, final-test, or production settings depending on where we're deploying to. I'm also almost done with some wrappers to upload assemblies to our servicehosts and custom task scheduling services we have. These will be a snap to integrate into TFS Deployer's powershell scripts when they're done.

The only major piece left is a utility that will interrogate the build's changesets to determine what files changed and what corresponding assemblies and websites to deploy. Deploying everything wholesale would be a bit too cumbersome with the amount of projects we have.

Really happy to see all of this starting to come together.

Automating deployment

2008-03-17T17:35:00.000-07:00

One problem with having lots of n-tier apps is deployment changes. For our environment, it means that a change to a single common assembly might be deployed to 4 different websites, remoting servicehosts, and any other places referenced. Keeping it all straight is a manual, error-prone process. We've been testing TFS Deployer as a means to allow automated deployment of Team Builds whenever QA accepts them. So far, it seems to be a very simple concept that works great. PowerShell is incredibly easy to write scripts for and it took about 10 minutes to get it deploying one of our websites to a test directory anytime I updated the build status to "Ready for Initial Testing".

One thing holding us back from going live with it is a way to automatically update configuration settings for the sites after deploying over them. We need to point the websites to the right databases, override values for testing, etc. Our original plan for this was to just never deploy web.config and xml configuration files that contained environment-specific values, but we ended up losing code changes that were made to them during deployments.

We've now decided to forcibly deploy every file, including web.config, and run a program to update all environment-specific values afterward. Since all our configuration files are XML-based, it seems like it should be real simple, but XML namespaces are giving me trouble. Hopefully there's a simple solution I'm missing, because once my XML problem's solved, I can get out of the CM business.

Fixed build step deletions

2008-03-10T15:15:00.000-07:00

I tried increasing the duration I run my "delete extraneous build steps" job and found that deleting the steps during particularly large project builds (such as web deployment projects) can cause the build to abort. I updated the query to only delete the build steps once the build is finished and that seems to have resolved the issue. Unfortunately, it means I still have to wait for the build to complete before I'm able to see what failed to build.

Stopping excessive build steps in TFS 2008

2008-03-07T20:40:00.001-08:00

EDIT: Updated the query below to only delete the steps of finished builds. Deleting steps while the build is in progress can cause the build to abort.

Since upgrading to TFS 2008, we've had two extremely annoying sources of pain:

the amount of msbuild output that goes into the BuildLog.txt
the excessive number of build steps generated during a Team Build if you use project references within a solution

The excessive output in BuildLog.txt is easy enough to fix. MSBuild 3.5 defaults to diagnostic logging, so appending "/fileLoggerParameters:verbosity=normal" to the TFSBuild.rsp will cut down on the file size considerably.

The Build Steps are unaffected by the verbosity option though and there appears to be no simple flag to cut down on the chatter generated. A Team Builds of 4 or 5 solutions which generated around 400 steps under TFS 2005 now generates over 3000 steps under TFS 2008.

The bulk of these steps are repetitive and rather useless information messages about building targets GetTargetPath, GetNativeManifest, and GetCopyToOutputDirectory. Rather than writing custom tasks to iterate through the BuildStep node tree, I wrote the query below and scheduled it to run every 2 minutes in TFSBuild. It will delete any successful buildstep nodes that mention one of the three targets above. Running this cut our build steps down to a much more manageable size. While I haven't had any issues with it, Microsoft doesn't like you touching their tables directly, so buyer beware.

DECLARE @nodes TABLE (NodeId int)

INSERT @nodes
 SELECT bif.NodeId
FROM tbl_BuildInformationField bif INNER JOIN tbl_BuildInformation bi
ON bif.NodeId = bi.NodeId INNER JOIN tbl_Build b
ON bi.BuildId = b.BuildId
 WHERE (
  bif.FieldValue LIKE '%"GetTargetPath".'
  OR bif.FieldValue LIKE '%"GetNativeManifest".'
  OR bif.FieldValue LIKE '%"GetCopyToOutputDirectoryItems".'
 )
 AND EXISTS( --only delete nodes that succeeded
 SELECT 1
 FROM tbl_BuildInformationField status
 WHERE status.NodeId = bif.NodeId
  AND status.FieldName = 'Status'
  AND status.FieldValue = 'Succeeded'
 )
AND b.FinishTime IS NOT NULL

DELETE tbl_BuildInformationField
WHERE NodeId IN (SELECT NodeId FROM @nodes)

DELETE tbl_BuildInformation
WHERE NodeId IN (SELECT NodeId FROM @nodes)

Team Building shelvesets

2008-03-05T20:29:00.000-08:00

With our smaller development team, we decided to try and minimize the number of branches we use to simplify our merge/release process. One drawback we ran into is how to handle pushing out an emergency fix when we've already got changes being tested by QA in trunk (our integration branch, for all intents and purposes). We could merge the change into the Release branch (a branch that contains code that's already been deployed to Production and gone live) and check it in, but the idea of release is to only check in once it's verified by QA and deployed.

Another solution is to have an interim branch between trunk and Release, a production integration branch of sorts. This adds extra complexity by having another merge to another branch you're responsible for.

To work around this, we decided to merge to the Release branch but rather than performing a check-in, we shelve our changes. We have set up a teambuild that will get the latest source from Release, unshelve a shelveset, build the newly modifed code, and then undo the shelveset. Can't remember where I yanked it from, but it simply involved adding the following targets to our build .proj definition:

<Target Name="AfterGet">
<Exec WorkingDirectory="$(SolutionRoot)" Command=""$(TeamBuildRefPath)\..\tf.exe" unshelve "Shelveset build test";mcooper" />
</Target>

<Target Name="BeforeDropBuild">
<Exec WorkingDirectory="$(SolutionRoot)" Command=""$(TeamBuildRefPath)\..\tf.exe" undo /recursive /noprompt $/" ContinueOnError="true" />
</Target>

Now all we have to do is merge to release but don't check in, shelve the changes, and update the proj file to have the shelveset name and owner before kicking off a build.

Trimming TFS database size

2008-03-05T06:38:00.000-08:00

Our TfsVersionControl database was at nearly 20 gigs when I started digging into why some source control actions were so slow. While 20 gigs is relatively small in terms of enterprise databases, it's huge for our small source tree. Simple tasks like getting latest on a branch had a noticeable 5-30 second spin-up time while they queried the database.

The largest table turned out to be tbl_LocalVersion, which clocked in at 10 gigs. This table tracks the links between versioned items and the local workspaces of the developers. Listing out the workspaces with "tf workspaces /owner:*" showed the problem, we had 500+ workspaces, most of them belonging to developers who were no longer here. Deleting these workspaces also deletes their associated LocalVersion records. A quick script to run "tf workspace /delete "my dev branch;johnsmith"" for all the old workspaces brought us down to about 40 workspaces and a LocalVersion table of less than 3 gb.

To get a rough idea of who's using what, run this in TfsVersionControl:

SELECT i.DisplayName, COUNT(DISTINCT w.WorkspaceId) AS Workspaces, COUNT(1) AS LocalVersionCount
FROM tbl_Identity i WITH (NOLOCK) INNER JOIN tbl_Workspace w WITH (NOLOCK)
ON i.IdentityId = w.OwnerId INNER JOIN tbl_LocalVersion v WITH (NOLOCK)
ON w.WorkspaceId = v.WorkspaceId
GROUP BY i.DisplayName

Trimming out these workspaces has made the single largest improvement in the perceived performance of TFS out of anything I've tried so far.

Error 32000 - tfsdb.exe /showui:590116

2008-03-05T04:27:00.000-08:00

Our upgrade from TFS 2005 to TFS 2008 took about 2 weeks from conception to completion. Surprisingly, most of this time was spent trying to get a new VM set up to mirror our existing TFS 2005 environment. VMware's snapshots are a lifesaver, and made the process so much easier. We simply rolled back over and over until we got it figured out, got the steps documented, and then tested our documented steps. Once we had TFS 2005 up and running on a separate server, upgrading the test server to TFS 2008 was simple and painless. A day of playing with the builds, getting everything compiling, checkins and checkouts, etc. and we could find nothing that didn't work under TFS 2008, even from a VS2005 client.

So that night we took the plunge and upgraded. 6:30pm rolled around and I confirmed the developers had gone home so I kicked it off. Naturally, what worked over and over in testing did not go so smoothly on Prod. After 20 minutes or so, an error popped up:

Error 32000.The Commandline 'TfsDb.exe upgrade [blah blah] /showui:590116' returned non-zero value: 100.

A few googles later, the best I could find was a suggestion to hit "Retry" and pray. After retrying a few times with no success, Profiler revealed that it was failing during the creation of the SQL Server Maintenance Jobs:

-- Create the job
EXEC msdb.dbo.sp_add_job
@job_name = @adminJobName,
@category_id = 3, -- Database maintenance
@description = 'Performs maintenance on the Microsoft Team Foundation Version Control database.',
@job_id = @job_id OUTPUT,
@owner_login_name = 'sa' -- should set owner login name as sa, incase user who runs setup account expires

Microsoft recommends you rename the "sa" account as an added security measure, but then hardcodes "sa" into the TFS upgrade? Renaming our account back to "sa" temporarily for the upgrade resolved this and the rest of the upgrade went off without a hitch.

Another one

2008-03-05T03:50:00.001-08:00

In an effort to keep the remaining friends I have on LJ, I've decided to start posting my work experiences in a separate blog. We recently had a pretty nasty shake-up (or lay-offs, or "trimming the fat", or "right-sizing") at my company (lost about 60% of IT) and as a result, I found myself unexpectedly thrust into the Configuration Manager/Build Guy role. This was on top of my existing DBA responsibilities.

We're a very small shop with only 5 developers now so some of our processes no longer made sense. Branching for every release to production, branching for every minor new project, queueing up tickets in our integration branch until the existing release was done so we could dump them onto our QA team all at once and give them 2 weeks to test it and get it out the door... none of this made sense anymore. This is especially true now that we've just gone from 1 QA lead to manage everything, 5 QA testers, and 2 dedicated CMs to just 1 QA manager/tester and me as CM/DBA.

All of this went down about a month ago. Since then, I've been slammed getting our CM stuff under control. We're unquestionably a Microsoft shop and we use VS2005/TFS2005. Unfortunately our idea of CM has been looking over a list of changesets we think might need to be released, figuring out which assemblies and/or websites to deploy based on that list (we're very n-Tier, so 1 assembly change might need to be deployed to 10 different apps), and then copying and pasting files around.

To try and get our heads out from under water, we've decided to upgrade to VS2008 and TFS2008. I'm not sure what VS2008 brings to the table for the developers beyond LINQ, but they seem pretty excited about just that aspect of it. TFS2008 though... that's where the money is for me. 99% of the improvements I've run across in TFS2008 are on the build side and they kick ass, so we've crammed through an upgrade in the span of about 2 weeks that went live yesterday morning. The developers didn't notice a thing... flawless.