A bit of IT in developer's world: services.exe high CPU usage

By jay at March 24, 2011 12:36 Tags: , , ,

One of the advantages of virtualization is the P2V (Physical to Virtual) process: Converting an "old" build machine to a VM so it can be moved around with the load as-is, snapshotted, backed-up and so on.

This is particularly useful when say, you have a build machine that's been there for a very long time, has a lot of dependencies over old third party software, has been customized by so many people (that have long left the company) that if you wanted to rebuild that machine from scratch, it would literally take you weeks of tweaking to get it to work properly. And that machine is running out of very old hardware that may break at any time. And that the edition of Windows that does not migrate easily to new hardware because of HAL or Mass Storage issues, requiring a reinstallation. That a lot of "ands".

That's the kind of choice you do not need to make: You just take the machine and virtualize it using SCVMM 2008 R2.

But still, even virtualized, the machine been there that long, and things have started falling apart, like having the services.exe process taking 100% of the CPU. And I did not want to have to rebuild that machine just because of that strange behavior.

If you read scott hanselman's blog, you've been recalled that Windows Server 2008 and later has the resource monitor that gives a wealth of information about the services running under services.exe. But if you're out of luck, like running under Windows Server 2003, you can still use Process Explorer. This will give you the similar kind of insight in the Windows Services that are running.

For my particular issue, this was actually the Event Log service that was taking all the resources.

 

How about I get my CPU back ?

After some digging around, I noticed that :

  • All Event Viewer logging sections in the MMC snap-in were all displaying the same thing, which was actually a mix of all the System, Application and Security logs.
  • Displaying any of these logs was taking a huge amount of time to display.
  • The HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System\Source was containing something like "System System System System System System System" a hundred of times, the same thing for the keys for the other event logs
  • A whole bunches (thousands) of interesting sources named like some .NET application domain created by the application being built on this machine

To fix it, a few steps in that order :

  • Disable the Event Log service and reboot. You won't be able to stop it, but at the next reboot it will not start.
  • In the C:\WINDOWS\system32\config folder, move the files *.evt to a temporary folder, so they don't get picked up by the service when it'll restart
  • In the registry, for each HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application]\Source, replace the content with the one found on the same key on another very similar Windows Server 2003 machine. You can install a brand new machine and pick up the content.
  • If you have, like I did, a whole bunch of sources that look familiar and should not be there under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application], remove their keys if you removed them from the "Source" value.
  • Set the Event Log service to "Automatic" and reboot.

 

The interesting part about the virtualization of that build machine is like in many other occasions, the snapshots, where you can make destructive changes and go back if they were actually too destructive.

 

What's with the event log "interesting sources" ?

The application being built and tested is running tests on the build machine, and it makes use of application domains and log4net. Log4net has an EventLogAppender that allows the push of specific content to the Windows Event Log. Log4net defaults the name of the source to the application domain name, if there is no entry assembly.

Those tests were actually using a default configuration, and were logging Critical messages to the event log, but the domains were created using a new GUID to avoid supposed name collisions. This is something that did actually more harm than good in the long run, because each new appdomain that was logging to the event log was creating a new event source.

And the build system has been there for a long time. Hence the thousands of "oddly named" event sources.

 

How to convert a (big) VMWare VMDK into an Hyper-V VHD

By Jay at October 16, 2008 19:36 Tags:

Cet article est disponible en francais ici.

I've been playing a lot with Hyper-V lately and quite frankly, I'm very pleased with it.

VMs a very responsive, the IO is not the bottleneck it was before, the impact of VMs on the host is far lower than with VPC or VMWare, it supports snapshots and fine grained ACLs on each VMs.The performance part is subjective as always, but I find it faster, and better integrated in the OS that other products. And now, there is a free version of Windows 2008 Server named Hyper-V Server, which allows to run a bare minimum text-mode only version of windows Just Hyper-V and the VMs.

By the way, I partially agree with Paul Thurrott on the "GUI user experience" of Hyper-V which is not very elegant, involving some scripting and information found on blogs, but hey, this is a server product. This is definitely not for the average joe that does not know a bit of what he is doing.

I've had recently to port a VMWare VM to Hyper-V and the main disk was created as a fixed length (flat) -- disk of 80GB, which is obviously a VMDK file of 80GB.

First, I tried converting the file with the VMDK to VHD converter, which unfortunately does not seem to support big flat disks. I already tried converting disks with this tool, I know for a fact that is does work, so it must be the size of the file.

Then I tried using the VMWare Virtual Disk Manager to convert the flat VMDK to a multiple 2GB VMDK spanned file. After the conversion, the VMDK to VHD converter worked perfectly by converting my spanned VMDK to a flat VHD disk compatible with Hyper-V.

This does not end here however, because mass storage drivers installed by Windows at install time for VMWare VMs are not compatible with the one Hyper-V is using. This leads to a nice BSOD saying INACCESSIBLE_BOOT_DEVICE, described here and by the KB314082.

There's two ways to fix this : Create the reg file from the KB article and merge it when the VM is running under VMWare, or mount the VHD disk into an other Hyper-V VM and merge the reg file in the SYSTEM hive of the target OS.

In my case the first possibility was out of the question; Moving the disk again to another machine would have been a waste of time.

That left me with the registry hive mounting solution. Here's how to integrate the registry file :

  • Use another VM to mount the target VHD, to be able to see the SYSTEM hive file.
  • Using the regedit, load the system hive from System32\config\system into HKLM\temp.
  • Modify the KB reg file replace every reference to "SYSTEM\CurrentControlSet" by "temp\ControlSet001", and import it to update the default boot configuration
  • Modify again the KB reg file to replace every reference to "SYSTEM\CurrentControlSet" by "temp\ControlSet002", to modify the "Last Known Good Configuration", and import it , just in case.
  • Unload the hive.
  • stop the current VM.
  • Boot the new VM using the converted VHD disk, and voilà !
This is time consuming, but worth the trouble, the VM is now working properly under Hyper-V.

Using Google Gears to find Montreal's Bus Stops

By Jay at August 31, 2008 20:08 Tags: , ,

Cet article est également disponible en francais ici.

It's been a while since I've posted on this blog. This time, I will not be talking about bluetooth, but still about some .NET powered code :)

I've been busy lately, but I've found some time to work on something that will help me a lot, and I think a lot of Windows Mobile users and mobile users in general.

Montreal's Bus network is somehow large, but its representation in the digital world is quite poor, and inexistent when talking about mobile internet. The web site in question is generating some quite large pages and is not suited for mobile web browsing.

Most of the time, you may want to know the schedule of the next bus, and this is quite hard to get this way.

There's been some effort lately to offer this kind of service on the iPhone, but I wanted to give the opportonity to other users to have the same information, with some Geo Localization features.

This is where google gears comes into action, where their latest release offers a Geo-Location API, which approximates a position using the nearest GSM cells location. Unfortunately, it only works on Windows Mobile devices. But don't worry, if you don't have a Gears enable device, it will still work ! You'll only have to type a bit, by entering your streets intersection.

After getting that location, I'm querying a database (using Linq to SQL) to get the nearest Bus Stops and their next schedule. I'm also querying Google Maps to get some markers pointing at the bus stops. That can be helpful since the Geo-Location is only an approximation by nature, because of the GSM 'triangulation'. It can also be used to query the schedule of a specific bus stop, using the number placed at the bottom of the bus stop signs. A small plus here, compared to the original site, is that schedules from the past half hour are still visible, making possible to have determine if a bus has missed its schedule using a great long street.

Anyway, if you're in Montreal and have an internet connected device (or a normal PC), give it a try by connecting to this adress : http://jaylee.org/stm

Any comments or suggestions are welcome !

Canadian Mobile Data Plans

By Jerome at April 13, 2008 20:06 Tags: ,

I've been interrogating myself a lot lately about the current state of mobile internet in Canada. I'm using my cell phone with a 25$ a month (excluding taxes) data plan for 5 Megabytes, which makes it almost useless for web browsing. Besides, using an HSDPA cell phone, I would deplete my data plan in about 5 minutes.

There's been a lot of buzz around the price of Canada data plans and the absence of the iPhone in Canada for the past year. Projections showed that using an iPhone would cost something like 300$ a month with data plans comparable to what AT&T is offering. I'm guessing that noone would be interested in paying that much to have a 500MB almost unlimited data plan. It seems that Rogers is not willing to let go of the current data plan rates to offer a service that appropriate to the iPhone.

I'm new to the Canadian environment, but for what I can tell when I sometimes hear that Canadians are not really into cell phones -- that they can live without it and do not really need it -- I have an impression of 'déjà vu'. French people had this kind of state of mind when there were only two carriers. People at that time also though they did not need cell phones. Except that it was not that they were technophics, it only was because it was darn too expensive !!

Now, prices in france have dropped a lot, and people are using a lot of services offered by the cell phone carriers. I'm insisting on the services word because I'm sensing that this is where canadians carriers are shooting themselves in the foot by only focusing on being "data pipes". They could expand their business by offering services that would be far more lucrative than only conveying data or voice. If I consider my own use of the voice plan, knowing that the person I am calling is paying the call when he did not initiate the call, makes me talk less.

Ok, there were some improvements the past few months, which Mark is pointing out, but which seem to have halted. Bell released a 7$/month plan which made Canada coming from the most expensive country for data plans to the less expensive in the world, which is a bit odd. With a twist though, it's HTC Touch only. The rest of "improvements" are crippled data plans that are only interesting if you're willing to use the internet that existed 10 years ago...

I'm guessing that breaking the monopoly will change the current state, and that the bidding for new frequencies will force existing carriers to lower their prices to keep their customer base. This is one bit of a stretch, but I'm comparing the state of the industry to the bad phase the music and movie industry is going through right now... There are now forced to understand that they can't sell their music as a product but as a service if they want to keep their business going.

The cell phone industry can be forced to do so to keep their customers, if some newcomer is not playing by the established rules by giving for instance, a flat rate of 45$/month, everything included. I'm not giving this example randomly; I'm referring to the French ISP Free.fr which made quite a perturbation when they offered for something like 45$ a month 100 TV channels, free phone calls to 100 countries in the world, 25Mbps internet, Wifi access point, and a lot more. All this with an excellent quality of service. They did not play by the established rules, and they are now the second most important player in the market and growing every day. I do not see any reason this would not happen for cell phone carriers the same way it did in France.

But maybe there is a good reason for all this though... Canada's a big "empty" space, and maybe expanding the cell coverage is not as money efficient as it is in France, or USA. I don't know all the details, so maybe I'm missing some things.

We'll see in the next few months...

Using a real USB bluetooth device in Virtual PC 2007

By Jay at March 25, 2008 20:03 Tags: , ,

This is one of the biggest missing features of VPC 2007, the lack of USB passthrough support. The lack of a proper snapshot feature is annoying too, even though it is possible to move Undo Disks to simulate snapshots. But I still prefer VPC 2007 over VMWare, mainly because I find the latter to be too intrusive and unstable...

Anyway, I was looking for any update to VPC 2007 for the support of USB devices, in one form or another, maybe with a beta VPC 2008 or 2009. This is still not happening but, I came across this software, Usb over Network, which does exactly what I need to do.

This software is particularly easy to install, and it works right after being installed. I'm able to "export" my integrated USB Bluetooth dongle to the VPC, and it works like a charm ! I'm able to run both bluetooth stacks in controlled environments, without crippling my main OS with multiple stacks.

A really nice software, not free, but really useful if you're depending on USB hardware and you want to test it in a virtual environment.

Windows Server 2008 and Microsoft Bluetooth Stack trouble

By Jerome at March 24, 2008 13:49 Tags: ,

There's been a lot of Buzz lately about a "Windows Workstation 2008", which actually does not quite exist, but that should. It is actually installing Windows Server 2008 and making it a workstation platform, by enabling every workstation component that is disabled by default.

From my point of view, Vista is definitely interesting, though it has too many services that are enabled by default and that do not make sense in every situation. For a computer savvy user, all this stuff is not really interesting, and Windows Server 2008 with its "do not enable unused components" policy, is quite interesting.

I decided to give it a shot by installing it as my main (and only) laptop OS, and quite frankly, I'm pleasantly surprised ! I do get the same user experience that I did have with Windows Vista with Aero, the nifty new features like the new start menu, and I seem to get a performance improvement over Vista. (Performance improvement is only a feeling; I don't have any numbers to show, though some did).

Everything works as expected, except for the bluetooth part, for which I do not seem to be the only one having problem with. The microsoft stack does not seem to install completely, as there are three "unknown devices" left : BTH\MS_RFCOMM, BTH\MS_BTHBRB and BTH\MS_BTHPAN. All three of them are core components of the bluetooth stack, and are obiously needed to get bluetooth related software working properly. The interesting part is that there are actually all the driver and metadata files required to install these devices, but for some reason, Win2008 does not want to use them. The driver files seem to be identical to the files Vista SP1, so this is one bit of a mystery to me. Added to that, this installation issue seems to be related to the KB940199 where the infcache.1 file could not be found. Screwing with that file did not help either...

So as a backup plan, I decided to fall back on the Widcomm/Broadcom Stack with this guide, which seems to work fine, at least for the part I'm interested in, Bluetooth Remote Control . I still don't understand the licensing policy on this software... You need the hardware to get that software to work, why bother having an licensing scheme over this, haven't you already paid for it buying the hardware ?

Anyway, if you're a tech savvy user, give Windows Server 2008 a try as your workstation OS, you might be surprised :)

Now, I'm going back to adding new features to Bluetooth Remote Control !

Broadcom/Widcomm Bluetooth Stack and SetSecurityLevel

By Jerome at September 05, 2007 15:37 Tags: ,

C++, my almost favorite language. This language had good intentions, really. But this is such a pain -- a  pit of despair -- to have to deal with library compatibility and compilation parameters matching issues, where it should be handled by the system.

Anyway, if you're trying to use the Broadcom Bluetooth SDK for Windows CE with VS2005 or VS2008, you might encounter this pretty message :

error LNK2019: unresolved external symbol "__declspec(dllimport) public: int __cdecl CRfCommIf::SetSecurityLevel(wchar_t *,unsigned char,int)" (__imp_?SetSecurityLevel@CRfCommIf@@QAAHPA_WEH@Z)

The symbol actually exists in the broadcom provided library (BtSdkCE30.lib or BtSdkCE30.lib)  but is defined like this :

SetSecurityLevel@CRfCommIf@@QAAHPAGEH@Z

which, undecorated, means :

int CRfCommIf::SetSecurityLevel(unsigned short *,unsigned char,int)

See the difference ? The wchar_t type the compiler is using should be an unsigned short. Basically, both types are binary equivalents but the compiler is treating them as different, hence the different signature.

To fix this, just set the option "Treat wchar_t as Built-in Type" to No.

This way, the wchar_t will go back to being a short, and then match the method built in the static library.

I'm going back to my C# code, now.

Unprotecting Protected Processes

By Jerome at April 07, 2007 15:09 Tags: ,

Alex Ionescu's been searching a bit about Protected Processes, and he's managed to get around that protected state.

I'm no expert on that part nor have I read enough documentation on how that works, but since a goal of that particular feature is the "Protected Media Path" (PMP) to prevent anyone from eavesdropping a protected media, this is not good.

Since that implementation is based on a driver, that won't work on Vista 64, well, as long as you don't boot in that particular mode that allows you to load unsigned drivers. That's a good news for malware protection, since a virus shoud not be able to hide itself under normal conditions, but this is not for PMP. It seems that protected processes can check for a "tainted" environment, but how long is it going to take for someone to fool programs into thinking the system is clean... ? As always, that won't prevent evil dvd rippers to copy the media... but that'll piss a legitimate user.

Moreover, since it is easily possible in Vista 32, as alex is pointing it out, it probably won't take long for viruses to hide themselves using this technique and just a bit longer for antiviruses to unprotected any running process.

What a mess :)

Vista's Support for External Displays

By Jerome at April 07, 2007 10:14 Tags:

Among the many hidden new features of vista which belong to the category of "why didn't they do this in the first place", there is the new and improved support for displays and external displays in particular for a laptop.

In Windows XP, when you were attaching a display to your laptop, the resolution (and color depth) used was attached to the physical port, not to the display itself. So if you were moving around a lot with your laptop attaching many displays with many different resolutions, you had to reconfigure each time.

Vista's way is a bit different since the resolution and color depth are now attached to each specific display, and are automatically recalled upon connecting the cable. Now, when you attach a display, you can hear the "device connected" sound, the same used when connecting an USB device or PCMCIA card.

This also goes along with the presentation mode which allows to hide your trolling background when hosting a presentation. This is particularly useful at Epitech :)

Rest in Peace, TuO

By Jerome at February 26, 2007 02:38 Tags:

Jean-Michel Hervé, aka TuO, has left us yesterday. I've worked with him a few times during my demomaking years when I was an active member of Orion, and I had a lot of respect for the guy.

He was a particularly gifted coder, and will be remembered for many things like the excellent VIP2 invitation demo released in June 2000 when he was a member of the PoPsY TeAm. Lately, he was working in lyon as a developer at Eden Games.

Rest in peace, my friend.

About me

My name is Jerome Laban, I am a Software Architect, C# MVP and .NET enthustiast from Montréal, QC. You will find my blog on this site, where I'm adding my thoughts on current events, or the things I'm working on, such as the Remote Control for Windows Phone.