Lesson learned during my business trip

I spent 5 weeks at customers site recently which also means that I was away from my team for 5 weeks.

The first thing I have learnt is that it is so easy to have miscommunications. I have been working closely with my team for 3 to 4 years. For some people, I have been working with them for around 5 years in the company. But when I was at my customer site and communicated with them what customer wanted and when the deadline was going to be, sometimes it was just difficult to make them understand the situation. At some moment I was a bit frustrated by it and then realized it is natural to have miscommunication. My solution to this is to be honest with the other party and respect them. As normal miscommunication problems, the technical solution is to listen and understand the other party first. And then try to let them understand your needs.The authority does not help in this case.

  

Resume the post now

I have not updated here for a long time. It does not mean I forgot about it. No. Work is part of my life. I would like to write them down.

Last year I have the second daughter come to my life. Amazing. Although it is the second time, but I am still surprised by the miracle of life. Now I almost forgot how painful the delivery was. She is almost 1 year now. So I think I am ready to pick up my 'old' life style a bit more, a bit more, and more..

After reading a post from Martin Fowler - Retreaded: Rigorous Agile

After reading a post from Martin Fowler - Retreaded: Rigorous Agile
I am a big fan of this Martin Fowler. Another reason for less of rigorous definition from what I understand is that people now still do not have a full concept regarding how to make software development with quality under constraint of schedule, people, and deliverable contents. Agile is still not a complete methodology. To me it is more like mixed guidelines from scrum, extreme programming, pair programming. You just tailor it when you want to use it in your team. People in your team, Process constraint in your company, release pressure from customer and so on decide how to tailor it.   

Why Sizeof object with empty class is not zero in C++?

My understanding would be creating an object means occupying some memory. No size, no memory, no way to tell it is a object or it is an array of objects in C++. So size of object with empty class should one byte. The value could be 1 or 4 depending on the machine.
I am not sure about C#. It is not difficult to know. I will test tomorrow.

Iterative vs. Recursive Approaches

I always do not like recursion. I will definitely use iteration if there is a way to use it instead of recursion. No theory. I just do not like recursion naturally.
Today the article I read tells that the performance of iteration is much better than recursion. Performance here includes both speed and memory usage. Here is the link http://www.codeproject.com/KB/recipes/Iterative_vs_Recursive.aspx so I will confidently continue to use iteration instead of recursion in my daily work.

Name of marshalling

Marshalling is so named because it was first studied in 1962 by Edward Waite Marshall, then with the General Electric corporation

Ridiculous Microsoft - Visual Studio Extensions does not support 64-bit platform

SharePoint 2007 is decided to be used as content management server for our application. It gives us lots of troubles when customized web services are needed. For me I think customized web services are necessary for SharePoint 2007 unless you only use web browser as client. SharePoint exposes its interfaces by web services but it only exposes very limited interfaces in this way. No file check in or check out is supported!

Something also ridiculous to me is that a tool from MS, Visual Studio Extensions for WSS 3.0, does not support 64-bit platforms. This tool has to work with SharePoint server and all MSDN documents tell you that SharePoint server is better installed on 64-bit platform and it only supports 64-bit machine from SharePoint 2010.

Stupid MS.

Dispose() vs Finalize()

Enough reading of Dispose, IDisposable, Finalize, SuppressFinalize, Finalized Queue, destructor, Garbage collector, weak reference ....... Hope I could fall in sleep without memory leak........ Managed type, Unmanaged Resources...

Dispose() vs Finalize() (Copied from MSDN after this line)

The primary use of this interface is to release unmanaged resources. The garbage collector automatically releases the memory allocated to a managed object when that object is no longer used. However, it is not possible to predict when garbage collection will occur. Furthermore, the garbage collector has no knowledge of unmanaged resources such as window handles, or open files and streams.

Use the Dispose method of this interface to explicitly release unmanaged resources in conjunction with the garbage collector. The consumer of an object can call this method when the object is no longer needed.

Class instances often encapsulate control over resources that are not managed by the runtime, such as window handles (HWND), database connections, and so on. Therefore, you should provide both an explicit and an implicit way to free those resources. Provide implicit control by implementing the protected Finalize on an object (destructor syntax in C# and C++). The garbage collector calls this method at some point after there are no longer any valid references to the object.
In some cases, you might want to provide programmers using an object with the ability to explicitly release these external resources before the garbage collector frees the object. If an external resource is scarce or expensive, better performance can be achieved if the programmer explicitly releases resources when they are no longer being used. To provide explicit control, implement the Dispose provided by the IDisposable. The consumer of the object should call this method when it is finished using the object. Dispose can be called even if other references to the object are alive.
Note that even when you provide explicit control using Dispose, you should provide implicit cleanup using the Finalize method. Finalize provides a backup to prevent resources from permanently leaking if the programmer fails to call Dispose.

There is no performance benefit in implementing the Dispose method on types that use only managed resources (such as arrays) because they are automatically reclaimed by the garbage collector. Use the Dispose method primarily on managed objects that use native resources and on COM objects that are exposed to the .NET Framework. Managed objects that use native resources (such as the FileStream class) implement the IDisposable interface.

Team leader's role

I have been working as a team leader's role for months. It is the second time of being team leader in my career although the first time only continued for two months before I left the company. It is totally different with being a software engineer. Personally I think the main difference is that as a software engineer I try my best to finish my task on the schedule while as a team leader I try my best to make my team members finish their tasks and then make the team's schedule. 'Make' means properly assigning work item, properly estimation of work items, progress checking, removing anything that blocks their development, and adjusting schedule.

Luckily I got support from my software manager, project manager and other team leaders. They share the experience with me and sometimes just point out the improper actions. Many thanks to them. Also our team provides books for learners. One book I am reading is TSP, Leading a Development Team.

Memory leak, Dispose() and Using statement (5)

Some conclusions. Just add something useful in the end of this series of post although most of people have known it.

Tip1:
c# provides a statement which provides a convenient syntax that ensures the correct use of IDisposable objects. It is 'using' statement.

Copied from MSDN "http://msdn.microsoft.com/en-us/library/yh598w02.aspx".
"As a rule, when you use an IDisposable object, you should declare and instantiate it in a using statement. The using statement calls the Disposemethod on the object in the correct way, and (when you use it as shown earlier) it also causes the object itself to go out of scope as soon as Dispose is called. Within the using block, the object is read-only and cannot be modified or reassigned."


An example from MSDN is like followings and MSDN also gives code of the example in the compile time.
Example about how to use 'using' statement.

C#

using (Font font1 = new Font("Arial", 10.0f)) 
{
    byte charset = font1.GdiCharSet;
}

Code in the compile time.

C#

{
  Font font1 = new Font("Arial", 10.0f);
  try
  {
    byte charset = font1.GdiCharSet;
  }
  finally
  {
    if (font1 != null)
      ((IDisposable)font1).Dispose();
  }
}

Tip2:

Call Dispose() explicitly if you can not use 'using' statement. I saw a case 'using' statement can not be used. At this time, use the try/finally pattern to make sure the unmanaged resources are disposed of even if an exception interrupts your application.

Tip3:

Do not dispose an passed in object. The reason is also obvious. The passed in parameter could be used by the calling method again. 

Memory leak, Dispose() and Using statement (4)

Step 3 run pair testing for over 12 hours to see the results

The final test result is much better than I expected. I run the original project and the modified one both over 12 hours. The result showed on average the original one increases memory at a speed of 3 GB per week. The modified one only increases at a speed of 500 MB per year.

I am happy now at this stage. And curious why it worked.

Almost everything is done here. I just want to know more about System.DirectoryServices because I have never worked on this part before. I was surfing in MSDN. And I got something interesting from MSDN.  

http://msdn.microsoft.com/en-us/library/system.directoryservices.searchresultcollection.aspx?PHPSESSID=7c61ntavm1dmu84rl10kephll5

"Due to implementation restrictions, the SearchResultCollection class cannot release all of its unmanaged resources when it is garbage collected. To prevent a memory leak, you must call the Dispose method when the SearchResultCollection object is no longer needed."

Damn Microsoft.

(Many thanks to Colin. He helped a lot and gave many suggestions when I was investigating this problem. And one of my funny colleague, Takashi, changed office when I was editing this post. 20% percent of fun on my job is gone. So sad.)

Memory leak, Dispose() and Using statement (3)

Step 2 Release the memory properly.
As I said in last post "It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. "

It is confirmed there is memory leak in one public method after several rounds of tests. I spent almost one hour to stare at this piece of code and no progress. I even doubt my tests because this project is big and I need mock up some code to filter the suspicious method. I might be wrong when doing this thing. It turns out what I have done is ok after reviewing the record. So you know it is important to write down whatever tests you did.

So now the piece of code is here.
        using System.DirectoryServices;
        ****
         public List DoSomething()
        {
            List results = new List();
            using (DirectorySearcher searcher = new DirectorySearcher(new DirectoryEntry(DefinedObject2.Instance.Path), "some string"))
            {
                foreach (SearchResult result in searcher.FindAll())
                {
                    using (DirectoryEntry someObject = result.GetDirectoryEntry())
                    {
                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));
                    }
                }
            }
            return results;
        }

Take a while to guess what code leaks memory. Do not need doubt the logic. The method works fine.

I did more tests in this method. Tried different things including mocking up some pieces of code. But I did not doubt the libraries from .Net until I got nothing more to try. So I just add some Dispose() method to the objects which implements IDisposable(). Magically it worked. Tests show that memory usage is much much better. Now I got a clue how to fix them but still do not know why it works since CLR is supposed to take care of the memory usage.

More tests are done during this stage to help figure out exactly where to use Dispose(). It turns out objects under namespace System.DirectoryServices are all suspicious and need call Dispose() explicitly . I am not sure if every object under that namespace needs do it but I am sure System.DirectoryServices.SearchResultCollection and System.DirectoryServices.DirectoryEntry definitely need call dispose() explicitly. Did not get enough time to test every object under that namespace. So I just simply call Dispose() to every object under that namespace and implement IDisposable().
The final code looks like this

        using System.DirectoryServices;

        ****

         public List DoSomething()

        {

            List results = new List();

            using (DirectoryEntry temp = new DirectoryEntry(DefinedObject2.Instance.Path))

        {

            using (DirectorySearcher searcher = new DirectorySearcher(temp, "some string"))

            {

               using (DirectorySearchCollections tempCollection  = searcher.FindAll())

               {

                foreach (SearchResult result in tempCollection)

                {

                    using (DirectoryEntry someObject = result.GetDirectoryEntry())

                    {

                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));

                    }

                }

               }

            }

          }

            return results;

        }

Authentication web service for SharePoint

To use the authentication web service, Form based authentication has to be enabled to your SharePoint server. Otherwise there will an error message about "LoginErrorCode.NotInFormsAuthenticationMode" back from server. It does not care about username or password is correct or not. In another word, Form based authentication is the first condition to check when logging through authentication web service.

Memory leak, Dispose() and Using statement (2)

Step 1: Diagnose your code. Find the evil piece.

Probably you know some tools which helps to measure performance and memory usage. Two types of tools can help you in this step. Memory measurement tool and memory explorer tool.

WMI (Windows Management Instruction) does a good job measuring private working set for a defined process or processes. It keeps tracking the memory usage with a configured time interval and saving to spreadsheet or other format files. There is also a user interface so you can see some graph of memory usage.

There are also several memory explorer tools in the market. The one I am using is ANTS memory profiler. It gives a good hint which objects are now inside the memory (heap), and also objects numbers, values and how far they are from the GC root. Even it can compare the memory snapshots at two different time. From my experience sometimes it helps, sometimes not. Memory usage is a complex issue especially when you have a complex system. You got tons of things in the memory. Usually unused objects stay in the memory for a while should be ok as long as they could be collected in the next garbage collection. But the thing is that GC , not you, decides when and how to collect the garbage. So it is really hard to tell if these unused objects in the memory are new or survive in the last garbage collection. You have to run the tool for a longer time to compare the several snapshots.

Go back to the memory leak problem I fixed. Some context. The project size is big and I am not familiar with every module. And no way to be familiar with every module of this big project.

I am a bit lucky this time because I paid attention to the memory usage of the system for a long time. Sometime I left the system running over night and there was no obvious memory leaking. So think about the developing environment and production environment and get some idea.

• Database size. I believe production environment has a much bigger database size.
• LDAP. Production environment uses LDAP for authentication. I do not.
• I did not do a real long test.

It did not take me very long time to narrow down to one suspicious module. I spent almost one day on some suspicious UI things because ANTS memory profiler told me that there are a lot of objects staying in the memory relevant to UI refreshing. But after some painful code checking and more tests I found out that number of that part of objects went down after it reached a limit. And it does it consistently. So forgot about it and moved on. I also did spend some time to increase my database size and did more tests. It turned out database size has nothing to do with memory leaking at this time.

I focused on several public methods after verifying that this module is the evil one with several tests. It became easy now. Tested one method each time after mocking up the other suspicious methods. I were almost there after several tests. I can tell which lines are leaking memory. But I can not tell if they are all of the evil pieces.

To be honest ANTS memory profiler does not help a lot in this step and drove me to the wrong direction sometimes.

Memory leak, Dispose() and Using statement (1)

The project I worked on has a huge memory leak in production environment. It was reported recently by onsite engineer. I believe this problem exists for a long time. It hid very well previously.

Memory leak is always not easy to fix. First step you have to find out which place has memory leak. It is painful to go through this process. Next step you have to release the memory properly. It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. Of course third step you have to run pair testing for over 12 hours to see the results. It is not compulsory to be 12 hours. 2 hours even 1 hour is still good for the middle test. It has to be long enough to make you confident that your modification works or it does not work. But the final test has to be more than 12 hours because this time you want to show your managers and team members that your modification works. Generally you will see some nice results after several times tests. Another annoy thing about memory leak problem is that memory leak usually does not happen at only one place. You have to find them piece by piece and fix them piece by piece. The test result will be a bit nicer, a bit nicer, a bit more nicer... Usually not big change at one time. It also means memory leak problem is time consuming.

So to fix memory leak problem, probably you will go through hopeless, depressed, disappointed, and annoyed. And most of the important hopefully you will feel happiness in the end . I reach the last one every time luckily.

Many thanks to my lovely and funny colleagues, Colin, Takashi, and Thura. They make my job more interesting.

A Win32Exception is thrown - The file size exceeds the limit allowed and cannot be saved

Currently we got an exception A Win32Exception is thrown - The file size exceeds the limit allowed and cannot be saved. It seems it is a bug from Windows after searching online for a while.

My colleague confirmed this with his change set although I do not know it works or not. And he referred to an article about this exception. The link is here http://daniel-richardson.blogspot.com/2008/12/handling-eventlogentrywritten-event-for.html

"A Win32Exception is thrown if EnableRaisingEvents is set to true on an empty EventLog. The exception message is "The file size exceeds the limit allowed and cannot be saved".
The workaround for this is to check if the EventLog is empty and write an entry if necessary before setting EnableRaisingEvents to true."

Or you do not set EnableRaisingEvents to true for no good reason.

ActiveMQ Message Routing

There are several things to think about when you design the ActiveMQ message routing.

1) JMS Selectors. Selectors are a way of attaching a filter to a subscription to perform content based routing. Selectors are defined using SQL 92 syntax and typically apply to message headers; whether the standard properties available on a JMS message or custom headers you can add via the JMS code.

http://activemq.apache.org/selectors.html

ActiveMQ message properties and JMS message properties should be considered when using JMS selectors.

http://activemq.apache.org/activemq-message-properties.html

2) Wildcards. ActiveMQ supports destination wildcards to provide easy support for federated name hierarchies. A subscriber could use wildcards to define hierarchial pattern matches to the destinations to subscribe from. Note wildcards are not part of the JMS specification so are custom enhancements.

I think this feature is quite useful when some messages are only interesting to part of the consumers and relation structure is complex.

http://activemq.apache.org/wildcards.html

3) Advisory Message. It is kind of administrative information of ActiveMQ which helps to watch the system by subscribing to regular JMS messages. By default this is disabled.

http://activemq.apache.org/advisory-message.html?bcsi-ac-20A76BC15DBD50F6=1AADD8DD00000503CzAfPhVn8T+sJ2asth9iskWerAsTAgAAAwUAAJQ8hQIIBwAAVQAAAMKDAwA=

4) Message Redelivery. Messages are redelivered to a client when any of the following occurs:

1. A transacted session is used and rollback() is called.
2. A transacted session is closed before commit is called.
3. A session is using CLIENT_ACKNOWLEDGE and Session.recover() is called.

http://activemq.apache.org/message-redelivery-and-dlq-handling.html

(DLQ means dead letter queue)

Host multiple services with multiple configuration files

Definitely it should be good if we could do this - Host multiple services with multiple configuration files. I found a nice post talking about the way the author implements this. Basic idea is to use different AppDomain for different services and handling loading, hosting service manually. Here is the link: http://blogs.microsoft.co.il/blogs/alon/archive/2008/03/12/hosting-plug-in-wcf-services.aspx

I am not sure if this is a good idea. Creating different AppDomains in one process is a little scaring. Do not know how to catch thread level exceptions for each domain. It might be possible to do it. Have not tried. And another thing is how to handle threading context. Not sure. To be honest, I did not we could have different App.Domain in one process. Let's find out more about App.Domain.

===later added

By reading a little bit about App.Domain. I think it might be a good idea. Process is a concept of Windows OS. App.Domain is a concept of .Net Framework. It is about CLR running time resource. It might be good to separate AppDomain for different services.

Properties.Settings.Default.Save() saves to which file?

This did bother me for a while.

There are two options for Scope when we try to add a setting. User and Application. Both default settings are stored in the app.config of this project.

Where will it be saved if user changes the setting in the running time?

The save will save to app.config of the project if it is a application level setting. But it will save to C:\Documents and Settings\[user]\Local Settings\Application Data\[application_name] folder if it is a User level setting.

You can only get the default setting back by deleting the XML files in the above folders and run your application.

SharePoint 2010 Preliminary System Requirements

I probably will work on SharePoint for a while.

Microsoft announced preliminary system requirements for SharePoint server 2010. Obviously it goes for some big enterprise with enough budget.

1. SharePoint Server 2010 will be 64-bit only.
2. SharePoint Server 2010 will require 64-bit Windows Server 2008 or 64-bit Windows Server 2008 R2.
3. SharePoint Server 2010 will require 64-bit SQL Server 2008 or 64-bit SQL Server 2005.

I copied it from http://blogs.msdn.com/sharepoint/archive/2009/05/07/announcing-sharepoint-server-2010-preliminary-system-requirements.aspx