Monday, 7 June 2010

Memory leak, Dispose() and Using statement (5)

Some conclusions. Just add something useful in the end of this series of post although most of people have known it.

Tip1:
c# provides a statement which provides a convenient syntax that ensures the correct use of IDisposable objects. It is 'using' statement.

Copied from MSDN "http://msdn.microsoft.com/en-us/library/yh598w02.aspx".
"As a rule, when you use an IDisposable object, you should declare and instantiate it in a using statement. The using statement calls the Disposemethod on the object in the correct way, and (when you use it as shown earlier) it also causes the object itself to go out of scope as soon as Dispose is called. Within the using block, the object is read-only and cannot be modified or reassigned."


An example from MSDN is like followings and MSDN also gives code of the example in the compile time.
Example about how to use 'using' statement.

C#
using (Font font1 = new Font("Arial", 10.0f)) 
{
    byte charset = font1.GdiCharSet;
}
Code in the compile time.

C#
{
  Font font1 = new Font("Arial", 10.0f);
  try
  {
    byte charset = font1.GdiCharSet;
  }
  finally
  {
    if (font1 != null)
      ((IDisposable)font1).Dispose();
  }
}
Tip2:
Call Dispose() explicitly if you can not use 'using' statement. I saw a case 'using' statement can not be used. At this time, use the try/finally pattern to make sure the unmanaged resources are disposed of even if an exception interrupts your application.

Tip3:
Do not dispose an passed in object. The reason is also obvious. The passed in parameter could be used by the calling method again. 

Memory leak, Dispose() and Using statement (4)

Step 3 run pair testing for over 12 hours to see the results

The final test result is much better than I expected. I run the original project and the modified one both over 12 hours. The result showed on average the original one increases memory at a speed of 3 GB per week. The modified one only increases at a speed of 500 MB per year.

I am happy now at this stage. And curious why it worked.

Almost everything is done here. I just want to know more about System.DirectoryServices because I have never worked on this part before. I was surfing in MSDN. And I got something interesting from MSDN.  

"Due to implementation restrictions, the SearchResultCollection class cannot release all of its unmanaged resources when it is garbage collected. To prevent a memory leak, you must call the Dispose method when the SearchResultCollection object is no longer needed."

Damn Microsoft.

(Many thanks to Colin. He helped a lot and gave many suggestions when I was investigating this problem. And one of my funny colleague, Takashi, changed office when I was editing this post. 20% percent of fun on my job is gone. So sad.)

Memory leak, Dispose() and Using statement (3)

Step 2 Release the memory properly.
As I said in last post "It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. "

It is confirmed there is memory leak in one public method after several rounds of tests. I spent almost one hour to stare at this piece of code and no progress. I even doubt my tests because this project is big and I need mock up some code to filter the suspicious method. I might be wrong when doing this thing. It turns out what I have done is ok after reviewing the record. So you know it is important to write down whatever tests you did.

So now the piece of code is here.
        using System.DirectoryServices;
        ****
         public List DoSomething()
        {
            List results = new List();
            using (DirectorySearcher searcher = new DirectorySearcher(new DirectoryEntry(DefinedObject2.Instance.Path), "some string"))
            {
                foreach (SearchResult result in searcher.FindAll())
                {
                    using (DirectoryEntry someObject = result.GetDirectoryEntry())
                    {
                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));
                    }
                }
            }
            return results;
        }

Take a while to guess what code leaks memory. Do not need doubt the logic. The method works fine.

I did more tests in this method. Tried different things including mocking up some pieces of code. But I did not doubt the libraries from .Net until I got nothing more to try. So I just add some Dispose() method to the objects which implements IDisposable(). Magically it worked. Tests show that memory usage is much much better. Now I got a clue how to fix them but still do not know why it works since CLR is supposed to take care of the memory usage.

More tests are done during this stage to help figure out exactly where to use Dispose(). It turns out objects under namespace System.DirectoryServices are all suspicious and need call Dispose() explicitly . I am not sure if every object under that namespace needs do it but I am sure System.DirectoryServices.SearchResultCollection and System.DirectoryServices.DirectoryEntry definitely need call dispose() explicitly. Did not get enough time to test every object under that namespace. So I just simply call Dispose() to every object under that namespace and implement IDisposable().
The final code looks like this

        using System.DirectoryServices;
        ****
         public List DoSomething()
        {
            List results = new List();
            using (DirectoryEntry temp = new DirectoryEntry(DefinedObject2.Instance.Path))
        {
            using (DirectorySearcher searcher = new DirectorySearcher(temp, "some string"))
            {
               using (DirectorySearchCollections tempCollection  = searcher.FindAll())
               {
                foreach (SearchResult result in tempCollection)
                {
                    using (DirectoryEntry someObject = result.GetDirectoryEntry())
                    {
                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));
                    }
                }
               }
            }
          }
            return results;
        }


Authentication web service for SharePoint

To use the authentication web service, Form based authentication has to be enabled to your SharePoint server. Otherwise there will an error message about "LoginErrorCode.NotInFormsAuthenticationMode" back from server. It does not care about username or password is correct or not. In another word, Form based authentication is the first condition to check when logging through authentication web service.

Thursday, 3 June 2010

Memory leak, Dispose() and Using statement (2)

Step 1: Diagnose your code. Find the evil piece.
Probably you know some tools which helps to measure performance and memory usage. Two types of tools can help you in this step. Memory measurement tool and memory explorer tool.
WMI (Windows Management Instruction) does a good job measuring private working set for a defined process or processes. It keeps tracking the memory usage with a configured time interval and saving to spreadsheet or other format files. There is also a user interface so you can see some graph of memory usage.
There are also several memory explorer tools in the market. The one I am using is ANTS memory profiler. It gives a good hint which objects are now inside the memory (heap), and also objects numbers, values and how far they are from the GC root. Even it can compare the memory snapshots at two different time. From my experience sometimes it helps, sometimes not. Memory usage is a complex issue especially when you have a complex system. You got tons of things in the memory. Usually unused objects stay in the memory for a while should be ok as long as they could be collected in the next garbage collection. But the thing is that GC , not you, decides when and how to collect the garbage. So it is really hard to tell if these unused objects in the memory are new or survive in the last garbage collection. You have to run the tool for a longer time to compare the several snapshots.

Go back to the memory leak problem I fixed. Some context. The project size is big and I am not familiar with every module. And no way to be familiar with every module of this big project.
I am a bit lucky this time because I paid attention to the memory usage of the system for a long time. Sometime I left the system running over night and there was no obvious memory leaking. So think about the developing environment and production environment and get some idea.
  • Database size. I believe production environment has a much bigger database size.
  • LDAP. Production environment uses LDAP for authentication. I do not.
  • I did not do a real long test.
It did not take me very long time to narrow down to one suspicious module. I spent almost one day on some suspicious UI things because ANTS memory profiler told me that there are a lot of objects staying in the memory relevant to UI refreshing. But after some painful code checking and more tests I found out that number of that part of objects went down after it reached a limit. And it does it consistently. So forgot about it and moved on. I also did spend some time to increase my database size and did more tests. It turned out database size has nothing to do with memory leaking at this time.

I focused on several public methods after verifying that this module is the evil one with several tests. It became easy now. Tested one method each time after mocking up the other suspicious methods. I were almost there after several tests. I can tell which lines are leaking memory. But I can not tell if they are all of the evil pieces.

To be honest ANTS memory profiler does not help a lot in this step and drove me to the wrong direction sometimes.


Memory leak, Dispose() and Using statement (1)

The project I worked on has a huge memory leak in production environment. It was reported recently by onsite engineer. I believe this problem exists for a long time. It hid very well previously.

Memory leak is always not easy to fix. First step you have to find out which place has memory leak. It is painful to go through this process. Next step you have to release the memory properly. It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. Of course third step you have to run pair testing for over 12 hours to see the results. It is not compulsory to be 12 hours. 2 hours even 1 hour is still good for the middle test. It has to be long enough to make you confident that your modification works or it does not work. But the final test has to be more than 12 hours because this time you want to show your managers and team members that your modification works. Generally you will see some nice results after several times tests. Another annoy thing about memory leak problem is that memory leak usually does not happen at only one place. You have to find them piece by piece and fix them piece by piece. The test result will be a bit nicer, a bit nicer, a bit more nicer... Usually not big change at one time. It also means memory leak problem is time consuming.

So to fix memory leak problem, probably you will go through hopeless, depressed, disappointed, and annoyed. And most of the important hopefully you will feel happiness in the end . I reach the last one every time luckily.

Many thanks to my lovely and funny colleagues, Colin, Takashi, and Thura. They make my job more interesting.