Dispose() vs Finalize()

Enough reading of Dispose, IDisposable, Finalize, SuppressFinalize, Finalized Queue, destructor, Garbage collector, weak reference ....... Hope I could fall in sleep without memory leak........ Managed type, Unmanaged Resources...

Dispose() vs Finalize() (Copied from MSDN after this line)

The primary use of this interface is to release unmanaged resources. The garbage collector automatically releases the memory allocated to a managed object when that object is no longer used. However, it is not possible to predict when garbage collection will occur. Furthermore, the garbage collector has no knowledge of unmanaged resources such as window handles, or open files and streams.

Use the Dispose method of this interface to explicitly release unmanaged resources in conjunction with the garbage collector. The consumer of an object can call this method when the object is no longer needed.

Class instances often encapsulate control over resources that are not managed by the runtime, such as window handles (HWND), database connections, and so on. Therefore, you should provide both an explicit and an implicit way to free those resources. Provide implicit control by implementing the protected Finalize on an object (destructor syntax in C# and C++). The garbage collector calls this method at some point after there are no longer any valid references to the object.
In some cases, you might want to provide programmers using an object with the ability to explicitly release these external resources before the garbage collector frees the object. If an external resource is scarce or expensive, better performance can be achieved if the programmer explicitly releases resources when they are no longer being used. To provide explicit control, implement the Dispose provided by the IDisposable. The consumer of the object should call this method when it is finished using the object. Dispose can be called even if other references to the object are alive.
Note that even when you provide explicit control using Dispose, you should provide implicit cleanup using the Finalize method. Finalize provides a backup to prevent resources from permanently leaking if the programmer fails to call Dispose.

There is no performance benefit in implementing the Dispose method on types that use only managed resources (such as arrays) because they are automatically reclaimed by the garbage collector. Use the Dispose method primarily on managed objects that use native resources and on COM objects that are exposed to the .NET Framework. Managed objects that use native resources (such as the FileStream class) implement the IDisposable interface.

Memory leak, Dispose() and Using statement (5)

Some conclusions. Just add something useful in the end of this series of post although most of people have known it.

Tip1:
c# provides a statement which provides a convenient syntax that ensures the correct use of IDisposable objects. It is 'using' statement.

Copied from MSDN "http://msdn.microsoft.com/en-us/library/yh598w02.aspx".
"As a rule, when you use an IDisposable object, you should declare and instantiate it in a using statement. The using statement calls the Disposemethod on the object in the correct way, and (when you use it as shown earlier) it also causes the object itself to go out of scope as soon as Dispose is called. Within the using block, the object is read-only and cannot be modified or reassigned."


An example from MSDN is like followings and MSDN also gives code of the example in the compile time.
Example about how to use 'using' statement.

C#

using (Font font1 = new Font("Arial", 10.0f)) 
{
    byte charset = font1.GdiCharSet;
}

Code in the compile time.

C#

{
  Font font1 = new Font("Arial", 10.0f);
  try
  {
    byte charset = font1.GdiCharSet;
  }
  finally
  {
    if (font1 != null)
      ((IDisposable)font1).Dispose();
  }
}

Tip2:

Call Dispose() explicitly if you can not use 'using' statement. I saw a case 'using' statement can not be used. At this time, use the try/finally pattern to make sure the unmanaged resources are disposed of even if an exception interrupts your application.

Tip3:

Do not dispose an passed in object. The reason is also obvious. The passed in parameter could be used by the calling method again. 

Memory leak, Dispose() and Using statement (4)

Step 3 run pair testing for over 12 hours to see the results

The final test result is much better than I expected. I run the original project and the modified one both over 12 hours. The result showed on average the original one increases memory at a speed of 3 GB per week. The modified one only increases at a speed of 500 MB per year.

I am happy now at this stage. And curious why it worked.

Almost everything is done here. I just want to know more about System.DirectoryServices because I have never worked on this part before. I was surfing in MSDN. And I got something interesting from MSDN.  

http://msdn.microsoft.com/en-us/library/system.directoryservices.searchresultcollection.aspx?PHPSESSID=7c61ntavm1dmu84rl10kephll5

"Due to implementation restrictions, the SearchResultCollection class cannot release all of its unmanaged resources when it is garbage collected. To prevent a memory leak, you must call the Dispose method when the SearchResultCollection object is no longer needed."

Damn Microsoft.

(Many thanks to Colin. He helped a lot and gave many suggestions when I was investigating this problem. And one of my funny colleague, Takashi, changed office when I was editing this post. 20% percent of fun on my job is gone. So sad.)

Memory leak, Dispose() and Using statement (3)

Step 2 Release the memory properly.
As I said in last post "It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. "

It is confirmed there is memory leak in one public method after several rounds of tests. I spent almost one hour to stare at this piece of code and no progress. I even doubt my tests because this project is big and I need mock up some code to filter the suspicious method. I might be wrong when doing this thing. It turns out what I have done is ok after reviewing the record. So you know it is important to write down whatever tests you did.

So now the piece of code is here.
        using System.DirectoryServices;
        ****
         public List DoSomething()
        {
            List results = new List();
            using (DirectorySearcher searcher = new DirectorySearcher(new DirectoryEntry(DefinedObject2.Instance.Path), "some string"))
            {
                foreach (SearchResult result in searcher.FindAll())
                {
                    using (DirectoryEntry someObject = result.GetDirectoryEntry())
                    {
                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));
                    }
                }
            }
            return results;
        }

Take a while to guess what code leaks memory. Do not need doubt the logic. The method works fine.

I did more tests in this method. Tried different things including mocking up some pieces of code. But I did not doubt the libraries from .Net until I got nothing more to try. So I just add some Dispose() method to the objects which implements IDisposable(). Magically it worked. Tests show that memory usage is much much better. Now I got a clue how to fix them but still do not know why it works since CLR is supposed to take care of the memory usage.

More tests are done during this stage to help figure out exactly where to use Dispose(). It turns out objects under namespace System.DirectoryServices are all suspicious and need call Dispose() explicitly . I am not sure if every object under that namespace needs do it but I am sure System.DirectoryServices.SearchResultCollection and System.DirectoryServices.DirectoryEntry definitely need call dispose() explicitly. Did not get enough time to test every object under that namespace. So I just simply call Dispose() to every object under that namespace and implement IDisposable().
The final code looks like this

        using System.DirectoryServices;

        ****

         public List DoSomething()

        {

            List results = new List();

            using (DirectoryEntry temp = new DirectoryEntry(DefinedObject2.Instance.Path))

        {

            using (DirectorySearcher searcher = new DirectorySearcher(temp, "some string"))

            {

               using (DirectorySearchCollections tempCollection  = searcher.FindAll())

               {

                foreach (SearchResult result in tempCollection)

                {

                    using (DirectoryEntry someObject = result.GetDirectoryEntry())

                    {

                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));

                    }

                }

               }

            }

          }

            return results;

        }

Memory leak, Dispose() and Using statement (2)

Step 1: Diagnose your code. Find the evil piece.

Probably you know some tools which helps to measure performance and memory usage. Two types of tools can help you in this step. Memory measurement tool and memory explorer tool.

WMI (Windows Management Instruction) does a good job measuring private working set for a defined process or processes. It keeps tracking the memory usage with a configured time interval and saving to spreadsheet or other format files. There is also a user interface so you can see some graph of memory usage.

There are also several memory explorer tools in the market. The one I am using is ANTS memory profiler. It gives a good hint which objects are now inside the memory (heap), and also objects numbers, values and how far they are from the GC root. Even it can compare the memory snapshots at two different time. From my experience sometimes it helps, sometimes not. Memory usage is a complex issue especially when you have a complex system. You got tons of things in the memory. Usually unused objects stay in the memory for a while should be ok as long as they could be collected in the next garbage collection. But the thing is that GC , not you, decides when and how to collect the garbage. So it is really hard to tell if these unused objects in the memory are new or survive in the last garbage collection. You have to run the tool for a longer time to compare the several snapshots.

Go back to the memory leak problem I fixed. Some context. The project size is big and I am not familiar with every module. And no way to be familiar with every module of this big project.

I am a bit lucky this time because I paid attention to the memory usage of the system for a long time. Sometime I left the system running over night and there was no obvious memory leaking. So think about the developing environment and production environment and get some idea.

• Database size. I believe production environment has a much bigger database size.
• LDAP. Production environment uses LDAP for authentication. I do not.
• I did not do a real long test.

It did not take me very long time to narrow down to one suspicious module. I spent almost one day on some suspicious UI things because ANTS memory profiler told me that there are a lot of objects staying in the memory relevant to UI refreshing. But after some painful code checking and more tests I found out that number of that part of objects went down after it reached a limit. And it does it consistently. So forgot about it and moved on. I also did spend some time to increase my database size and did more tests. It turned out database size has nothing to do with memory leaking at this time.

I focused on several public methods after verifying that this module is the evil one with several tests. It became easy now. Tested one method each time after mocking up the other suspicious methods. I were almost there after several tests. I can tell which lines are leaking memory. But I can not tell if they are all of the evil pieces.

To be honest ANTS memory profiler does not help a lot in this step and drove me to the wrong direction sometimes.

Memory leak, Dispose() and Using statement (1)

The project I worked on has a huge memory leak in production environment. It was reported recently by onsite engineer. I believe this problem exists for a long time. It hid very well previously.

Memory leak is always not easy to fix. First step you have to find out which place has memory leak. It is painful to go through this process. Next step you have to release the memory properly. It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. Of course third step you have to run pair testing for over 12 hours to see the results. It is not compulsory to be 12 hours. 2 hours even 1 hour is still good for the middle test. It has to be long enough to make you confident that your modification works or it does not work. But the final test has to be more than 12 hours because this time you want to show your managers and team members that your modification works. Generally you will see some nice results after several times tests. Another annoy thing about memory leak problem is that memory leak usually does not happen at only one place. You have to find them piece by piece and fix them piece by piece. The test result will be a bit nicer, a bit nicer, a bit more nicer... Usually not big change at one time. It also means memory leak problem is time consuming.

So to fix memory leak problem, probably you will go through hopeless, depressed, disappointed, and annoyed. And most of the important hopefully you will feel happiness in the end . I reach the last one every time luckily.

Many thanks to my lovely and funny colleagues, Colin, Takashi, and Thura. They make my job more interesting.

Lifetime of static variable

From MSDN http://msdn.microsoft.com/en-us/library/aa645766%28VS.71%29.aspx

A field declared with the static modifier is called a static variable. A static variable comes into existence before execution of the static constructor (Section 10.11) for its containing type, and ceases to exist when the associated application domain ceases to exist.

The initial value of a static variable is the default value (Section 5.2) of the variable's type.

For purposes of definite assignment checking, a static variable is considered initially assigned.

How accurate of DateTime.Now

Copied from here http://www.red-gate.com/products/ants_performance_profiler/dotnet_challenge_question_4_v2.htm?utm_source=simpletalk&utm_medium=email&utm_content=dotnetchallengeq4&utm_campaign=antsperformanceprofiler
DateTime.Now is not intended to be a high-precision timer. For Windows NT 3.5, Windows 2000 and later, it has a resolution of approximately 10–15 milliseconds. Resolution may be lower on other platforms. For more accurate timing, the System.Diagnostics.StopWatch class is available. Other high-precision timers include the kernel32.dll QueryPerformanceTimer and QueryPerformanceFrequency native calls. If you are trying to time the execution speed of code, an even better solution is just to use ANTS Performance Profiler.

Static constructor

From MSDN.
http://msdn.microsoft.com/en-us/library/k9x6w0hc(VS.80).aspx

Static constructors have the following properties:

• A static constructor does not take access modifiers or have parameters.

• A static constructor is called automatically to initialize the class before the first instance is created or any static members are referenced.

• A static constructor cannot be called directly.

• The user has no control on when the static constructor is executed in the program.

• A typical use of static constructors is when the class is using a log file and the constructor is used to write entries to this file.

• Static constructors are also useful when creating wrapper classes for unmanaged code, when the constructor can call the LoadLibrary method.