Name of marshalling

Marshalling is so named because it was first studied in 1962 by Edward Waite Marshall, then with the General Electric corporation

Ridiculous Microsoft - Visual Studio Extensions does not support 64-bit platform

SharePoint 2007 is decided to be used as content management server for our application. It gives us lots of troubles when customized web services are needed. For me I think customized web services are necessary for SharePoint 2007 unless you only use web browser as client. SharePoint exposes its interfaces by web services but it only exposes very limited interfaces in this way. No file check in or check out is supported!

Something also ridiculous to me is that a tool from MS, Visual Studio Extensions for WSS 3.0, does not support 64-bit platforms. This tool has to work with SharePoint server and all MSDN documents tell you that SharePoint server is better installed on 64-bit platform and it only supports 64-bit machine from SharePoint 2010.

Stupid MS.

Dispose() vs Finalize()

Enough reading of Dispose, IDisposable, Finalize, SuppressFinalize, Finalized Queue, destructor, Garbage collector, weak reference ....... Hope I could fall in sleep without memory leak........ Managed type, Unmanaged Resources...

Dispose() vs Finalize() (Copied from MSDN after this line)

The primary use of this interface is to release unmanaged resources. The garbage collector automatically releases the memory allocated to a managed object when that object is no longer used. However, it is not possible to predict when garbage collection will occur. Furthermore, the garbage collector has no knowledge of unmanaged resources such as window handles, or open files and streams.

Use the Dispose method of this interface to explicitly release unmanaged resources in conjunction with the garbage collector. The consumer of an object can call this method when the object is no longer needed.

Class instances often encapsulate control over resources that are not managed by the runtime, such as window handles (HWND), database connections, and so on. Therefore, you should provide both an explicit and an implicit way to free those resources. Provide implicit control by implementing the protected Finalize on an object (destructor syntax in C# and C++). The garbage collector calls this method at some point after there are no longer any valid references to the object.
In some cases, you might want to provide programmers using an object with the ability to explicitly release these external resources before the garbage collector frees the object. If an external resource is scarce or expensive, better performance can be achieved if the programmer explicitly releases resources when they are no longer being used. To provide explicit control, implement the Dispose provided by the IDisposable. The consumer of the object should call this method when it is finished using the object. Dispose can be called even if other references to the object are alive.
Note that even when you provide explicit control using Dispose, you should provide implicit cleanup using the Finalize method. Finalize provides a backup to prevent resources from permanently leaking if the programmer fails to call Dispose.

There is no performance benefit in implementing the Dispose method on types that use only managed resources (such as arrays) because they are automatically reclaimed by the garbage collector. Use the Dispose method primarily on managed objects that use native resources and on COM objects that are exposed to the .NET Framework. Managed objects that use native resources (such as the FileStream class) implement the IDisposable interface.

Team leader's role

I have been working as a team leader's role for months. It is the second time of being team leader in my career although the first time only continued for two months before I left the company. It is totally different with being a software engineer. Personally I think the main difference is that as a software engineer I try my best to finish my task on the schedule while as a team leader I try my best to make my team members finish their tasks and then make the team's schedule. 'Make' means properly assigning work item, properly estimation of work items, progress checking, removing anything that blocks their development, and adjusting schedule.

Luckily I got support from my software manager, project manager and other team leaders. They share the experience with me and sometimes just point out the improper actions. Many thanks to them. Also our team provides books for learners. One book I am reading is TSP, Leading a Development Team.

Memory leak, Dispose() and Using statement (5)

Some conclusions. Just add something useful in the end of this series of post although most of people have known it.

Tip1:
c# provides a statement which provides a convenient syntax that ensures the correct use of IDisposable objects. It is 'using' statement.

Copied from MSDN "http://msdn.microsoft.com/en-us/library/yh598w02.aspx".
"As a rule, when you use an IDisposable object, you should declare and instantiate it in a using statement. The using statement calls the Disposemethod on the object in the correct way, and (when you use it as shown earlier) it also causes the object itself to go out of scope as soon as Dispose is called. Within the using block, the object is read-only and cannot be modified or reassigned."


An example from MSDN is like followings and MSDN also gives code of the example in the compile time.
Example about how to use 'using' statement.

C#

using (Font font1 = new Font("Arial", 10.0f)) 
{
    byte charset = font1.GdiCharSet;
}

Code in the compile time.

C#

{
  Font font1 = new Font("Arial", 10.0f);
  try
  {
    byte charset = font1.GdiCharSet;
  }
  finally
  {
    if (font1 != null)
      ((IDisposable)font1).Dispose();
  }
}

Tip2:

Call Dispose() explicitly if you can not use 'using' statement. I saw a case 'using' statement can not be used. At this time, use the try/finally pattern to make sure the unmanaged resources are disposed of even if an exception interrupts your application.

Tip3:

Do not dispose an passed in object. The reason is also obvious. The passed in parameter could be used by the calling method again. 

Memory leak, Dispose() and Using statement (4)

Step 3 run pair testing for over 12 hours to see the results

The final test result is much better than I expected. I run the original project and the modified one both over 12 hours. The result showed on average the original one increases memory at a speed of 3 GB per week. The modified one only increases at a speed of 500 MB per year.

I am happy now at this stage. And curious why it worked.

Almost everything is done here. I just want to know more about System.DirectoryServices because I have never worked on this part before. I was surfing in MSDN. And I got something interesting from MSDN.  

http://msdn.microsoft.com/en-us/library/system.directoryservices.searchresultcollection.aspx?PHPSESSID=7c61ntavm1dmu84rl10kephll5

"Due to implementation restrictions, the SearchResultCollection class cannot release all of its unmanaged resources when it is garbage collected. To prevent a memory leak, you must call the Dispose method when the SearchResultCollection object is no longer needed."

Damn Microsoft.

(Many thanks to Colin. He helped a lot and gave many suggestions when I was investigating this problem. And one of my funny colleague, Takashi, changed office when I was editing this post. 20% percent of fun on my job is gone. So sad.)

Memory leak, Dispose() and Using statement (3)

Step 2 Release the memory properly.
As I said in last post "It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. "

It is confirmed there is memory leak in one public method after several rounds of tests. I spent almost one hour to stare at this piece of code and no progress. I even doubt my tests because this project is big and I need mock up some code to filter the suspicious method. I might be wrong when doing this thing. It turns out what I have done is ok after reviewing the record. So you know it is important to write down whatever tests you did.

So now the piece of code is here.
        using System.DirectoryServices;
        ****
         public List DoSomething()
        {
            List results = new List();
            using (DirectorySearcher searcher = new DirectorySearcher(new DirectoryEntry(DefinedObject2.Instance.Path), "some string"))
            {
                foreach (SearchResult result in searcher.FindAll())
                {
                    using (DirectoryEntry someObject = result.GetDirectoryEntry())
                    {
                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));
                    }
                }
            }
            return results;
        }

Take a while to guess what code leaks memory. Do not need doubt the logic. The method works fine.

I did more tests in this method. Tried different things including mocking up some pieces of code. But I did not doubt the libraries from .Net until I got nothing more to try. So I just add some Dispose() method to the objects which implements IDisposable(). Magically it worked. Tests show that memory usage is much much better. Now I got a clue how to fix them but still do not know why it works since CLR is supposed to take care of the memory usage.

More tests are done during this stage to help figure out exactly where to use Dispose(). It turns out objects under namespace System.DirectoryServices are all suspicious and need call Dispose() explicitly . I am not sure if every object under that namespace needs do it but I am sure System.DirectoryServices.SearchResultCollection and System.DirectoryServices.DirectoryEntry definitely need call dispose() explicitly. Did not get enough time to test every object under that namespace. So I just simply call Dispose() to every object under that namespace and implement IDisposable().
The final code looks like this

        using System.DirectoryServices;

        ****

         public List DoSomething()

        {

            List results = new List();

            using (DirectoryEntry temp = new DirectoryEntry(DefinedObject2.Instance.Path))

        {

            using (DirectorySearcher searcher = new DirectorySearcher(temp, "some string"))

            {

               using (DirectorySearchCollections tempCollection  = searcher.FindAll())

               {

                foreach (SearchResult result in tempCollection)

                {

                    using (DirectoryEntry someObject = result.GetDirectoryEntry())

                    {

                        results.Add(new DefinedObject3( Get Something information from 'someObject' to construct this object ));

                    }

                }

               }

            }

          }

            return results;

        }

Authentication web service for SharePoint

To use the authentication web service, Form based authentication has to be enabled to your SharePoint server. Otherwise there will an error message about "LoginErrorCode.NotInFormsAuthenticationMode" back from server. It does not care about username or password is correct or not. In another word, Form based authentication is the first condition to check when logging through authentication web service.

Memory leak, Dispose() and Using statement (2)

Step 1: Diagnose your code. Find the evil piece.

Probably you know some tools which helps to measure performance and memory usage. Two types of tools can help you in this step. Memory measurement tool and memory explorer tool.

WMI (Windows Management Instruction) does a good job measuring private working set for a defined process or processes. It keeps tracking the memory usage with a configured time interval and saving to spreadsheet or other format files. There is also a user interface so you can see some graph of memory usage.

There are also several memory explorer tools in the market. The one I am using is ANTS memory profiler. It gives a good hint which objects are now inside the memory (heap), and also objects numbers, values and how far they are from the GC root. Even it can compare the memory snapshots at two different time. From my experience sometimes it helps, sometimes not. Memory usage is a complex issue especially when you have a complex system. You got tons of things in the memory. Usually unused objects stay in the memory for a while should be ok as long as they could be collected in the next garbage collection. But the thing is that GC , not you, decides when and how to collect the garbage. So it is really hard to tell if these unused objects in the memory are new or survive in the last garbage collection. You have to run the tool for a longer time to compare the several snapshots.

Go back to the memory leak problem I fixed. Some context. The project size is big and I am not familiar with every module. And no way to be familiar with every module of this big project.

I am a bit lucky this time because I paid attention to the memory usage of the system for a long time. Sometime I left the system running over night and there was no obvious memory leaking. So think about the developing environment and production environment and get some idea.

• Database size. I believe production environment has a much bigger database size.
• LDAP. Production environment uses LDAP for authentication. I do not.
• I did not do a real long test.

It did not take me very long time to narrow down to one suspicious module. I spent almost one day on some suspicious UI things because ANTS memory profiler told me that there are a lot of objects staying in the memory relevant to UI refreshing. But after some painful code checking and more tests I found out that number of that part of objects went down after it reached a limit. And it does it consistently. So forgot about it and moved on. I also did spend some time to increase my database size and did more tests. It turned out database size has nothing to do with memory leaking at this time.

I focused on several public methods after verifying that this module is the evil one with several tests. It became easy now. Tested one method each time after mocking up the other suspicious methods. I were almost there after several tests. I can tell which lines are leaking memory. But I can not tell if they are all of the evil pieces.

To be honest ANTS memory profiler does not help a lot in this step and drove me to the wrong direction sometimes.

Memory leak, Dispose() and Using statement (1)

The project I worked on has a huge memory leak in production environment. It was reported recently by onsite engineer. I believe this problem exists for a long time. It hid very well previously.

Memory leak is always not easy to fix. First step you have to find out which place has memory leak. It is painful to go through this process. Next step you have to release the memory properly. It sounds simple, but not because CLR is supposed to take care of the memory stuff. It works as a memory manager for .Net application, right? Something must be wrong if CLR can not do its job well which means somewhere in your code breaks the default garbage collection rules. Of course third step you have to run pair testing for over 12 hours to see the results. It is not compulsory to be 12 hours. 2 hours even 1 hour is still good for the middle test. It has to be long enough to make you confident that your modification works or it does not work. But the final test has to be more than 12 hours because this time you want to show your managers and team members that your modification works. Generally you will see some nice results after several times tests. Another annoy thing about memory leak problem is that memory leak usually does not happen at only one place. You have to find them piece by piece and fix them piece by piece. The test result will be a bit nicer, a bit nicer, a bit more nicer... Usually not big change at one time. It also means memory leak problem is time consuming.

So to fix memory leak problem, probably you will go through hopeless, depressed, disappointed, and annoyed. And most of the important hopefully you will feel happiness in the end . I reach the last one every time luckily.

Many thanks to my lovely and funny colleagues, Colin, Takashi, and Thura. They make my job more interesting.

A Win32Exception is thrown - The file size exceeds the limit allowed and cannot be saved

Currently we got an exception A Win32Exception is thrown - The file size exceeds the limit allowed and cannot be saved. It seems it is a bug from Windows after searching online for a while.

My colleague confirmed this with his change set although I do not know it works or not. And he referred to an article about this exception. The link is here http://daniel-richardson.blogspot.com/2008/12/handling-eventlogentrywritten-event-for.html

"A Win32Exception is thrown if EnableRaisingEvents is set to true on an empty EventLog. The exception message is "The file size exceeds the limit allowed and cannot be saved".
The workaround for this is to check if the EventLog is empty and write an entry if necessary before setting EnableRaisingEvents to true."

Or you do not set EnableRaisingEvents to true for no good reason.

ActiveMQ Message Routing

There are several things to think about when you design the ActiveMQ message routing.

1) JMS Selectors. Selectors are a way of attaching a filter to a subscription to perform content based routing. Selectors are defined using SQL 92 syntax and typically apply to message headers; whether the standard properties available on a JMS message or custom headers you can add via the JMS code.

http://activemq.apache.org/selectors.html

ActiveMQ message properties and JMS message properties should be considered when using JMS selectors.

http://activemq.apache.org/activemq-message-properties.html

2) Wildcards. ActiveMQ supports destination wildcards to provide easy support for federated name hierarchies. A subscriber could use wildcards to define hierarchial pattern matches to the destinations to subscribe from. Note wildcards are not part of the JMS specification so are custom enhancements.

I think this feature is quite useful when some messages are only interesting to part of the consumers and relation structure is complex.

http://activemq.apache.org/wildcards.html

3) Advisory Message. It is kind of administrative information of ActiveMQ which helps to watch the system by subscribing to regular JMS messages. By default this is disabled.

http://activemq.apache.org/advisory-message.html?bcsi-ac-20A76BC15DBD50F6=1AADD8DD00000503CzAfPhVn8T+sJ2asth9iskWerAsTAgAAAwUAAJQ8hQIIBwAAVQAAAMKDAwA=

4) Message Redelivery. Messages are redelivered to a client when any of the following occurs:

1. A transacted session is used and rollback() is called.
2. A transacted session is closed before commit is called.
3. A session is using CLIENT_ACKNOWLEDGE and Session.recover() is called.

http://activemq.apache.org/message-redelivery-and-dlq-handling.html

(DLQ means dead letter queue)

Host multiple services with multiple configuration files

Definitely it should be good if we could do this - Host multiple services with multiple configuration files. I found a nice post talking about the way the author implements this. Basic idea is to use different AppDomain for different services and handling loading, hosting service manually. Here is the link: http://blogs.microsoft.co.il/blogs/alon/archive/2008/03/12/hosting-plug-in-wcf-services.aspx

I am not sure if this is a good idea. Creating different AppDomains in one process is a little scaring. Do not know how to catch thread level exceptions for each domain. It might be possible to do it. Have not tried. And another thing is how to handle threading context. Not sure. To be honest, I did not we could have different App.Domain in one process. Let's find out more about App.Domain.

===later added

By reading a little bit about App.Domain. I think it might be a good idea. Process is a concept of Windows OS. App.Domain is a concept of .Net Framework. It is about CLR running time resource. It might be good to separate AppDomain for different services.

Properties.Settings.Default.Save() saves to which file?

This did bother me for a while.

There are two options for Scope when we try to add a setting. User and Application. Both default settings are stored in the app.config of this project.

Where will it be saved if user changes the setting in the running time?

The save will save to app.config of the project if it is a application level setting. But it will save to C:\Documents and Settings\[user]\Local Settings\Application Data\[application_name] folder if it is a User level setting.

You can only get the default setting back by deleting the XML files in the above folders and run your application.

SharePoint 2010 Preliminary System Requirements

I probably will work on SharePoint for a while.

Microsoft announced preliminary system requirements for SharePoint server 2010. Obviously it goes for some big enterprise with enough budget.

1. SharePoint Server 2010 will be 64-bit only.
2. SharePoint Server 2010 will require 64-bit Windows Server 2008 or 64-bit Windows Server 2008 R2.
3. SharePoint Server 2010 will require 64-bit SQL Server 2008 or 64-bit SQL Server 2005.

I copied it from http://blogs.msdn.com/sharepoint/archive/2009/05/07/announcing-sharepoint-server-2010-preliminary-system-requirements.aspx

Threads allocated in ActiveMQ and options

I found a very nice post about threads allocated in ActiveMQ consumer, broker and producer. http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ

In the testing I did, Connection is really created in a new thread. Session part I am not sure. Do not know how to test it. OnMessage() always comes to same thread (different with connection thread). It is wired

Two wire protocol supported in ActiveMQ

Default wire protocol supported by ActiveMQ is Java OpenWire transport. OpenWire is used to marshal objects to byte arrays and back. http://activemq.apache.org/openwire.html

Another one is Stomp. The Stomp project is the Streaming Text Orientated Messaging Protocol site (or the Protocol Briefly Known as TTMP and Represented by the symbol :ttmp).

Stomp provides an interoperable wire format so that any of the available Stomp Clients can communicate with any Stomp Message Broker to provide easy and widespread messaging interop among languages, platforms and brokers.

http://stomp.codehaus.org/

I think wire protocol is the way ActiveMQ pack messages to byte arrays. OpenWire and Stomp are two different ways here. As ActiveMQ website says "OpenWire is designed for maximum performance and features; its the protocol used inside ActiveMQ. If you want a simpler protocol to work with to get started with a cross language client then tryStomp which is designed for ease-of-implementation so its easy to support many clients." So it is your choice.

Why UI application hangs and top possible reasons

These days I am investigating freezing UI application problem. It is very painful. No clues.
I tried reading logs of UI application, checking socket connection status, memory and CPU usage. But it does not help. Everything seems fine.
I tried to add logs when entering and leaving locks in UI application. It seems fine. Entering and leaving match.
I also tried Windows debugging tool and ANTS performance profiler. They are not really helpful since freezing does not happen all the time. And Windows debugging tool is kind of difficult to use. You need spend some time to learn how to use it. It is not straightforward telling you what is wrong with your codes.

The problem was finally solved 'by accident'. Somebody changed something and magically UI stopps hangging any more.
Take a look at change history, we found out hanging is caused by cross thread problem of UI application. UI thread creates a new UI object. The handler on the new UI object starts to doing something. The later change is to put these codes in a safe invoke scope. It looks like as followings.
UIObject b = new UIObject(); //For example a button
if ( b.InvokeRequired )
{
// fall into here if new thread is needed
}
else
{
//fall into here if no new thread is needed
}
So top issues I will focus on if unfortunately I have to investigate this problem again
1) CPU and memory usage. They are always the first suspect and easy to check.
2) Threads number of this application. Easy to check.
3) UI update should be covered in safe Invoke/BeginInvoke/EndInvoke scope. Just like the above example.
4) lock problems.

Lifetime of static variable

From MSDN http://msdn.microsoft.com/en-us/library/aa645766%28VS.71%29.aspx

A field declared with the static modifier is called a static variable. A static variable comes into existence before execution of the static constructor (Section 10.11) for its containing type, and ceases to exist when the associated application domain ceases to exist.

The initial value of a static variable is the default value (Section 5.2) of the variable's type.

For purposes of definite assignment checking, a static variable is considered initially assigned.

InstanceContextMode and ConcurrencyMode

There are three InstanceContextMode in WCF.

• PerCall: a new InstanceContext object is created and recycled succeeding each call.
• PerSession: A new InstanceContext object is created per session and instance is not sharable by multiple sessions.
• Single: A single instance is created and used for all incoming calls.
  

There are three ConcurrencyMode supported by WCF.

• Single: The service instance is single threaded and does not accept reentrance calls. If a new message comes when another message is being processed, this message has to wait there until the first one is finished.
• Multiple: The service is multi-threaded. It is fast. No waiting. But developer has to make sure the code is thread-safe.
• Reentrance: The service is single threaded and can accept reentrant calls. It implies that the service processes only one message at a given time. To ensure thread safety, WCF locks the InstanceContext processing a message so that no other messages can be processed. In case of Reentrant mode, the InstanceContext is unlocked just before the service makes an outgoing call thereby allowing the subsequent call to get the lock next time it comes in to the service.

There are many factors we should think about when choosing service behavior.

• Constructor of service. Probably InstanceConxt. Single should be used if there are lots of stuff to load and initialize. InstanceContext.PerSession should be used if there are states to maintain for one client. No thing to maintain for different calls, different clients, InstanceContext.PerCall should be used.
• ConcurrencyMode does not matter if InstanceContextMode.PerCall is adopted. New available thread will be used if new call comes.
  
• InstanceContext.PerCall and InstanceContext.PerSession mean same thing if there is no session maintained for this service
• ConcurrencyMode.Reentrant does not mean multiple thread. It is single thread in most cases except there is outgoing service call.
• Take care of thread-saftety if InstanceContextMode.Single and ConcurrencyMode.Multiple are considered to use. It will be faster and it will also be tricky.
  

We could put some logs to know when the object is initialized and which thread is used in the constructor. Just for confirmation.

OptimisticConcurrencyException-When it is triggered?

It could be triggered in two cases:
1)The entity property is defined in the conceptual layer with an attribute of ConcurrencyMode="fixed"(A property of Entity Framework, Not SQL Server).When this attribute is used, Object Services checks for changes in the database before saving changes to the database. Any conflicting changes will cause an OptimisticConcurrencyException.
By default, however, Object Services saves object changes to the database without checking for concurrency.

2)An OptimisticConcurrencyException can also occur when you define an Entity Data Model that uses stored procedures to make updates to the data source. In this case, the exception is raised when the stored procedure that is used to perform updates reports that zero rows were updated.