Correct, Pali. That (blocked finalizer) is in fact exactly what MS Support has diagnosed (and what I'd suspected, based on that same article you linked to and some others).

I received info from their "esacalation team" saying that we should "Please investigate the logic inside SerializableObject.RemoveHandler()". I don't believe we have that code, however, plus obviously the issue could be related to one of our objects that's based on that.

The support person said that this is the call stack from the blocked finalization thread, and it seems to be because some finalizer is allocating something (looks like an array), which is causing a GC, for which the finalizer has to wait:

"The finalizer thread triggered the GC and it is blocked waiting for GC to complete.

…

..

00000000`04e9e850 000007fe`f9584d85 clr!SVR::gc_heap::allocate_large_object+0x65

00000000`04e9e8b0 000007fe`f93e407c clr!AllocateArrayEx+0xa1a

00000000`04e9ea20 000007fe`f725e5df clr!JIT_NewArr1+0x45c

00000000`04e9ebf0 000007fe`f72428ee mscorlib_ni!System.MulticastDelegate.DeleteFromInvocationList(System.Object[], Int32, Int32, Int32)+0x5f

00000000`04e9ec60 000007ff`00c4945b mscorlib_ni!System.MulticastDelegate.RemoveImpl(System.Delegate)+0x10e

00000000`04e9ed00 000007ff`010cc4a6 Gizmox_WebGUI_Common!Gizmox.WebGUI.Forms.SerializableObject.RemoveHandler(Int32, System.Delegate)+0x9b

00000000`04e9ed80 000007fe`f93e9666 Gizmox_WebGUI_Common!Gizmox.WebGUI.Forms.ComponentBase.Finalize()+0x26

Obviously, we should be able to avoid that happening by calling .Dispose() to release resources and suppress finalization -- if we knew what objects this is not being done for. I've added event-hanlder unwiring code and calls to .Dispose throughout our app, but so far have not made the type of event pictured in my previous post go away. On average, they  last ~10-15 minutes, but the signature is similar in them all.

-Kelly

Thanks, Pali. On our side, we happened to have deployed our latest release Tuesday evening. Monitoring the web server on Wednesday showed a definite difference in the perfmon counters pattern than prior to this release. Additionally, the number of ListViewItem and ...SubItem objects in memory at the time that we collected dumps of w3wp process memory in the morning and afternoon were "reasonable" compared to the number typically in memory while using our previous release. 

Part of my coding for this release focused on unwiring menuclick event handlers for listview items as well as calling ListViewItem.Dispose()  prior to calling ListView.Items.Clear(). I also added a call to base.Dispose() in the overridden Dispose method of at least one of our VWG subclasses.

The caveat here is that we've had single days before where we did not see the issue on our server. They have been fairly rare, however, so my hope is that the code changes have helped to alleviate this issue that we've been having with ListViewItems not collecting. We'll continue to monitor for the next week or so and see whether yesterday was a fluke or the new baseline.

Also, wrt the reproducing sample app, I definitely have noticed that the issue we've had, even in our production app, is MUCH more difficult to reproduce when running locally than when on the server. I have not yet been able to do this, but my theory is that is either you or I were to deploy the sample app,  run it outside of the VS environment, and use memory dumps for analysis vs the debugger, we'd more easily repro the scenario.

Hi Kelly,

I'm glad that things are looking better for your at this point.

A ratio of 1 : 1 : 1 is probably a very typical symtom where you have a stuck Finalizer thread.

The results you got from the MS escalation team do indicate that a RemoveHandler called from within a Finalize() method of a ComponentBase or derivatives (like ListViewItem etc.) does under some conditions need to allocate a chunk of memory (could be array resize or something like that). My short code review I did earlier did reveal that explicitly calling Dispose() on ComponentBase derivative objects will avoid this particular problem, so while we are researching, it's a good approach you have taken to do exactly that.

As a good rule of thumb, while we are researching, you should at least call expllicit dispose() on object instances you use in huge numbers, like the ListViewItem in your case.

Based on all this, then I can't help but think that if this is indeed the case, then a very specific memory conditions must be met for this to occur, which would indeed explain the hard time we have reproducing it.

Let's see what the developers have to say after they have read my report.

Palli