Author Topic: .Net Plugin Causes High Memory Consumption / Leak (not in Managed Heap)  (Read 4054 times)

0 Members and 1 Guest are viewing this topic.

kbrown@wynright.com

  • Guest
I have a very large and complex .net Plugin for AutoCAD. I'm trying to avoid explaining the workings of this plugin in detail since my question is a general one. I apologize if I have not used the terms AppDomain and Managed Heap correctly. I lack detailed knowledge of the memory allocation system in .net and how the .net plugin system in autocad is implemented, so I have done my best.

The plugin utilizes a set of complex dynamic blocks. The properties of these dynamic blocks are manipulated programatically. The plugin subscribes to Document.ImpliedSelectionChanged in order to detect when one or more dynamic blocks created by the plugin are selected. I also subscribe to each BlockReference's ObjectClosed event to detect when a block is modified. When the ImpliedSelectionChanged event is fired for relevant blocks, the plugin reads the values of the selected block's properties. The properties may also be set. Essentially I expose a custom block property editor so that complex engineering rules can be enforced on which block properties are available given other factors. Also, these rules may dictate that if a block property is set to a certain value, other property values are also automatically changed. Thus most block properties are hidden from the user and can only be edited via my property editor control.

The issue I have is that large amounts of memory are allocated by AutoCAD when a number of these blocks are selected repeatedly. This memory is never released, resulting in GBs of memory being consumed by acad.exe. This occurs even if one selects/deselects the same set of blocks over and over.

I've analysed the managed heap for the AppDomain my plugin runs in using a memory profiler (JustTrace). The memory leak is not present in this AppDomain. Acad.exe could be using 4GB of memory, and the managed heap is only using a few hundred MBs. Comparing Snapshots of the heap before and after the problematic actions are performed does not reveal the source of this memory consumption.

I have painstakingly wrapped all ACAD API class instances which implement IDisposable in Using blocks or otherwise ensured that Dispose is explicitly called on the main UI thread to eliminate access violations caused by the GC calling dispose on a different thread, so I do not believe this is the source of the problem. And the GC should call Dispose when I force collection, which I have tried.

I'm seeking general insight about finding the cause of this memory consumption, which seems to be occurring in AutoCAD's native code. Saving the drawing and closing/opening it causes the memory to be deallocated. It seems I have caused a memory leak (well many memory leaks), but I am at a loss as to how to track down the source. Any advice would be much appreciated.

Note that I have tried disabling UNDO but there was no decernible change in memory consumption.

Thanks,
Kevin
« Last Edit: May 13, 2015, 06:00:15 PM by k_brown »

BlackBox

  • King Gator
  • Posts: 3770
In order to open the ObjectId ForRead, you're presumably using a Database Transaction... You mention Using blocks, but did you ever Commit() the Transaction?

Also, I'm sure your app makes sense to you, but it sounds like your logic of reading the Object's properties at SelectionChanged is premature (unless some modeless dialog is dynamically updated?)... Selection is a primary function, so unless you're then acting on a Click event (ContextMenu for example), or Command, you're inviting an abundance of effort upon your app, IMO.

More information (and code) is needed.


Cheers
"How we think determines what we do, and what we do determines what we get."

kbrown@wynright.com

  • Guest
"In order to open the ObjectId ForRead, you're presumably using a Database Transaction... You mention Using blocks, but did you ever Commit() the Transaction?"

So, you are suggesting that it is necessary to commit a transaction even when objects are only opened ForRead? And this could account for the memory consumption? I do not currently commit when I do not write to the drawing database. When I began this project, all of the example code I encountered utilized transactions (and not even StartOpenCloseTransaction, but StartTransaction). I have since learned that it is possible to open objectIDs for read or write without the use of a transaction. Either avoiding a transaction on using StartOpenCloseTransaction are both much more performant (as an aside).

"Also, I'm sure your app makes sense to you, but it sounds like your logic of reading the Object's properties at SelectionChanged is premature (unless some modeless dialog is dynamically updated?)... Selection is a primary function, so unless you're then acting on a Click event (ContextMenu for example), or Command, you're inviting an abundance of effort upon your app, IMO. "

When I began work on this plugin, I knew very little about AutoCAD and its APIs. Two years or so later I probably still don't know much about either subject. As I explained, I had to replace the Properties window in AutoCAD (specifically its functionality for editing dynamic block properties). I found an approach that worked, which is to subscribe to ImpliedSelectionChanged. In the handler method, I first determine if the selected block is a part of my application. I should note that there is a button which can be used to disable this integration. The replacement block property editor should only be active when the user intends to edit block properties. Anyway, likely there is a better way to implement this functionality. I only described the workings of this part of the plugin to provide an idea of what might be causing the memory consumption.

Posting code.... there are tens of thousands of lines of code which directly call AutoCAD's .net API, and much more code which drives these calls or does... many other things. I cannot zero in on a specific bit of code which is causing the issue.

In any case I am grateful for your reply and any further advice you might have (especially about the question of the need to call Commit when only reading from the database).

I made the same post to the AutoDesk forums (http://forums.autodesk.com/t5/net/net-plugin-causes-high-memory-consumption-leak-not-in-managed/td-p/5633915). This was the only reply I received so far (the responder is a person I have come to respect. He is very knowledgeable.)

"I have no idea on how the native code wrapped by .NET API handle memory, but one thing surely results in more memory user: manipulating BlockReference's DynamicBlockReferenceProperties, especially when the dynamic block is quite complicated, because whenever a dynamic propery is set to a different value, AutoCAD creats a new anonymous block definition so that the instance of the blockreference with that specific dynamic property value can have a block definition to reference to. After using your app to manipulate the complicated dynamic blocks for a while, if you go to "Purge" dialog box, you would see tons of purge-able anonymous blocks, which surely take up momeries. It is known side effect that using many dynamic blocks with many dynamic properties slows AutoCAD down noticeably."

He is correct that many purgeable anonymous blocks are created. I automatically purge the BlockTable before each save because of this problem, which otherwise eventually causes drawing files to become very large. There is also an option to purge the BlockTable at regular intervals in my application. However, even running the purge command and doing a 'purge all' on the drawing barely dents the amount of memory consumed. I also tried combining purge all and having UNDO completely disabled from the time a drawing is first created. To no avail.

BlackBox

  • King Gator
  • Posts: 3770
So, you are suggesting that it is necessary to commit a transaction even when objects are only opened ForRead? And this could account for the memory consumption? I do not currently commit when I do not write to the drawing database. When I began this project, all of the example code I encountered utilized transactions (and not even StartOpenCloseTransaction, but StartTransaction). I have since learned that it is possible to open objectIDs for read or write without the use of a transaction. Either avoiding a transaction on using StartOpenCloseTransaction are both much more performant (as an aside).

<snip>

Posting code.... there are tens of thousands of lines of code which directly call AutoCAD's .net API, and much more code which drives these calls or does... many other things. I cannot zero in on a specific bit of code which is causing the issue.

In any case I am grateful for your reply and any further advice you might have (especially about the question of the need to call Commit when only reading from the database).

<snip>

Sorry - very busy day for production (client decided they want to reconfigure the entire 1,000 AC development, Grr), so I'm cherry picking some of your response for expediency -

I'm suggesting that you properly release the Objects you open, most easily done via Commit()-ing the Transaction used to open said Objects (yes, even OpenMode.ForRead). Opening ForRead is less costly in memory, performance, etc. but it still consumes as you've noticed; it's not free so-to-speak.

You can certainly do all of this without Transactions; it's just more work on your part, and disastrous when you neglect to implement the code logic needed to mitigate all negative potentialities. Transactions, are more costly in terms of performance, etc., but are simpler to code, and maintain - particularly for those new to AutoCAD .NET API, IMHO. The fact that you're adept at .NET _before_ diving into AutoCAD .NET API is a good thing, most start bassackwards (like me), being a CAD Monkey, who learns Scripts, LISP, etc. and then attempts to step up to .NET API.

In any event, a Using block on your Transaction, and calling Commit() *should* clean up your memory consumption problem... *IF* that is the only problem; without code to look at, impossible to know, test, etc.



To help qualify my suggestion that you should call Commit() when using Transactions, here's a snippet from Kean:

Quote
It should also be noted – and this can cause a big impact – that you should always call Commit() on a successful Transaction, even when it’s nominally read-only in nature. If Commit() is not called then the Transaction will be aborted, and depending on what you’re doing this can be a performance killer.


Cheers
"How we think determines what we do, and what we do determines what we get."

kbrown@wynright.com

  • Guest
Thanks. I'm working on making sure Commit is always called now. I'll post an update about how this change affects memory use.

Jeff H

  • Needs a day job
  • Posts: 6150
When using AutoCAD API 9 out 10 times calling dispose does not release any memory it calls close on the in-memory object.

Could be Selection Sets not being released, or number of other things, but with information given all I would know to do is narrow down cause by removing code that subscribes to selection set change events, and see if still happens, then remove code that subscribes close events, etc... to see what area is causing problem then remove smaller portions of area causing problem until found.

kbrown@wynright.com

  • Guest
Calling Commit for all transactions, even those that only read data, had no discernible impact on memory consumption or performance.

kbrown@wynright.com

  • Guest
Jeff H,

The selection set event drives the code in question. So yes, this would eliminate the memory leak.

The close even handler is not the cause of the memory leak. I tried disabling this part of the system as you suggested, and there was no change or only a slight change to memory consumption.

I believe the AutoDesk forum member is correct in that the memory consumption is caused by a combination of the complexity of the dynamic blocks used in my application and the manipulation of the properties of these blocks by my code. It just makes no sense to me that AutoCAD never releases the memory used for these complex operations. I can literally cause AutoCAD to use 12GB of memory by selecting/deseleting the same set of blocks over and over again. Each time the blocks are selected and my code is invoked, more memory is allocated. The majority of the newly allocated memory is never released.

I'm considering creating a test plugin which contains only a small part of my code base but which can be used to replicate this memory leak. This will take me a few hours, but this way I can provide the plugin here and also send it to ADN and ask for their take.

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
< .. >
I'm considering creating a test plugin which contains only a small part of my code base but which can be used to replicate this memory leak. This will take me a few hours, but this way I can provide the plugin here and also send it to ADN and ask for their take.

Kevin, That would be my way to treat the issue.
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

kbrown@wynright.com

  • Guest
I've prepared a runnable plugin which can be used to reproduce the memory leak I've described. The attached solution was writing in VS 2010 and tested with AutoCAD 2014 and 2016 (I only debugged in 2014). It should run in any version between 2013 and 2016. I included a working dll in bin\Debug.

I've included a small subset of the dynamic blocks utilized by my plugin in order to reproduce the problem. There is a single command defined by the plugin (RunAutomatedMemoryLeakTest). The plugin creates 10 instances of each dynamic block, then sets every available property value 3 times, where the property type code is 5. The amount of memory is directly proportional to the number of blocks/times each property is set. One can easily adjust the amount of memory consumed by altering the for loops.

Neither inserting the blocks nor reading the block properties seems to cause memory consumption. The memory consumption is caused by calls to DynamicBlockHelper.SetParameter.

Here is the code for the command (sorry, my employer requires vb.net):

Code: [Select]

    <CommandMethod("RunAutomatedMemoryLeakTest")> _
    Public Shared Sub AutoMemoryLeakTestCommand()
        If Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument IsNot Nothing Then
            Dim blockObjectIDs As New List(Of ObjectId)
            Dim directoryPath As String = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location)
            For x As Integer = 1 To 10
                For Each filePath In Directory.GetFiles(directoryPath)
                    If Path.GetExtension(filePath).Equals(".dwg", StringComparison.OrdinalIgnoreCase) Then
                        DynamicBlockHelper.AddDynBlockToDrawingFromFile(filePath, Path.GetFileNameWithoutExtension(filePath))
                        blockObjectIDs.Add(DynamicBlockHelper.InsertBlock(HostApplicationServices.WorkingDatabase, BlockTableRecord.ModelSpace, New Point3d(0, 0, 0), Path.GetFileNameWithoutExtension(filePath), 1, 1, 1, "NONE"))
                    End If
                Next
            Next
            For x As Integer = 1 To 3
                For Each objID In blockObjectIDs
                    Dim blockProperties = DynamicBlockHelper.GetBlockProperties(objID)
                    For Each blockProp In blockProperties
                        Dim propertyAttribs() As String = blockProp.Key.Split("~")
                        Dim propertyName As String = propertyAttribs(0)
                        Dim propertyType As String = propertyAttribs(2)
                        If propertyType = 5 AndAlso blockProp.Value.Count > 1 Then
                            For Each propValue As String In blockProp.Value
                                DynamicBlockHelper.SetParameter(objID, propertyName, propValue)
                            Next
                        End If
                    Next
                Next
            Next
        End If
    End Sub



And here is the code for the SetParameter method:

Code: [Select]

    Public Shared Function SetParameter(ByVal BlockID As ObjectId, ByVal ParameterName As String, ByVal Value As String) As Boolean
        Dim doc As Document = ApplicationServices.Application.DocumentManager.GetDocument(BlockID.Database)
        Using lock As DocumentLock = doc.LockDocument()
            Using myTrans As Transaction = BlockID.Database.TransactionManager.StartOpenCloseTransaction()
                Try
                    Using myBRef As BlockReference = myTrans.GetObject(BlockID, OpenMode.ForWrite)
                        For Each myDynamProp As DynamicBlockReferenceProperty In _
                           myBRef.DynamicBlockReferencePropertyCollection
                            If myDynamProp.PropertyName.Equals( _
                               ParameterName, StringComparison.OrdinalIgnoreCase) = True Then
                                myDynamProp.Value = Value
                                myTrans.Commit()
                                Return True
                            End If
                        Next
                    End Using
                Catch ex As System.Exception
                    myTrans.Abort()
                    Try
                        EventLog.WriteEntry("MemLeakTest", ex.GetType().Name + ": " + ex.Message + Environment.NewLine + ex.StackTrace, EventLogEntryType.Error)
                    Catch eex As System.Exception

                    End Try
                End Try
                Return False
            End Using
        End Using
    End Function


Thanks,
Kevin

kbrown@wynright.com

  • Guest
I received a response from ADN saying they could reproduce the problem (the message follows). This is my first "Change Request". Does anyone have advice on how to get the fix expedited?

Quote
Yes, I am able to reproduce the issue. I feel issue is related to not releasing the memory of objects created during the dynamic block update operations as same memory is getting released when drawing is closed.
 
We have logged Change Request number 70788 with our development team as this issue requires a modification to our software. Please make a note of this number for future reference. You are welcome to request an update on the status of this issue, or to provide us with additional information, at any time by submitting a new Case through DevHelp Online quoting the Change Request number or this Case number.
 
This issue is important to me. What can I do to help?
This issue needs to be assessed by our engineering team, and prioritized against all of the other change requests that are outstanding. As a result any information that you can provide to influence this assessment will help. Please provide the following where possible:
•   Impact on your application and/or your development.
•   The number of users affected.
•   The potential revenue impact to you.
•   The potential revenue impact to Autodesk.
•   Realistic timescale over which a fix would help you.
•   In the case of a request for a new feature or a feature enhancement, please also provide detailed Use Cases for the workflows that this change would address.
This information is extremely important. Our engineering team have limited resources, and so must focus their efforts on the highest impact Change Requests. We do understand that this will cause you delays and affect your development planning, and we appreciate your cooperation and patience.
 
Best Regards,
Virupaksha Aithal
 
Developer Technical Services
http://adn.autodesk.com
« Last Edit: May 20, 2015, 12:21:49 PM by k_brown »

Jeff_M

  • King Gator
  • Posts: 4094
  • C3D user & customizer
I've had 2 change requests, that I'm aware of, implemented in the very next Service Pack. Both of them were things that crashed Civil3D without warning, so were pretty high on the priority list. I've had a total of ~11 or 12 change requests that either have not yet made it into the product, or did and I missed the change....this is over a period of the past 8 years or so.

kbrown@wynright.com

  • Guest
This issue affects AutoCAD 2013-2016 and probably older versions as well. Will they patch all of the versions affected? Should I try to document that all of these versions are affected and provide them with the information?

You (Jeff_M) mention having changes implemented. Did they provide hot fixes? Or did you have to wait for the next service pack?

They ask for further information, but I don't want to waste my time if it isn't likely to produce a more favorable outcome. I suppose extra information cannot hurt, but I have real work to do...

I've also seen the plugin I created to reproduce the problem crash AutoCAD multiple times. Usually CAD will crash just after the command is executed. My real plugin crashes occasionally also. I've been unsuccessful in finding a cause. I sent multiple minidumps to ADN and they couldn't provide me with any useful guidance. I now wonder if the crashing and the memory leak are related.

Jeff_M

  • King Gator
  • Posts: 4094
  • C3D user & customizer
From my experience it's possible they will patch 2015, but not anything older than that. My Change Requests appeared in the next Service Pack, I don't recall if there were any Hotfixes released prior to that.