Code Red > .NET

Serializing xml files to memory.

(1/4) > >>

Keith Brown:
Hi,


I have some functions where I am serializing some large xml files to store in the named object dictionary.  Everything works great except for speed.  The deserializing speed is not bad and is something that I can live with but the actual serializing of the memory stream takes an incredible amount of time.  The slow down appears to be when breaking the memory stream into small chunks to store in a result buffer.  When the file is fairly small, it is exremely fast, but as the file size increases the speed slows down in a non linear fashion.  The time it takes to do the first 200 result buffers is slower by factors larger than a magnitude of 10 to do the next 200 and so on.  I believe that this is due to the how the xrecord is storing the result buffer.  I have attempted to break my memory stream down into smaller chunks before passing to the code that places them into the result buffer but I have not had alot of success with rebuilding the stream.

Has anyone had success serializing xml files to drawing memory?  Do you use large files?

Here are the two methods that I use to serialize to resultbuffers.  I did not write the original code and I am not sure where I got it from.  It might have been a class at AU from Jerry Winters but not 100% sure.  The code to convert to and from a memory stream is boilerplate and is not shown here.  However, i think the key to success is to break the memorystream into small chunks before calling the streamtoresultbuffer.


--- Code - C#: ---private ResultBuffer StreamToResultBuffer(MemoryStream memoryStream, string applicationName){   ResultBuffer resultBuffer = new ResultBuffer(new TypedValue(Convert.ToInt32(DxfCode.ExtendedDataRegAppName), applicationName));   var i = 0;   memoryStream.Seek(0, SeekOrigin.Begin);   while (i < memoryStream.Length) {      var length = Convert.ToInt32(Math.Min(memoryStream.Length - i, MaxChunkSize));      var datachunk = new byte[length];      memoryStream.Read(datachunk, 0, length);      resultBuffer.Add(new TypedValue(Convert.ToInt32(DxfCode.ExtendedDataBinaryChunk), datachunk));      i += MaxChunkSize;   }   return resultBuffer;} private MemoryStream ResultBufferToStream(ResultBuffer resultBuffer){   MemoryStream memoryStream = new MemoryStream();   TypedValue[] values = resultBuffer.AsArray();     // Start from 1 to skip application name   for (int i = 1; i <= values.Length - 1; i++) {      byte[] datachunk = (byte[])values(i).Value;      memoryStream.Write(datachunk, 0, datachunk.Length);   }   memoryStream.Position = 0;   return memoryStream;}

dgorsman:
I don't push raw data to XRecords (although, is that XData you are using in the code??); I break it up appropriately to what it is/does i.e. a length gets a DXF code for a double, and its stored as a double.  Then there's no need to convert it from XML fragments every time its needed.  On the occasion that I do need access to truly large-scale XML content I keep the XML object at the document or application scope, and only store the filename and path of the XML source file.

Keith Brown:

I am using the NOD and storing in dictionaries and XRecords.  The resultbuffer gets pushed to the XRecord.

I am not really storing specific entity information.  It is basically data and can be approximately 75,000 kb in size or even larger depending on the number of objects in the drawing.


I have tried to stay away from using an external xml file due to the headaches of keeping track of its location, duplicating it, renaming it upon saveas command, etc.


However, some tests have shown that I can write the file in about 5 seconds whereas writing it to drawing memory takes upwards of 20 minutes which is a less than ideal time frame.


The specs for the program required that the data be stored in memory, but I think I might push back against that if I cannot get the time lowered.

dgorsman:
If it doesn't come from a file in the first place, where does the XML content come from?  Database query?  Constructed from scratch?

Keith Brown:
It is processed from the drawing itself.  I am reading CADWorx piping entities and determining how the entities are connected to each other.  This information is then used to determine stress on the systems and then highlight the high stress areas.


I did not anticipate the customers using the software with hundreds of piping systems in a single drawing. (thousands of entities, 30k+)  So I have added a flag to the settings section that allows them to use either external or internal memory with a recommendation of using external memory for larger systems.  I would like to optimize my routine however as I find it useful and do use it to save information to an entity.  I find it quicker and easier to serialize a class and then store that information as xml in the objects extension dictionary rather than create result buffers with dxf codes.  This is time consuming to me and not worth the payoff when I can achieve the same result with a much faster design time using xml.

Navigation

[0] Message Index

[#] Next page