Author Topic: Reading and writing Dwg properties  (Read 10478 times)

0 Members and 1 Guest are viewing this topic.

Jeff H

  • Needs a day job
  • Posts: 6144
Reading and writing Dwg properties
« on: January 07, 2013, 05:35:30 PM »
After seeing this article http://www.qsinformatica.it/index.php?option=com_content&view=article&id=157%3Aread-dwg-properties-part-1&catid=45%3Ablog&Itemid=72&lang=it
 
I am able to read and write them back but not writing them back correctly.
Any ideas on writing them back or how to set the bits?
 
Here is what I threw together quickly for testing to read them.
Does not take into account if it is a 2004 file and if so should not mutiply bits by 2, or if earlier than 2004 then not valid.
 
Code - C#: [Select]
  1.  
  2.  using System;
  3. using System.Collections.Generic;
  4. using System.Linq;
  5. using System.Text;
  6. using System.IO;
  7. namespace CsDwgProps
  8. {
  9.     class Program
  10.     {
  11.         static string[] standardDwgProps = { "Title", "Subject", "Author", "Keywords", "Comments", "LastSavedBy", "Revisions", "Hyperlink" };
  12.        
  13.         static void Main(string[] args)
  14.         {
  15.             using (BinaryReader br = new BinaryReader(File.Open(@"C:\Test\Test1.dwg", FileMode.Open, FileAccess.Read, FileShare.ReadWrite)))
  16.             {
  17.                 Console.WriteLine("**********Year********");
  18.                 Console.WriteLine(new string(br.ReadChars(6)));    
  19.            
  20.                 br.BaseStream.Seek(32, SeekOrigin.Begin);
  21.                 int propAdress = br.ReadInt32();
  22.                 br.BaseStream.Seek(propAdress, SeekOrigin.Begin);
  23.                 Console.WriteLine("**********Standard Properties********");
  24.                 foreach (string s in standardDwgProps)
  25.                 {
  26.                         DwgProp dp = new DwgProp(s, readProperty(br));                      
  27.                         Console.WriteLine("{0} = {1}", dp.Name, dp.Value);
  28.                 }
  29.  
  30.                 br.BaseStream.Seek(24, SeekOrigin.Current);
  31.                 short numOfcustProps = br.ReadInt16();
  32.                 Console.WriteLine("**********Custom Properties********");
  33.                 for (int i = 0; i < numOfcustProps; i++)
  34.                 {
  35.                     DwgProp dp = new DwgProp(readProperty(br), readProperty(br));
  36.                     Console.WriteLine("{0} = {1}", dp.Name, dp.Value);
  37.                 }
  38.             }
  39.             Console.ReadKey();
  40.         }
  41.        static string readProperty(BinaryReader br)
  42.         {
  43.            //For 2007 & up is unicode so Multiply by 2
  44.             int byteSize = br.ReadInt16() * 2;
  45.             if (byteSize > 0)
  46.             {
  47.                 int byteIndex = 0;
  48.                 byte[] bts = new byte[byteSize];
  49.                 while (byteIndex < byteSize)
  50.                 {
  51.                     bts[byteIndex] = br.ReadByte();
  52.                     byteIndex++;
  53.                 }
  54.              
  55.                 return Encoding.Unicode.GetString(bts);
  56.            
  57.             }
  58.             return string.Empty;
  59.         }
  60.     }
  61.  
  62.     struct DwgProp
  63.     {
  64.         string _name;
  65.         string _value;
  66.         public string Name { get { return _name; } }
  67.         public string Value { get { return _value; } }
  68.         public DwgProp(string name, string val)
  69.         {
  70.             _name = name;
  71.             _value = val;
  72.         }
  73.     }
  74.  
  75.  
  76. }
  77.  

pkohut

  • Bull Frog
  • Posts: 483
Re: Reading and writing Dwg properties
« Reply #1 on: January 07, 2013, 08:36:06 PM »
Almost all IO on drawings files goes through CRC. So changing any value in the summary will also change the calced/expected CRC.

See page 21, 33, and 84.
http://opendesign.com/files/guestdownloads/OpenDesign_Specification_for_.dwg_files.pdf


http://code.google.com/p/drawgin/
(volatile code base, can read R13-R15, does not decode objects yet)
New tread (not retired) - public repo at https://github.com/pkohut

Jeff H

  • Needs a day job
  • Posts: 6144
Re: Reading and writing Dwg properties
« Reply #2 on: January 07, 2013, 10:23:04 PM »
Thanks Paul!
 
Will take a look at that and was not aware of that document.
 
Thanks again

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #3 on: May 19, 2013, 03:21:47 PM »
Almost all IO on drawings files goes through CRC. So changing any value in the summary will also change the calced/expected CRC.
...

That's not a problem. The problem is that if we modify this section we will have to modify the section map and the page map (offsets).

The Open Design Specification for .dwg files is very confusing (maybe it's because I do not speak English) and contains several errors. So far i have found these errors:

This is confusing:

4.5 2004 Data section map

Quote
Maximum section page size appears to be 0x7400 bytes in the normal case. If a logical section of the file (the database objects, for example) exceeds this size, then it is broken up into pages of size 0x7400. In this case, the PageCount value above will contain the number of 0x7400 byte pages, and the data from the pages can be appended together in order and treated as a single logical section.

And if the size does not exceed the maximum size, then what contains PageCount ?
Do not worry, in any case create or buffer of size PageCount * pagesize (0x7400) and append the data using something like this:

Code - C#: [Select]
  1. byte[] buffer = new byte[PageCount * pagesize];
  2.  
  3. fixed(byte* pB =  buffer){
  4.   for (i = 0; i < PageCount; i++) {
  5.     decompress(dat, &pB[i * pagesize]);
  6.   }
  7. }
  8.  


Errors:

19.4.1 Common Entity Data (Page 104)
Wrong:
Quote
R2000+ Only:
 Obj size RL  size of object in bits, not including end handles

Correct:
R2000 - R2007 only


19.4.1 Common Entity Data  (Page 105)
Wrong:
Quote
Common:
 Nolinks B  1 if major links are assumed +1, -1, else 0
    For R2004+ this always has value 1
    (links are not used)

Correct:
For R2004+ It can be 0.(This affects as one reads the color)


19.4.1 Common Entity Data  (Page 105)
Wrong:
Quote
Color CMC(B) 62

Correct:
ENC(B)
If Nolinks == 0 then
 read EMC
else
 read B


19.4.1 Common Entity Data  (Page 105)
Wrong:
Quote
Common:
 Invisibility BS 60
Between this and the previous field missing specific data for R2010 or higher

Correct:
See: 19.2 Common entity format


Take a look at this,  :
http://savannah.gnu.org/projects/libredwg/

The project seems abandoned, and the code (for reading R2004 files) have some bugs: reading Data section map is wrong (Correct: see "This is confusing" inside this post) , reading xdata is wrong and the code to decompress does not work with large files (tested with a file of 30 MB), but it is a (very) good starting point.




« Last Edit: May 19, 2013, 03:42:50 PM by whoknows »

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #4 on: August 31, 2013, 03:51:27 AM »
Seems that the author has lost interest in this subject, at any case and about "IO" CRC's

Open Design Specification for .dwg files Version 5.3
2.14.1 8-bit CRC


Code - C#: [Select]
  1. internal static unsafe ushort Crc8(ushort seed, byte* adr, int lenght) {
  2.     byte al;
  3.     for (; lenght > 0; lenght--) {
  4.         al = (byte)((*adr) ^ ((byte)(seed & 0xFF)));
  5.         seed = (ushort)((seed >> 8) & 0xFF);
  6.         seed = (ushort)(seed ^ _crc8table [al & 0xFF]);
  7.         adr++;
  8.     }
  9.     return (seed);
  10. }
  11.  
  12. private static readonly ushort[] _crc8table = {
  13.     0x0000, 0xC0C1, 0xC181, 0x0140, 0xC301, 0x03C0, 0x0280, 0xC241, 0xC601,
  14.     0x06C0, 0x0780, 0xC741, 0x0500, 0xC5C1, 0xC481, 0x0440, 0xCC01, 0x0CC0,
  15.     0x0D80, 0xCD41, 0x0F00, 0xCFC1, 0xCE81, 0x0E40, 0x0A00, 0xCAC1, 0xCB81,
  16.     0x0B40, 0xC901, 0x09C0, 0x0880, 0xC841, 0xD801, 0x18C0, 0x1980, 0xD941,
  17.     0x1B00, 0xDBC1, 0xDA81, 0x1A40, 0x1E00, 0xDEC1, 0xDF81, 0x1F40, 0xDD01,
  18.     0x1DC0, 0x1C80, 0xDC41, 0x1400, 0xD4C1, 0xD581, 0x1540, 0xD701, 0x17C0,
  19.     0x1680, 0xD641, 0xD201, 0x12C0, 0x1380, 0xD341, 0x1100, 0xD1C1, 0xD081,
  20.     0x1040, 0xF001, 0x30C0, 0x3180, 0xF141, 0x3300, 0xF3C1, 0xF281, 0x3240,
  21.     0x3600, 0xF6C1, 0xF781, 0x3740, 0xF501, 0x35C0, 0x3480, 0xF441, 0x3C00,
  22.     0xFCC1, 0xFD81, 0x3D40, 0xFF01, 0x3FC0, 0x3E80, 0xFE41, 0xFA01, 0x3AC0,
  23.     0x3B80, 0xFB41, 0x3900, 0xF9C1, 0xF881, 0x3840, 0x2800, 0xE8C1, 0xE981,
  24.     0x2940, 0xEB01, 0x2BC0, 0x2A80, 0xEA41, 0xEE01, 0x2EC0, 0x2F80, 0xEF41,
  25.     0x2D00, 0xEDC1, 0xEC81, 0x2C40, 0xE401, 0x24C0, 0x2580, 0xE541, 0x2700,
  26.     0xE7C1, 0xE681, 0x2640, 0x2200, 0xE2C1, 0xE381, 0x2340, 0xE101, 0x21C0,
  27.     0x2080, 0xE041, 0xA001, 0x60C0, 0x6180, 0xA141, 0x6300, 0xA3C1, 0xA281,
  28.     0x6240, 0x6600, 0xA6C1, 0xA781, 0x6740, 0xA501, 0x65C0, 0x6480, 0xA441,
  29.     0x6C00, 0xACC1, 0xAD81, 0x6D40, 0xAF01, 0x6FC0, 0x6E80, 0xAE41, 0xAA01,
  30.     0x6AC0, 0x6B80, 0xAB41, 0x6900, 0xA9C1, 0xA881, 0x6840, 0x7800, 0xB8C1,
  31.     0xB981, 0x7940, 0xBB01, 0x7BC0, 0x7A80, 0xBA41, 0xBE01, 0x7EC0, 0x7F80,
  32.     0xBF41, 0x7D00, 0xBDC1, 0xBC81, 0x7C40, 0xB401, 0x74C0, 0x7580, 0xB541,
  33.     0x7700, 0xB7C1, 0xB681, 0x7640, 0x7200, 0xB2C1, 0xB381, 0x7340, 0xB101,
  34.     0x71C0, 0x7080, 0xB041, 0x5000, 0x90C1, 0x9181, 0x5140, 0x9301, 0x53C0,
  35.     0x5280, 0x9241, 0x9601, 0x56C0, 0x5780, 0x9741, 0x5500, 0x95C1, 0x9481,
  36.     0x5440, 0x9C01, 0x5CC0, 0x5D80, 0x9D41, 0x5F00, 0x9FC1, 0x9E81, 0x5E40,
  37.     0x5A00, 0x9AC1, 0x9B81, 0x5B40, 0x9901, 0x59C0, 0x5880, 0x9841, 0x8801,
  38.     0x48C0, 0x4980, 0x8941, 0x4B00, 0x8BC1, 0x8A81, 0x4A40, 0x4E00, 0x8EC1,
  39.     0x8F81, 0x4F40, 0x8D01, 0x4DC0, 0x4C80, 0x8C41, 0x4400, 0x84C1, 0x8581,
  40.     0x4540, 0x8701, 0x47C0, 0x4680, 0x8641, 0x8201, 0x42C0, 0x4380, 0x8341,
  41.     0x4100, 0x81C1, 0x8081, 0x4040
  42. };
  43.  

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #5 on: August 31, 2013, 03:55:38 AM »
Open Design Specification for .dwg files Version 5.3
2.14.2 32-bit CRC


Code - C#: [Select]
  1. internal static unsafe uint CRC32(byte* p, uint n, uint seed) {
  2.     uint invertedCrc = ~seed;
  3.     while (n-- > 0) {
  4.         byte bytee = (byte)*p++;
  5.         invertedCrc = (invertedCrc >> 8) ^ crc32Table[(invertedCrc ^ bytee) & 0xff];
  6.     }
  7.     return ~invertedCrc;
  8. }
  9.  
  10. private static readonly uint[] crc32Table =
  11. {
  12.     0x00000000, 0x77073096, 0xee0e612c, 0x990951ba,
  13.     0x076dc419, 0x706af48f, 0xe963a535, 0x9e6495a3,
  14.     0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988,
  15.     0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91,
  16.     0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de,
  17.     0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7,
  18.     0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec,
  19.     0x14015c4f, 0x63066cd9, 0xfa0f3d63, 0x8d080df5,
  20.     0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172,
  21.     0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b,
  22.     0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940,
  23.     0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59,
  24.     0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116,
  25.     0x21b4f4b5, 0x56b3c423, 0xcfba9599, 0xb8bda50f,
  26.     0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924,
  27.     0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d,
  28.     0x76dc4190, 0x01db7106, 0x98d220bc, 0xefd5102a,
  29.     0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433,
  30.     0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818,
  31.     0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01,
  32.     0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e,
  33.     0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457,
  34.     0x65b0d9c6, 0x12b7e950, 0x8bbeb8ea, 0xfcb9887c,
  35.     0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65,
  36.     0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2,
  37.     0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb,
  38.     0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0,
  39.     0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9,
  40.     0x5005713c, 0x270241aa, 0xbe0b1010, 0xc90c2086,
  41.     0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f,
  42.     0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4,
  43.     0x59b33d17, 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad,
  44.     0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a,
  45.     0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683,
  46.     0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8,
  47.     0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1,
  48.     0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe,
  49.     0xf762575d, 0x806567cb, 0x196c3671, 0x6e6b06e7,
  50.     0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc,
  51.     0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5,
  52.     0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252,
  53.     0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b,
  54.     0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60,
  55.     0xdf60efc3, 0xa867df55, 0x316e8eef, 0x4669be79,
  56.     0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236,
  57.     0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f,
  58.     0xc5ba3bbe, 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04,
  59.     0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d,
  60.     0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a,
  61.     0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713,
  62.     0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38,
  63.     0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21,
  64.     0x86d3d2d4, 0xf1d4e242, 0x68ddb3f8, 0x1fda836e,
  65.     0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777,
  66.     0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c,
  67.     0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45,
  68.     0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2,
  69.     0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db,
  70.     0xaed16a4a, 0xd9d65adc, 0x40df0b66, 0x37d83bf0,  
  71.     0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9,  
  72.     0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6,  
  73.     0xbad03605, 0xcdd70693, 0x54de5729, 0x23d967bf,
  74.     0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94,  
  75.     0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d
  76. };
  77.  

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #6 on: August 31, 2013, 04:01:23 AM »
Open Design Specification for .dwg files Version 5.3
5.12 64-bit CRC calculation
« Last Edit: August 31, 2013, 04:19:51 AM by jar »

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #7 on: August 31, 2013, 04:04:13 AM »
Open Design Specification for .dwg filesVersion 5.3
4.2 Section sectionPage Checksum


Code - C#: [Select]
  1. internal static unsafe uint SectionPageChecksum(Bit_Chain* chain, uint seed , uint size) {
  2.     uint sum1 = seed & 0xffff;
  3.     uint sum2 = seed >> 0x10;
  4.     while (size != 0) {
  5.         uint chunkSize = Math.Min(0x15b0, size); size -= chunkSize;
  6.         for (int i = 0; i < chunkSize; i++) {
  7.             sum1 += bits.bit_read_RC(chain);
  8.             sum2 += sum1;
  9.         }
  10.         sum1 %= 0xFFF1;
  11.         sum2 %= 0xFFF1;
  12.     }
  13.     return (sum2 << 0x10) | (sum1 & 0xffff);
  14. }

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #8 on: August 31, 2013, 05:38:44 AM »
I know how to read the most important sections (page map, section map, object map...) of .dwg files, from r2000 to r2013 and also i know how to access to the "string stream" (2007+). If anyone is interested I can post the (C#) code

But... Still don't know how to fully decode drawing objects/entities in 2004+ files,  seems that I have a problem with the "Common Entity Handle Data", but with the code we can:
- Read the file dependencies
- Read the drawing variables
- Verify the password if the file is encrypted (using CryptoAPI: CryptAcquireContext, CryptCreateHash, CryptHashData, CryptDeriveKey and finally CryptDecrypt),
- Check if the file is corrupted
- Search strings

Some parts (bit reader and r2007 decompression) are based on libredwg

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Re: Reading and writing Dwg properties
« Reply #9 on: August 31, 2013, 05:59:42 AM »
I know how to read the most important sections (page map, section map, object map...) of .dwg files, from r2000 to r2013 and also i know how to access to the "string stream" (2007+). If anyone is interested I can post the (C#) code

[< .. >

please post it jar, you're on a roll here :)
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #10 on: August 31, 2013, 06:18:09 AM »
please post it jar, you're on a roll here :)

I am traveling and the code needs to be refactored (now it's an complete chaos) but will try to post it in less than two weeks.

Maybe for you is a stupid question, but i do not speak English and i do not understand a part of your answer (I'm using google translator):
That means "you're on a roll here"?

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Re: Reading and writing Dwg properties
« Reply #11 on: August 31, 2013, 06:53:31 AM »
It means
You are winning
You are progressing well
experiencing a period of success or good luck
in the midst of a series of successes
to be having a successful or lucky period
(in a gambling game) having a continuing winning streak.


I could have said
You have been posting a lot, please continue.

 :-)
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #12 on: September 01, 2013, 08:02:08 AM »
After a quick refactoring I've posted the code here:
http://www.theswamp.org/index.php?topic=45234.new#new

and thanks for your clarification...

Jeff H

  • Needs a day job
  • Posts: 6144
Re: Reading and writing Dwg properties
« Reply #13 on: September 02, 2013, 08:05:32 PM »
Seems that the author has lost interest in this subject, at any case and about "IO" CRC's
I have not lost interest, partly has to do with time and ADDHD.
I have refactored code for reading the properties and that is all is about it and will post it.

exmachina

  • Guest
Re: Reading and writing Dwg properties
« Reply #14 on: September 02, 2013, 08:54:53 PM »
After a first approach I believe the only problem is figuring out how to compress (r2010 and r2013 compression are the same as in R2004) and the Reed-Solomon encoding in the r2007:
After modifying the summaryinfo section we need  modify the offsets in the page mapa and in the section maps and finally modify the file header data . The new offsets can be calculated easily but the maps should be compressed and still not know how to do.

Next weekend shall begin to investigate how to compress.