Topic: Slow iteration on ObjectIds

[Andrey Bushman]

Hello,

• Windows XP SP3 x86 Rus
• .Net Framework 3.5 SP1
• AutoCAD 2009 SP3 x86 Enu

I have are problem - slow ObjectIds iteration in Database. The size of a checked dwg-file is 23 Mb.

Slow code:

Code: [Select]

[CommandMethod("SlowIteration", CommandFlags.Modal)]
public void SlowIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    long amount = 0;//common counter
    long exceptCount = 0;//exception counter

    using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
        for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i++) {
            ++amount;
            ObjectId id = ObjectId.Null;

            //Slow code begin:
            try {
                id = TargetDb.GetObjectId(false, new Handle(i), 0);
            }
            catch {
                ++exceptCount;
                continue;
            }
            //Slow code end.

        }
    }
    TimeSpan len = DateTime.Now - start;
    ed.WriteMessage(string.Format("Amount: {0}\nSlow iteration time: {1} min. {2} sec.\n",
        amount, len.Minutes, len.Seconds));
}
SlowIteration command result:

Command: SlowIteration
Amount: 517579
Slow iteration time: 4 min. 36 sec.

I have marked problem code location comments.
If to comment out the slow code the result will be another. Below sample.

Fast code:

Code: [Select]

[CommandMethod("FastIteration", CommandFlags.Modal)]
public void FastIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    long amount = 0;//common counter
    //long exceptCount = 0;//exception counter

    using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
        for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i++) {
            ++amount;
            ObjectId id = ObjectId.Null;
            //Below slow code is commented:
            //try {
            //    id = TargetDb.GetObjectId(false, new Handle(i), 0);
            //}
            //catch {
            //    ++exceptCount;
            //    continue;
            //}
        }
    }
    TimeSpan len = DateTime.Now - start;
    ed.WriteMessage(string.Format("Amount: {0}\nFast iteration time: {1} min. {2} sec.\n",
        amount, len.Minutes, len.Seconds));
}

FastIteration command result:

Command: FastIteration
Amount: 517579
Fast iteration time: 0 min. 0 sec.

I need get all ObjectIds from Database object. How can I get it without slow iteration?

I thank for attention.

[kaefer]

Hi Andrey,

what you're doing there doesn't look like it's guaranteed to work at all. So you're basically on your own and you've been lucky so far.

That being said, I could think of a few optimizations:
- Get rid of try/catch in a  tight loop, and
- Try to enumerate ObjectId instead of Handle, using the fact that ObjectId is IntPtr under the hood.

A quick mock-up in F# gets the same "Approx. number of objects" as MgdDbgSnoopDb would display. What's missing is the code for generating a new Entity to ensure (entlast) gets the highest number.

Code: [Select]

let test1() =
    let ed = Application.DocumentManager.MdiActiveDocument.Editor
    let db = HostApplicationServices.WorkingDatabase
    use tr = db.TransactionManager.StartTransaction()

    let bt = tr.GetObject(db.BlockTableId, OpenMode.ForRead) :?> BlockTable
    let entLast = Autodesk.AutoCAD.Internal.Utils.EntLast()
    if entLast.IsNull then
        ed.WriteMessage("\nNo objects in drawing. ")
    else
        let start = (int64) bt.ObjectId.OldIdPtr
        let stop = (int64) entLast.OldIdPtr
        let cnt = ref 0
        let sw = System.Diagnostics.Stopwatch.StartNew()
        for ptr in start .. 4L .. stop do
            let oid = new ObjectId(nativeint ptr)
            if oid.IsValid then
                incr cnt
        sw.Stop()       
       
        ed.WriteMessage("\n{0} objects in drawing, time elapsed: {1} ", !cnt, sw.Elapsed)
           
    tr.Commit()


[Andrey Bushman]

what you're doing there doesn't look like it's guaranteed to work at all. So you're basically on your own and you've been lucky so far.

I use this approach for a long time and it always works successfully, but slowly... ((

Which library I must add to my project for Autodesk.AutoCAD.Internal.Utils.EntLast() using?

[Andrey Bushman]

I has found: C:\Program Files\AutoCAD 2009\acmgdinternal.dll

[kaefer]

Which library I must add to my project for Autodesk.AutoCAD.Internal.Utils.EntLast() using?

Eh, acmgdinternal.dll, perhaps?  (You said you're on 17.2. Since18.0 it's integrated in AcMgd.dll.)

[Andrey Bushman]

Eh, acmgdinternal.dll, perhaps?  (You said you're on 17.2. Since18.0 it's integrated in AcMgd.dll.)

Thank you.

Code: [Select]

[CommandMethod("KaefersIteration", CommandFlags.Modal)]
public void KaefersIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    ObjectId lastId = Autodesk.AutoCAD.Internal.Utils.EntLast();

    if (lastId != ObjectId.Null) {
        long amount = 0;

        using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
            BlockTable bt = (BlockTable) t.GetObject(TargetDb.BlockTableId, OpenMode.ForRead);

            for (Int64 i = (Int64) bt.ObjectId.OldIdPtr; i <= (Int64) lastId.OldIdPtr; ++i) {
                IntPtr ptr = new IntPtr(i);
                ObjectId id = new ObjectId(ptr);
                if (id.IsValid)
                    ++amount;
            }
        }
        TimeSpan len = DateTime.Now - start;
        ed.WriteMessage(string.Format("Amount: {0}\nFast iteration time: {1} min. {2} sec.\n",
            amount, len.Minutes, len.Seconds));
    }
    else
        ed.WriteMessage("No objects in drawing.\n");
}

result:

Command: KaefersIteration
Amount: 0
Fast iteration time: 0 min. 0 sec.

nothing not found.

[kaefer]

nothing not found.

Did you add an Entity to your drawing?

[Andrey Bushman]

nothing not found.

Did you add an Entity to your drawing?

The size of a checked dwg-file is 23 Mb. It has 517 579 Entities.

[kaefer]

Did you add another Entity to your drawing?

[Andrey Bushman]

Did you add another Entity to your drawing?

No, I need all database entities, not new only 

But if I add new Entities - it not work too.

How in your variant the code learns, with what database it is necessary to work?

Oops...

bt.ObjectId.OldIdPtr: 2 128 931 848
lastId.OldIdPtr:  2 124 488 384

second more then first.

[Jeff H]

Are you saying that
 
(LastId - FirstId) / 8 = Total number of objects
 
Since Int64 is 8 bytes and taking the difference between first and last would be the total number bytes and dividing by eight gives total number of objects.
 
 

Code: [Select]

        [CommandMethod("FasterIteration")]
        public void FasterIteration()
        {
            Document doc = Application.DocumentManager.MdiActiveDocument;
            Database db = doc.Database;
            Editor ed = doc.Editor; ;
            long amount = 0;//common counter

            Int64 firstId = db.BlockTableId.OldIdPtr.ToInt64();
            Int64 lastId = Autodesk.AutoCAD.Internal.Utils.EntLast().OldIdPtr.ToInt64();

            ed.WriteMessage("\n{0} {1}\n", firstId.ToString(), lastId.ToString());

            amount = (long)((lastId - firstId) /;
            ed.WriteMessage("\n{0}\n", amount.ToString());
        }

but modifying Andreys code to output the difference between ObjectIds 98% are 8 but some will be 64, 16, 2400 etc.........
 
I guess the processor breaks it up into available areas?!???
 

Code: [Select]


        [CommandMethod("SlowIteration", CommandFlags.Modal)]
        public void SlowIteration()
        {
            Document dwg = Application.DocumentManager.MdiActiveDocument;
            Database TargetDb = dwg.Database;
            Editor ed = dwg.Editor;
            DateTime start = DateTime.Now;
            long amount = 0;//common counter
            long exceptCount = 0;//exception counter
            Int64 Current = 0;
            Int64 previous = 0;
            using (Transaction t = TargetDb.TransactionManager.StartTransaction())
            {
                for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i  )
                {
                      amount;
                    ObjectId id = ObjectId.Null;
                    //Slow code begin:
                    try
                    {
                        id = TargetDb.GetObjectId(false, new Handle(i), 0);
                        previous = Current;
                        Current = id.OldIdPtr.ToInt64();
                        if (previous != 0)
                        {
                            ed.WriteMessage("\n{0}\n", (Current - previous) .ToString());
                        }
                    }
                    catch
                    {
                          exceptCount;
                        continue;
                    }
                    //Slow code end.
                }
            }
            TimeSpan len = DateTime.Now - start;
            ed.WriteMessage("\n{0}\n", exceptCount.ToString());
            ed.WriteMessage(string.Format("Amount: {0}\nSlow iteration time: {1} min. {2} sec.\n",
                amount, len.Minutes, len.Seconds));
        }

 
If it is working you could do for or while loop going backwards from the Handseed property and the first one that does not throw error set to last then break out of the loop.
 
 
 

[Andrey Bushman]

(LastId - FirstId) / 8 = Total number of objects
 
Since Int64 is 8 bytes and taking the difference between first and last would be the total number bytes and dividing by eight gives total number of objects.

But entity can be removed, so in an expected place it can not appear the identifier...
P.S. You not truly quote my code (increments have disappeared).

[kaefer]

Int64 is 8 bytes

You've got me there. Andrey got me with another unwarranted assumption: That (entlast) after creating a new object returns a new ObjectId whose pointer is bigger than other existing pointers.

How well-founded are the assumptions that
1) BlockTable is the "first" object in the drawing, because it has Handle "1"
2) Handseed is after the "last" object in the drawing (well, it's settable).

[Alexander Rivilis]

1) BlockTable is the "first" object in the drawing, because it has Handle "1"

It is not always true. I've seen dwg-file with Handle of BlockTable such as "FEDCBA", but it's handle really was "first" in dwg

[Andrey Bushman]

You've got me there. Andrey got me with another unwarranted assumption: That (entlast) after creating a new object returns a new ObjectId whose pointer is bigger than other existing pointers.

Online-translator translates it not clearly. I haven't understood that sense that you have written.

[Andrey Bushman]

Code: [Select]

//Database's ObjectIds iteration
[CommandMethod("FastIteration", CommandFlags.Modal)]
public void FastIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    long amount = 0;//Database's ObjectIds common counter

    using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
        for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i++) {
            ObjectId id = ObjectId.Null;

            //next line - Fatal error!!!
            int x = getAcDbObjectId17(TargetDb.UnmanagedObject, ref id, false, new Handle(i), 0);
            if (x != 0 && id.IsValid)
                ++amount;
        }
    }
    TimeSpan len = DateTime.Now - start;
    ed.WriteMessage(string.Format("Amount: {0}\nFast iteration time: {1} min. {2} sec.\n",
        amount, len.Minutes, len.Seconds));
}

//P/Invoke method for AutoCAD 2009 x86 (Alexander Rivilis's code)
[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("acdb17.dll", CallingConvention = CallingConvention.ThisCall,
   EntryPoint = "?getAcDbObjectId@AcDbDatabase@@QAE?AW4ErrorStatus@Acad@@AAVAcDbObjectId@@_NABVAcDbHandle@@K@Z")]
extern static int getAcDbObjectId17(IntPtr db, ref ObjectId id, bool createnew, Handle h, Int32 reserved);

I get fatall error

About my topic:

I have some methods, which operation I want to accelerate. Here one of them:

Code: [Select]

public Dictionary<Type, List<ObjectId>> GetByTypes(Func<Transaction, ObjectId, bool> requirement) {
    Dictionary<Type, List<ObjectId>> dict = new Dictionary<Type, List<ObjectId>>();
    using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
        for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i++) {
            ObjectId id = ObjectId.Null;
            try {
                id = TargetDb.GetObjectId(false, new Handle(i), 0);
            }
            catch { continue; }

            if (id != ObjectId.Null &&  id.IsValid && !id.IsErased && requirement(t, id)) {
                Type type = t.GetObject(id, OpenMode.ForRead).GetType();
                if (!dict.Keys.Contains(type))
                    dict.Add(type, new List<ObjectId>());
                dict[type].Add(id);
            }
        }
    }
    return dict;
}

It method return ObjectIds, grouped on Type, which is key of dictionary record.

Somebody can help with P/Invoke?

 

[Andrey Bushman]

Problem is solved!

Code: [Select]

//WARNING: P/Invoke method FOR AUTOCAD 2009 x86 ONLY!!! (Alexander Rivilis's code)
[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("acdb17.dll", CallingConvention = CallingConvention.ThisCall,
   EntryPoint = "?getAcDbObjectId@AcDbDatabase@@QAE?AW4ErrorStatus@Acad@@AAVAcDbObjectId@@_NABVAcDbHandle@@K@Z")]
extern static int getAcDbObjectId17(IntPtr db, ref ObjectId id, bool createnew, ref Handle h, Int32 reserved);
// Define Command "AsdkCmd1"
[CommandMethod("AsdkCmd1")]
static public void test() // This method can have any name
{
    // Put your command code here
    Database db = HostApplicationServices.WorkingDatabase;
    Editor ed = Application.DocumentManager.MdiActiveDocument.Editor;
    DateTime start = DateTime.Now;

    long amount = 0;//common counter

    using (Transaction t = db.TransactionManager.StartTransaction()) {
        ObjectId id = new ObjectId();
        for (long i = db.BlockTableId.Handle.Value; i < db.Handseed.Value; i++) {
            Handle h = new Handle(i);
            if (getAcDbObjectId17(db.UnmanagedObject, ref id, false, ref h, 0) == 0) {
                ++amount;
                id = db.GetObjectId(false, new Handle(i), 0);
                DBObject dbo= t.GetObject(id, OpenMode.ForRead);
            }
        }
    }
    TimeSpan len = DateTime.Now - start;
    ed.WriteMessage(string.Format("Amount: {0}\nFast iteration time: {1} min. {2} sec.\n",
        amount, len.Minutes, len.Seconds));

}
for other AutoCAD versions:

Code: [Select]

//Ниже приведён набор методов, с помощью которых можно получить значение идентификатора, минуя при этом генерацию
//исключения, если запрашиваемый идентификатор отсутствует в базе данных чертежа (это существенно ускоряет
//производительность)
//Низкий поклон Александру Ривилису, написавшему эти методы!!!!!
[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("acdb17.dll", CallingConvention = CallingConvention.ThisCall,
   EntryPoint = "?getAcDbObjectId@AcDbDatabase@@QAE?AW4ErrorStatus@Acad@@AAVAcDbObjectId@@_NABVAcDbHandle@@K@Z")]
extern static int getAcDbObjectId17x32(IntPtr db, ref ObjectId id, bool createnew, ref Handle h, Int32 reserved);

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("acdb17.dll", CallingConvention = CallingConvention.ThisCall,
  EntryPoint = "?getAcDbObjectId@AcDbDatabase@@QEAA?AW4ErrorStatus@Acad@@AEAVAcDbObjectId@@_NAEBVAcDbHandle@@K@Z")]
extern static int getAcDbObjectId17x64(IntPtr db, ref ObjectId id, bool createnew, ref Handle h, Int32 reserved);

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("acdb18.dll", CallingConvention = CallingConvention.ThisCall,
   EntryPoint = "?getAcDbObjectId@AcDbDatabase@@QAE?AW4ErrorStatus@Acad@@AAVAcDbObjectId@@_NABVAcDbHandle@@K@Z")]
extern static int getAcDbObjectId18x32(IntPtr db, ref ObjectId id, bool createnew, ref Handle h, Int32 reserved);

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("acdb18.dll", CallingConvention = CallingConvention.ThisCall,
  EntryPoint = "?getAcDbObjectId@AcDbDatabase@@QEAA?AW4ErrorStatus@Acad@@AEAVAcDbObjectId@@_NAEBVAcDbHandle@@K@Z")]
extern static int getAcDbObjectId18x64(IntPtr db, ref ObjectId id, bool createnew, ref Handle h, Int32 reserved);

public static int getAcDbObjectId(IntPtr db, ref ObjectId id, bool createnew, ref Handle h, Int32 reserved) {
    switch (Application.Version.Major) {
        case 17: {
                if (IntPtr.Size == 4) return getAcDbObjectId17x32(db, ref id, createnew, ref h, reserved);
                else return getAcDbObjectId17x64(db, ref id, createnew, ref h, reserved);
            }
        case 18: {
                if (IntPtr.Size == 4) return getAcDbObjectId18x32(db, ref id, createnew, ref h, reserved);
                else return getAcDbObjectId18x64(db, ref id, createnew, ref h, reserved);
            }
    }
    return -1;
}

Thank you all, and Alexander Rivilis!!!

[kaefer]

Code: [Select]

           if (getAcDbObjectId17(db.UnmanagedObject, ref id, false, ref h, 0) == 0) {
                ++amount;
                id = db.GetObjectId(false, new Handle(i), 0);  // <- Could you dump this line?


Firstly, congratulations!

What caught my attention now was the fact that you're doing the same thing twice, via p/invoke and for good measure the .NET way too. If I'm not completely mistaken; the second argument is actually out, and the fourth is ref. Not sure if this distinction carries over into C++ land.

Acad::ErrorStatus getAcDbObjectId(
    AcDbObjectId& retId,
    bool createIfNotFound,
    const AcDbHandle& objHandle,
    Adesk::UInt32 xRefId = 0
);

[Andrey Bushman]

What caught my attention now was the fact that you're doing the same thing twice

Yes it my mistake. Thank you!

I have published the code and tests here (the text in Russian, but the code it is possible to read and understand).

[Jeff H]

One thing that might help instead of storing types you could use strings and not have to open each object.
 
For example this just prints all the
ObjectId's in modelSpace ObjectID.ObjectClass.Name & ObjectID.ObjectClass.DxfName
 
For a line will print

AcDbLine LINE

Code: [Select]


              [CommandMethod("PrintAllModelSpaceTypes")]
        public void PrintAllModelSpaceTypes()
        {
            Document doc = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument;
            Database db = doc.Database;
            Editor ed = doc.Editor;
            using (Transaction trx = db.TransactionManager.StartTransaction())
            {
               
                    BlockTable bt = db.BlockTableId.GetObject(OpenMode.ForRead) as BlockTable;
                    BlockTableRecord modelBtr = (BlockTableRecord)trx.GetObject(bt[BlockTableRecord.ModelSpace], OpenMode.ForRead);
                    foreach (ObjectId objId in modelBtr)
                    {               
                            ed.WriteMessage("\n{0} {1}", objId.ObjectClass.Name, objId.ObjectClass.DxfName);                       
                    }
                    trx.Commit();
                             
            }
        }


[kaefer]

One thing that might help instead of storing types you could use strings and not have to open each object.

It does help indeed. For the benefit of those being on a platform slightly more up-to-date then there's another thing that helps: Database.TryGetObjectId. Fast it is too.

Code: [Select]

        [CommandMethod("Test")]
        static public void test()
        {
            Database db = HostApplicationServices.WorkingDatabase;
            Editor ed = Application.DocumentManager.MdiActiveDocument.Editor;
            Stopwatch sw = System.Diagnostics.Stopwatch.StartNew();
            long amount = 0;
            Dictionary<string, int> d = new Dictionary<string, int>();
            ObjectId id = ObjectId.Null;
            for (long i = db.BlockTableId.Handle.Value; i < db.Handseed.Value; i++)
            {
                Handle h = new Handle(i);
                if (db.TryGetObjectId(h, out id) && !id.IsNull && id.IsValid && !id.IsErased)
                    {
                        string t = id.ObjectClass.DxfName;
                        amount++;       
                        if(d.ContainsKey(t))
                            d[t]++;
                        else
                            d.Add(t, 1);
                    }
            }
            sw.Stop();
            foreach(KeyValuePair<string,int> kvp in d)
                ed.WriteMessage("\n{0}: {1} ", kvp.Key, kvp.Value);
            ed.WriteMessage("\nTotal {0} objects in drawing, time elapsed: {1} ", amount, sw.Elapsed);
        }


[Andrey Bushman]

db.TryGetObjectId(h, out id)

You are magician!!!! Many thanks!!!!! It is the fastest variant, fastest all previous!!! (3 seconds).

[Andrey Bushman]

One thing that might help instead of storing types you could use strings and not have to open each object.

Thank you! It really very much has accelerated operation.