Topic: Slow iteration on ObjectIds

[Andrey Bushman]

Hello,

• Windows XP SP3 x86 Rus
• .Net Framework 3.5 SP1
• AutoCAD 2009 SP3 x86 Enu

I have are problem - slow ObjectIds iteration in Database. The size of a checked dwg-file is 23 Mb.

Slow code:

Code: [Select]

[CommandMethod("SlowIteration", CommandFlags.Modal)]
public void SlowIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    long amount = 0;//common counter
    long exceptCount = 0;//exception counter

    using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
        for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i++) {
            ++amount;
            ObjectId id = ObjectId.Null;

            //Slow code begin:
            try {
                id = TargetDb.GetObjectId(false, new Handle(i), 0);
            }
            catch {
                ++exceptCount;
                continue;
            }
            //Slow code end.

        }
    }
    TimeSpan len = DateTime.Now - start;
    ed.WriteMessage(string.Format("Amount: {0}\nSlow iteration time: {1} min. {2} sec.\n",
        amount, len.Minutes, len.Seconds));
}
SlowIteration command result:

Command: SlowIteration
Amount: 517579
Slow iteration time: 4 min. 36 sec.

I have marked problem code location comments.
If to comment out the slow code the result will be another. Below sample.

Fast code:

Code: [Select]

[CommandMethod("FastIteration", CommandFlags.Modal)]
public void FastIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    long amount = 0;//common counter
    //long exceptCount = 0;//exception counter

    using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
        for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i++) {
            ++amount;
            ObjectId id = ObjectId.Null;
            //Below slow code is commented:
            //try {
            //    id = TargetDb.GetObjectId(false, new Handle(i), 0);
            //}
            //catch {
            //    ++exceptCount;
            //    continue;
            //}
        }
    }
    TimeSpan len = DateTime.Now - start;
    ed.WriteMessage(string.Format("Amount: {0}\nFast iteration time: {1} min. {2} sec.\n",
        amount, len.Minutes, len.Seconds));
}

FastIteration command result:

Command: FastIteration
Amount: 517579
Fast iteration time: 0 min. 0 sec.

I need get all ObjectIds from Database object. How can I get it without slow iteration?

I thank for attention.

[kaefer]

Hi Andrey,

what you're doing there doesn't look like it's guaranteed to work at all. So you're basically on your own and you've been lucky so far.

That being said, I could think of a few optimizations:
- Get rid of try/catch in a  tight loop, and
- Try to enumerate ObjectId instead of Handle, using the fact that ObjectId is IntPtr under the hood.

A quick mock-up in F# gets the same "Approx. number of objects" as MgdDbgSnoopDb would display. What's missing is the code for generating a new Entity to ensure (entlast) gets the highest number.

Code: [Select]

let test1() =
    let ed = Application.DocumentManager.MdiActiveDocument.Editor
    let db = HostApplicationServices.WorkingDatabase
    use tr = db.TransactionManager.StartTransaction()

    let bt = tr.GetObject(db.BlockTableId, OpenMode.ForRead) :?> BlockTable
    let entLast = Autodesk.AutoCAD.Internal.Utils.EntLast()
    if entLast.IsNull then
        ed.WriteMessage("\nNo objects in drawing. ")
    else
        let start = (int64) bt.ObjectId.OldIdPtr
        let stop = (int64) entLast.OldIdPtr
        let cnt = ref 0
        let sw = System.Diagnostics.Stopwatch.StartNew()
        for ptr in start .. 4L .. stop do
            let oid = new ObjectId(nativeint ptr)
            if oid.IsValid then
                incr cnt
        sw.Stop()       
       
        ed.WriteMessage("\n{0} objects in drawing, time elapsed: {1} ", !cnt, sw.Elapsed)
           
    tr.Commit()


[Andrey Bushman]

what you're doing there doesn't look like it's guaranteed to work at all. So you're basically on your own and you've been lucky so far.

I use this approach for a long time and it always works successfully, but slowly... ((

Which library I must add to my project for Autodesk.AutoCAD.Internal.Utils.EntLast() using?

[Andrey Bushman]

I has found: C:\Program Files\AutoCAD 2009\acmgdinternal.dll

[kaefer]

Which library I must add to my project for Autodesk.AutoCAD.Internal.Utils.EntLast() using?

Eh, acmgdinternal.dll, perhaps?  (You said you're on 17.2. Since18.0 it's integrated in AcMgd.dll.)

[Andrey Bushman]

Eh, acmgdinternal.dll, perhaps?  (You said you're on 17.2. Since18.0 it's integrated in AcMgd.dll.)

Thank you.

Code: [Select]

[CommandMethod("KaefersIteration", CommandFlags.Modal)]
public void KaefersIteration() {
    Document dwg = acad.DocumentManager.MdiActiveDocument;
    Database TargetDb = dwg.Database;
    Editor ed = dwg.Editor;
    DateTime start = DateTime.Now;

    ObjectId lastId = Autodesk.AutoCAD.Internal.Utils.EntLast();

    if (lastId != ObjectId.Null) {
        long amount = 0;

        using (Transaction t = TargetDb.TransactionManager.StartTransaction()) {
            BlockTable bt = (BlockTable) t.GetObject(TargetDb.BlockTableId, OpenMode.ForRead);

            for (Int64 i = (Int64) bt.ObjectId.OldIdPtr; i <= (Int64) lastId.OldIdPtr; ++i) {
                IntPtr ptr = new IntPtr(i);
                ObjectId id = new ObjectId(ptr);
                if (id.IsValid)
                    ++amount;
            }
        }
        TimeSpan len = DateTime.Now - start;
        ed.WriteMessage(string.Format("Amount: {0}\nFast iteration time: {1} min. {2} sec.\n",
            amount, len.Minutes, len.Seconds));
    }
    else
        ed.WriteMessage("No objects in drawing.\n");
}

result:

Command: KaefersIteration
Amount: 0
Fast iteration time: 0 min. 0 sec.

nothing not found.

[kaefer]

nothing not found.

Did you add an Entity to your drawing?

[Andrey Bushman]

nothing not found.

Did you add an Entity to your drawing?

The size of a checked dwg-file is 23 Mb. It has 517 579 Entities.

[kaefer]

Did you add another Entity to your drawing?

[Andrey Bushman]

Did you add another Entity to your drawing?

No, I need all database entities, not new only 

But if I add new Entities - it not work too.

How in your variant the code learns, with what database it is necessary to work?

Oops...

bt.ObjectId.OldIdPtr: 2 128 931 848
lastId.OldIdPtr:  2 124 488 384

second more then first.

[Jeff H]

Are you saying that
 
(LastId - FirstId) / 8 = Total number of objects
 
Since Int64 is 8 bytes and taking the difference between first and last would be the total number bytes and dividing by eight gives total number of objects.
 
 

Code: [Select]

        [CommandMethod("FasterIteration")]
        public void FasterIteration()
        {
            Document doc = Application.DocumentManager.MdiActiveDocument;
            Database db = doc.Database;
            Editor ed = doc.Editor; ;
            long amount = 0;//common counter

            Int64 firstId = db.BlockTableId.OldIdPtr.ToInt64();
            Int64 lastId = Autodesk.AutoCAD.Internal.Utils.EntLast().OldIdPtr.ToInt64();

            ed.WriteMessage("\n{0} {1}\n", firstId.ToString(), lastId.ToString());

            amount = (long)((lastId - firstId) /;
            ed.WriteMessage("\n{0}\n", amount.ToString());
        }

but modifying Andreys code to output the difference between ObjectIds 98% are 8 but some will be 64, 16, 2400 etc.........
 
I guess the processor breaks it up into available areas?!???
 

Code: [Select]


        [CommandMethod("SlowIteration", CommandFlags.Modal)]
        public void SlowIteration()
        {
            Document dwg = Application.DocumentManager.MdiActiveDocument;
            Database TargetDb = dwg.Database;
            Editor ed = dwg.Editor;
            DateTime start = DateTime.Now;
            long amount = 0;//common counter
            long exceptCount = 0;//exception counter
            Int64 Current = 0;
            Int64 previous = 0;
            using (Transaction t = TargetDb.TransactionManager.StartTransaction())
            {
                for (long i = TargetDb.BlockTableId.Handle.Value; i < TargetDb.Handseed.Value; i  )
                {
                      amount;
                    ObjectId id = ObjectId.Null;
                    //Slow code begin:
                    try
                    {
                        id = TargetDb.GetObjectId(false, new Handle(i), 0);
                        previous = Current;
                        Current = id.OldIdPtr.ToInt64();
                        if (previous != 0)
                        {
                            ed.WriteMessage("\n{0}\n", (Current - previous) .ToString());
                        }
                    }
                    catch
                    {
                          exceptCount;
                        continue;
                    }
                    //Slow code end.
                }
            }
            TimeSpan len = DateTime.Now - start;
            ed.WriteMessage("\n{0}\n", exceptCount.ToString());
            ed.WriteMessage(string.Format("Amount: {0}\nSlow iteration time: {1} min. {2} sec.\n",
                amount, len.Minutes, len.Seconds));
        }

 
If it is working you could do for or while loop going backwards from the Handseed property and the first one that does not throw error set to last then break out of the loop.
 
 
 

[Andrey Bushman]

(LastId - FirstId) / 8 = Total number of objects
 
Since Int64 is 8 bytes and taking the difference between first and last would be the total number bytes and dividing by eight gives total number of objects.

But entity can be removed, so in an expected place it can not appear the identifier...
P.S. You not truly quote my code (increments have disappeared).

[kaefer]

Int64 is 8 bytes

You've got me there. Andrey got me with another unwarranted assumption: That (entlast) after creating a new object returns a new ObjectId whose pointer is bigger than other existing pointers.

How well-founded are the assumptions that
1) BlockTable is the "first" object in the drawing, because it has Handle "1"
2) Handseed is after the "last" object in the drawing (well, it's settable).

[Alexander Rivilis]

1) BlockTable is the "first" object in the drawing, because it has Handle "1"

It is not always true. I've seen dwg-file with Handle of BlockTable such as "FEDCBA", but it's handle really was "first" in dwg

[Andrey Bushman]

You've got me there. Andrey got me with another unwarranted assumption: That (entlast) after creating a new object returns a new ObjectId whose pointer is bigger than other existing pointers.

Online-translator translates it not clearly. I haven't understood that sense that you have written.