TheSwamp

Code Red => AutoLISP (Vanilla / Visual) => Topic started by: qjchen on April 02, 2010, 11:42:44 AM

Title: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 02, 2010, 11:42:44 AM
Attachment is a dwg contains 10000 random points.

Then the challenge is to connect each two points whose distance is shorter than given value, in this dwg the value is 70.

My first code (as follows) is very very slow for 10000 points, when for few points, it can solve. And I think I can improve it more quickly.

I hope you can give more effective way.

Code: [Select]
(vl-load-com)
;;stdlib
(defun std-sslist (ss / n lst)
  (if (eq 'pickset (type ss))
    (repeat (setq n (fix (sslength ss))) ; fixed
      (setq lst (cons (ssname ss (setq n (1- n))) lst))
    )
  )
)
;;make layer or turn on current layer
(defun mylayer (str)
  (if (tblsearch "LAYER" str)
    (command "._-Layer" "_set" str "")
    (command "._-Layer" "_Make" str "")
  )
)
;;
;;entmake line
(defun entmakeline (pt1 pt2 layer)
  (entmake (list (cons 0 "LINE") ;***
(cons 6 "BYLAYER") (cons 8 layer) (cons 10 pt1) ;***
(cons 11 pt2) ;***
(cons 39 0.0) (cons 62 256) (cons 210 (list 0.0 0.0 1.0))
  )
  )
);;
;;get the point coordinate
(defun mygetpoint (lst / plst)
  (foreach x lst
    (setq plst (append
plst
(list (cdr (assoc 10 (entget x))))
      )
    )
  )
  plst
)
;;Main
(defun c:gap ( / c i j len len1 limdis pj plst x)
  (mylayer "temp")
  (setq limdis 70)
  (setq c (ssget '((0 . "POINT"))))
  (setq c (std-sslist c))
  (setq plst (mygetpoint c))
  (setq i 0
len (length plst)
  )
  (foreach x plst
    (princ "\n               i = ") (princ i);Erase_DV
    (setq j (1+ i)
 len1 (- len j)
    )
    (repeat len1
      (setq pj (nth j plst))
      (if (< (distance x pj) limdis)
(entmakeline x pj "temp")
      )
      (setq j (1+ j))
    )
    (setq i (1+ i))
  )
)


Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: MP on April 02, 2010, 11:43:38 AM
zomg it's full of starz!!
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ronjonp on April 02, 2010, 12:42:30 PM
Well ... here's mine to start it off  :-)
Code: [Select]
(defun c:test (/ d e n out pt ss)
  (and
    (setq n -1)
    (setq d 70.)
    (setq ss (ssget '((0 . "point"))))
    (while (setq e (ssname ss (setq n (1+ n))))
      (setq out (cons (cdr (assoc 10 (entget e))) out))
    )
    (setq out (vl-sort out (function (lambda (x1 x2) (< (car x1) (car x2))))))
    (while (setq pt (car out))
      (mapcar (function
(lambda (x) (entmakex (list '(0 . "LINE") (cons 10 pt) (cons 11 x))))
      )
      (vl-remove-if
(function (lambda (x) (>= (distance pt x) d)))
(setq out (cdr out))
      )
      )
    )
  )
  (princ)
)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: Lee Mac on April 02, 2010, 12:48:51 PM
Not sure how quick this'll be:

Code: [Select]
(defun PointGap (gap / GetPoints Pts tmp x y)

  (defun GetPoints (ss / e i l)
    (setq i -1)
    (if ss
      (while (setq e (ssname ss (setq i (1+ i))))
        (setq l (cons (cdr (assoc 10 (entget e))) l))))
    l)

  (if (setq Pts (GetPoints (ssget "_X" '((0 . "POINT")))))
    (while (setq x (car Pts))

      (setq tmp (cdr Pts))
      (while (setq y (car tmp))

        (if (< (distance x y) gap)
          (entmakex (list (cons 0 "LINE")
                          (cons 10 x) (cons 11 y))))
       
        (setq tmp (cdr tmp)))

      (setq Pts (cdr Pts))))

  (princ))
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 02, 2010, 02:31:55 PM
 :|


Code: [Select]
(_connect (_GetPnts) 70.))
(_connect2 (_GetPnts) 70.))


Code: [Select]
(defun _GetPnts (/ ss i)
  (setq i -1)
  (if (setq ss (ssget "_X" '((0 . "POINT"))))
    (while (setq e (ssname ss (setq i (1+ i))))
      (setq l (cons (cdr (assoc 10 (entget e))) l))
    ) ;_ while
  ) ;_ if
) ;_ defun


(defun _Line (a b) (entmakex (list '(0 . "LINE") (cons 10 b) (cons 11 a))))


(defun _connect (lst d / lst)
  (foreach x lst
    (foreach i (setq lst (vl-remove x lst))
      (and (< (distance x i) d)
           (_Line x i)
      ) ;_ and
    ) ;_ foreach
  ) ;_ foreach
) ;_ defun


(defun _connect2 (lst d / lst)
  (while (setq a (car lst))
    (foreach x (setq lst (cdr lst))
      (and (< (distance a x) d)
           (_Line a x)
      ) ;_ and
    ) ;_ foreach
  ) ;_ while
) ;_ defun
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gile on April 02, 2010, 04:00:28 PM
Hi,

Here's my way, quite similar to those posted before.

Code: [Select]
(defun foo (dist / n ss ent lst pt)
  (if (setq n  -1
    ss (ssget "_X" '((0 . "POINT")))
      )
    (while (setq ent (ssname ss (setq n (1+ n))))
      (setq lst (cons (cdr (assoc 10 (entget ent))) lst))
    )
  )
  (while lst
    (setq pt  (car lst)
  lst (cdr lst)
    )
    (foreach p lst
      (if (<= (distance pt p) dist)
(entmake (list '(0 . "LINE") (cons 10 pt) (cons 11 p)))
      )
    )
  )
)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 02, 2010, 04:51:50 PM
Hi,

Here's my way, quite similar to those posted before.
Code: [Select]
<=

Would have been smart of me to do the same.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 02, 2010, 05:01:10 PM
Ug, to busy to play, but can't resist   ^-^. Will post ARX code in a bit.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gile on April 02, 2010, 06:00:13 PM
Hi,

A C# solution
Much more faster than LISP. With the sample drawing and a 70.0 length:
LISP = about 45 seconds
C# = about 2 seconds
Maybe ARX shoulde be faster...

Code: [Select]
using System;
using Autodesk.AutoCAD.ApplicationServices;
using Autodesk.AutoCAD.DatabaseServices;
using Autodesk.AutoCAD.EditorInput;
using Autodesk.AutoCAD.Geometry;
using Autodesk.AutoCAD.Runtime;

namespace ChallengeConnectPoints
{
    public class Connect
    {

        [CommandMethod("Test")]
        public void Test()
        {
            Editor ed = Application.DocumentManager.MdiActiveDocument.Editor;
            PromptDistanceOptions pdo = new PromptDistanceOptions("\nConnection distance: ");
            pdo.AllowNegative = false;
            pdo.AllowZero = false;
            PromptDoubleResult pdr = ed.GetDistance(pdo);
            if (pdr.Status == PromptStatus.OK)
                ConnectPoints(pdr.Value);
        }

        private void ConnectPoints(double d)
        {
            Document doc = Application.DocumentManager.MdiActiveDocument;
            Database db = doc.Database;
            Editor ed = doc.Editor;
            TypedValue[] tv = new TypedValue[1] { new TypedValue(0, "POINT") };
            PromptSelectionResult psr = ed.SelectAll(new SelectionFilter(tv));
            if (psr.Status != PromptStatus.OK)
                return;
            using (Transaction tr = db.TransactionManager.StartTransaction())
            {
                Point3d[] pts = Array.ConvertAll<ObjectId, Point3d>(
                    psr.Value.GetObjectIds(), x => ((DBPoint)x.GetObject(OpenMode.ForRead)).Position);
                int cnt = pts.Length;
                BlockTableRecord btr =
                    (BlockTableRecord)tr.GetObject(db.CurrentSpaceId, OpenMode.ForWrite);
                for (int i = 0; i < cnt; i++)
                {
                    for (int j = i + 1; j < cnt; j++)
                    {
                        if (pts[i].DistanceTo(pts[j]) <= d)
                        {
                            Line l = new Line(pts[i], pts[j]);
                            btr.AppendEntity(l);
                            tr.AddNewlyCreatedDBObject(l, true);
                        }
                    }
                }
                tr.Commit();
            }
        }
    }
}

You can try tne DLL in the ZIP file.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 02, 2010, 06:34:55 PM
Hi,

A C# solution
Much more faster than LISP. With the sample drawing and a 70.0 length:
LISP = about 45 seconds
C# = about 2 seconds
Maybe ARX shoulde be faster...

Nice Gile.

I'm taking a none brute force approach, initial results look promising  ;-) Have an hour or so of coding left to do.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 02, 2010, 07:25:47 PM
Have an hour or so of coding left to do.
Man, one really pays for performance. I'm not knocking C/Arx, just taken aback at the amount of time.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 02, 2010, 07:32:37 PM
Have an hour or so of coding left to do.
Man, one really pays for performance. I'm not knocking C/Arx, just taken aback at the amount of time.

Your right, and this is even repurposing code I've already written. The solution though is non-trivial, but screams. Tested with a million points and it's under a second. Still coding though so back to work  :lmao:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 02, 2010, 07:35:12 PM
Have an hour or so of coding left to do.
Man, one really pays for performance. I'm not knocking C/Arx, just taken aback at the amount of time.

Your right, and this is even repurposing code I've already written. The solution though is non-trivial, but screams. Tested with a million points and it's under a second. Still coding though so back to work  :lmao:
:lol:

Can't wait to see it!!
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: Kerry on April 02, 2010, 09:13:40 PM
Have an hour or so of coding left to do.
Man, one really pays for performance. I'm not knocking C/Arx, just taken aback at the amount of time.

Your right, and this is even repurposing code I've already written. The solution though is non-trivial, but screams. Tested with a million points and it's under a second. Still coding though so back to work  :lmao:
:lol:

Can't wait to see it!!

Yes you can, really, you can.  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 02, 2010, 10:53:04 PM
Thank you all :), Your algorithms are all faster than me.
In my PC, (E7500 CPU, 4G Mem)
Ronjonp 's code, about 34.812967s
Lee Mac's code, about 40.733018s
Gile's code 42.111982s
alanjt's code is about 50s
Mine first code is very slow.   :-D, (could you tell me which cause the very slow)
Not very accurate test.

And I hope to see pkohut's super code :)

I am thinking about the "Divide and conquer algorithm", but I need some time to sort the point in some manner, and not so sure about whether it can be improved, because the sort also need some times.


Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 02, 2010, 11:15:37 PM
brute force with arx

doit
GetDist: 70
1.036883 Seconds

Code: [Select]
static void ArxDist_doit(void)
  {
    double distance = 0.0;
    acedGetDist(0,ACRX_T("\nGetDist: "),&distance);

    __int64 freq, start, end, diff;
    QueryPerformanceFrequency((LARGE_INTEGER*)&freq);
    QueryPerformanceCounter((LARGE_INTEGER*)&start);

    std::vector<AcGePoint3d> points;
    AcDbBlockTableRecord *pRec;
    AcDbDatabase *pDb = acdbHostApplicationServices()->workingDatabase();
   
    if(acdbOpenAcDbObject((AcDbObject *&) pRec,pDb->currentSpaceId(), AcDb::kForWrite) == Acad::eOk)
    {
      AcDbBlockTableRecordIterator *pIter = NULL;
      if(pRec->newIterator(pIter) == Acad::eOk)
      {
        for(pIter->start(); !pIter->done(); pIter->step())
        {
          AcDbEntity *pEnt = NULL;
          if(pIter->getEntity(pEnt, AcDb::kForRead) == Acad::eOk)
          {
            AcDbPoint *pPoint = AcDbPoint::cast(pEnt);
            if(pPoint)
              points.push_back(pPoint->position());
            pEnt->close();
          }
        }
        delete pIter;
      }
      size_t len = points.size();
      for (size_t i = 0; i < len; i++)
      {
        for (size_t j = i + 1; j < len; j++)
        {
          if ( (points[i].distanceTo(points[j])) <= distance)
          {
            AcDbLine *pLine = new AcDbLine(points[i],points[j]);
            if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
              delete pLine;
            pLine->close();
          }
        }
      }
      pRec->close();
    }
    QueryPerformanceCounter((LARGE_INTEGER*)&end);
    acutPrintf(_T("\n%g Seconds\n"), (double)(end - start) / (double) freq);
  }

arx for 07
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 02, 2010, 11:20:16 PM
Thank you all :), Your algorithms are all faster than me.
In my PC, (E7500 CPU, 4G Mem)
Ronjonp 's code, about 34.812967s
Lee Mac's code, about 40.733018s
Gile's code 42.111982s
alanjt's code is about 50s
Mine first code is very slow.   :-D, (could you tell me which cause the very slow)
Not very accurate test.

And I hope to see pkohut's super code :)

I am thinking about the "Divide and conquer algorithm", but I need some time to sort the point in some manner, and not so sure about whether it can be improved, because the sort also need some times.

That's interesting that Gile's is 8 seconds faster. Unless I'm overlooking something, we used the same method (comparing with my _connect2), except that I had it broken up into 3 separate subs. I didn't think it would effect performance, but I guess it does. Food for thought.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 02, 2010, 11:22:02 PM
brute force with arx

doit
GetDist: 70
1.036883 Seconds

arx for 07

That's incredible Daniel! I have got to put Lisp down and learn C.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 02, 2010, 11:27:46 PM
I'm guessing Paul is pulling apart his triangle routine.. right?
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: Kerry on April 02, 2010, 11:37:54 PM
brute force with arx

doit
GetDist: 70
1.036883 Seconds

arx for 07

That's incredible Daniel! I have got to put Lisp down and learn C.

:)

Lisp can't be 'put down' it just refuses to die  :-D

I agree about the idea of learning C, C++.  Great if we can make the time and if the wet stuff is still receptive.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 03, 2010, 12:05:12 AM
The 80/20 rule kicked in and it took me a bit longer to write than estimated.

My ARX entry is attached.  ConnectDots2004.arx is for Autocad 2004-2006, and ConnectDots2007 is for 2007-2008(9?).  Just load the appropriate version and at the command line type
Code: [Select]
connectdotsYou will them be prompted for the maximum segment length to connect, and to select the points to operate on.

Here the processing times (counted in my head) -
10,000 points, uh' its done.
100,000 points, little less then 2 seconds.
500,000 points, about 15 seconds.

The heart of the program is the KD-Tree data structure which holds point data selected by the user. Given any point, we can ask the structure to return a set of points within a radius of the source point. Line segments can then be created from each point in the set to the source point. Doing this for all the points in the collection, results in at best 2 line segments for each qualified set of points. There are a number of methods to find duplicate line segments, but they all have some performance cost associated with them. Instead I set a flag for the source point, marking it as already processed and don't have to worry about duplicate segments.

Points of interest in the code (since there is a lot of code)-
Create an instance of CKdTree and initialize it to the number of dimensions. kdTree is set to 3 dimensions, though for the purposes of this program 2 would have been fine since the Z is ignored in all the calculates.
Code: [Select]
CKdTree kdTree;
kdTree.Create(3);

Add a node to the tree. The forth parameter is of type void *, and is for user defined data. In this case I'm using it as a flag to determine if the node has been processed. Initial value is NULL or false.
Code: [Select]
kdTree.Insert(pt3d.x, pt3d.y, pt3d.z, NULL);
Query for all the gathered points (need to add another public function to CKdTree to get the head position).
Get a pointer to the X, Y, Z data
Query for another result set around the X, Y, Z, and radius.
Code: [Select]
RESULT_SET * pResAllPoints = kdTree.NearestRange(0.0, 0.0, 0.0, HUGE_VAL);
...
double * pPos = pResAllPoints->riter->item->pos;
RESULT_SET * pRes = kdTree.NearestRange(pPos[0], pPos[1], 0.0, dRadius);

The KD-Tree code I got off the internet about 5 years ago, though I've not been able to find it again and it had no copyright notice or name associated with it. Any changes I did to the code was so it would compile in C++, IIRC work with doubles instead of floats, and a couple helper functions.  I've used the code in about 7 projects, and it's always fun when another opportunity arises to use it again.

The rest of the code should be straight forward, if not a bit messy.  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 03, 2010, 12:08:20 AM
Have an hour or so of coding left to do.
Man, one really pays for performance. I'm not knocking C/Arx, just taken aback at the amount of time.

Your right, and this is even repurposing code I've already written. The solution though is non-trivial, but screams. Tested with a million points and it's under a second. Still coding though so back to work  :lmao:

When I posted that message I misread the number of points in the selection set. I should have wrote 100,000. sry
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 03, 2010, 12:15:05 AM
blistering fast  8-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 03, 2010, 12:35:46 AM
Thank you Daniel for your arx, but I have only 2004 for this PC, I will test your codes later.

And thank you pkohut, Your arx code is very very fast :), I will learn your article later.

After 1 hour programming, I wrote this first lisp revision.

It is about  3.2s for 10000 points, and 6.5s for 20000 points and 17.233984s for 100000 points.

I use many idea of ronjonp, Lee mac ,gile and alanjt, and then, I divide the pointlist to many sub pointlist according to the x coordinate

((points  xmin<x<xmin+70) (points  xmin+70<x<xmin+140) .........)

Then I connect the shortline in each sub pointlist.

Second, I connect the shortline between the adjacent pointlist.

Maybe it can be called "Divide and conquer" method.

In the next revision, I think I should divide the pointlist to x y grid, maybe it will more quickly.


Code: [Select]

(vl-load-com)
;;;start of Time, from theswamp
(defun startTimer ()
  (setq time (getvar "DATE"))
)
;;;end of Time, from theswamp
(defun endTimer (/ seconds totaltime)
  (setq time (- (getvar "DATE") time)
seconds (* 86400.0 (- time (fix time)))
  )
  (setq totaltime (rtos seconds 2 6))
  (princ (strcat "\nTimed:" (rtos seconds 2 6)))
)
;;entmake line
(defun entmakeline (pt1 pt2 layer)
  (entmake (list (cons 0 "LINE") ;***
(cons 6 "BYLAYER") (cons 8 layer) (cons 10 pt1) ;***
(cons 11 pt2) ;***
(cons 39 0.0) (cons 62 256) (cons 210 (list 0.0 0.0 1.0))
   )
  )
)
;;The idea of gile
(defun getpointscor ( / ent n plst ss)
  (if (setq n -1
    ss (ssget '((0 . "POINT")))
      )
    (while (setq ent (ssname ss (setq n (1+ n))))
      (setq plst (cons (cdr (assoc 10 (entget ent))) plst))
    )
  )
)
(defun dividex (plst dis / a b div end index len lst1 plstmp res start xend
     xstart
       )
  (setq start (car plst)
xstart (car start)
end (last plst)
len (- (car end) (car start))
div (+ (fix (/ len dis)) 1)
plstmp plst
res nil
  )
  (repeat div 
    (grdraw (list xstart 0 0) (list xstart 10000 0) 8)
    (setq xend (+ xstart dis))
    (setq index T
  lst1 nil
    )
    (while (and
     plstmp
     index
   )
      (if (< (setq a (car plstmp)
   b (car a)
     )
     xend
  )
(setq lst1 (append
     lst1
     (list a)
   )
      plstmp (cdr plstmp)
)
(setq index nil)
      )
    )
    (setq res (append
res
(list lst1)
      )
    )
    (setq xstart (+ xstart dis))
  )
  res
)
;;;draw short line in each x region
;;;The idea of Leemac and gile
(defun drawshortline (plst limdis / a b tmp)
  (while (setq a (car plst))
    (setq plst (cdr plst)
  tmp plst
    )
    (while (setq b (car tmp))
      (setq tmp (cdr tmp))
      (if (< (distance a b) limdis)
(entmakeline a b "temp")
      )
    )
  )
)
;;;draw short line in each two Neighbor x region
(defun drawshortline12 (plst1 plst2 limdis / a b x)
  (while (setq a (car plst1))
    (foreach x plst2
      (if (< (distance a x) limdis)
(entmakeline a x "temp")
      )
    )
    (setq plst1 (cdr plst1))
  )
)


;;Connect short line between each two points
;;by qjchen@gmail.com
;; Thanks to ronjonp, Lee Mac, and Gile
;;Version 1.0
(defun c:gap ( / a b limdis plst plst1)  ;(mylayer "temp")
  (starttimer)
  (setq limdis 70)
  ;;The idea of gile
  (setq plst (getpointscor))
  ;;The idea of ronjonp
  (setq plst (vl-sort plst (function (lambda (x1 x2)(< (car x1) (car x2))))))
  ;;Divde points list to several region
  ;;such like ((0, contain x=0~70),(1, contain x=70~140)........)
  (setq plst1 (dividex plst limdis))
  ;;Draw shortline in each x region ,such like (0, contain x=0~70))
  (foreach x plst1
    (drawshortline x limdis)
  )
  ;;Draw shortline between two Neighbor x region, such between (0, contain x=0~70),(1, contain x=70~140))
  (while (setq a (car plst1) b (cadr plst1) plst1 (cdr plst1))
    (drawshortline12 a b limdis)
  )
  (endtimer)
)


Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 03, 2010, 01:14:22 AM
brute force with arx

doit
GetDist: 70
1.036883 Seconds

arx for 07

That's incredible Daniel! I have got to put Lisp down and learn C.

:)

Lisp can't be 'put down' it just refuses to die  :-D

I agree about the idea of learning C, C++.  Great if we can make the time and if the wet stuff is still receptive.

I just meant to put it down long enough to start with a new language. It's far too useful to abandon.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ElpanovEvgeniy on April 03, 2010, 05:16:56 AM
my variant:

Code: [Select]
(defun el (d / A L LL TI X)
 ;;(el 70.)
 (setq ti (car (_VL-TIMES))
       a  -1
       ll (ssget "_x" '((0 . "point")))
 )
 (while (setq x (ssname ll (setq a (1+ a))))
  (setq l (cons (cdr (assoc 10 (entget x))) l))
 )
 (setq l (vl-sort l (function (lambda (a b) (<= (car a) (car b))))))
 (while (setq a (car l)
              l (cdr l)
        )
  (setq x  (car a)
        ll l
  )
  (while (and ll (<= (- (caar ll) x) d))
   (if (and (equal (cdar ll) (cdr a) d) (<= (distance a (car ll)) d))
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 (car ll))))
   )
   (setq ll (cdr ll))
  )
 )
 (princ (strcat "\n " (rtos (/ (- (car (_VL-TIMES)) ti) 1000.) 2 4) "sec."))
 (princ)
)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 03, 2010, 09:28:22 AM
Command: doit
497.0208 Milliseconds 8-)

Code: [Select]
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Diagnostics;

using Autodesk.AutoCAD.GraphicsInterface;
using Autodesk.AutoCAD.Runtime;
using Autodesk.AutoCAD.Windows;
using Autodesk.AutoCAD.Geometry;
using Autodesk.AutoCAD.EditorInput;
using Autodesk.AutoCAD.LayerManager;
using Autodesk.AutoCAD.ApplicationServices;
using Autodesk.AutoCAD.DatabaseServices;
using Autodesk.AutoCAD.Colors;
using System.Threading.Tasks;


using AcAp = Autodesk.AutoCAD.ApplicationServices;
using AcEd = Autodesk.AutoCAD.EditorInput;
using AcGe = Autodesk.AutoCAD.Geometry;
using AcRx = Autodesk.AutoCAD.Runtime;
using AcDb = Autodesk.AutoCAD.DatabaseServices;
using AcWd = Autodesk.AutoCAD.Windows;

namespace Test
{
  public class ExecMethod
  {
    [CommandMethod("doit")]
    public static void MyCommand()
    {

      Stopwatch stopWatch = new Stopwatch();
      stopWatch.Start();

      SynchronizedCollection<Line>
        lines = new SynchronizedCollection<Line>();

      List<Point3d> points = getPoints();
      int cnt = points.Count;


      Parallel.For(0, cnt, delegate(int i)
      {
        Point3d pnt = points[i];
        for (int j = i + 1; j < cnt; j++)
        {
          if (pnt.DistanceTo(points[j]) <= 70.0)
          {
            lines.Add(new Line(pnt, points[j]));
          }
        }
      });
      addtoDb(lines);

      stopWatch.Stop();
      TimeSpan ts = stopWatch.Elapsed;
      AcAp.Application.DocumentManager.MdiActiveDocument.
        Editor.WriteMessage("\n{0} Seconds", ts.TotalMilliseconds);
    }

    static void addtoDb(SynchronizedCollection<Line> lines)
    {
      List<Point3d> points = new List<Point3d>();
      Database database = HostApplicationServices.WorkingDatabase;
      using (Transaction transaction = database.TransactionManager.StartTransaction())
      {
        BlockTableRecord record =
          transaction.GetObject(
            database.CurrentSpaceId, OpenMode.ForWrite) as BlockTableRecord;

        foreach (Line line in lines)
        {
          record.AppendEntity(line);
          transaction.AddNewlyCreatedDBObject(line, true);
        }
        transaction.Commit();
      }
    }


    static List<Point3d> getPoints()
    {
      List<Point3d> points = new List<Point3d>();
      Database database = HostApplicationServices.WorkingDatabase;
      using (Transaction transaction = database.TransactionManager.StartTransaction())
      {
        BlockTableRecord record =
          transaction.GetObject(
            database.CurrentSpaceId, OpenMode.ForRead) as BlockTableRecord;

        foreach (ObjectId id in record)
        {
          DBPoint point = transaction.GetObject(id, OpenMode.ForRead) as DBPoint;
          if (point != null)
          {
            points.Add(point.Position);
          }
        }
        transaction.Commit();
      }
      return points;
    }
  }
}
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 03, 2010, 09:38:32 AM
 :-D
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: alanjt on April 03, 2010, 09:39:59 AM
Command: doit
497.0208 Milliseconds 8-)
:lol: :lol:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 03, 2010, 10:02:02 AM
my variant:

Code: [Select]
(defun el (d / A L LL TI X)
 ;;(el 70.)
 (setq ti (car (_VL-TIMES))
       a  -1
       ll (ssget "_x" '((0 . "point")))
 )
 (while (setq x (ssname ll (setq a (1+ a))))
  (setq l (cons (cdr (assoc 10 (entget x))) l))
 )
 (setq l (vl-sort l (function (lambda (a b) (<= (car a) (car b))))))
 (while (setq a (car l)
              l (cdr l)
        )
  (setq x  (car a)
        ll l
  )
  (while (and ll (<= (- (caar ll) x) d))
   (if (and (equal (cdar ll) (cdr a) d) (<= (distance a (car ll)) d))
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 (car ll))))
   )
   (setq ll (cdr ll))
  )
 )
 (princ (strcat "\n " (rtos (/ (- (car (_VL-TIMES)) ti) 1000.) 2 4) "sec."))
 (princ)
)

you're good  8-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 03, 2010, 11:08:27 AM
my variant:

Code: [Select]
(defun el (d / A L LL TI X)
 ;;(el 70.)
 (setq ti (car (_VL-TIMES))
       a  -1
       ll (ssget "_x" '((0 . "point")))
 )
 (while (setq x (ssname ll (setq a (1+ a))))
  (setq l (cons (cdr (assoc 10 (entget x))) l))
 )
 (setq l (vl-sort l (function (lambda (a b) (<= (car a) (car b))))))
 (while (setq a (car l)
              l (cdr l)
        )
  (setq x  (car a)
        ll l
  )
  (while (and ll (<= (- (caar ll) x) d))
   (if (and (equal (cdar ll) (cdr a) d) (<= (distance a (car ll)) d))
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 (car ll))))
   )
   (setq ll (cdr ll))
  )
 )
 (princ (strcat "\n " (rtos (/ (- (car (_VL-TIMES)) ti) 1000.) 2 4) "sec."))
 (princ)
)

Evgeniy, you are always so cool.   "(<= (- (caar ll) x) d)" is very effective

In a 100000 points dwg generate by this code

Code: [Select]
;;from internet
(defun random ()
  (setq seed (if seed
       (rem (+ (* seed 15625.7) 0.21137152) 1)
       0.3171943
     )
  )
)
(defun random-n (n)
  (* n (random))
)
(defun entmakepoint (pt layer)
  (entmake (list (cons 0 "POINT")
                 (cons 8 layer);***
(cons 6 "BYLAYER")
(cons 10 pt) ;***
(cons 39 0.0)
(cons 50 0.0)
(cons 62 256)
(cons 210 (list 0.0 0.0 1.0))
   )
  )
)
;;generate random points
(defun c:test1 ()
  (repeat 100000
    (entmakepoint (list (random-n 50000) (random-n 30000) 0) "0")
  )
)

your code is about 22s, My revision is about 28s, but you are in a so short code. I will try my best to make my code more quick. :)

Thank you very much.

Daniel, thanks for your c# code, it is rather quick. ~~, Thank you.

Pkohut, thanks for teaching me about the kd-tree algorithm, I will study it carefully and I hope I can also finish such code in Lisp.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gile on April 03, 2010, 11:48:39 AM
A new C# variant (using .NET Framework 2.0 and inspired by Evgeniy LISP code)

Code: [Select]
using System;
using System.Collections.Generic;
using Autodesk.AutoCAD.ApplicationServices;
using Autodesk.AutoCAD.DatabaseServices;
using Autodesk.AutoCAD.EditorInput;
using Autodesk.AutoCAD.Geometry;
using Autodesk.AutoCAD.Runtime;

namespace ChallengeConnectPoints
{
    public class Connect
    {

        [CommandMethod("Test")]
        public void Test()
        {
            Editor ed = Application.DocumentManager.MdiActiveDocument.Editor;
            PromptDistanceOptions pdo = new PromptDistanceOptions("\nConnection distance: ");
            pdo.AllowNegative = false;
            pdo.AllowZero = false;
            PromptDoubleResult pdr = ed.GetDistance(pdo);
            if (pdr.Status == PromptStatus.OK)
                ConnectPoints(pdr.Value);
        }

        private void ConnectPoints(double d)
        {
            DateTime t0 = DateTime.Now;
            Document doc = Application.DocumentManager.MdiActiveDocument;
            Database db = doc.Database;
            Editor ed = doc.Editor;
            List<Point3d> pts = new List<Point3d>();
            using (Transaction tr = db.TransactionManager.StartTransaction())
            {
                BlockTableRecord btr =
                    (BlockTableRecord)tr.GetObject(db.CurrentSpaceId, OpenMode.ForWrite);
                foreach (ObjectId id in btr)
                {
                    DBPoint pt = tr.GetObject(id, OpenMode.ForRead) as DBPoint;
                    if (pt != null)
                        pts.Add(pt.Position);
                }
                pts.Sort((p1,p2) => p1.X.CompareTo(p2.X));
                int cnt = pts.Count;
                for (int i = 0; i < cnt; i++)
                {
                    Point3d p1 = pts[i];
                    double x1 = p1.X;
                    for (int j = i+1; j < cnt; j++)
                    {
                        Point3d p2 = pts[j];
                        if ((p2.X - x1) > d)
                            break;
                        if (p1.DistanceTo(p2) <= d)
                        {
                            Line l = new Line(p1, pts[j]);
                            btr.AppendEntity(l);
                            tr.AddNewlyCreatedDBObject(l, true);
                        }
                    }
                }
                tr.Commit();
            }
            DateTime t1 = DateTime.Now;
            TimeSpan ts = t1 - t0;
            ed.WriteMessage("\nEllapsed milliseconds : {0}", ts.TotalMilliseconds);
        }
    }
}
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 03, 2010, 01:30:23 PM
Wow, wake up and what do I see but blazing speed. Nice coding Gile, Daniel, and master Evgeniy.  Now I gotta put timers in mine, test your codes, and see how your guys code scale to 100,000 and 500,000 points. Results should be interesting. But first, coffee and lots of it.

edit: btw, can someone post a .Net project of theirs. txs
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 03, 2010, 02:56:31 PM
Added time code to mine (project and ARX attached), and ran Giles for comparison.
Did tests for 10,000, 100,000, 500,000, and 1,000,000 points

Test Results
Quote
Intel Core 2 Q6600 @ 2.4GHz
3.25 GB Ram
Tested Ac2008

Gile's
10,000 pts    -   328 ms - 0.328s
100,000 pts   -  4859 ms - 4.86s
500,000 pts   - 26015 ms - 26.02s
1,000,000 pts - 84500 ms - 84.5s

Paul's
10,000 pts    -    78 ms - 0.078s
100,000 pts   -  1297 ms - 1.30s
500,000 pts   - 14766 ms - 14.77s
1,000,000 pts - 34625 ms - 34.63s

Daniel's
Can't recreate because I don't have MSVS 2010/.Net 4
also note his reported 497 ms is on his smoking hot i7 machine
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 03, 2010, 11:09:23 PM
Paul, Try this ARX , the command is doit... using Evgeniy's optimization
 
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 03, 2010, 11:39:42 PM
Times using Evgeniy's optimization, .NET 4,  running in parallel.
Just a note, but the 50% of the time is spent adding the lines to the DB as this procedure cannot run in parallel

10,000  - 106.7622 Milliseconds
100,000 - 1127.0502 Milliseconds
1,000,000 - 18485.5115 Milliseconds
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 04, 2010, 12:35:31 AM
Paul, Try this ARX , the command is doit... using Evgeniy's optimization

Nice numbers! Code please  :-D

   10,000   -  0.0637 s
  100,000  -  1.7528 s
  500,000  - 10.215 s
1,000,000 - 36.993 s

Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 04, 2010, 12:37:48 AM
Times using Evgeniy's optimization, .NET 4,  running in parallel.
Just a note, but the 50% of the time is spent adding the lines to the DB as this procedure cannot run in parallel

10,000  - 106.7622 Milliseconds
100,000 - 1127.0502 Milliseconds
1,000,000 - 18485.5115 Milliseconds


Typo ?
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 12:47:27 AM
Times using Evgeniy's optimization, .NET 4,  running in parallel.
Just a note, but the 50% of the time is spent adding the lines to the DB as this procedure cannot run in parallel

10,000  - 106.7622 Milliseconds
100,000 - 1127.0502 Milliseconds
1,000,000 - 18485.5115 Milliseconds


Typo ?

no
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 12:49:05 AM
Paul, Try this ARX , the command is doit... using Evgeniy's optimization

Nice numbers! Code please  :-D

   10,000   -  0.0637 s
  100,000  -  1.7528 s
  500,000  - 10.215 s
1,000,000 - 36.993 s




Code: [Select]
static void ArxDist_doit(void)
  {
    double distance = 0.0;
    if(acedGetDist(0,ACRX_T("\nGetDist: "),&distance) != RTNORM)
      return;

    __int64 freq, start, end, diff;
    QueryPerformanceFrequency((LARGE_INTEGER*)&freq);
    QueryPerformanceCounter((LARGE_INTEGER*)&start);

    std::vector<AcGePoint3d> points;
    AcDbBlockTableRecord *pRec;
    AcDbDatabase *pDb = acdbHostApplicationServices()->workingDatabase();

    if(acdbOpenAcDbObject((AcDbObject *&) pRec,pDb->currentSpaceId(), AcDb::kForWrite) == Acad::eOk)
    {
      AcDbBlockTableRecordIterator *pIter = NULL;
      if(pRec->newIterator(pIter) == Acad::eOk)
      {
        for(pIter->start(); !pIter->done(); pIter->step())
        {
          AcDbEntity *pEnt = NULL;
          if(pIter->getEntity(pEnt, AcDb::kForRead) == Acad::eOk)
          {
            AcDbPoint *pPoint = AcDbPoint::cast(pEnt);
            if(pPoint)
              points.push_back(pPoint->position());
            pEnt->close();
          }
        }
        delete pIter;
      }
      std::sort(points.begin(),points.end(),psort);
      size_t len = points.size();
      for (size_t i = 0; i < len; i++)
      {
        for (size_t j = i + 1; j < len; j++)
        {
          if ((points[j].x - points[i].x) > distance)
          {
            break;
          }
          else if ( (points[i].distanceTo(points[j])) <= distance)
          {
            AcDbLine *pLine = new AcDbLine(points[i],points[j]);
            if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
              delete pLine;
            pLine->close();
          }
        }
      }
      pRec->close();
    }
    QueryPerformanceCounter((LARGE_INTEGER*)&end);
    acutPrintf(_T("\n%g Seconds\n"), (double)(end - start) / (double) freq);
  }

Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 12:52:51 AM
 8-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 04, 2010, 01:04:22 AM
Thanks for Daniel and pkohut, your c code all very fast.

But I am not familiar with c, so I only have to do this in Lisp.

Now I use xy grid to sort the point. It seems that now it is quick a little than Evgeiny's. But my code is toooooooo long. Sorry, I need to improve my coding skill. But at least, I think this algorithm is effective.  The next step, I want to study pkohut's kd-tree algorithm,  it seems so amazing.

Code: [Select]
(vl-load-com)
;;entmake line
(defun entmakeline (pt1 pt2 layer)
  (entmake (list (cons 0 "LINE") ;***
(cons 6 "BYLAYER") (cons 8 layer) (cons 10 pt1) ;***
(cons 11 pt2) ;***
(cons 39 0.0) (cons 62 256) (cons 210 (list 0.0 0.0 1.0))
   )
  )
)
;;The idea of gile
(defun getpointscor (/ ent n plst ss)
  (if (setq n -1
    ss (ssget '((0 . "POINT")))
      )
     (while (setq pubtime (car (_VL-TIMES)) ent (ssname ss (setq n (1+ n))))
      (setq plst (cons (cdr (assoc 10 (entget ent))) plst))
    )
  )
)

;;the bounding box of the points set
(defun getboundingboxpointset (a)
  (list (apply 'min (mapcar '(lambda (x) (car x)) a))
  (apply 'max (mapcar '(lambda (x) (car x)) a))
  (apply 'min (mapcar '(lambda (x) (cadr x)) a))
  (apply 'max (mapcar '(lambda (x) (cadr x)) a))
  )
)
;;;divide the points set according to a certain x or y or z space
(defun dividexyz (plst dis vmin vmax stringxyz / a b div end index lst1 res start)
  (setq start vmin
div (+ (fix (/ (- vmax vmin) dis)) 1)
res nil
  )
  (repeat div
;    (grdraw (list start 0 0) (list start 10000 0) 8)
    (setq end (+ start dis)
    index T
  lst1 nil
    )
    (while (and plst index)
      (if (< (setq a (car plst)
                   b (cond ((= stringxyz "x") (car a))
                     ((= stringxyz "y") (cadr a))
                     ((= stringxyz "z") (caddr a))
                     (T nil)
                     )
             )
     end
  )
(setq lst1 (append
     lst1
     (list a)
   )
      plst (cdr plst)
)
(setq index nil)
      )
    )
    (setq res (append
res
(list lst1)
      )
    )
    (setq start (+ start dis))
  )
  res
)
;;;draw short line in each x region
;;;The idea of Leemac and gile
(defun drawshortline (plst limdis / a b tmp)
  (if plst
    (while (setq a (car plst))
      (setq plst (cdr plst)
    tmp plst
      )
      (while (setq b (car tmp))
(setq tmp (cdr tmp))
(if (< (distance a b) limdis)
  (entmakeline a b "temp")
)
      )
    )
  )
)
;;;draw short line in each two Neighbor x region
(defun drawshortline12 (plst1 plst2 limdis / a b x)
  (if (and
plst1
plst2
      )
    (while (setq a (car plst1))
      (foreach x plst2
(if (< (distance a x) limdis)
  (entmakeline a x "temp")
)
      )
      (setq plst1 (cdr plst1))
    )
  )
)

;;draw line between two pointsets and manner
(defun dl12(a b limdis stringabc / ab lst1 lst2)
  (cond ((= stringabc "90")  (setq lst1 (reverse (cdr (reverse a))) lst2 (cdr b)))
        ((= stringabc "0")  (setq lst1 a lst2 b))
        ((= stringabc "-45")  (setq lst1 (cdr a) lst2 (reverse (cdr (reverse b)))))
        ((= stringabc "45")  (setq lst1 (reverse (cdr (reverse a))) lst2 (cdr b)))
        (T nil)
  )
  (setq ab (mapcar '(lambda (x y) (list x y)) lst1 lst2))
  (foreach x ab
      (if x
(drawshortline12 (car x) (cadr x) limdis)
      )
  )        
)
;;draw line between the own pointsets
(defun dl1(pl1 limdis)
        (foreach x pl1
  (drawshortline x limdis)

 
)

;;Connect short line between each two points
;;by qjchen@gmail.com
;; Thanks to ronjonp, Lee Mac, and Gile and alanjt and Evgeniy
;;Version 1.0
(defun c:gap ( / a b bbox limdis plst plst1 plst2)  ;(mylayer "temp")
  ;(starttimer)
  (setq limdis 70 )  ;;The idea of gile
  (setq plst (getpointscor))  ;;The idea of ronjonp
  ;;getting the boundingbox of all the points set
  (setq bbox (getboundingboxpointset plst) )
  ;;;sort the point according to the x coordinate
  (setq plst (vl-sort plst (function (lambda (x1 x2)
       (< (car x1) (car x2))
     )
   )
     )
  )  ;;Divde points list to several region according to the x value=>plst1
  ;;such like ((lst 0, contain x=0~70),(lst 1, contain x=70~140)........)
  (setq plst1 (dividexyz plst limdis (nth 0 bbox) (nth 1 bbox) "x" )
plst2 nil
  )
  ;;Divde each plst1 to several region according to the y value=>plst2
  (foreach x plst1
    (if x
      (setq x (vl-sort x (function (lambda (x1 x2)
     (< (cadr x1) (cadr x2))
   )
)
      )
    plst2 (append
    plst2
    (list (dividexyz x limdis (nth 2 bbox) (nth 3 bbox) "y"))
  )
      )
      (setq plst2 (append
    plst2
    (list nil)
  )
      )
    )
  ) 
  ;;Draw shortline in each x region,such like (0, contain x=0~70))
  (while (setq a (car plst2)
       b (cadr plst2)
       plst2 (cdr plst2)
)
    (if a
      (progn
        (dl1 a limdis)
(dl12 a a limdis "90")
(if b
  (progn
    (dl12 a b limdis "0")
    (dl12 a b limdis "-45")
    (dl12 a b limdis "45")
  )
  (progn
  )
)
      )  ;else continue
     
    )
  )
  (dl1 a limdis)
  (dl12 a a limdis "90")
  (princ (strcat "\n " (rtos (/ (- (car (_VL-TIMES)) pubtime) 1000.) 2 4) "sec."))
  ;(endtimer)
)





Now In my PC, it is about 0.79s for 10,000 points, and 160s for 500,000 points.
Evgeniy's code is about 1.6s for  10,000 points, and 288s for 500,000 points.

Sure master Evgeniy can easily improve the code and quick quick than me, but  I am just try to study this funny algorithm.  :-P

The algorithm is as following picture.



Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 04, 2010, 01:07:10 AM
Paul, Try this ARX , the command is doit... using Evgeniy's optimization

Nice numbers! Code please  :-D

   10,000   -  0.0637 s
  100,000  -  1.7528 s
  500,000  - 10.215 s
1,000,000 - 36.993 s

Nice code! For balls out speed I think this is the winner, and for a smart brute force routine this kicks azzstroids. You could squeeze even more by enabling OpenMP and surround the work loop with the proper #pragma omp statement, there for getting the most out of available processors.  Noticed that somewhere along the line the times converged between this one and mine, probably in the 800,000 range.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 01:42:18 AM
wait not done yet...
ARX
1,000,000
14.569079 Seconds
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 02:01:56 AM
last one ... maybe  :-D

Code: [Select]
static inline bool psort(const AcGePoint3d &a, const AcGePoint3d &b)
  {
    if(a.x == b.x)
      return a.y < b.y;
    return a.x < b.x;
  }

  static inline double distsqrd(const AcGePoint3d &a, const AcGePoint3d &b)
  {
    double dx = a.x-b.x;
    double dy = a.y-b.y;
    double dz = a.z-b.z;
    return dx*dx+dy*dy+dz*dz;
  }

  static void ArxDist_doit(void)
  {
    double distance = 0.0;

    if(acedGetDist(0,ACRX_T("\nGetDist: "),&distance) != RTNORM)
      return;

    const double distancesqrd = pow(distance,2) + 1.192092896e-07F;

    __int64 freq, start, end, diff;
    QueryPerformanceFrequency((LARGE_INTEGER*)&freq);
    QueryPerformanceCounter((LARGE_INTEGER*)&start);

    std::vector<AcGePoint3d> points;
    AcDbBlockTableRecord *pRec;
    AcDbDatabase *pDb = acdbHostApplicationServices()->workingDatabase();

    if(acdbOpenAcDbObject((AcDbObject *&) pRec,pDb->currentSpaceId(), AcDb::kForWrite) == Acad::eOk)
    {
      AcDbBlockTableRecordIterator *pIter = NULL;
      if(pRec->newIterator(pIter) == Acad::eOk)
      {
        for(pIter->start(); !pIter->done(); pIter->step())
        {
          AcDbEntity *pEnt = NULL;
          if(pIter->getEntity(pEnt, AcDb::kForRead) == Acad::eOk)
          {
            AcDbPoint *pPoint = AcDbPoint::cast(pEnt);
            if(pPoint)
              points.push_back(pPoint->position());
            pEnt->close();
          }
        }
        delete pIter;
      }
      std::sort(points.begin(),points.end(),psort);
      size_t len = points.size();
      for (size_t i = 0; i < len; i++)
      {
        for (size_t j = i + 1; j < len; j++)
        {
          if ((points[j].x - points[i].x) > distance)
          {
            break;
          }
          else if ( distsqrd(points[i],points[j]) < distancesqrd)
          {
            AcDbLine *pLine = new AcDbLine(points[i],points[j]);
            if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
              delete pLine;
            pLine->close();
          }
        }
      }
      pRec->close();
    }
    QueryPerformanceCounter((LARGE_INTEGER*)&end);
    acutPrintf(_T("\n%g Seconds\n"), (double)(end - start) / (double) freq);
  }

Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 02:06:29 AM
gile, your lambda expression to sort the points is awesome!
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 04, 2010, 02:25:18 AM
last one ... maybe  :-D

There ain't much skin left on that cat.  :-D

Nice clean code and then -
Code: [Select]
const double distancesqrd = pow(distance,2) + 1.192092896e-07F;Eww! The fabs function is like 1 clock cycle, so your better to use it and an epsilon value of 1e-8 for your comparisons.  Otherwise NICE.


Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ElpanovEvgeniy on April 04, 2010, 03:23:37 AM
Sure master Evgeniy can easily improve the code and quick quick than me, but  I am just try to study this funny algorithm.  :-P

The algorithm is as following picture.

An interesting algorithm, similar to I use in many of its programs.
Hopefully, tonight will be free time and I will write the new version of the code.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gile on April 04, 2010, 03:34:37 AM
gile, your lambda expression to sort the points is awesome!

Thanks.
Maybe a 'LISP like' way of thinking/writing
Code: [Select]
(vl-sort pts '(lambda (p1 p2) (< 'car p1) (car p2))))
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ElpanovEvgeniy on April 04, 2010, 03:48:53 PM
new version:

Code: [Select]
(defun el-1 (l)
 (mapcar (function (lambda (a) (cons (fix (/ (car a) d)) (cons (fix (/ (cadr a) d)) a))))
         l
 )
)
(defun el-2 (l / a x)
 (setq x (caar l)
       y (cadar l)
       a (list (list (cdar l)))
       l (cdr l)
 )
 (while l
  (while (and l (= x (caar l)))
   (setq a (cons (cons (cdar l) (car a)) (cdr a))
         l (cdr l)
   )
  )
  (setq y (min y (caaar a)))
  (cond (l
         (repeat (- x (caar l) 1) (setq a (cons (list (setq x (1- x))) a)))
         (setq x (1- x)
               a (cons (list (cdar l)) a)
         )
        )
  )
 )
 a
)
(defun el-3 (l y / a)
 (while l
  (repeat (- (caar l) y)
   (setq a (cons nil a)
         y (1+ y)
   )
  )
  (setq a (cons (list (cdar l)) a)
        l (cdr l)
  )
  (while (and l (= y (caar l)))
   (setq a (cons (cons (cdar l) (car a)) (cdr a))
         l (cdr l)
   )
  )
  (setq y (1+ y))
 )
 (reverse a)
)
(defun el-4 (l / A LL X)
 (while (setq a (car l)
              l (cdr l)
        )
  (foreach b l
   (if (< 0. (distance a b) d)
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 b)))
   )
  )
 )
)
(defun el-5 (l la / A)
 (foreach a l
  (foreach b la
   (if (< 0. (distance a b) d)
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 b)))
   )
  )
 )
)
(defun c:el (/ D E I L S TI Y)
 (setq ti (car (_VL-TIMES))
       i  -1
       d  70.
       s  nil
       s  (ssget "_x" '((0 . "point")))
       l  nil
 )
 (while (setq e (ssname s (setq i (1+ i)))) (setq l (cons (cdr (assoc 10 (entget e))) l)))
 (setq l (mapcar (function (lambda (a) (el-3 a y)))
                 (el-2 (vl-sort (el-1 l)
                                (function (lambda (a b)
                                           (if (= (car a) (car b))
                                            (>= (cadr a) (cadr b))
                                            (> (car a) (car b))
                                           )
                                          )
                                )
                       )
                 )
         )
 )
 (mapcar
  (function (lambda (a) (mapcar (function el-4) a) (mapcar (function el-5) a (cdr a))))
  l
 )
 (mapcar (function (lambda (a b)
                    (mapcar (function el-5) a (mapcar (function append) b (cdr b)))
                    (mapcar (function el-5) a b)
                   )
         )
         l
         (cdr l)
 )
 (princ (strcat "\n " (rtos (/ (- (car (_VL-TIMES)) ti) 1000.) 2 4) "sec."))
 (princ)
)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ElpanovEvgeniy on April 04, 2010, 04:06:30 PM
new version:

recommend testing after compiling:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gile on April 04, 2010, 05:21:16 PM
Awesome Evgeniy (as usual) it's so fast compared to the first LISP codes in this thread. :kewl:

I'll have to study your algorithm...
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ElpanovEvgeniy on April 04, 2010, 05:28:47 PM
Awesome Evgeniy (as usual) it's so fast compared to the first LISP codes in this thread. :kewl:

I'll have to study your algorithm...

I used an algorithm similar to the picture of the messages Chen

(http://www.theswamp.org/index.php?action=dlattach;topic=32874.0;attach=15418;image)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 04, 2010, 07:48:26 PM
Super cool, Evgeniy, very very quick.

Now about a 500,000 points dwg generated by the following code.

Your code is about only 48s (vlx) or 58s (lsp), my code is about 160s.

sure, Daniel and pkohut and Gile 's  code is much quick. But maybe C can 10 times quick than Lisp?

I need to deep study for your code structure, Thank you :)

---------------------------------

Just find a bugs, when in a only two points dwg, p1=(60,0,0) p2=(180,0,0), the code seems to get error, it seems that the error happen in the sortting data into xy grid.

----------------------------------

Recently, I buy the book you recommend to me :"Structure and interpretation of computer programs" by Harold Abeleson in MIT.

It is a rather good book for LISP, I am learning now about the recursion and so on.

I read your article about "recursion" in dwg.ru and learn a lot, could you post it here also for theswamp friends.   :-D

Code: [Select]
;;from internet
(defun random ()
  (setq seed (if seed
       (rem (+ (* seed 15625.7) 0.21137152) 1)
       0.3171943
     )
  )
)
(defun random-n (n)
  (* n (random))
)
(defun entmakepoint (pt layer)
  (entmake (list (cons 0 "POINT")
                 (cons 8 layer);***
(cons 6 "BYLAYER")
(cons 10 pt) ;***
(cons 39 0.0)
(cons 50 0.0)
(cons 62 256)
(cons 210 (list 0.0 0.0 1.0))
   )
  )
)
;;generate random points
(defun c:test1 ()
  (repeat 500000
    (entmakepoint (list (random-n 100000) (random-n 60000) 0) "0")
  )
)

Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 04, 2010, 07:50:28 PM
last one ... maybe  :-D

Times a staggering -
Quote
10,000     -   0.0449 s
100,000    -  0.885 s
500,000    -  5.829 s
1,000,000 - 19.376 s

edit: what is the project optimizations? I say the numbers last night, but being so tired they didn't register.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 04, 2010, 11:19:22 PM
I was able to cut another 25% of the time using a bit of caching. only has an effect on larger sets


@ 1,000,000
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 05, 2010, 12:08:28 AM
I was able to cut another 25% of the time using a bit of caching. only has an effect on larger sets


@ 1,000,000

Yep, 15.92 seconds over here.  Nice.

I reorg'ed mine earlier (prior to seeing these monster numbers) and was pumped that I shaved 5 seconds so it runs about 29.5 s. While writing a message to brag, fired up your code so I could get some current numbers and BAM 19 freaking seconds, and now 15 seconds.  Tremendous job...goes for everyone else too.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: MP on April 05, 2010, 12:32:43 AM
Just to throw another challenge to you keeners ... fill a volume with random 3D points rather than an area with 2D points, connecting them with 3D lines wherever the 3D distance is less than or equal to a specified value. (http://www.theswamp.org/screens/mp/wave.gif)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 05, 2010, 12:40:38 AM
I was able to cut another 25% of the time using a bit of caching. only has an effect on larger sets


@ 1,000,000

Yep, 15.92 seconds over here.  Nice.

I reorg'ed mine earlier (prior to seeing these monster numbers) and was pumped that I shaved 5 seconds so it runs about 29.5 s. While writing a message to brag, fired up your code so I could get some current numbers and BAM 19 freaking seconds, and now 15 seconds.  Tremendous job...goes for everyone else too.


the only thing I did was add temps inside the loop.. I thought the compiler would have done this but..

Code: [Select]
...
    for (size_t i = 0; i < len; i++)
      {
        const AcGePoint3d &a = points[i];
        for (size_t j = i + 1; j < len; j++)
        {
          const AcGePoint3d &b = points[j];
          if ((b.x - a.x) > distance)
          {
            break;
          }
          else if ( distsqrd(a,b) < distancesqrd)
          {
            AcDbLine *pLine = new AcDbLine(a,b);
            if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
              delete pLine;
            pLine->close();
          }
        }
      }
...
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 05, 2010, 01:05:23 AM
Just to throw another challenge to you keeners ... fill a volume with random 3D points rather than an area with 2D points, connecting them with 3D lines wherever the 3D distance is less than or equal to a specified value. (http://www.theswamp.org/screens/mp/wave.gif)

This is my reorg'ed version (2).  To meet your requirements (I think) the only change needed just now was to add and use a Distance3DSquared function, everything else is already in place.

Code notes: This version follows Daniel's version and layout to facilitate post mortem comparisions.

Just replace the contents acrxEntryPoint.cpp in the VC++ project at
http://www.theswamp.org/index.php?topic=32874.msg383657#msg383657
and recompile.

Code: [Select]
// acrxEntryPoint.cpp
// Copyright 2010 Paul Kohut
//

//-----------------------------------------------------------------------------
//----- acrxEntryPoint.h
//-----------------------------------------------------------------------------
#include "StdAfx.h"
#include "resource.h"
#include "KdTree.h"
#include "LapseTime.h"
#include <list>
#include <algorithm>

//-----------------------------------------------------------------------------
#define szRDS _RXST("rpk")

struct PT3D{
    double x, y, z;
};

inline double Distance2DSquared(PT3D * pt1, PT3D * pt2)
{
    return ((pt2->x - pt1->x) * (pt2->x - pt1->x) + (pt2->y - pt1->y) * (pt2->y - pt1->y));
}

inline double Distance3DSquared(PT3D * pt1, PT3D * pt2)
{
    return ((pt2->x - pt1->x) * (pt2->x - pt1->x) + (pt2->y - pt1->y) * (pt2->y - pt1->y) + (pt2->z - pt1->z) * (pt2->z - pt1->z));
}

void ConnectDotsWithin(AcDbBlockTableRecord * pRcd, CKdTree & kdTree, double dRadius)
{
    RESULT_SET * pResAllPoints = kdTree.NearestRange(0.0, 0.0, 0.0, HUGE_VAL);
    while(pResAllPoints->riter) {
        RESULT_SET * pRes = kdTree.NearestRange(pResAllPoints->riter->item->pos[0],
            pResAllPoints->riter->item->pos[1], pResAllPoints->riter->item->pos[2], dRadius);
        while(pRes->riter && pRes->size > 1) {
            if(!pRes->riter->item->data && Distance3DSquared((PT3D *) pResAllPoints->riter->item->pos, (PT3D *) pRes->riter->item->pos) > 0.0) {
                AcDbLine * pLine = new AcDbLine(asPnt3d(pResAllPoints->riter->item->pos), asPnt3d(pRes->riter->item->pos));
                if(pRcd->appendAcDbEntity(pLine) != Acad::eOk) {
                    delete pLine;
                } else {
                    pLine->close();
                }
            }
            pRes->riter = pRes->riter->next;
        }
        kdTree.ResultFree(pRes);
        pResAllPoints->riter->item->data = (void *) 1;
        pResAllPoints->riter = pResAllPoints->riter->next;
    }
    kdTree.ResultFree(pResAllPoints);
}

void ConnectDotsOptimized(void)
{
    double dRadius = 0.0;
    if(acedGetDist(0, _T("\nMax segment length: "), &dRadius) != RTNORM)
        return;

    CKdTree kdTree;
    kdTree.Create(3);

    CLapseTime timer;

    AcDbDatabase * pDb = acdbHostApplicationServices()->workingDatabase();
    AcDbBlockTableRecord * pRec;
    if(acdbOpenAcDbObject((AcDbObject *&) pRec, pDb->currentSpaceId(), AcDb::kForWrite) == Acad::eOk) {
        AcDbBlockTableRecordIterator * pIter = NULL;
        if(pRec->newIterator(pIter) == Acad::eOk) {
            AcGePoint3d pt;
            for(pIter->start(); !pIter->done(); pIter->step()) {
                AcDbEntity * pEnt = NULL;
                if(pIter->getEntity(pEnt, AcDb::kForRead) == Acad::eOk) {
                    AcDbPoint * pPoint = AcDbPoint::cast(pEnt);
                    if(pPoint) {
                        pt = pPoint->position();
                        kdTree.Insert(pt.x, pt.y, pt.z, NULL);
                    }
                    pEnt->close();
                }
            }
            delete pIter;
        }
        ConnectDotsWithin(pRec, kdTree, dRadius);
        pRec->close();
    }
    acutPrintf(_T("\n%.3f milliseconds."), timer.LapseTimeMS());
}

//////////////////////////////////////////////////////////////////////////
// Begin ARX Initialization
//////////////////////////////////////////////////////////////////////////

//-----------------------------------------------------------------------------
//----- ObjectARX EntryPoint
class CConnectDotsApp: public AcRxArxApp {

public:
    CConnectDotsApp (): AcRxArxApp () {}

    virtual AcRx::AppRetCode On_kInitAppMsg (void *pkt){
        // TODO: Load dependencies here

        // You *must* call On_kInitAppMsg here
        AcRx::AppRetCode retCode =AcRxArxApp::On_kInitAppMsg (pkt) ;

        // TODO: Add your initialization code here

        return (retCode) ;
    }

    virtual AcRx::AppRetCode On_kUnloadAppMsg (void *pkt){
        // TODO: Add your code here

        // You *must* call On_kUnloadAppMsg here
        AcRx::AppRetCode retCode =AcRxArxApp::On_kUnloadAppMsg (pkt) ;

        // TODO: Unload dependencies here

        return (retCode) ;
    }

    virtual void RegisterServerComponents (){
    }


    // - rpkConnectDots.ConnectDots command (do not rename)
    static void rpkConnectDotsConnectDots(void)
    {
        ConnectDotsOptimized();
    }
} ;

//-----------------------------------------------------------------------------
IMPLEMENT_ARX_ENTRYPOINT(CConnectDotsApp)

ACED_ARXCOMMAND_ENTRY_AUTO(CConnectDotsApp, rpkConnectDots, ConnectDots, ConnectDots, ACRX_CMD_TRANSPARENT, NULL)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on April 05, 2010, 09:06:23 AM
This version I posted here
http://www.theswamp.org/index.php?topic=32874.msg383742#msg383742
seems to work with 3d
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on April 05, 2010, 02:02:30 PM
I was able to cut another 25% of the time using a bit of caching. only has an effect on larger sets

@ 1,000,000

Using multiple cores to make Daniel's flame thrower code even hotter (AC2008 binary attached)

The multi core times are with OMP. It's interesting that the referenced temp variables caused a significant slow down on OMP, and changing them to objects had a small performance hit compared to not using temp variables. Maybe there is an OMP setting that could help, but OMP can get complicated and this is beyond what I know. Not using temp variables with OMP always ran faster than the version with temp variables.

Quote
Single core
No temps 19.304 s
Temps - 15.685 s

Multi core use
No temps - 14.704 s (about 65% of 4 core processor utilization)
With temps as object - 15.790 s (about 65% 4 C PU)
With temps as reference type - 20.770 s (about 50% 4 C PU)
With temps as pointer - 20.074 s (about 50% 4 C PU)

A good OMP reference
http://www.codeguru.com/cpp/cpp/cpp_mfc/general/print.php/c15419

OMP enabled code listing, this is the fastest version. Note the use of #pragma omp critical, this ensures that adding the entity to the DB is done by a single thread (otherwise Autocad vaporizes immediately), and it's critical that the newed AcDbLine is inside the block too, otherwise Autocad will throw a thread exception when the function exits.
Code: [Select]
#pragma omp parallel
            {           
#pragma omp for
                for (long i = 0; i < len; i++)
                {
                    for (long j = i + 1; j < len; j++)
                    {
                        if ((points[j].x - points[i].x) > distance)
                        {
                            break;
                        }
                        else if ( distsqrd(points[i],points[j]) < distancesqrd)
                        {
#pragma omp critical
                            {
                                AcDbLine *pLine = new AcDbLine(points[i],points[j]);
                                if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
                                    delete pLine;
                                pLine->close();
                            }
                        }
                    }
                }
            }

Full ArxDist_doit replacement for playing with parallel processing.  #if 1 = no temp variables, #if 0 = temp variables
Code: [Select]
#include <omp.h>
...
    static void ArxDist_doit(void)
    {
        double distance = 0.0;

        if(acedGetDist(0,_T("\nGetDist: "),&distance) != RTNORM)
            return;

        const double distancesqrd = pow(distance,2) + 1.192092896e-07F;

        __int64 freq, start, end, diff;
        QueryPerformanceFrequency((LARGE_INTEGER*)&freq);
        QueryPerformanceCounter((LARGE_INTEGER*)&start);

        std::vector<AcGePoint3d> points;
        AcDbBlockTableRecord *pRec;
        AcDbDatabase *pDb = acdbHostApplicationServices()->workingDatabase();

        if(acdbOpenAcDbObject((AcDbObject *&) pRec,pDb->currentSpaceId(), AcDb::kForWrite) == Acad::eOk)
        {
            AcDbBlockTableRecordIterator *pIter = NULL;
            if(pRec->newIterator(pIter) == Acad::eOk)
            {
                for(pIter->start(); !pIter->done(); pIter->step())
                {
                    AcDbEntity *pEnt = NULL;
                    if(pIter->getEntity(pEnt, AcDb::kForRead) == Acad::eOk)
                    {
                        AcDbPoint *pPoint = AcDbPoint::cast(pEnt);
                        if(pPoint)
                            points.push_back(pPoint->position());
                        pEnt->close();
                    }
                }
                delete pIter;
            }
            std::sort(points.begin(),points.end(),psort);
            size_t len = points.size();

#if 1
            // No temp variables
#pragma omp parallel
            {           
#pragma omp for
                for (long i = 0; i < len; i++)
                {
                    for (long j = i + 1; j < len; j++)
                    {
                        if ((points[j].x - points[i].x) > distance)
                        {
                            break;
                        }
                        else if ( distsqrd(points[i],points[j]) < distancesqrd)
                        {
#pragma omp critical
                            {
                                AcDbLine *pLine = new AcDbLine(points[i],points[j]);
                                if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
                                    delete pLine;
                                pLine->close();
                            }
                        }
                    }
                }
            }
#else
            // With Temp variables as object
#pragma omp parallel
            {           
#pragma omp for
                for (long i = 0; i < len; i++)
                {
                    const AcGePoint3d a = points[i];
                    for (long j = i + 1; j < len; j++)
                    {
                        const AcGePoint3d b = points[j];
                        if ((b.x - a.x) > distance)
                        {
                            break;
                        }
                        else if ( distsqrd(a,b) < distancesqrd)
                        {
#pragma omp critical
                            {
                                AcDbLine *pLine = new AcDbLine(a,b);
                                if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
                                    delete pLine;
                                pLine->close();
                            }
                        }
                    }
                }
            }

#endif
            pRec->close();
        }
        QueryPerformanceCounter((LARGE_INTEGER*)&end);
        acutPrintf(_T("\n%g Seconds\n"), (double)(end - start) / (double) freq);
    }

} ;
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: ElpanovEvgeniy on April 08, 2010, 03:56:59 AM
new version:

Code: [Select]
(defun el-1 (l / A)
 (while (setq a (car l)
              l (cdr l)
        ) ;_  setq
  (foreach b l
   (if (< 0. (distance a b) d)
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 b)))
   ) ;_  if
  ) ;_  while
 ) ;_  while
) ;_  defun

(defun el-2 (l la)
 (foreach a l
  (foreach b la
   (if (< 0. (distance a b) d)
    (entmakex (list '(0 . "LINE") (cons 10 a) (cons 11 b)))
   ) ;_  if
  ) ;_  foreach
 ) ;_  foreach
) ;_  defun
(defun c:el (/ A AA B C D E L S TI X Y YY)
 ;;(repeat 3(c:el))
 (setq ti (car (_VL-TIMES))
       a  -1
       d  70.
       s  (ssget "_x" '((0 . "point")))
       y  (mapcar (function (lambda (a) (atoi (rtos (/ a d) 2 0)))) (getvar "EXTMIN"))
       x  (if (minusp (car y))
           (- (car y))
           0
          ) ;_  if
       y  (if (minusp (cadr y))
           (- (cadr y))
           0
          ) ;_  if
       yy y
 ) ;_  setq
 (while (setq e (ssname s (setq a (1+ a))))
  (setq e (cdr (assoc 10 (entget e)))
        e (list (car e) (cadr e))
        l (cons (cons (fix (+ x (/ (car e) d))) (cons (fix (+ y (/ (cadr e) d))) e)) l)
  ) ;_  setq
 ) ;_  while
 (setq l (vl-sort l (function (lambda (a b) (<= (car a) (car b)))))
       a (list (cdar l))
       l (cdr l)
 ) ;_  setq
 (while (and l (= x (caar l)))
  (setq a (cons (cdar l) a)
        l (cdr l)
  ) ;_  setq
 ) ;_  while
 (setq a (vl-sort a (function (lambda (a b) (<= (car a) (car b)))))
       x (1+ x)
 ) ;_  setq
 (while a
  (repeat (- (caar a) y)
   (setq c (cons nil c)
         y (1+ y)
   ) ;_  setq
  ) ;_  repeat
  (while (and a (= y (caar a)))
   (while (and a (= y (caar a)))
    (setq b (cons (cdar a) b)
          a (cdr a)
    ) ;_  setq
   ) ;_  while
   (el-1 b)
   (el-2 b (car c))
   (setq c (cons b c)
         b nil
         y (1+ y)
   ) ;_  setq
  ) ;_  while
 ) ;_  while
 (setq c (reverse c))
 (while l
  (setq aa c
        a  (list (cdar l))
        y  yy
        l  (cdr l)
        b  nil
        c  nil
  ) ;_  setq
  (while (and l (= x (caar l)))
   (setq a (cons (cdar l) a)
         l (cdr l)
   ) ;_  setq
  ) ;_  while
  (setq a (vl-sort a (function (lambda (a b) (<= (car a) (car b)))))
        x (1+ x)
  ) ;_  setq
  (while a
   (repeat (- (caar a) y)
    (setq c (cons nil c)
          y (1+ y)
    ) ;_  setq
   ) ;_  repeat
   (while (and a (= y (caar a)))
    (while (and a (= y (caar a)))
     (setq b (cons (cdar a) b)
           a (cdr a)
     ) ;_  setq
    ) ;_  while
    (el-1 b)
    (el-2 b (car c))
    (setq c (cons b c)
          b nil
          y (1+ y)
    ) ;_  setq
   ) ;_  while
  ) ;_  while
  (setq c (reverse c))
  (mapcar (function el-2) c (mapcar (function append) aa (cdr aa)))
  (mapcar (function el-2) c aa)
 ) ;_  while
 (princ (strcat "\n " (rtos (/ (- (car (_VL-TIMES)) ti) 1000.) 2 4)))
;;; (vl-cmdf "_erase" (ssget "_x" '((0 . "line"))) "")
 (princ)
) ;_  defun
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on April 08, 2010, 08:43:40 AM
Dear Evgeniy, thank you very much.

Now the little bug in last version is fixed.

And the speed is more quick.

About 42s (vlx)  ---> last time 48s (vlx) for 500,000 pts.~

Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on April 18, 2010, 02:20:01 AM
Hi,here is  my routine.
I used recursion.
Code: [Select]
;;; the main routine
;;; 主程序
(defun C:test (/ TOL sl to ss Pts T0 pp)
  ;;set or input Tolerance
  ;; 输入容差
  (if (not (setq TOL (getdist "\nPlease input the tolerance:")))
    (setq TOL 70)
  )
  ;;也可以用其他方式取得点集
  ;;Get the points
  (setq sl '((0 . "POINT")))
  (setq ss (ssget sl))
  (setq Pts (getpt ss))
  ;;按照X值排序
  ;;sort by X value
  (setq t0 (getvar "TDUSRTIMER"))
  (setq Pts (sortx Pts))
  (princ "\nThe time for sorting these points:")
  (princ (* (- (getvar "TDUSRTIMER") t0) 86400))
  (princ "s.")
  ;;函数用时估算,以了解函数性能
  ;;how long it will take
  (setq t0 (getvar "TDUSRTIMER"))
  (setq pp (f2 Pts TOL))
  (princ "\nThe time for finding the pairs of points:")
  (princ (* (- (getvar "TDUSRTIMER") t0) 86400))
  (princ "s.")
  (if pp
    ;;画最短距离的点对集的连线,可能有多条
    ;;draw the line between these pairs.
    (foreach n pp
      (entmake
(list
  '(0 . "LINE")
  (cons 10 (car n))
  (cons 11 (cadr n))
  (cons 62 1)
)
      )
    )
  )
  (princ)
)
;;; 取点函数,其中i为点的编号
;;; get points.
(defun getpt (ss / i l a b c)
  (setq i 0)
  (if ss
    (repeat (sslength ss)
      (setq a (ssname ss i))
      (setq b (entget a))
      (setq c (cdr (assoc 10 b)))
      (setq l (cons c l))
      (setq i (1+ i))
    )
  )
  (reverse l)
)
;;; 从0到k的点集
;;; the elments of points from 0 to k
(defun Array (Pts k / L1 L2)
  (setq L1 Pts)
  (repeat k
    (setq L2 (cons (car L1) L2))
    (setq L1 (cdr L1))
  )
  (list (reverse L2) L1)
)
;;; 对X排序
;;; sort points by X value.
(defun sortX (pts)
  (vl-sort pts (function (lambda (e1 e2) (< (car e1) (car e2))))
  )
)
;;; 在带形区域查找
;;; search these points in the Tolerance
(defun searchX (Pts x1 x2 / pp)
  (foreach n Pts
    (if (and (>= (car n) x1)
     (<= (car n) x2)
)
      (setq pp (cons n pp))
    )
  )
  (reverse pp)
)

;;; 用递归解决
;;; use recursion to solve this problem (divide-and-conquer)

(defun f2 (Pts TOL / l m n a b pt1 pt2 ppp Pts0 Pts1 Pts2 Pts3 Pts4
   midptx mind1 mind2 qqq)
  (setq l (length Pts))
  (cond
    ((= l 1) nil)
    ((= l 2)
     ;; 两点还用说
     ;; judge directly if two points
     (setq pt1 (car Pts))
     (setq pt2 (cadr Pts))
     (if (and (equal (car pt1) (car pt2) TOL)
      (equal (cadr pt1) (cadr pt2) TOL)
      (< (distance pt1 pt2) TOL)
)
       (list Pts)
     )
    )   
    ((> l 2)
     (setq n (/ l 2)
   m (- l n)
     )
     ;;分治
     ;;divide
     (setq Pts0 (Array Pts n))
     (setq Pts1 (car Pts0))
     (setq Pts2 (cadr Pts0))
     ;;递归左边
     ;;recurse left
     (setq mind1 (f2 Pts1 TOL))
     ;;递归右边
     ;;recurse right
     (setq mind2 (f2 Pts2 TOL))
     ;;合并左右集合
     ;;merge them
     (setq ppp (append mind1 mind2))
     ;;对带形区域的合并
     ;;consider the points in the Tolerance
     (setq midptx (caar Pts2))
     (setq a (- midptx TOL)
   b (+ midptx TOL)
     )
     (if (setq Pts3 (searchX Pts1 a midptx))
       (if (setq Pts4 (searchx Pts2 midptx b))
(foreach n Pts3
   (foreach e Pts4
     (if (equal (cadr n) (cadr e) TOL)
       (if (<= (distance n e) TOL)
(setq ppp (cons (list n e) ppp))
       )
     )
   )
)
       )
     )
     ppp
    )
  )
)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on April 18, 2010, 02:32:01 AM
by the way, if it is compiled, it runs 10 times faster.

and compare mine to ElpanovEvgeniy's ,in this sample, if the tolerance is smaller or very small,my algorithm has more advantages.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on May 08, 2010, 09:40:03 AM
I improved my program,so now it's faster ,and I made an arx routine. it runs 10 times faster than VLX.
I has compiled the Arx code .see the attachments, maybe it's faster than pkohut's.
I learned a lot from pkohut's. KD-tree,is advanced and hard to understand.
Any ideas, thanks.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on May 08, 2010, 10:35:34 AM
Highflybird, my dear friend.

Thank you for your good program, it is rather quick.

Your arx code is about 0.5s for 100,000 pts in My PC, and about 3.393s for 500,000 pts in My PC.

I will try 1,000,000 pts later.

I cant load pkohut's arx in my AUTOCAD 2004 and 2007. So I cant test.

and your Lisp code is also very quick,  yours and Evgeniy's lisp is the two most quick lisp code.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on May 08, 2010, 11:20:52 AM
I improved my program,so now it's faster ,and I made an arx routine. it runs 10 times faster than VLX.
I has compiled the Arx code .see the attachments, maybe it's faster than pkohut's.
I learned a lot from pkohut's. KD-tree,is advanced and hard to understand.
Any ideas, thanks.

Excellent work! It's very fast  8-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on May 08, 2010, 11:25:54 AM
 I compiled a faster  arx version.
if somebody has visual studio 2005,or 2008, he can compile my attachment to a version for autocad2007-2010

(I corrected a bug in my code. -(GMT +8) 2010.05.09. 05:00 ,Please re-download.)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on May 11, 2010, 05:46:31 AM
Your CreatLine function is really heavy, you ought to move all the code needed to get the AcDbBlockTableRecord out to the top level function and pass a pointer.

Code: [Select]
Acad::ErrorStatus CreatLine(const AcGePoint3d & ptStart,const AcGePoint3d & ptEnd
                             AcDbBlockTableRecord *pBlockTableRecord)
{
    AcDbLine *pLine = new AcDbLine(ptStart, ptEnd);
    es=pBlockTableRecord->appendAcDbEntity(pLine);
    pLine->close();
    // pBlockTableRecord->close(); close this later
    return es;
}
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on May 12, 2010, 08:35:02 AM
Your CreatLine function is really heavy, you ought to move all the code needed to get the AcDbBlockTableRecord out to the top level function and pass a pointer.
...
Daniel,you  are right.Actually ,I realized this question when I begun to optimize my code,but in that time , I was busy, so I didn't post my modified code.
Now ,I finished it. At the begining , I wanted to complish a C++,not just for ARX,so I used the STL, and AcGePoint3d replaced by point2d,and used the deque to store the result(so it will take a longer time). I designed my quick sort function.(because it's faster). And then, I found a bug in my program. I corrected it.
Also I found a bug in Pkohut's,(he doesn't consider the duplicate points.)
Thanks your help,Daniel,I am a C++ beginner,but I found a lot of fun. Now I crazy about it.

Now I attached the new code.
p.s  I finished this program I felt very happy,because it's my algorithm,it's made by myself
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on May 14, 2010, 05:01:42 AM
Code: [Select]
;;; 用递归解决
;;; use recursion to solve this problem (divide-and-conquer)

(defun f2 (Pts TOL / l m n pt1 pt2 Pts1 Pts2 Pts3 Pts4 Mx x1 x2)
  (setq l (length Pts))
  (cond
    ((= l 1) nil)
    ((= l 2)
     ;; 两点还用说
     ;; judge directly if 2 points
     (setq pt1 (car Pts))
     (setq pt2 (cadr Pts))
     (if (and (equal (car pt1) (car pt2) TOL)
      (equal (cadr pt1) (cadr pt2) TOL)
      (<= (distance pt1 pt2) TOL)
)
       (setq pairs (cons pt1 pairs) ;add the start point into pairs;
     pairs (cons pt2 pairs) ;add the end point into pairs;
       )
     )
    )
    ((> l 2)
     (setq n (/ l 2)
   m (- l n)
     )
     (setq Pts1 (Array Pts n)) ;Divide
     (setq Pts2 (cadr Pts1)) ;R points
     (setq Pts1 (car Pts1)) ;L points
     (f2 (reverse Pts1) TOL) ;Recurse Left
     (f2 Pts2 TOL) ;Recurse Right
     (setq Mx (caar Pts2)) ;Midpoint
     (setq x1 (- Mx tol)) ;L points in the TOL
     (setq x2 (+ Mx tol)) ;R points in the TOL
     (while (and pts2 (<= (caar pts2) x2)) ;gather the points in the Right
       (setq Pts3 (cons (cons (car pts2) 1) Pts3)) ;1 means in the Right
       (setq pts2 (cdr pts2))
     )
     (setq pts3 (reverse pts3))
     (while (and pts1 (>= (caar pts1) x1)) ;gather the points in the Left
       (setq Pts3 (cons (cons (car pts1) 0) Pts3)) ;0 means in the Left
       (setq pts1 (cdr pts1))
     )
     (setq Pts3 (sorty pts3 tol)) ;sort these points by Y value
     (setq pts4 (cdr pts3)) ;consider these points one by one
     (if pts4
       (while pts3
(setq pt1 (caar pts3))
(setq pt2 (caar pts4))
(while (and pts4 (<= (- (cadr pt2) (cadr pt1)) tol))
   (if (and (/= (cdar pts3) (cdar pts4)) ;different side;
    (equal (car pt1) (car pt2) tol) ;x value in tol;
    (<= (distance pt1 pt2) tol) ;distance <= tol;
       )
     (setq pairs (cons pt1 pairs) ;add the start point into pairs;
   pairs (cons pt2 pairs) ;add the end point into pairs;
     )
   )
   (setq pts4 (cdr pts4))
   (setq pt2 (caar pts4))
)
(setq pts3 (cdr pts3))
(setq pts4 (cdr pts3))
       )
     )
     Pairs
    )
  )
)
Ok ,this is the main code. I considered the duplicate points. So this program can be used for removing duplicate elements, or finding those closed points,etc.  these kinds of finding will get easier and quicklier. I have no idea for analysing the time complexity of this algorithm. maybe it's  in O(n*log(n)),because I tested it many times, found that it's faster than pkohut's. I think his algorithm is  in O(n*log(n)).
by the way, I posted these LSP files,and  a program for generating  random points(it's real random,not fake random). just for 2004-2006,if  someone can help me to compile these to the other versions,I will thank him very much.
Code: [Select]
//-----------------------------------------------------------------------------
//----- acrxEntryPoint.h
//-----------------------------------------------------------------------------
#include "StdAfx.h"
#include "resource.h"
#include "tchar.h"
#include <stdlib.h>
#include <stdio.h>
#include <time.h>

//-----------------------------------------------------------------------------
#define szRDS _RXST("")

//-----------------------------------------------------------------------------
//----- ObjectARX EntryPoint
class CCreatRandomPointsApp : public AcRxArxApp {

public:
CCreatRandomPointsApp () : AcRxArxApp () {}

virtual AcRx::AppRetCode On_kInitAppMsg (void *pkt) {
// TODO: Load dependencies here

// You *must* call On_kInitAppMsg here
AcRx::AppRetCode retCode =AcRxArxApp::On_kInitAppMsg (pkt) ;

// TODO: Add your initialization code here
acutPrintf(_T("\nThe command is:rand."));

return (retCode) ;
}

virtual AcRx::AppRetCode On_kUnloadAppMsg (void *pkt) {
// TODO: Add your code here

// You *must* call On_kUnloadAppMsg here
AcRx::AppRetCode retCode =AcRxArxApp::On_kUnloadAppMsg (pkt) ;

// TODO: Unload dependencies here

return (retCode) ;
}

virtual void RegisterServerComponents () {
}


// - CreatRandomPoints._Rand command (do not rename)
static void CreatRandomPoints_Rand(void)
{
// Add your code for command CreatRandomPoints._Rand here

// 输入范围
ads_point pt1,pt2;
int ret=acedGetPoint(NULL,_T("\nPlease drag the First coner of region: "),pt1);
if (ret!= RTNORM)
{
acutPrintf(_T("\nInvalid!"));
return;
}
ret=acedGetCorner(pt1,_T("\nPlease drag the region: "),pt2) ;
if (ret!= RTNORM)
{
acutPrintf(_T("\nInvalid!"));
return;
}

//转化坐标系
acdbUcs2Wcs(pt1,pt1,0);
acdbUcs2Wcs(pt2,pt2,0);

        //防止点过于密集
double deltaX = (pt2[X]-pt1[X]);
double deltaY = (pt2[Y]-pt1[Y]);
if (abs(deltaX) < 1e-4 || abs(deltaY) < 1e-4 )
{
acutPrintf(_T("\nYour region is too small,Please drag it again."));
return;
}
deltaX = deltaX / 32767.0;
deltaY = deltaY / 32767.0;

//输入数量
double Number;
acedInitGet(7,NULL);
ret = acedGetReal(_T("\nPlease Enter the Number of Points:"),&Number);
if (ret!= RTNORM)
{
acutPrintf(_T("\nInvalid input!"));
return;
}
       
//初始化种子
srand((unsigned int)time(NULL));

//获得块表
Acad::ErrorStatus es;
AcDbBlockTable * pBlockTable;
es=acdbHostApplicationServices()->workingDatabase()->getBlockTable(pBlockTable, AcDb::kForRead);
if(es!=Acad::eOk)
return ;
pBlockTable->close();

//获得当前空间
resbuf tilemode;
AcDbBlockTableRecord * pBlockTableRecord;
acedGetVar(_T("TILEMODE"),&tilemode);
if(tilemode.resval.rint )
es=pBlockTable->getAt(ACDB_MODEL_SPACE, pBlockTableRecord,AcDb::kForWrite);
else
es=pBlockTable->getAt(ACDB_PAPER_SPACE, pBlockTableRecord,AcDb::kForWrite);
if (es!=Acad::eOk)
return;

AcDbPoint * pPoint;
AcGePoint3d  pt;
double Cx,Cy;
Cx = pt1[X];
Cy = pt1[Y];
pt.z = 0;
       
//画点
int n = (int) Number;
for (int i = 0;i<n;i++)
{
pt.x = Cx + deltaX * rand();
pt.y = Cy + deltaY * rand();
pPoint = new AcDbPoint(pt);
es=pBlockTableRecord->appendAcDbEntity(pPoint);
pPoint->close();
}
pBlockTableRecord->close();
}
} ;

//-----------------------------------------------------------------------------
IMPLEMENT_ARX_ENTRYPOINT(CCreatRandomPointsApp)

ACED_ARXCOMMAND_ENTRY_AUTO(CCreatRandomPointsApp, CreatRandomPoints, _Rand, Rand, ACRX_CMD_TRANSPARENT, NULL)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on June 15, 2010, 10:49:04 AM
Thank you, Highflybird

I test that when the distance is small, just like 1, not 70
your code is the most quickest one.

your recursion is very cool, thank you~
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 17, 2010, 05:19:20 AM
Now I attached the new code.

Just had a chance to go through it a bit. First let me express good job getting this to work and it made for a good exercise.  Here's a few items to fix or change.

The CreateLine function has a potential memory leak, consider what would happen if appendAcDbEntity returns something other than Acad::eOk.

The PointArray::equal function should simply be be
Code: [Select]
BOOL PointArray::equal(const double & v1, const double & v2, const double & e)
{
    return (fabs(v2 - v1) <= e);
}
even better would be, this thrown into the point2d class
Code: [Select]
                strikeout this code.
BOOL PointArray::equal const point2d & pt1, const point2d & pt2, const double & e) const
{
    return distance(pt1, pt2) <= e); return;
}

This line

Code: [Select]
// line is OK, checking if x and y are in range before doing the more expensive distance calculation.
if (equal (pt1.x, pt2.x,TOL) && equal (pt1.y ,pt2.y,TOL) && distance(pt1,pt2) <= TOL)
should simply be
Code: [Select]
           strike this code too
if (distance(pt1,pt2) <= TOL)

========================================================================
Now to that all important timing section.  In order to get just the balls out speed I removed the AcDbLine creation from your's and Daniel's code. I also adjusted where Daniel's time starts to more closely match how you've done it.  With those adjustments on 1,000,000 points.
Your's  - 1.43 seconds    (1.45 seconds when setup to use and calc 3D data and distances)
Daniels - 7.69 seconds

For the next test I change where the timer is started in your code to more closely match Daniel's original, and I subtracted 1 from your total time as an allowance to type "all" at the select objects prompt and hit return twice.
Your's  - 22.63 seconds
Daniel's - 8.24 seconds

And finally enable the AcDbLine creation.
Your's  - 31.48 seconds.
Daniel's - 18.90 seconds  (oops forgot to cache a value, now down to 16.4)

========================================================================
So overall very good job. Now I challenge you to make your whole program faster, cause 20 seconds acedSSGet is killing it.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 17, 2010, 02:38:21 PM
Hey Highflybird,

I've taken the liberty to use your QuickSortY and GetClosestPairs (renamed to DandC, for Divide and Conquer) routines to put another test together.  They've been reworked to get max speed, at least as much as I could squeeze out of 'em.  More speed could probably be had if the routines used only 1 or 2 parameters, that way the parameters would fit in the CPU registers.  That would require setting up a structure to hold all the needed data for the routines and pass it by reference.

Other notable changes - arrays are replaced with std::vector, the quick sort routine replaced with a faster vertex sort, and the collection of points is done in a manner similar to what Dan and I did previously.

These timings are the total run times after entering the input distance and include the time spent collecting and sorting 1,000,000 points using a distance of 70.
AcDbLine creation code disabled
1.464 seconds

AcDbLine creation code enabled
11.147 seconds

VS project file attached.

Code: [Select]
#include "StdAfx.h"
#include <vector>
#include <algorithm>

struct IndexAndSide
{
    int N; // index of point
    bool R; // is right side
};

/////////////////////////////////////////////////////////////////////////////
//* Neat little sort routine, shuffling an array of IndexAndSide objects. *//
/////////////////////////////////////////////////////////////////////////////
void QuickSortY(const AcGePoint3d * pts, IndexAndSide * data, int low, int high)
{

    if(low < high) {
        IndexAndSide tmp = data[low];
        int i = low;
        int j = high;
        while(i < j) {
            while(i < j && pts[data[j].N].y > pts[tmp.N].y)
                --j;
            if(i < j)
                data[i++] = data[j];
            while(i < j && pts[data[i].N].y < pts[tmp.N].y)
                ++i;
            if(i < j)
                data[j--] = data[i];
        }
        data[i] = tmp;

        QuickSortY(pts, data, low, i - 1);
        QuickSortY(pts, data, i + 1, high);
    }
}

///////////////////////////////////////////////////////////////////////
//* Hmm, append a line (from pt1 to pt2) to the block table record. *//
///////////////////////////////////////////////////////////////////////
void AddLine(AcDbBlockTableRecord * pRec, const AcGePoint3d & pt1, const AcGePoint3d & pt2)
{
    AcDbLine *pLine = new AcDbLine(pt1, pt2);
    if(pRec->appendAcDbEntity(pLine) != Acad::eOk)
        delete pLine;
    pLine->close();
}

/////////////////////////////////////////////////////////////////////////////////
//* DandC (Divide and Conquer), this is where the actual work-work begins and *//
//* its smokin'                                                               *//
/////////////////////////////////////////////////////////////////////////////////
void DandC( AcDbBlockTableRecord * pRec, const AcGePoint3d * pts, int low, int high, const double & distance )
{
    int index = high - low + 1;

    if(index < 3) {
        if(index == 2) {
            // only 2 points in the array, are they less than distance appart?
            if( fabs(pts[high].x - pts[low].x) <= distance &&
                fabs(pts[high].y - pts[low].y) <= distance &&
                pts[low].distanceTo(pts[high]) <= distance) {
                    // yep, build a line
                    AddLine(pRec, pts[low], pts[high]);
            }
        }
        return;
    }

    int u = low + index / 2;

    DandC(pRec, pts, low, u - 1, distance);
    DandC(pRec, pts, u, high, distance);

    int k1 = 0, k2 = 0;

    // note pts[u].x is middle x
    for(int i = u - 1; i >= low && pts[i].x >= (pts[u].x - distance); --i, ++k1);

    if(k1 == 0)
        return;

    for(int i = u; i <= high && pts[i].x <= (pts[u].x + distance); ++i, ++k2);

    std::vector<IndexAndSide> pts3(k1 + k2);

    for(int i = 0; i < k1; ++i) {
        pts3[i].N = u - 1 - i;
        pts3[i].R = false;
    }

    for(int i = 0; i < k2; ++i) {
        pts3[i + k1].N = u + i;
        pts3[i + k1].R = true;
    }

    QuickSortY(pts, &pts3.front(), 0, k1 + k2 - 1);

    // look at points in array from 0 to k1 + k2
    for(int i = 0; i < k1 + k2 - 1; ++i) {
        AcGePoint3d pt1 = pts[pts3[i].N];
        // look at points in array from i + 1 to k1 + k2
        for(int j = i + 1; j < k1 + k2; ++j) {
            // compare the points to see if they are within distance of each other
            AcGePoint3d pt2 = pts[pts3[j].N];
            if(fabs(pt2.y - pt1.y) > distance)
                break;
            if(fabs(pt2.x - pt1.x) <= distance && fabs(pt2.y - pt1.y) <= distance &&
                pt1.distanceTo(pt2) <= distance &&
                (pts3[i].R ^ pts3[j].R)) {
                    // yep, build a line
                    AddLine(pRec, pt1, pt2);
            }
        }
    }
}

//////////////////////////////////////////////////////////////////////////
//* Coolect all the points in the block table record reference in pRec *//
//////////////////////////////////////////////////////////////////////////
size_t CollectPoints(AcDbBlockTableRecord * pRec, std::vector<AcGePoint3d> & points)
{
    AcDbBlockTableRecordIterator *pIter;
    if(pRec->newIterator(pIter) == Acad::eOk)
    {
        for(pIter->start(); !pIter->done(); pIter->step())
        {
            AcDbEntity *pEnt = NULL;
            if(pIter->getEntity(pEnt, AcDb::kForRead) == Acad::eOk)
            {
                AcDbPoint *pPoint = AcDbPoint::cast(pEnt);
                if(pPoint)
                    points.push_back(pPoint->position());
                pEnt->close();
            }
        }
        delete pIter;
    }
    return points.size();
}

/////////////////////////////////////////////////////////////////////////////////
//* A very fast vertex sort routine written by Jonathan Richard Shewchuk      *//
//* and is part of his "Triangle, A Two-Dimensional Quality Mesh Generator    *//
//* and Delaunay Triangulator, copyright 2005                                 *//
//////                                                                     //////
//*  This is part of the insertvertex routine starting at line 6507 - 6525    *//
//* of version 1.5 of trinalge.  More specific it from about line 7707 - 7822 *//
/////////////////////////////////////////////////////////////////////////////////
unsigned long randomseed = 0;
size_t randomnation(size_t choices)
{

    randomseed = (randomseed * 13661 + 1508891) % 7140251;
    return randomseed / (7140251 / choices + 1);
}

void VertexSortX(AcGePoint3d * sortarray, int arraysize)
{
    int left, right;
    int pivot;
    double pivotx, pivoty;


    if (arraysize == 2) {
        /* Recursive base case. */
        if ((sortarray[0].x > sortarray[1].x) ||
            ((sortarray[0].x == sortarray[1].x) &&
            (sortarray[0].y > sortarray[1].y))) {
                std::swap(sortarray[1], sortarray[0]);
        }
    }
    /* Choose a random pivot to split the array. */
    pivot = (int) randomnation((unsigned int) arraysize);
    pivotx = sortarray[pivot].x;
    pivoty = sortarray[pivot].y;
    /* Split the array. */
    left = -1;
    right = arraysize;
    while (left < right) {
        /* Search for a vertex whose x-coordinate is too large for the left. */
        do {
            left++;
        } while ((left <= right) && ((sortarray[left].x < pivotx) ||
            ((sortarray[left].x == pivotx) &&
            (sortarray[left].y < pivoty))));
        /* Search for a vertex whose x-coordinate is too small for the right. */
        do {
            right--;
        } while ((left <= right) && ((sortarray[right].x > pivotx) ||
            ((sortarray[right].x == pivotx) &&
            (sortarray[right].y > pivoty))));
        if (left < right) {
            /* Swap the left and right vertices. */
            std::swap(sortarray[left], sortarray[right]);
        }
    }

    if (left > 1) {
        /* Recursively sort the left subset. */
        VertexSortX(sortarray, left);
    }
    if (right < arraysize - 2) {
        /* Recursively sort the right subset. */
        VertexSortX(&sortarray[right + 1], arraysize - right - 1);
    }
}

/////////////////////////////////////////////////////////////////
//* This is where all the work begins getting user input,     *//
//* collecting and sorting the points then finally qualifying *//
//* the points. Oh ya, and the all important timer reporting  *//
/////////////////////////////////////////////////////////////////
void ConnectPoints(void)
{
    double distance = 0.0;

    if(acedGetDist(0,_T("\nGetDist: "),&distance) != RTNORM)
        return;

    const double distancesqrd = pow(distance,2);

    __int64 freq, start, end, diff;
    QueryPerformanceFrequency((LARGE_INTEGER*)&freq);
    QueryPerformanceCounter((LARGE_INTEGER*)&start);

    std::vector<AcGePoint3d> points;
    AcDbBlockTableRecord *pRec;
    AcDbDatabase *pDb = acdbHostApplicationServices()->workingDatabase();

    if(acdbOpenAcDbObject((AcDbObject *&) pRec,pDb->currentSpaceId(), AcDb::kForWrite) == Acad::eOk)
    {           
        CollectPoints(pRec, points);
        VertexSortX(&points.front(), points.size());

        // QueryPerformanceCounter((LARGE_INTEGER*)&end);
        // acutPrintf(_T("\n%g Seconds collecting and sorting points\n"), (double)(end - start) / (double) freq);

        DandC(pRec, &points.front(), 0, points.size(), distance);

        pRec->close();
    }

    QueryPerformanceCounter((LARGE_INTEGER*)&end);
    acutPrintf(_T("\n%g Seconds\n"), (double)(end - start) / (double) freq);
}
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on June 17, 2010, 08:43:59 PM
Pkonut,you are very good man,I like to talk to you.
you gave me  some very good advice and a big challenge,I am happy  to accept  it.
Today I am very busy, so I won't be able to take a look,tomorrow,I am free.
BTW,I have a msn, highflybird@msn,would you make friend with me ?
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 17, 2010, 10:22:00 PM
you gave me  some very good advice and a big challenge,I am happy  to accept  it.

Well I just want to say thanks for sharing your research and code, that approach to the problem is very interesting.

The very best way to contact me through email (see profile).
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on June 18, 2010, 12:14:13 AM
I analyzed my algorithm  carefully a few days ago, I found  it takes O(n*log(n) *log(n) + K) time,This is in the worst situation. and I learned the Algorithm of  KDtree, it take  O (n*sqrt(n)+k) time ,so I think my algorithm is better than the algorithm of KDtree ,  :-),maybe  my analyzing is wrong. 
Yesterday, I thought a better way to improver my code, I will realize it  after I finish my work.
pkohut, your email address is hidden.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 18, 2010, 01:41:23 AM
I analyzed my algorithm  carefully a few days ago, I found  it takes O(n*log(n) *log(n) + K) time,This is in the worst situation. and I learned the Algorithm of  KDtree, it take  O (n*sqrt(n)+k) time ,so I think my algorithm is better than the algorithm of KDtree ,  :-),maybe  my analyzing is wrong. 
Yesterday, I thought a better way to improver my code, I will realize it  after I finish my work.
pkohut, your email address is hidden.


Setting up the KDTree is about as fast as doing a quick sort.  I was working on a remove node function, but kept muffing it up, which I think once implemented would allow it to perform at least as fast as Daniel's code (gut feeling is it would smoke Daniel's entry).  But your algorithm will be faster for this challenge.
Code: [Select]
   while kd-tree has points {
         process root node
         remove root node
    }
This would keep the total iterations to an absolute minimum.  As it is now it must visit work that has already been completed in order to find a leaf that needs to be worked on, which takes time.

But, KD-tree is a spatial container for N-dimensional objects and allows some types of spatial queries, especially if the data to queried is stored on a drive, network, or even in a SQL database.  I use the KDTree code presented here in a few programs , and it's generic enough for easy reuse. What's very nice about the KDTree routine presented is that each leaf node can contain user defined data.

When trying to speed up my entry I used the extra data as a flag indicating the leaf has been previously processed (speedup was only a couple seconds over my best posted speed).

For this challenge I wanted to show a different way to accomplish the work.  Because my best posted speed of 29 seconds (about 20 seconds raw speed) people may think its not very good, but it's pretty much left in its generic state. Adding the remove node will speed it up dramatically, and tweaking the code a bit for this challenge could get a little more speed.

My email address is paulkohut at hotmail dot com
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 21, 2010, 12:17:32 AM
<...snip...>
Setting up the KDTree is about as fast as doing a quick sort.  I was working on a remove node function, but kept muffing it up, which I think once implemented would allow it to perform at least as fast as Daniel's code (gut feeling is it would smoke Daniel's entry).
<...snip...>
For this challenge I wanted to show a different way to accomplish the work.  Because my best posted speed of 29 seconds (about 20 seconds raw speed)


Erase node implemented and first pass clocks in at 12.24 seconds raw speed. Looks like I'll be eatin' gut and breathing Daniel's smoke  :kewl:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: Kerry on June 21, 2010, 01:17:26 AM


That statement may be a little difficult for 'bing' to translate :) (sensibly, anyway)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 21, 2010, 05:08:37 AM


That statement may be a little difficult for 'bing' to translate :) (sensibly, anyway)

Ya, that dawned on me a little while after reading your post. All of a sudden there was an ugly picture in my mind.

Without some major restructuring (beyond that CKdTree is now a template and does conditional compiling) I've got it down to 11.20 seconds RS.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 22, 2010, 05:52:59 AM
For the next test I change where the timer is started in your code to more closely match Daniel's original, and I subtracted 1 from your total time as an allowance to type "all" at the select objects prompt and hit return twice.
Your's  - 22.63 seconds
Daniel's - 8.24 seconds
(note: Your's is HighFlyBird's code)

And finally enable the AcDbLine creation.
Your's  - 31.48 seconds.
Daniel's - 18.90 seconds  (oops forgot to cache a value, now down to 16.4)

New numbers for my KDTree entry  :-)
Prior to my final optimization it was running raw speed at 11.20. Could not get it any faster unless I turned on SSE2 and favor speed as a compiler option which then dropped it to 10.83 seconds raw speed for the million point test.  I tried 5 or 6 methods of tracking previous nodes and such and non provided any speed improvement, two methods I thought would work for sure added about 5 seconds to the elapse time (making the starting point in the tree the current node position, then walk backward up the tree till a candidate is found, current method starts at the root node and walks forward).  Which got me thinking, is my KDTree Optimized, in other words balanced and the answer to that was, not in a million freaking years.

So, after implementing a tree optimization routine the final numbers are:
6.15 seconds raw speed, with the 1 million point test. (SSE2 off, favor speed or size: neither, 5.91 with them on)
16.46 seconds adding AcDbLine's to the drawing.

For kicks I turned off the routine that erases previously visited points (to reduce redundant work) and it runs 7.99 seconds RS, prior to that it was in the 15 second range.

I've got to do some code clean up then I'll post. Probably after 9pm PST Tuesday.

2008 ARX attached. Load it and test drawing with points, type "doit" at the command prompt.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 23, 2010, 11:42:10 AM
So, after implementing a tree optimization routine the final numbers are:
6.15 seconds raw speed, with the 1 million point test.

While documenting the code found a spot to further optimize by delaying when an expensive set of calcs is done.
Now down to 5.79 seconds.  :lol:

There's 14 functions left to document, once finished will release code. Got a busy day today, so shooting for midnight tonight PST.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on June 23, 2010, 04:41:27 PM
Paul try this arx (07) and see if there is any difference, note: its not creating the lines
the command is doit
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on June 23, 2010, 05:09:34 PM
Hey Highflybird,

I've taken the liberty to use your QuickSortY ...

Woa! this one is fast!  8-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 23, 2010, 05:32:00 PM
Paul try this arx (07) and see if there is any difference, note: its not creating the lines
the command is doit

Ya, thats the raw speed test, it reports how many lines would have been created.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on June 23, 2010, 05:52:41 PM
Paul try this arx (07) and see if there is any difference, note: its not creating the lines
the command is doit

Ya, thats the raw speed test, it reports how many lines would have been created.

Sorry, I just compiled this.. I wanted to know if there was any difference on your machine.
...This is before I tested the DandC version which crushes mine so there is no need....  :laugh:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on June 23, 2010, 09:18:35 PM
Quote
         
            if(fabs(pt2.x - pt1.x) <= distance && fabs(pt2.y - pt1.y) <= distance &&
                pt1.distanceTo(pt2) <= distance &&
                (pts3.R ^ pts3[j].R)) {
                    // yep, build a line
                    AddLine(pRec, pt1, pt2);
            }

I think these codes can be improved.
           if(fabs(pt2.x - pt1.x) <= distance &&
               //fabs(pt2.y - pt1.y) <= distance &&         //This line can be cut. because
                                                                         // we have "if(fabs(pt2.y - pt1.y) > distance) break; "    
                (pts3.R ^ pts3[j].R)  &&                                  
                pt1.distanceTo(pt2) <= distance ){         //The distance calculation is more expensive.
                    // yep, build a line
                    AddLine(pRec, pt1, pt2);
            }
I found an another way for this topics
that is called "layered range tree"

http://cgi.di.uoa.gr/~vfisikop/compgeom/Layered_Range_trees/doc_tex/report.pdf
use this algorithm ,we can get the time O(n*log(n)) .  I wanted to change it to arx,but I failed.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on June 23, 2010, 09:36:50 PM
I found a bug In this forum?
when I use the word
Code: [Select]
"pts[i]" ---the "[ i ]",it 's dispeared. it can't be displayed?  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 23, 2010, 09:38:52 PM
Quote
         
            if(fabs(pt2.x - pt1.x) <= distance && fabs(pt2.y - pt1.y) <= distance &&
                pt1.distanceTo(pt2) <= distance &&
                (pts3.R ^ pts3[j].R)) {
                    // yep, build a line
                    AddLine(pRec, pt1, pt2);
            }

I think these codes can be improved.
           if(fabs(pt2.x - pt1.x) <= distance &&
               //fabs(pt2.y - pt1.y) <= distance &&         //This line can be cut. because
                                                                         // we have "if(fabs(pt2.y - pt1.y) > distance) break; "    
                (pts3.R ^ pts3[j].R)  &&                                  
                pt1.distanceTo(pt2) <= distance ){         //The distance calculation is more expensive.
                    // yep, build a line
                    AddLine(pRec, pt1, pt2);
            }

Based on the code snippet you show, I don't think so. The original is better as it does a quick Manhattan Distance check before doing the more expensive Pythagorean calc.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on June 23, 2010, 10:05:02 PM
Based on the code snippet you show, I don't think so. The original is better as it does a quick Manhattan Distance check before doing the more expensive Pythagorean calc.
in my test, yep,the distance check is faster,but just a little,nearly the same speed.I don't know the why.
in my old opinion,it's slower.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 23, 2010, 10:46:20 PM
Based on the code snippet you show, I don't think so. The original is better as it does a quick Manhattan Distance check before doing the more expensive Pythagorean calc.
in my test, yep,the distance check is faster,but just a little,nearly the same speed.I don't know the why.
in my old opinion,it's slower.

Floating point subtraction and addition are faster operations than multiplication and division. fabs, abs, etc take a single machine clock cycle to perform.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on June 23, 2010, 11:07:50 PM
Floating point subtraction and addition are faster operations than multiplication and division. fabs, abs, etc take a single machine clock cycle to perform.

but
Code: [Select]
(Pts3[ i ].R ^ Pts3[ j ].R) is a logistic calculation, distanceto includes substracts,additions,and multiplication and squrare root calculation.Ok,this is another topic.pkohut, thanks your replay.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 23, 2010, 11:35:16 PM
As promised, the source code to the new KDTree version, there are also 2 2008 arx builds for doing the full test and the raw speed test, each one has commands "doit" and "doit2".  The first runs with an optimized (balanced KDTree), the second with an unbalanced tree.

The 1 million point test runs are:
5.77 seconds raw speed.
15.92 seconds with drawing the AcDbLine's (full test)

The full test is taking a performance hit because the KDNODE position data must be converted to AcGePoint3d for the line to be built. I have been so concentrated on RS that I overlooked this important issue.  The fix is easy, probably take an hour to implement and another hour to test, but I promised to get the code out today, so that's where it stands (for now). Never mind, this is in line with the performance with DandC.

Notes about this code (almost worth its own thread) -
The source code is heavily documented, and I ran Doxygen on it to produce developer documentation in the Doc directory. Just load the "index.html" file to browse the developer docs.

The code uses C++ templates to help the compiler optimize the code.

Total this is about 4 days effort to put together into this form. Was it worth it?  I think so and I hope others do too. Those that have been following this thread, and seeing how everyone's code has progressed, can probably learn something from our work.

As I already mentioned, the full test can be sped up by having KDNodes store AcGePoint3d data, that will take care of the hit when creating AcDbLines.   The code is not thread safe so utilizing multiple processors is out of the question (actually there is one method I can think of to queue up a bunch of point data and have another thread draw the AcDbLines, this would take 4 to 6 seconds off the full test run time.  And I'm thinking with a major rewrite I can get this to between 2 to 4 seconds raw speed (dawned on me yesterday while I was away).

Thanks  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on June 23, 2010, 11:42:35 PM
Floating point subtraction and addition are faster operations than multiplication and division. fabs, abs, etc take a single machine clock cycle to perform.

but
Code: [Select]
(Pts3[ i ].R ^ Pts3[ j ].R) is a logistic calculation, distanceto includes substracts,additions,and multiplication and squrare root calculation.Ok,this is another topic.pkohut, thanks your replay.

This is better, check if point 3 i and j are on the right side first, then do the calculations.

Code: [Select]
if((pts3[i].R ^ pts3[j].R) &&
    fabs(pt2.x - pt1.x) <= distance && fabs(pt2.y - pt1.y) <= distance &&
    pt1.distanceTo(pt2) <= distance) {
        // yep, build a line
        AddLine(pRec, pt1, pt2);
}
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 14, 2010, 08:29:22 AM
Floating point subtraction and addition are faster operations than multiplication and division. fabs, abs, etc take a single machine clock cycle to perform.

but
Code: [Select]
(Pts3[ i ].R ^ Pts3[ j ].R) is a logistic calculation, distanceto includes substracts,additions,and multiplication and squrare root calculation.Ok,this is another topic.pkohut, thanks your replay.

This is better, check if point 3 i and j are on the right side first, then do the calculations.

Code: [Select]

if((pts3[i].R ^ pts3[j].R) &&
    fabs(pt2.x - pt1.x) <= distance && fabs(pt2.y - pt1.y) <= distance &&
    pt1.distanceTo(pt2) <= distance) {
        // yep, build a line
        AddLine(pRec, pt1, pt2);
}

How to get an easy 5% speed boost from DandC
Been playing around with some code optimizations today and revisited the DandC and KdTree code bases (originals can be found in this thread). For the DandC version, there are 2 if statements similar to the one above. That's a lot of if processing, especially in the inner loop of the main work function. Here is what the code looks like without the && operators -
Code: [Select]
if(pts3[i].R ^ pts3[j].R)
  if(fabs(pt2.x - pt1.x) <= distance)
    if(fabs(pt2.y - pt1.y) <= distance)
      if(pt1.distanceTo(pt2) <= distance) {
        // yep, build a line
        AddLine(pRec, pt1, pt2);
       }
The first 3 compares can all be combined into a single logical & statement which will get rid of 2 if's, and we come out ahead in clock cycles. So the above if statement can be rewritten to -
Code: [Select]
if( (fabs(pt2.x - pt1.x) <= distance) & (fabs(pt2.y - pt1.y) <= distance) &
    (pts3[i].R ^ pts3[j].R) &&
    (pt1.distanceTo(pt2) <= distance))

When run with the core algorithm only (no line work drawing) you get about a 3.5% speed boost with the 1,000,000 dot drawing. Not a whole bunch, especially when the program does the work in under 2 seconds, but we'll take the reduced runtime all the same.

For an addition 1.5% speed boost, add void Set(int n, bool r) { N = n; R = r; } to the struct IndexAndSide, then call Set instead of accessing the members indirectly. See source code.

Code: [Select]

struct IndexAndSide
{
    int N; // index of point
    bool R; // is right side
    void Set(int n, bool r) { N = n; R = r; }
};

void DandC( AcDbBlockTableRecord * pRec, const AcGePoint3d * pts, int low, int high, const double & distance )
{
    int index = high - low + 1;

    if(index < 3) {
        if(index == 2) {
            // only 2 points in the array, are they less than distance appart?
            //if( fabs(pts[high].x - pts[low].x) <= distance &&
            // fabs(pts[high].y - pts[low].y) <= distance &&
            // pts[low].distanceTo(pts[high]) <= distance) {
            if( (fabs(pts[high].x - pts[low].x) <= distance) &
                (fabs(pts[high].y - pts[low].y) <= distance) &&
                pts[low].distanceTo(pts[high]) <= distance) {

                    // yep, build a line
                    AddLine(pRec, pts[low], pts[high]);
            }
        }
        return;
    }

    int u = low + index / 2;

    DandC(pRec, pts, low, u - 1, distance);
    DandC(pRec, pts, u, high, distance);

    int k1 = 0, k2 = 0;

    // note pts[u].x is middle x
    for(int i = u - 1; i >= low && pts[i].x >= (pts[u].x - distance); --i, ++k1);

    if(k1 == 0)
        return;

    for(int i = u; i <= high && pts[i].x <= (pts[u].x + distance); ++i, ++k2);

    std::vector<IndexAndSide> pts3(k1 + k2);

    for(int i = 0; i < k1; ++i) {
        pts3[i].Set(u - 1 - i, false);
        //pts3[i].N = u - 1 - i;
        //pts3[i].R = false;
    }

    for(int i = 0; i < k2; ++i) {
        pts3[i + k1].Set(u + i, true);
        //pts3[i + k1].N = u + i;
        //pts3[i + k1].R = true;
    }

    QuickSortY(pts, &pts3.front(), 0, k1 + k2 - 1);

    // look at points in array from 0 to k1 + k2
    for(int i = 0; i < k1 + k2 - 1; ++i) {
        AcGePoint3d pt1 = pts[pts3[i].N];
        // look at points in array from i + 1 to k1 + k2
        for(int j = i + 1; j < k1 + k2; ++j) {
            // compare the points to see if they are within distance of each other
            AcGePoint3d pt2 = pts[pts3[j].N];
            if(fabs(pt2.y - pt1.y) > distance)
                break;
            //if(fabs(pt2.x - pt1.x) <= distance && fabs(pt2.y - pt1.y) <= distance &&
            //    pt1.distanceTo(pt2) <= distance &&
            //    (pts3[i].R ^ pts3[j].R))
            if( (fabs(pt2.x - pt1.x) <= distance) & (fabs(pt2.y - pt1.y) <= distance) &
                (pts3[i].R ^ pts3[j].R) &&
                (pt1.distanceTo(pt2) <= distance))
            {
                // yep, build a line
                AddLine(pRec, pt1, pt2);
            }

        }
    }
}


For the KdTree version applying the same principles reduced the runtime from about 6.4 seconds to 5.4 seconds, about 18% (earlier reports of 5.7 seconds was based on different OS and system configuration). That will be a different report.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 14, 2010, 12:02:49 PM
How 18% speed increase was achieved in KdTree version.

Most of the code changes were made in the inner most FindNearest function. Here's the original code for the function (Changed area is red, noted areas purple)
Code: [Select]
int __fastcall FindNearest(KDNODE<T, DIM> * pNode, RESULT_NODE<T, DIM> *pList)
{
if(!pNode)
return 0;

int added_res = 0; // track number of nodes are added to the pList
// get the Manhattan distance from pNode to the queried position. If Manhattan distance is <= query range
// the go ahead and do more expensive KDTree distance check.
[color=purple]if( fabs(pNode->pos[0] - m_dQueryPos[0]) <=m_dQueryRange && fabs(pNode->pos[1] - m_dQueryPos[1]) <= m_dQueryRange) {[/color]
// get the KDTree distance from pNode to the queried position
double dist_sq = 0.0;
for(int i = 0; i < DIM; i++) {
dist_sq += SQ(pNode->pos[i] - m_dQueryPos[i]);
}

// Only add nodes that are within the query range
if(dist_sq <= SQ(m_dQueryRange)) {
if(RlistInsert(pList, pNode, dist_sq) < 0) {
// error
return -1;
}
added_res = 1;
}
}

// Get the KDTree direction distance from the queryPosition to pNode,
// which determines which leaf node to traverse down next.
double dx = m_dQueryPos[pNode->dir] - pNode->pos[pNode->dir];
[color=red]int ret = FindNearest(dx <= 0.0 ? pNode->left : pNode->right, pList);
if(ret >= 0 && fabs(dx) < m_dQueryRange) {[/color]
added_res += ret;
[color=red]ret = FindNearest(dx <= 0.0 ? pNode->right : pNode->left, pList);[/color]
}
if(ret < 0) {
return -1;
}

added_res += ret;
return added_res;
}

The FindNearest function is the most used function in the KdTree algorithm and makes lots of recursive calls, so speeding it up is an obvious choice. In such a tight loop function, simplifying if statements is a good start, we've got 4 candidates. For know lets concentrate on the second one highlighted in red.
Code: [Select]
[color=red]if(ret >= 0 && fabs(dx) < m_dQueryRange)[/color]
This basically says
Code: [Select]
if(ret >= 0)
    (if(fabs(dx) < m_dQueryRange) {
        ...
    }       
Changing the && operator to & makes the statement a boolean check from a single if -
Code: [Select]
[color=green]if(ret >= 0 & fabs(dx) < m_dQueryRange)[/color]
Run benchmarks to determine overall effect. In this case we got a positive speed boost. Notice the purple highlighted line in the original code. Making to same change to it cause a negative impact on speed. After lots of testing I'm guessing that it is a memory cache miss that caused the slow down, so the line won't be changed.

Let's turn our attention to the first red highlighted line -
Code: [Select]
int ret = FindNearest([color=red]dx <= 0.0 ? pNode->left : pNode->right[/color], pList);For those not familiar with C or Java syntax this may look strange, it's called a Ternary operator, and basically we put a big ol if/then/else statement right in the function call parameter. It reads "if dx is less than or equal to 0.0 then use pNode->left else use pNode->right as the first parameter to FindNearest".

The goal of this exercise is to speed up the program by removing if/then/else statements or ternay operators from this heavily called function. If speed is not an issue then by all means use if/then/else and ternay, they help make the code maintainable and readable.

For us, the problem with if/then/else or ternay is that they force a CPU jump instruction no matter what and those cost CPU clock cycles. Depending on the CPU there is also branch prediction and pipelines to contend with, look it up on wiki if your interested. From a simplified CPU point of view here's how if/then/else and if/then may look.
Code: [Select]
[color=navy]if/then/else type statement[/color]
    Compare dx <=  0.0
    Jump to Label1 if false
    Use pNode->left
    Jump to Label2
Label1:
    Use pNode->right
Label2:

[color=navy]if/then type statement[/color]
    Compate dx <= 0.0
    Jump to Label3 if false
    Use pNode->left;
Label3:
See the 2 jumps in the if/then/else statement, for each pass through the if statement one of them must be executed, and those add up. Depending on the condition we will use either left or right. It would be nice if we could use the simpler if/then style statement and still have the proper left or right value set. By preloading a variable we can. The idea being, preload a variable with left or right (we'll use right), then do the check and change the preloaded variable if necessary. Here it is in code -
Code: [Select]
[color=green]        KDNODE<T, DIM> * pNode1 = pNode->right; // preload variable
        if(dx <= 0.0) {
            pNode1 = pNode->left; // oops, needed left
        }
        int ret = FindNearest(pNode1, pList);[/color]
Test the changes. Nice boost in speed. So when you have 2 conditions and the variables are simple, then preloading may be beneficial (watch out for memory cache misses).

Ok, let's do the same thing to the third red highlighted line from the original code -
Code: [Select]
[color=red]ret = FindNearest(dx <= 0.0 ? pNode->right : pNode->left, pList);[/color]
... becomes ...
Code: [Select]
[color=purple]            pNode1 = pNode->left;
            if(dx <= 0.0)
                pNode1 = pNode->right;
            ret = FindNearest(pNode1, pList);[/color]
Test the changes, oh ya, very nice speed boost. However, -
Code: [Select]
[color=green]            pNode1 = pNode->right;
            if(dx > 0.0)
                pNode1 = pNode->left;
            ret = FindNearest(pNode1, pList);[/color]
Gives a massive speed boost. And for a final speed boost to 18% remove the nCount++ from the ConnectDotsWithin function as it causes memory cache misses.

Getting these results only took an hour or so of messing with the code. After every little change benchmarks were checked. I also explored hand written SSE2 instructions, but could not get it correct. I especially wanted that purple highlighted line in the original code to use SSE2 as that would have been nice to parallel compute those values.

Final source code for the FindNearest function -
Code: [Select]
[color=green]    int __fastcall FindNearest(KDNODE<T, DIM> * pNode, RESULT_NODE<T, DIM> *pList)
    {
        if(!pNode)
            return 0;

        int added_res = 0; // track number of nodes are added to the pList
        // get the Manhattan distance from pNode to the queried position. If Manhattan distance is <= query range
        // the go ahead and do more expensive KDTree distance check.
        if( fabs(pNode->pos[0] - m_dQueryPos[0]) <= m_dQueryRange &&
            fabs(pNode->pos[1] - m_dQueryPos[1]) <= m_dQueryRange) {
                // get the KDTree distance from pNode to the queried position
                double dist_sq = 0.0;
                for(int i = 0; i < DIM; i++) {
                    dist_sq += SQ(pNode->pos[i] - m_dQueryPos[i]);
                }

                // Only add nodes that are within the query range
                if(dist_sq <= SQ(m_dQueryRange)) {
                    if(RlistInsert(pList, pNode, dist_sq) < 0) {
                        // error
                        return -1;
                    }
                    added_res = 1;
                }
        }

        // Get the KDTree direction distance from the queryPosition to pNode,
        // which determines which leaf node to traverse down next.
        double dx = m_dQueryPos[pNode->dir] - pNode->pos[pNode->dir];


        KDNODE<T, DIM> * pNode1 = pNode->right;
        if(dx <= 0.0) {
            pNode1 = pNode->left;
        }
        int ret = FindNearest(pNode1, pList);

        if(ret >= 0 & fabs(dx) < m_dQueryRange) {
            added_res += ret;
            // ret = FindNearest(dx <= 0.0 ? pNode->right : pNode->left, pList);
            pNode1 = pNode->right;
            if(dx > 0.0)
                pNode1 = pNode->left;
            ret = FindNearest(pNode1, pList);
        }
        if(ret < 0) {
            return -1;
        }

        added_res += ret;
        return added_res;
    }[/color]
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on September 15, 2010, 12:01:11 AM
Thanks,pkohut.
I'll read your code carefully.KDTree is a very good algorithm,I wish I can understand it completely.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 17, 2010, 07:51:35 AM
KdTree breaks 5 second barrier for 1,000,000 point drawing  ^-^ (without SSE2, or special compiler settings)

From the previous message, it was mentioned that KdTree was running about 6.4 seconds and after some code changes down to 5.4 seconds. It's now down to 4.977 seconds (4.742 seconds with SSE2 enabled).

For this version the first change dropped about 3 tenths of a second from the running time. The busy FindNearest routine was restructured to eliminate recursive calls when a child node is NULL. This saves 43,000,000 recursive calls that would have immediately returned (see code below). Each recursive call causes some stack space to be reserved and initialized, then unreserved upon return. So that's 43 million times that doesn't have to be done. For each of the recursive functions, except ClearRecord, the same refactoring was done. After the changes the program was running just under 5.1 seconds.

Those changes were easy, but the 5 second barrier was just right there. Mocking me. The next couple hours resulted in obtaining 5.022 seconds, and for the next 4 hours that was the best I could achieve. While outside having a smoke, I was thinking if only I could find 1 more if statement to refactor. Came back in and 5 minutes later found one, right there in the FindNearest function. Perfect. 4.977 seconds, mission accomplished.

In summary, after the tree optimization routine, the next biggest impact was inner loop and if statement refactoring. The tiny details of chasing 2 hundredths of a second here and there are not important, and probably don't apply from application to application.

The attached binaries are for 2008 32bit. One is just the raw speed test, no lines are drawn.  The other draws the lines in Autocad, which takes an additional 8 seconds or so. Source code will be posted after some code clean up.

Refactored FindNearest function.
Code: [Select]
    int __declspec(nothrow) __fastcall FindNearest(KDNODE<T, DIM> * pNode, RESULT_NODE<T, DIM> *pList)
    {
        // This function should never be called if pNode is NULL,
        // otherwise program will fault

        int added_res = 0; // track number of nodes are added to the pList
        // get the Manhattan distance from pNode to the queried position. If Manhattan distance is <= query range
        // the go ahead and do more expensive KDTree distance check.
        if( fabs(pNode->pos[0] - m_dQueryPos[0]) <= m_dQueryRange &&
            fabs(pNode->pos[1] - m_dQueryPos[1]) <= m_dQueryRange) {

                // get the KDTree distance from pNode to the queried position
                double dist_sq = 0.0;
                for(int i = 0; i < DIM; i++) {
                    dist_sq += SQ(pNode->pos[i] - m_dQueryPos[i]);
                }

                // Only add nodes that are within the query range
                if(dist_sq <= SQ(m_dQueryRange)) {
                    if(RlistInsert(pList, pNode, dist_sq) < 0) {
                        // error
                        return -1;
                    }
                    added_res = 1;
                }
        }

        // Get the KDTree direction distance from the queryPosition to pNode,
        // which determines which leaf node to traverse down next.
        double dx = m_dQueryPos[pNode->dir] - pNode->pos[pNode->dir];

        // preload pNode1
        KDNODE<T, DIM> * pNode1 = pNode->right;
        if(dx <= 0.0) {
            pNode1 = pNode->left;
        }
       
        // preload ret
        int ret = 0;
        if(pNode1)
            ret = FindNearest(pNode1, pList);

        if(ret < 0)
            return -1;

        if(fabs(dx) < m_dQueryRange) {
            added_res += ret;
            ret = 0; // preload ret

            pNode1 = pNode->right; // preload pNode1
            if(dx > 0.0)
                pNode1 = pNode->left;

            if(pNode1)
                if( (ret = FindNearest(pNode1, pList)) < 0)
                    return -1;
        }

        added_res += ret;
        return added_res;
    }


Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: Kerry on September 17, 2010, 08:02:55 AM

Your attitude and persistance should be a shining example to us all Paul.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 17, 2010, 08:15:11 AM

Your attitude and persistance should be a shining example to us all Paul.


Thank you.

The back story might change your view though. I spent about 12 hours writing assembler code to replace the FindNearest function, but could not beat VC's generated code.  It ain't like the old Z80 and 6502 days.

edit: what resurrected my interest is the SSE2 intrinsics and running computations in parallel. Come to find out MSVC 2008 doesn't do a very good job with these, which lead me to the back story. Read a MS blog that MSVC 2010 generates much better code from SSE2 intrinsics.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on September 17, 2010, 09:22:08 AM
Your attitude and persistance should be a shining example to us all Paul.
Thank you.
The back story might change your view though. I spent about 12 hours writing assembler code to replace the FindNearest function, but could not beat VC's generated code.  It ain't like the old Z80 and 6502 days.

edit: what resurrected my interest is the SSE2 intrinsics and running computations in parallel. Come to find out MSVC 2008 doesn't do a very good job with these, which lead me to the back story. Read a MS blog that MSVC 2010 generates much better code from SSE2 intrinsics.
I admire you very much. 12 hours,assembler code, you work very hard!
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 17, 2010, 07:03:51 PM
One simple change since the last post got me to 4.82 seconds. Simply make the ManhattenDistance calc it's own function allowed the compiler to do a little better optimization. The function is still inline, however a couple instructions in the disassembly were different. Most noticable the original had a fstp (0) instruction instead of the faster fabs.  Here's the code -

Code: [Select]
template<class T>
static int __fastcall ManhattenDistance2dArray( T * pos1,  T * pos2,
                                               const T & range)
{
    return (fabs(pos1[0] - pos2[0]) <= range) &&
        (fabs(pos1[1] - pos2[1]) <= range);
}

// Replace the if statement for the Manhattan distance calc in FindNearest with this -
if(ManhattenDistance2dArray<T>(pNode->pos, m_dQueryPos, m_dQueryRange)) {

Seeing that improvement, I wanted to get an assembly SSE2 function implemented. Since the calculation is data bound and not computation bound, the overhead is pretty big, pushing 64 bit data to 128 registers in the tightest loop of the whole program.

Code: [Select]
   // add these to the CKdTreeT class

    // define some typedefs for use withing the __asm sections. MSVC complains
    // about the brackets in the templates.
    typedef KDNODE<T, DIM> tKdNode;
    typedef CKdTreeT<T, DIM, ORDERED> tKtree;

    FORCEINLINE int __fastcall ManhattenDistance2dArray_2(const double * v1) const
    {
        __asm {
            mov eax, offset absMask    
            movapd xmm4, [eax]          // load mask data to r0 and r1 of xmm4

            mov eax, dword ptr [ecx]    // ecx is pointer to "this" object
            lea eax, [ecx + tKtree.m_dQueryPos] // this + CKdTreeT::m_dQueryPos
            movapd xmm1, [eax]          // load queryPos to r0 and r1 of xmm1

            lea eax, [ecx + tKtree.m_dQueryRange]   // m_dQueryRange is a single
            movsd xmm2, [eax]                       // double, so have to first
                                                    // stuff it into r0 of xmm1.
                                                    // Need to stuff it into r1
                                                    // too, but do something else
                                                    // ...

            mov eax, [edx]tKdNode.pos   // edx points to pNode
            movapd xmm0, [eax]          // load to r0 and r1 of xmm0

            shufpd xmm2, xmm2, 0        // ... we're back with xmm2. copy r0 to r1

            subpd xmm0, xmm1            // subtrack xmm0{r0:r1} - xmm1{r0:r1}
                                        // results go to xmm0

            andpd xmm0, xmm4            // make 'em positive with absMask
            cmpltpd xmm0, xmm2          // compare the results of the subtraction
                                        // m_dQueryRange. r0 and r1 will be -1 for true
                                        // and 0 for false

            movmskpd eax, xmm0          // pack the compare results to an int. They
                                        // will be in 2's complements, so later bitwise
                                        // checks can be applied to see which was true
                                        // or false.

                                        // eax is the return value.
        };
    }


///////////////////////////////////

// Substitute this if statement for the Manhattan distance calc in FindNearest

if(ManhattenDistance2dArray_2(pNode->pos)) {

edit: 2 good SSE2 sources (easy reads), out of about 20 I've been looking at
http://developer.apple.com/hardwaredrivers/ve/sse.html#Translation_Cmp
http://www.gamedev.net/reference/articles/article1987.asp


Anyways, it's been fun, but the compiler is way to good to be mucking around in this neck of the woods. Time to come up for air  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 18, 2010, 07:54:03 AM
An hour to implement a simple memory pool and its down to 4.695 seconds. No SSE2, nasty assembly, or crazy tweaks.  :-)
Not as much of speed improvement as I'd hoped. Probably because the memory pool is biased towards working with arrays,
and KdTree is a binary tree. So there is probably a lot of memory cache misses. Need to find a tool that can report that type
stuff of the CPU.

edit: Dawned on me a few minutes ago, that the dynamic memory used to hold the leaf node X, Y, Z position data isn't necessary since any more, since the code now uses templates. Changed it from dynamic to a standard array...and booYa - 4.228 seconds.

edit: For comparison sake, KdTree w/ SSE2 enabled 3.994 seconds. Divide and Concur (DandC) w/ SSE2 enabled 1.839 seconds.
3.994
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gskelly on September 18, 2010, 09:12:17 AM
Wow... a lot of info in this thread, most of it way over my head   :-o

Cool to follow the progression and helps to open my mind a bit to how to look at code differently.

Thanks
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 18, 2010, 12:17:51 PM
Wow... a lot of info in this thread, most of it way over my head   :-o

Cool to follow the progression and helps to open my mind a bit to how to look at code differently.

Thanks

Thanks for the comments. I enjoy doing these write ups.  Oh ya, chasing the 4 second barrier know. Down to 4.165.

edit:
In my code I tend to write a function like so

Code: [Select]
double DoHeavyCalc(const double & d1, const double & d2, const double & d3, const double & d4) { ... } Thinking being that on a 32 bit machine it is more efficient to pass a 4 byte reference and make it const to let the compiler do any optimizations it can. For giggles I changed the function signatures to
Code: [Select]
double DoHeavyCalc(double d1, double d2, double d3, double d4) { ... } which turns out to be slightly more efficient. Running with the 1 million point shaved about 0.025 seconds, but in my new quest to 4, I'll take it.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 18, 2010, 02:06:08 PM
Found a piece in the code that has the potential to shave off 0.578 seconds. The structure ResultNode is being dynamically allocated and released about 4,500,000 times. At most probably only a thousand or so need to be allocated. My thought is to revamp the simple memory pool introduced earlier and use it for both the ResultNode's and tree leaf nodes.

What's nice about all these optimizations (so far) is they don't rely on tricks or slight of hand. So far it's just been simple refactoring and I'm learning a great deal.  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: Lee Mac on September 18, 2010, 02:10:47 PM
So far it's just been simple refactoring and I'm learning a great deal.  :-)

That's good to hear, I really admire your persistence Paul, but I'm afraid with every post it is going further and further above my head  :oops:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 18, 2010, 02:39:47 PM
So far it's just been simple refactoring and I'm learning a great deal.  :-)

That's good to hear, I really admire your persistence

Making up for lost seat time at the computer. Summer is over and the kids are back in school, so I'm left with way to much time on my hands. Happens every year  :-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 20, 2010, 03:55:14 PM
The final optimization from the last post was a simple memory pool and the code was running 4.165 seconds for the test, and there might be a potential to shave off another 0.578 seconds. Implementing the changes dropped the running time to 3.5233 seconds, shaving off 0.6417 seconds (3.2719 seconds with SSE2 enabled). Very happy with the results since it only took an hour or so to implement. (All tests run on 1,000,000 point drawings, benchmarks attached in Excel spread sheet below)

At this point I figure I'm done trying to optimize this thing to the hilt and decide additional benchmarks are in order using different query ranges. The first benchmarks were to compare the KdTree version with the SSE2 compiler setting off and on. The results were a bit of a surprise in that the SSE2 enabled version only ever so slightly edged out the none SSE2 version, consistently running less than a second faster. The timings for 4000, 5000, and 10,000 show the non-SSE2 winning, but I suppose that could have been some OS activity taking place in the background when the test was running.  Given that the KdTree is more data driven than computation driven explains some of why SSE2 didn't provide more of an edge.

The next task was to run benchmarks on the DandC (Divide and Conquer algorithm) version (SSE2 enabled). For query ranges less than 200 the DandC beats out the KdTree version. DandC has very little overhead in the way of data structures so it can start cranking on the problem right away. As the query ranges increase the speed of DandC becomes very bad. For example, querying a range of 10,000 with DandC takes about 2 hrs 15 minutes to complete. The same query range with KdTree took about 19 minutes (15.8 minutes after another optimization was done).
(http://lh3.ggpht.com/_gZ879aDg7Ms/TJeotddK1II/AAAAAAAAAFQ/7DhRAXLwwYM/Timings2.png)
(http://lh4.ggpht.com/_gZ879aDg7Ms/TJeotPkISDI/AAAAAAAAAFM/gCAuP1zjCxI/Timings1.png)

After the benchmarks were run I found another piece in the code to optimize. So I ran another set of benchmarks for comparison. The optimization was to simplify 2 if statements and precompute a value that had the potential to be used (computed) twice.
(http://lh6.ggpht.com/_gZ879aDg7Ms/TJeotS-5-VI/AAAAAAAAAFU/zaEZGBa4SnI/Timings3.png)

A sampling of the query ranges performed
(http://lh3.ggpht.com/_gZ879aDg7Ms/TJeot3WwUUI/AAAAAAAAAFY/ACGMaNO9gSg/Timings4.png)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 20, 2010, 10:53:14 PM
C++ Source code and binary attached to this post.

Had a chance to clean up the code and update the comments. I've also regenerated the programmers docs and added those as a separate attachment.

For those that want to compare changes made since the prior project posting, look at these areas in the code. These changed areas had the biggest impact.
Code: [Select]
Functions -
int __fastcall CKdTreeT::FindNearest(KDNODE<T, DIM> * pNode, RESULT_NODE<T, DIM> *pList);
KDNODE<T, DIM> * __fastcall CKdTreeT::FindMinimum(const KDNODE<T, DIM> * pNode, int axis, int dir) const;
KDNODE<T, DIM> * __fastcall CKdTreeT::EraseNode(KDNODE<T, DIM> * pNode, const T * pos, int dir) const;
struct KDNODE

Header file
MemoryPool.h (and few functions in KdTreeT that use the memory pool)

Most of the other changes whether be code structure, or data layout. contributed to the low running times, but only a small amount. I did lots of benchmarking, testing, and looking at the assembly output. If the change reduced the running time, no matter how small, it was a keeper. BTW, during code cleanup, made another change (included in source code) and know the 10,000 query range sample runs 26 seconds faster.

Thanks everyone for putting up with my posts, especially all the non-C++ types. This was more an educational exercise for myself, and what I've documented here is only a small fraction of what I got out of it.

Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 20, 2010, 11:09:14 PM
Figured I'd better split this info from the previous post -

...And just to restate, almost every major improvement came from simplifying the code, mostly if statements. For instance take the if statement below, where pNode->right and pNode->left are pointers, and we want to see if the are both NULL.
Code: [Select]
if(!pNode->right && !pNode->left) { ... }As explained in an earlier post. This is doing 2 if compares; if pNode->right is NULL AND if pNode->left is NULL. This can be simplified (in terms of the computer) to
Code: [Select]
if(!pNode->right & !pNode->left) { ... }
Now it's a single if statement doing a "logical and" of right and left, which provides a nice little speed increase. Though the assembly output is better with this change, it didn't look very efficent
Code: [Select]
; 1257 :             if( !pNode->right & !pNode->left) {
  000ba 33 c9 xor ecx, ecx
  000bc 85 c0 test eax, eax
  000be 0f 94 c1 sete cl
  000c1 33 db xor ebx, ebx
  000c3 85 d2 test edx, edx
  000c5 0f 94 c3 sete bl
  000c8 85 cb test ecx, ebx
  000ca 74 0a je SHORT $LN5@EraseNode

At some point a forgotten memory was ebbing it's way to the surface. Something about doing a "not not" operation on a pointer being fast.  Ok, reform the if statement to use !! on the pointers ...
Code: [Select]
if(!( !!pNode->right & !!pNode->left)) { ... }
The !!SomePointer tells the compiler to first negate the pointer to true/false based on the value of the pointer, and then negate the true/false result. The results for both pointers are "logically and" together then that result is inverted for the final answer.

Running the benchmark showed a very nice improvement. The "KdTree w/SSE2 & mod*" section in the timings chart reflects this change.

So what's the assembly look like for this statement?
Code: [Select]
; 1257 :             if(!( !!pNode->right & !!pNode->left)) {
  000aa 8b 5c 24 1c mov ebx, DWORD PTR tv271[esp+16]
  000ae 85 c3 test eax, ebx
  000b0 75 0a jne SHORT $LN5@EraseNode
That's rockin'!

Again, thanks everyone.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: It's Alive! on September 21, 2010, 03:06:00 AM
Now you need to make a version for X64. doubles are stuck into the XMM registers by default, all you need to do is pack them  :evil:
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 21, 2010, 03:55:21 AM
Now you need to make a version for X64. doubles are stuck into the XMM registers by default, all you need to do is pack them  :evil:

The seeds for that thought are already planted. Probably on the next rain of boredom will they germinate.  :-D
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: CAB on September 21, 2010, 09:27:47 AM
Programmer and a poet!  8-)
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 22, 2010, 03:11:18 PM
Now you need to make a version for X64. doubles are stuck into the XMM registers by default, all you need to do is pack them  :evil:

The program know runs from the command prompt.

Timings for a couple different ranges and compilers
MSVC 2008 32    - (70) 2.94975, (100) 3.49471
MSVC 2008 x64   - (70) 3.03533, (100) 3.60556
X64 version is running slower than 32 bit. For the ranges 70 = -0.15, 100 = -0.18, 500 = 0.85. So it's actually getting worse the larger the range.

MSVC 2010 32    - (70) 3.09298, (100) 3.72199
MSVC 2010 x64   - (70) 3.04821, (100) 3.63488

32 bit VS 2008 version blows the doors off any version. Though that should be taken with a grain of salt, as I just compiled and run the 2010 moments ago and haven't dig into the compiler options.

In the meantime, going through the code to make sure the correct width variables are being passed around. Also, have gone through different for compiler settings VC 2008 x64, so far nothing jumps out.

After I pack the data to a manageable size I'll release it and the code.

Found this on MSDN
Quote
When you compile applications as 64-bit, the calculations get more complicated. A 64-bit program uses 64-bit pointers, and its instructions are slightly larger, so the memory requirement is slightly increased. This can cause a slight drop in performance. On the other hand, having twice as many registers and having the ability to do 64-bit integer calculations in a single instruction will often more than compensate. The net result is that a 64-bit application might run slightly slower than the same application compiled as 32-bit, but it will often run slightly faster.




Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: highflyingbird on September 23, 2010, 11:32:36 AM
It looks like  Divide and Conquer has a good performance with a small distance.
and  it can be converted to a LISP code.but It's very difficult for KDtree algorithm.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: pkohut on September 23, 2010, 01:03:39 PM
It looks like  Divide and Conquer has a good performance with a small distance.
and  it can be converted to a LISP code.but It's very difficult for KDtree algorithm.

Each has their place. The divide and conquer routine is tightly coupled to the problem statement (non-reusable), and the Kd-tree version is loosely coupled (reusable).

The challenge for this thread offered an opportunity to present the Kd-Tree algorithm, something I've reused on a number of occasions and where it ran fast enough. Except for the major refactoring (to templates) and optimizations of the code, it has been plug and play technology for several years.

For this challenge, "fast enough" in the past wasn't good at all. Based on the criteria of the challenge, the routine initially performed really bad, being pushed past it's limits. All of the performance issues were implementation details that were easily fixed.

As far as implementing in lisp, I don't see why not. I wouldn't want to do it, but those scary looking data structures with pointers, those are just nicely wrapped C structures. Lay 'em out flat and index, like we use to do before OOP.
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: gile on June 18, 2013, 06:39:41 PM
Hi,

I know this is a (very) old thread, but after trying the k-d tree route (http://www.theswamp.org/index.php?topic=44312.msg496460#msg496460) with C#, I've been playing with the divide and conquer one (using xy grid (http://www.theswamp.org/index.php?topic=32874.msg383679#msg383679) as shown by qjchen) and F#.

The code is quite concise and performances not so bad 8-).

Code - F#: [Select]
  1. module ConnectPointChallenge
  2.  
  3. open Autodesk.AutoCAD.DatabaseServices
  4. open Autodesk.AutoCAD.EditorInput
  5. open Autodesk.AutoCAD.Geometry
  6. open Autodesk.AutoCAD.Runtime
  7.  
  8. type AcAp = Autodesk.AutoCAD.ApplicationServices.Application
  9.  
  10. let connectAll (pts: Point3d[]) dist =
  11.     let getExtents pts =
  12.         Seq.fold(fun (s: Extents3d) p -> s.AddPoint(p); s) (new Extents3d()) pts
  13.     let ext = getExtents pts
  14.     let xMax, yMax = ext.MaxPoint.X, ext.MaxPoint.Y
  15.     let xMin, yMin = ext.MinPoint.X, ext.MinPoint.Y
  16.     let col, row = int ((xMax - xMin) / dist) + 2, int ((yMax - yMin) / dist) + 3
  17.     let table = Array2D.create col row []
  18.     pts |> Seq.iter(fun p ->
  19.                 let i, j = int ((p.X - xMin) / dist), int ((p.Y - yMin) / dist) + 1
  20.                 table.[i, j] <- p :: table.[i, j])
  21.     let dist = dist * dist
  22.  
  23.     let getConnexions i j =
  24.         let sqrDist2d (p1: Point3d) (p2: Point3d) =
  25.             (p1.X - p2.X) * (p1.X - p2.X) + (p1.Y - p2.Y) * (p1.Y - p2.Y)
  26.         let connect pt pts acc =
  27.             List.fold(fun s p -> if sqrDist2d pt p <= dist then (pt, p) :: s else s) acc pts
  28.         let connectInside pts =
  29.             let rec loop acc = function
  30.                 | [] -> acc
  31.                 | h :: t -> loop (connect h t acc) t
  32.             loop [] pts
  33.         let connectOutside pts acc =
  34.             List.fold(fun s p -> connect p pts s) acc table.[i,j]
  35.         table.[i,j]
  36.         |> connectInside
  37.         |> connectOutside table.[i + 1, j - 1]
  38.         |> connectOutside table.[i + 1, j]
  39.         |> connectOutside table.[i + 1, j + 1]
  40.         |> connectOutside table.[i, j + 1]
  41.  
  42.     [| for i in 0 .. col - 2 do
  43.         for j in 1 .. row - 2 do yield! getConnexions i j |]
  44.  
  45. [<CommandMethod("ConnectFs")>]
  46. let connectFs () =
  47.     let sw = new System.Diagnostics.Stopwatch()
  48.     let doc = AcAp.DocumentManager.MdiActiveDocument
  49.     let db = doc.Database
  50.     let ed = doc.Editor
  51.     sw.Start()
  52.     use ms = SymbolUtilityServices.GetBlockModelSpaceId(db).Open(OpenMode.ForWrite)
  53.              :?> BlockTableRecord
  54.     let rxc = RXClass.GetClass(typeof<DBPoint>)
  55.     let pts = [| for id in ms -> id |]
  56.               |> Array.filter(fun id -> id.ObjectClass = rxc)
  57.               |> Array.map(fun id ->
  58.                     use pt = id.Open(OpenMode.ForRead) :?> DBPoint
  59.                     pt.Position)
  60.     let t0 = sw.ElapsedMilliseconds
  61.     let pairs = connectAll pts 70.0
  62.     let t1 = sw.ElapsedMilliseconds
  63.     pairs
  64.     |> Array.iter (fun (p1, p2) ->
  65.         use line = new Line(p1, p2)
  66.         ms.AppendEntity(line) |> ignore)
  67.     sw.Stop()
  68.     let t2 = sw.ElapsedMilliseconds
  69.     ed.WriteMessage(
  70.         "\nGet points:{0,9}\nConnect points:{1,5}\nDraw lines:{2,9}\nTotal:{3,14}",
  71.         t0, t1 - t0, t2 - t1, t2);

Code: [Select]
10K points => 12,718 lines
Elapsed milliseconds:
Get points:       35
Connect points:   21
Draw lines:       56
Total:           112


100k points => 128,026 lines
Elapsed milliseconds:
Get points:      254
Connect points:  272
Draw lines:      633
Total:          1159


1M points => 1,280,741 lines
Elapsed milliseconds:
Get points:     2453
Connect points: 2792
Draw lines:    11748
Total:         16993
Title: Re: {Challenge} connect each two points whose distance is shorter than given value
Post by: qjchen on June 20, 2013, 06:53:29 AM
Thank you very much, gile.

It is really quite concise.

I am regret that recently I am too busy to write programs.

I hope I could learn your code carefully for study.