Author Topic: Remove duplicate Strings in List  (Read 12226 times)

0 Members and 1 Guest are viewing this topic.

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #15 on: February 22, 2005, 11:09:52 AM »
Michael,

Code: [Select]
(benchmark '((removeduplicatestrings full_list)
             (RemoveDuplicates full_list)
             (remove_dupSe7en full_list)
             (remove_dup2 full_list)            
             (remove_dup full_list)
            )
)
Benchmarking ...............Elapsed milliseconds / relative speed for 4096 iteration(s):

    (REMOVE_DUP FULL_LIST).................3075 / 3.0719 <slowest>
    (REMOVE_DUP2 FULL_LIST)................2193 / 2.1908
    (REMOVE_DUPSE7EN FULL_LIST)............1902 / 1.9001
    (REMOVEDUPLICATES FULL_LIST)...........1532 / 1.5305
    (REMOVEDUPLICATESTRINGS FULL_LIST).....1001 / 1.0000 <fastest>


(setq lst '(1 2 3 4 5 6 7 8))
(repeat 10
    (setq lst
        (append lst lst)
    )
)

(benchmark '((removeduplicatestrings lst)
             (RemoveDuplicates lst)
             (remove_dupSe7en lst)
             (remove_dup2 lst)            
             (remove_dup lst)
            )
)
Benchmarking ...........Elapsed milliseconds / relative speed for 256 iteration(s):

    (REMOVE_DUP LST).................10004 / 5.3440 <slowest>
    (REMOVE_DUP2 LST).................8072 / 4.3120
    (REMOVE_DUPSE7EN LST).............6119 / 3.2687
    (REMOVEDUPLICATESTRINGS LST)......3756 / 2.0064
    (REMOVEDUPLICATES LST)............1872 / 1.0000 <fastest>
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #16 on: February 22, 2005, 11:19:47 AM »
Michael, looks like your REMOVEDUPLICATESFOO is a screamer for the String List.
.. and your REMOVEDUPLICATES for the numberlist

The differences are significant, heh ..

Code: [Select]

Benchmarking ...........Elapsed milliseconds / relative speed for 256 iteration(s):

    (REMOVE_DUP LST).................10496 / 5.4020 <slowest>
    (REMOVE_DUP2 LST).................8302 / 4.2728
    (REMOVE_DUPSE7EN LST).............6359 / 3.2728
    (REMOVEDUPLICATESTRINGS LST)......3936 / 2.0257
    (REMOVEDUPLICATESFOO LST).........2233 / 1.1493
    (REMOVEDUPLICATES LST)............1943 / 1.0000 <fastest>

Benchmarking ................Elapsed milliseconds / relative speed for 8192 iteration(s):

    (REMOVE_DUP FULL_LIST).................6129 / 3.7555 <slowest>
    (REMOVE_DUP2 FULL_LIST)................4476 / 2.7426
    (REMOVE_DUPSE7EN FULL_LIST)............3886 / 2.3811
    (REMOVEDUPLICATES FULL_LIST)...........2834 / 1.7365
    (REMOVEDUPLICATESTRINGS FULL_LIST).....1903 / 1.1661
    (REMOVEDUPLICATESFOO FULL_LIST)........1632 / 1.0000 <fastest>
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #17 on: February 22, 2005, 11:25:55 AM »
Quote from: MP
Also (Kerry) please tell me about the speedometer in your vehicle.

:cheesy:


You mean this ?
System:
     Microsoft Windows XP
     Professional

     Version 2002
     Service Pack 2

 Computer:
                   Intel(R)
     Pentium(R) 4 CPU 2.00GHz
     2.02 GHz, 1.00 GB of RAM
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #18 on: February 22, 2005, 11:29:01 AM »
Quote from: Se7en
< .. >But it looks like it did. If I'm reading your benchmark correct. I imporved the preformance of the procedure by over 60% by just changing one thing. (using a NAMED procedure instead of an UNNAMED.) Am i right?


ahh , I see what you mean now  .. Yes, That change is better.
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

MP

  • Seagull
  • Posts: 17750
  • Have thousands of dwgs to process? Contact me.
Remove duplicate Strings in List
« Reply #19 on: February 22, 2005, 11:50:05 AM »
Small tweak makes it faster yet --

Code: [Select]
(defun RemoveDuplicates ( lst / foo index )
    (defun foo (x)
        (cond
            ((vl-position x index))
            ((null (setq index (cons x index))))
        )
    )
    (vl-remove-if
       'foo
        lst
    )
)
Engineering Technologist • CAD Automation Practitioner
Automation ▸ Design ▸ Drafting ▸ Document Control ▸ Client
cadanalyst@gmail.comhttp://cadanalyst.slack.comhttp://linkedin.com/in/cadanalyst

MP

  • Seagull
  • Posts: 17750
  • Have thousands of dwgs to process? Contact me.
Remove duplicate Strings in List
« Reply #20 on: February 22, 2005, 06:16:53 PM »
And still greater performance is realized by separating the functions --

Code: [Select]
(defun RemoveDuplicatesAux ( x )
    (cond
        ((vl-position x index))
        ((null (setq index (cons x index))))
    )
)

(defun RemoveDuplicates ( lst / index )
    (vl-remove-if
       'RemoveDuplicatesAux
        lst
    )
)

I'm guessing you'd like to mention that variable 'index' is not local in RemoveDuplicatesAux. Tell me what you think the implications are. :)
Engineering Technologist • CAD Automation Practitioner
Automation ▸ Design ▸ Drafting ▸ Document Control ▸ Client
cadanalyst@gmail.comhttp://cadanalyst.slack.comhttp://linkedin.com/in/cadanalyst

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #21 on: February 22, 2005, 10:33:28 PM »
Thanks for your continued input on this topic Michael.

I think we may have discussed the nature of the personality aberration that manifests itself in this sort of dedication.


:D


My results indicate that this approach is not as fast as  the  wrapped foo approach for your number lists and is a winner for the Strings, which is where we started. The relative speed differences are that minimal now that other issues will come into consideration.

The declaration of the local variable fascinated me.  I was presumptuous enough to consider it redundant, untill I saw the results of removing it and realised why it was required.
..that would be a nightmare to debug for the unwary and is an excellent demonstration of variable scope.

Thanks Michael

Kerry.
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

MP

  • Seagull
  • Posts: 17750
  • Have thousands of dwgs to process? Contact me.
Remove duplicate Strings in List
« Reply #22 on: February 22, 2005, 11:40:00 PM »
Ahhh, I see you read the small print; excellent. I'll resist participating in dialogues about my abberant fascinations, we banned someone for that yesterday. Anyway ...

For those that may desire more illumination about variable scope consider the following (observe variable 'temp') ...

Code: [Select]
;;  demo function
;;  works with variable
;;  temp appropriate to
;;  lexical scope

(defun ConsXToVariableTemp ( x )
    (princ
        (strcat
            "=> temp : "
            (vl-princ-to-string
                (setq temp
                    (cons x temp)
                )
            )    
            "\n"
        )
    )
)

;;  separate function definition
;;  with nested (local) functions

(defun c:Test ( / implicit explicit temp )

    ;;  variable temp is local to function
    ;;  because it is an argument to the
    ;;  function
    (defun implicit ( temp )
       (ConsXToVariableTemp 1)
    )

    ;;  variable temp is local to function
    ;;  because it is formally declared as
    ;;  having local scope in the function
    ;;  definition
    (defun explicit ( / temp )
        (setq temp '(7 8 9))
        (ConsXToVariableTemp 6)
    )

    ;;  call function implicit, passing a
    ;;  small list as the parameter
    (implicit '(2 3 4))

    ;;  call function explicit, no parameters
    (explicit)

    ;;  define and invoke an annonymous function
    ;;  that explicitly declares temp as having
    ;;  local scope; parameterless
    (   (lambda ( / temp )
            (setq temp '(b c d))
            (ConsXToVariableTemp 'a)
        )
    )

    ;;  define and invoke an annonymous function
    ;;  that implicitly declares temp as having
    ;;  local scope because it's a parameter
    (   (lambda (temp)
            (ConsXToVariableTemp '+)
        )
       '(- * /) ;; passed as the parameter
    )

    ;;  call ConsXToVariableTemp directly after
    ;;  setting variable temp that has scope local
    ;;  to function c:Test
    (setq temp '(@ # $))
   
    (ConsXToVariableTemp '!)
   
    ;;  shhh
    (princ)
   
)

;;  before we call c:Test let's set
;;  global variable 'temp' to something

(setq temp "Lexical Scope, whoa.")

;;  show whats bound to temp

!temp

=> "Lexical Scope, whoa."

;;  perform the test run

(c:Test)

=> temp : (1 2 3 4)
=> temp : (6 7 8 9)
=> temp : (A B C D)
=> temp : (+ - * /)
=> temp : (! @ # $)

;;  show whats bound to temp

!temp

=> "Lexical Scope, whoa."

Kerry is 100% correct - code like this can be difficult to debug if you lose track of the scope. As such I normally avoid it myself and do not recommend it as normal practice.

And now for something different ...

Code: [Select]
(defun ConsToXQuotedVariable ( x QuotedVariable )
    (set QuotedVariable
        (cons x
            (eval QuotedVariable)
        )
    )
)

(setq temp '(2 3 4))

!temp

=> (2 3 4)

(ConsToXQuotedVariable 1 'temp)

!temp

=> (1 2 3 4)

Cheers.
Engineering Technologist • CAD Automation Practitioner
Automation ▸ Design ▸ Drafting ▸ Document Control ▸ Client
cadanalyst@gmail.comhttp://cadanalyst.slack.comhttp://linkedin.com/in/cadanalyst

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #23 on: February 22, 2005, 11:47:16 PM »
Quote
Ahhh, I see you read the small print;


ohhh, no, sneaky ..
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

MP

  • Seagull
  • Posts: 17750
  • Have thousands of dwgs to process? Contact me.
Remove duplicate Strings in List
« Reply #24 on: February 22, 2005, 11:49:54 PM »
Guilty as charged.
Engineering Technologist • CAD Automation Practitioner
Automation ▸ Design ▸ Drafting ▸ Document Control ▸ Client
cadanalyst@gmail.comhttp://cadanalyst.slack.comhttp://linkedin.com/in/cadanalyst

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Remove duplicate Strings in List
« Reply #25 on: February 23, 2005, 12:14:14 AM »
re the ConsToXQuotedVariable

I use these complimentary routines where applicable
Code: [Select]

(defun pop_in (val qvar)
  (set qvar (cons val (eval qvar)))
)
(defun pop_out (val qvar)
  (set qvar (vl-remove val (eval qvar)))
)
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

MP

  • Seagull
  • Posts: 17750
  • Have thousands of dwgs to process? Contact me.
Remove duplicate Strings in List
« Reply #26 on: February 23, 2005, 07:29:36 AM »
It's like you can see my library; scary.
Engineering Technologist • CAD Automation Practitioner
Automation ▸ Design ▸ Drafting ▸ Document Control ▸ Client
cadanalyst@gmail.comhttp://cadanalyst.slack.comhttp://linkedin.com/in/cadanalyst

Andrea

  • Water Moccasin
  • Posts: 2372
Remove duplicate Strings in List
« Reply #27 on: February 24, 2005, 12:20:14 PM »
this is my code...


Code: [Select]
(setq full_list (list "c" "a" "b" "a" "d" "b" "c" "e"))
(setq #list (vl-list-length full_list))
(setq ts1 "")
(repeat #list
  (if (not (vl-string-search (car full_list) ts1))
    (progn
    (setq ts1 (strcat ts1 (car full_list)))
    (setq full_list (cdr full_list))
    )
    (progn
    (setq #list (+ #list 1))
    (setq full_list (cdr full_list))
    )
))
Keep smile...