Author Topic: Remove duplicates in a list  (Read 3459 times)

0 Members and 1 Guest are viewing this topic.

whdjr

  • Guest
Remove duplicates in a list
« on: December 02, 2005, 09:39:24 AM »
I have this list:

an example:
Code: [Select]
(list '("me" "5" "3" "4" "5") '("you" "6" "7" "8" "10") '("him" "6" "7" "8" "10")
      '("our" "6" "7" "8" "10") '("you" "6" "7" "8" "10") '("her" "6" "7" "8" "10")
      '("we" "6" "7" "8" "10"))
)
All I am concerned about is the first element.  I want to be able to scan this list and remove all but but one copy of the items that are duplicate only.

Using the above list as an example I want it to return this:
Code: [Select]
(list '("me" "5" "3" "4" "5")
      '("you" "6" "7" "8" "10")
      '("him" "6" "7" "8" "10")
      '("our" "6" "7" "8" "10")
      '("her" "6" "7" "8" "10")
      '("we" "6" "7" "8" "10"))

The second item that started with "you" got removed because it was a duplicate.

Can anyone help me out here? Maybe some suggestions?

Thanks,

MP

  • Seagull
  • Posts: 17529
Re: Remove duplicates in a list
« Reply #1 on: December 02, 2005, 09:55:26 AM »
Code: [Select]
(defun unique ( lst / result )
    (foreach item lst
        (if (null (member item result))
            (setq result (cons item result))
        )
    )
    (reverse result)
)

Coded blind but should do it.
\|// Set goal. Experiment tirelessly until
|Oo| practice has become expertise.  Loop.
|- | LinkedIn | Dropbox

whdjr

  • Guest
Re: Remove duplicates in a list
« Reply #2 on: December 02, 2005, 10:06:08 AM »
Thanks MP.

Very close but needs a little 'bit 'o tweeking.
It removes a list if every item is the two lists are the same but not if only some are the same.

Using this example list:
Code: [Select]
(list '("me" "5" "3" "4" "5")
       '("you" "6" "7" "8" "10")
       '("him" "6" "7" "8" "10")
       '("you" "6" "7" "8" "10")
       '("her" "6" "7" "8" "10")
       '("me" "6" "7" "8" "10"))

It removes the second 'you' because each element is the same but it doesn't remove the second 'me'.

whdjr

  • Guest
Re: Remove duplicates in a list
« Reply #3 on: December 02, 2005, 10:24:35 AM »
MP upon further thinking I decided yo use your tool just the way it is.  I decided that my duplicate lists would always be the same and if they weren't then I needed to know that.

Thanks,

MP

  • Seagull
  • Posts: 17529
Re: Remove duplicates in a list
« Reply #4 on: December 02, 2005, 10:26:15 AM »
I thought you wanted duplicates removed??

Do you want items removed on the basis of duplicated cars (first item)? Then perhaps --

Code: [Select]
(defun unique2 ( lst / result )
    (foreach item lst
        (if (null (assoc (car item) result))
            (setq result (cons item result))
        )
    )
    (reverse result)
)

\|// Set goal. Experiment tirelessly until
|Oo| practice has become expertise.  Loop.
|- | LinkedIn | Dropbox

Fatty

  • Guest
Re: Remove duplicates in a list
« Reply #5 on: December 02, 2005, 10:51:38 AM »
And another one

 
Code: [Select]
   
(defun remove-first (lst /)
  (while
    (member (caar lst) (mapcar 'car (cdr lst)))
     (setq lst (vl-remove-if
(function (lambda (x)
     (eq (car x) (caar lst))
   )
)
lst
       )
     )
  )
  lst
)

Fatty

  • Guest
Re: Remove duplicates in a list
« Reply #6 on: December 02, 2005, 10:55:36 AM »
Oops

Sorry, forget about it
I did misplaced

Fatty

whdjr

  • Guest
Re: Remove duplicates in a list
« Reply #7 on: December 02, 2005, 10:59:46 AM »
I thought you wanted duplicates removed??

Do you want items removed on the basis of duplicated cars (first item)? Then perhaps --

...All I am concerned about is the first element. I want to be able to scan this list and remove all but but one copy of the items that are duplicate only...

This is what I asked for MP but you knew better and provided what I needed -- I just didn't realize it then. :-)

Thanks a bunch MP,
 :-) :-)

MP

  • Seagull
  • Posts: 17529
Re: Remove duplicates in a list
« Reply #8 on: December 02, 2005, 11:04:38 AM »
Thanks a bunch MP,
 :-) :-)

My pleasure Will.

:) :) :)
\|// Set goal. Experiment tirelessly until
|Oo| practice has become expertise.  Loop.
|- | LinkedIn | Dropbox

MP

  • Seagull
  • Posts: 17529
Re: Remove duplicates in a list
« Reply #9 on: December 02, 2005, 11:20:46 AM »
Mr. Oleg my friend. I think yours needs a tweak (perhaps that's what you meant by "I did misplaced"), it removes all items that have duplicates, that is, it returns --

(   
    ("him" "6" "7" "8" "10")
    ("her" "6" "7" "8" "10")
)


for Will's test list of --

'( 
    ("me" "5" "3" "4" "5")
    ("you" "6" "7" "8" "10")
    ("him" "6" "7" "8" "10")
    ("you" "6" "7" "8" "10")
    ("her" "6" "7" "8" "10")
    ("me" "6" "7" "8" "10")
)


When I believe he desired --

(   
    ("me" "5" "3" "4" "5")
    ("you" "6" "7" "8" "10")
    ("him" "6" "7" "8" "10")
    ("her" "6" "7" "8" "10")
)


Also, a little incidental note here, while vl-remove is convenient, and indeed I used it (lots), in cases like this its performance sometimes lags behind a roll your own variant using "classic" lisp equivalents (primarilly because of the insitu mapcar calls I'm guessing).

An informal test, based on unique2 and remove-first (as is) and Will's test list suggests unique2 is about 60% faster, though admittedly, sometimes this matters not.

Hope you don't mind my observations.

:)
\|// Set goal. Experiment tirelessly until
|Oo| practice has become expertise.  Loop.
|- | LinkedIn | Dropbox

MP

  • Seagull
  • Posts: 17529
Re: Remove duplicates in a list
« Reply #10 on: December 02, 2005, 11:29:32 AM »
!! If you use a list like this --

(setq lst
   '(   
        ("me" "5" "3" "4" "5")
        ("you" "6" "7" "8" "10")
        ("him" "6" "7" "8" "10")
        ("you" "6" "7" "8" "10")
        ("her" "6" "7" "8" "10")
        ("me" "6" "7" "8" "10")
    )
)

(repeat 8
    (setq lst
        (append lst lst)
    )
)


The difference in performance is even more dramatic, like > 300% !!
\|// Set goal. Experiment tirelessly until
|Oo| practice has become expertise.  Loop.
|- | LinkedIn | Dropbox

CAB

  • Global Moderator
  • Seagull
  • Posts: 10376
Re: Remove duplicates in a list
« Reply #11 on: December 02, 2005, 11:40:54 AM »
Just to see if I could find another way.
Not better mind you just different. :-)

Code: [Select]
  (defun unique (lst / result)
    (while (setq itm (car lst))
      (setq lst    (vl-remove itm lst)
            result (cons itm result)
      )
    )
  )
I've reached the age where the happy hour is a nap. (°¿°)
Windows 10 core i7 4790k 4Ghz 32GB GTX 970
Please support this web site.

Fatty

  • Guest
Re: Remove duplicates in a list
« Reply #12 on: December 02, 2005, 11:47:37 AM »
Quote
..............

Hope you don't mind my observations.

:)

   Thank you Michael, I certainly shall not begin to object, especially
    that yours explanations very detailed and correct.
    In addition I have incorrectly understood a problem that it was necessary to solve
    and because of it I has given the wrong answer to a question :oops:

    Oleg

MP

  • Seagull
  • Posts: 17529
Re: Remove duplicates in a list
« Reply #13 on: December 02, 2005, 11:48:17 AM »
Interesting performance Alan!

Elapsed milliseconds / relative speed for 256 iteration(s):

    (UNIQUE_CAB LST).....1172 / 2.52 <fastest>
    (UNIQUE_MP LST)......2953 / 1.00 <slowest>
\|// Set goal. Experiment tirelessly until
|Oo| practice has become expertise.  Loop.
|- | LinkedIn | Dropbox

CAB

  • Global Moderator
  • Seagull
  • Posts: 10376
Re: Remove duplicates in a list
« Reply #14 on: December 02, 2005, 12:34:36 PM »
Wow, that was unexpected! :-o
Although the loop is shortened by using the while and therefore
duplicates are not tested, I think it must be the vl-remove
I've reached the age where the happy hour is a nap. (°¿°)
Windows 10 core i7 4790k 4Ghz 32GB GTX 970
Please support this web site.