Author Topic: Findfile recursive ..  (Read 11300 times)

0 Members and 1 Guest are viewing this topic.

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8659
  • AKA Daniel
Re: Findfile recursive ..
« Reply #15 on: December 21, 2008, 11:24:53 PM »
I’m in 17T but will be in 51R in February.

It's too cold in 17T   :lol:

TonyT

  • Guest
Re: Findfile recursive ..
« Reply #16 on: December 21, 2008, 11:25:27 PM »
Quote

   SearchOption.AllDirectories


From the docs: ''...follows symbolic links...', which is
microsoft-ese for 'follows shortcuts'.

IOW, when it finds a shortcut to a folder that was
already processed in a child of that folder, it will
process the parent folder again.

IOW, this option is dangerous because it can lead
to infinite recursion.


Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Re: Findfile recursive ..
« Reply #17 on: December 21, 2008, 11:27:29 PM »
Dan,
that  sounds like really bad poetry lyrics
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8659
  • AKA Daniel
Re: Findfile recursive ..
« Reply #18 on: December 21, 2008, 11:38:46 PM »
WooahYa I’m down and out in 17T where the wind chill is -17°C   :laugh:

Kerry

  • Mesozoic relic
  • Seagull
  • Posts: 11654
  • class keyThumper<T>:ILazy<T>
Re: Findfile recursive ..
« Reply #19 on: December 21, 2008, 11:42:41 PM »
Quote

   SearchOption.AllDirectories


From the docs: ''...follows symbolic links...', which is
microsoft-ese for 'follows shortcuts'.

IOW, when it finds a shortcut to a folder that was
already processed in a child of that folder, it will
process the parent folder again.

IOW, this option is dangerous because it can lead
to infinite recursion.



First Time I've read that Tony ... and the SearchOption.AllDirectories has been around for a while ... surprised it's not plastered all over the web.
kdub, kdub_nz in other timelines.
Perfection is not optional.
Everything will work just as you expect it to, unless your expectations are incorrect.
Discipline: None at all.

TonyT

  • Guest
Re: Findfile recursive ..
« Reply #20 on: December 22, 2008, 05:29:17 AM »
Actually, I think my translation of Microsoft-ese may have been
incorrect. Reparse points (like mapped drives) that can create a
circular or cyclical reference, is the problem, not shortcuts.

http://msdn.microsoft.com/en-us/library/ms143448.aspx   

(See the remarks)

The following link also has some useful information on traversing
the file system, and other potential pitfalls like directory access
rights, and so on:

http://msdn.microsoft.com/en-us/library/bb513869.aspx

As far as 'stack-o-phobia' goes :D, the depth of recursion equals
the depth of folder nesting and from my calculations, unless there
are folders nested ~250 levels deep or more, I think recursion is
a safe approach. The use of the stack can also be minimized by
using non-static methods and storing the current folder in a non
static member variable, rather than passing it as an argument to
the recursive method (which is stored in each call's stack frame).

CLR 4.0 has tail-recursion optimization, but only for F#, by way of
that compiler. From what I've read, the problem with tail-recursion
optimization in .NET is that the security model relies heavily on call
stack introspection to detect the calling context of methods (e.g.,
examine the call stack to see who is calling your method), which
can preclude optimizing tail-recursive calls into iterative loops.

One reason that purely-recursive or functional approaches with no
side-effects are desirable for solving problems like this, is that they
are better suited for parallel execution (even though in this case,
disk I/O is the likely bottleneck and limiting factor).

A stack-based approach could not easily run on mutliple CPU cores.

Quote

   SearchOption.AllDirectories


From the docs: ''...follows symbolic links...', which is
microsoft-ese for 'follows shortcuts'.

IOW, when it finds a shortcut to a folder that was
already processed in a child of that folder, it will
process the parent folder again.

IOW, this option is dangerous because it can lead
to infinite recursion.



First Time I've read that Tony ... and the SearchOption.AllDirectories has been around for a while ... surprised it's not plastered all over the web.
« Last Edit: December 22, 2008, 06:06:41 AM by TonyT »

Draftek

  • Guest
Re: Findfile recursive ..
« Reply #21 on: December 22, 2008, 07:52:11 AM »
Excellent Thread!
Thanks guys.

T.Willey

  • Needs a day job
  • Posts: 5251
Re: Findfile recursive ..
« Reply #22 on: December 22, 2008, 03:29:34 PM »
Couldn't you just use ' DirectoryInfo.GetFiles ' and have the search option set to ' AllDirectories '?  Untested, but I have this in my batch plot program, learned from Glenn, and it works with a filter for ' *.dwg '.

MSDN page for DirectoryInfo.GetFiles
[ http://msdn.microsoft.com/en-us/library/ms143327.aspx ]
Tim

I don't want to ' end-up ', I want to ' become '. - Me

Please think about donating if this post helped you.

TonyT

  • Guest
Re: Findfile recursive ..
« Reply #23 on: December 23, 2008, 09:27:04 AM »
Couldn't you just use ' DirectoryInfo.GetFiles ' and have the search option set to ' AllDirectories '?  Untested, but I have this in my batch plot program, learned from Glenn, and it works with a filter for ' *.dwg '.

MSDN page for DirectoryInfo.GetFiles
[ http://msdn.microsoft.com/en-us/library/ms143327.aspx ]

Yes you can, but as mentioned on the page at the link I posted,
the entire operation fails if you don't have sufficient access rights
(as rare as that may be) for every sub directory.

If you traverse the folder tree manually, you can handle the security
exceptions that are thrown if you don't have sufficient rights for a
given folder, log the error or do whatever else you want to do, and
continue processing folders, without the entire operation failing.


T.Willey

  • Needs a day job
  • Posts: 5251
Re: Findfile recursive ..
« Reply #24 on: December 23, 2008, 11:01:05 AM »
Couldn't you just use ' DirectoryInfo.GetFiles ' and have the search option set to ' AllDirectories '?  Untested, but I have this in my batch plot program, learned from Glenn, and it works with a filter for ' *.dwg '.

MSDN page for DirectoryInfo.GetFiles
[ http://msdn.microsoft.com/en-us/library/ms143327.aspx ]

Yes you can, but as mentioned on the page at the link I posted,
the entire operation fails if you don't have sufficient access rights
(as rare as that may be) for every sub directory.

If you traverse the folder tree manually, you can handle the security
exceptions that are thrown if you don't have sufficient rights for a
given folder, log the error or do whatever else you want to do, and
continue processing folders, without the entire operation failing.



Thanks Tony.  When I get a chance I'll go and read the pages you linked to.
Tim

I don't want to ' end-up ', I want to ' become '. - Me

Please think about donating if this post helped you.

TonyT

  • Guest
Re: Findfile recursive ..
« Reply #25 on: December 23, 2008, 11:56:56 AM »
And if you are only searching for files based on a pattern, it may not
matter if you have access rights, so AllDirectories may work in any
case. In my own file search class, I can do more specialized searches
that involve accessing each file (e.g., modified after a certain date,
or what have you), which means that my code must have the needed
access rights.

Couldn't you just use ' DirectoryInfo.GetFiles ' and have the search option set to ' AllDirectories '?  Untested, but I have this in my batch plot program, learned from Glenn, and it works with a filter for ' *.dwg '.

MSDN page for DirectoryInfo.GetFiles
[ http://msdn.microsoft.com/en-us/library/ms143327.aspx ]

Yes you can, but as mentioned on the page at the link I posted,
the entire operation fails if you don't have sufficient access rights
(as rare as that may be) for every sub directory.

If you traverse the folder tree manually, you can handle the security
exceptions that are thrown if you don't have sufficient rights for a
given folder, log the error or do whatever else you want to do, and
continue processing folders, without the entire operation failing.



Thanks Tony.  When I get a chance I'll go and read the pages you linked to.

gile

  • Gator
  • Posts: 2507
  • Marseille, France
Re: Findfile recursive ..
« Reply #26 on: November 10, 2010, 05:53:50 PM »
Hi,

I know this is an old topic, but while I'm trying to have a look around F#, I thaught it was a good exercice.

A 'try ... with' ( as C# 'try ... catch') statements used to ignore the limited access folders

Code: [Select]
let FindFiles folder pattern =
    let lst = ref List.Empty
    let rec foo dir pat =
        for f in Directory.GetFiles(dir, pat) do lst := f :: !lst
        for d in Directory.GetDirectories(dir) do
            try foo d pat with | ex -> ignore()
    foo folder pattern
    List.toArray(List.rev(!lst))
Speaking English as a French Frog

kaefer

  • Guest
Re: Findfile recursive ..
« Reply #27 on: November 11, 2010, 04:21:54 AM »
Hi,

I know this is an old topic, but while I'm trying to have a look around F#, I thaught it was a good exercice.

A 'try ... with' ( as C# 'try ... catch') statements used to ignore the limited access folders

Code: [Select]
let FindFiles folder pattern =
    let lst = ref List.Empty
    let rec foo dir pat =
        for f in Directory.GetFiles(dir, pat) do lst := f :: !lst
        for d in Directory.GetDirectories(dir) do
            try foo d pat with | ex -> ignore()
    foo folder pattern
    List.toArray(List.rev(!lst))

Hi Gile,

welcome to the club! (Cue J.H.Marx: "I don’t care to belong to any club that will have me as a member.")

The canonical example of recursive directory search is of course here: http://www.expert-fsharp.com/CodeSamples/Chapter03/Example08/script.fsx, from Expert F# by Don Syme, Adam Granicz and Antonio Cisternino.

Modifying that with the requirement to ignore dirs with insufficient permission and to return an array yields:
Code: [Select]
let FindFiles folder pat =
    let rec foo dir =
        try
            Directory.GetDirectories dir |> Array.collect foo
            |> Array.append (Directory.GetFiles(dir, pat))
        with _ -> Array.empty
    foo folder

Have lots of fun! Thorsten

gile

  • Gator
  • Posts: 2507
  • Marseille, France
Re: Findfile recursive ..
« Reply #28 on: November 11, 2010, 05:17:45 AM »
Thank you very much kaefer :-o
Speaking English as a French Frog

gile

  • Gator
  • Posts: 2507
  • Marseille, France
Re: Findfile recursive ..
« Reply #29 on: November 12, 2010, 02:41:15 AM »
Hi,

It seems there's no need to use a sub function:
Code: [Select]
let rec FindFiles dir pat =
    try
        Directory.GetDirectories dir
        |> Array.collect (fun x -> FindFiles x pat)
        |> Array.append (Directory.GetFiles (dir, pat))
    with _ -> Array.empty
Speaking English as a French Frog