Author Topic: Reducing the file size to make large contour set... help  (Read 13511 times)

0 Members and 1 Guest are viewing this topic.

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #15 on: March 22, 2010, 12:47:24 PM »
The test is relevant to show that weeding can indeed introduce error in the resultant surface built from contours.
LIDAR data is another animal, and one will in most instances need to use judgement in reducing that data to create a usable surface model from.

Well, of course weeding can introduce error.  That's a given, and we don't need a test to prove it.

The goal in this sort of task is simply to reduce our Civil 3D surface to a point where we have an acceptable trade-off between performance and accuracy.

And the only thing that's important for that is, after we try some weeding factors, to compare the resulting contours with the starting contours.  If the results are "close enough" for whatever purposes we intend, then we can proceed.  We do not need to worry at all about the fact that the contours originally came from Lidar data.

sourdough

  • Bull Frog
  • Posts: 367
Re: Reducing the file size to make large contour set... help
« Reply #16 on: March 22, 2010, 01:05:42 PM »
Hi Sinc
    I have tried some variations of the weeding and the results of what the contours look like or don't seem hard to submit
in a plat that I would want to turn in.  My example was taken from Lidar and made into a 3d Poly. So, wanting to make a surface
with it is where it got to be challenge. I tried weeding and the contours on the top of mesa's started to make nonsense contours.
So, this is a working from a 1/2 section of data. I am hoping to use data shortcuts to work with the surface and that seems to work.
What was happening in the experience over this weekend of experimenting was that I had to switch from Win 7 32bit to Win 7 64bit
because of memory limitations. Now after playing with the large file of contours it died after 4.7 gig of memory used. I have plenty of memory
and set page file to 12 gig max and wondered why it died so quickly. I thought Civil 3d 2010 could handle it. Am I the only one finding a
another limitation with filesize and Civil 3D?  This has to be answered first before I can start to understand the procedures to use for
working on large projects and phasing. Your thoughts?

MJP


The test is relevant to show that weeding can indeed introduce error in the resultant surface built from contours.
LIDAR data is another animal, and one will in most instances need to use judgement in reducing that data to create a usable surface model from.

Well, of course weeding can introduce error.  That's a given, and we don't need a test to prove it.

The goal in this sort of task is simply to reduce our Civil 3D surface to a point where we have an acceptable trade-off between performance and accuracy.

And the only thing that's important for that is, after we try some weeding factors, to compare the resulting contours with the starting contours.  If the results are "close enough" for whatever purposes we intend, then we can proceed.  We do not need to worry at all about the fact that the contours originally came from Lidar data.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14444
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #17 on: March 22, 2010, 01:27:26 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

pkohut

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #18 on: March 22, 2010, 01:50:52 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.

Don't know how fast C3D is for doing this, but LDD is extreamly slow. I've a standalone routine to create LDD surface from grid data that could be imported to C3D.

mjfarrell

  • Seagull
  • Posts: 14444
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #19 on: March 22, 2010, 01:56:07 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.

Don't know how fast C3D is for doing this, but LDD is extreamly slow. I've a standalone routine to create LDD surface from grid data that could be imported to C3D.

All well and good however, the user is NOT running LDD (to the best of my knowledge).
Further I have not noticed any issues within C3D, and using external point database(s) for surfaces.
Additionally, being this is LIDAR data, or the origins of the polylines he is attempting to use was, there is NO GRID of data.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

pkohut

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #20 on: March 22, 2010, 02:06:08 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.

Don't know how fast C3D is for doing this, but LDD is extreamly slow. I've a standalone routine to create LDD surface from grid data that could be imported to C3D.

All well and good however, the user is NOT running LDD (to the best of my knowledge).
Further I have not noticed any issues within C3D, and using external point database(s) for surfaces.
Additionally, being this is LIDAR data, or the origins of the polylines he is attempting to use was, there is NO GRID of data.


Fair enough, just providing alternatives to import large data sets.  I use Global Mapper to convert lidar data to ESRI GRD data, then import that to LDD. Takes 5 to 10 minutes total.  The routines are custom and if SourDog can use them for his purpose, they're available.

mjfarrell

  • Seagull
  • Posts: 14444
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #21 on: March 22, 2010, 02:08:15 PM »
Global Mapper is always in my tool kit....so powerful, so effective....for such a small investment!
However; I like to keep the 'solutions' I offer to the public within the abilities of the application they are trying to use.  As I am sure that you are aware of reluctance to employ third party solutions by some.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 367
Re: Reducing the file size to make large contour set... help
« Reply #22 on: March 22, 2010, 02:09:11 PM »
Hi Mike
   The file given wasn't in a point format to do that. I was told it was done by Lidar, and put into 3D poly format in a dwg.
Sorry, that I didn't make that more clear.
Mike


It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14444
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #23 on: March 22, 2010, 02:12:20 PM »
Hi Mike
   The file given wasn't in a point format to do that. I was told it was done by Lidar, and put into 3D poly format in a dwg.
Sorry, that I didn't make that more clear.
Mike


It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.
That part was clear...however I was ever hopeful that the source data could be at hand.
Could be a good time to contact the source and request the original data IF possible.  Just state your challenges, and ask for the original information as a bridge to the solution.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #24 on: March 22, 2010, 02:23:42 PM »
Now after playing with the large file of contours it died after 4.7 gig of memory used. I have plenty of memory
and set page file to 12 gig max and wondered why it died so quickly. I thought Civil 3d 2010 could handle it. Am I the only one finding a
another limitation with filesize and Civil 3D?  This has to be answered first before I can start to understand the procedures to use for
working on large projects and phasing. Your thoughts?

Not sure what you mean by "4.7 gig of memory used".  C3D 2010 is a 32-bit app, and as such, it can only access a max of 4GB of RAM.  However, due to other constraints, you won't see it go that high.  On a 64-bit OS, I generally seem to hit an "out of memory" error once C3D is using about 3GB of memory.

If you have at least 6GB of RAM in your system, the size of your pagefile is relatively immaterial, since pretty much everything C3D uses will be able to stay in RAM (unless you have other programs running simultaneously, and those other programs are also using lots of RAM).

As far as your problems, I think a lot of it is simply because you are trying to generate a surface from contours.  That's simply the worst way to go about it.  If you have no other choice but to use the contours you have, then I'm not sure what you need to do.  I can think of all kinds of nasty solutions, such as building C3D surfaces in chunks and using them to generate points, which you then put into a Point File and use to generate a surface from an external Point File, and things like that.  But they're all kind of nasty solutions, of the "last resort" kind.

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #25 on: March 22, 2010, 02:30:58 PM »
As another possibility, you might get something useful from running the Map Cleanup tools on your contours, to weed down the polylines before trying to add them to the surface.  Possibly, using a combination of Map Cleanup and the surface weeding options will get you something usable.

mjfarrell

  • Seagull
  • Posts: 14444
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #26 on: March 22, 2010, 02:43:05 PM »
going the MAP path....one could query the data, extracting coordinates from the polylines, to external file...then ADD that point file to the surface...
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 367
Re: Reducing the file size to make large contour set... help
« Reply #27 on: March 22, 2010, 02:48:09 PM »
Sinc
    Thanks for that info. I am using Win 7 64 and I really did get it to use just for that program 4.something worth of memory according
to my resource manager. I turned off most all my other programs to do it. So, it probably did max out. I thought even though the
program is 32bit that in a 64 bit os it would go further. Tell me that it might be fixed in 2011 C3D, please. I'll try your ideas to. The info
originally came from city and for preliminary design concept, NOT Final, is supposed to be accurate to within 1/2'. But I can't get the original
point data, so stuck until they actually put a survey crew on the ground. The idea behind the original question and follow up and reaching
limits of Civil 3D 2010 are becoming clearer though. One day maybe, someday and could be would be really nice today, though.

MJP

As another possibility, you might get something useful from running the Map Cleanup tools on your contours, to weed down the polylines before trying to add them to the surface.  Possibly, using a combination of Map Cleanup and the surface weeding options will get you something usable.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #28 on: March 22, 2010, 02:58:28 PM »
Tell me that it might be fixed in 2011 C3D, please.

OK...  "It might be fixed in 2011 C3D."    :-)

We're still in the "those who know can't say, those who say don't know" phase.  All I can tell you is that on the other forums recently, Peter Funk strongly encouraged people who are buying new machines right now to go with Win 7 x64 as their OS, because it will position them for "future versions of Civil 3D".  Read that however you will.

sourdough

  • Bull Frog
  • Posts: 367
Re: Reducing the file size to make large contour set... help
« Reply #29 on: March 22, 2010, 05:13:17 PM »
Sinc,

Well, that is good to know. Thanks for the hint. (On the upgrade path though I'm going to go with
Xeon Processor next time out. I did some research on the latest and for the money (not 6 core) the
Xeon X5520 is going to cost less then $350 and have 5.8 GT/s output, where as lesser including the I7 core less
than 960 series only do 4.8 GT/s. So, on the upward path I'm sure everyone is going to do even more
research to make C3D 20** can do. Just a passing thought.

Mike
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit