Author Topic: Reducing the file size to make large contour set... help  (Read 6924 times)

0 Members and 1 Guest are viewing this topic.

sourdough

  • Bull Frog
  • Posts: 362
Reducing the file size to make large contour set... help
« on: March 21, 2010, 08:34:36 PM »
Hi All
    I am working on really large files of 2d Polyline (were 3D poly) that are contours. My guess is that they were
part of a Lidar for topo. The file I'm working with is about 42 meg (2d poly) of data. When I try to make this into
a Surface in Civil 3D 2010 I have been killing my machine. I'm using Win 7 64bit with 8 Gig of memory with a
Intel I7 860 and I'm finding after creating contours for the surface that it turns into
about 200 meg file and is running out of memory, even though it only reaches 4.7gig used. What sort of weeding
factors should I use. I used the default 15 and 100 main numbers. I don't want to lose to much of the detail. So,
if anyone has some suggestions. This is just part of a really big project so the future is that I'm going to have to
link all these surface pieces today over time. I am going to use data shortcuts for sure, but just making the surfaces
with this large amount of data is just crazy. This sample is over 150 meg in snapshot.

MJP
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #1 on: March 21, 2010, 09:52:11 PM »
Mike,

From where I'm sitting, I would like to have the data and run a few experiments.

Absent that my best suggestion would be that you GRID the project area building the surfaces per grid region of a reasonable size, and then create Data Shortcuts for each Grid area, that you can then pull into the final design file.  Short of purchasing a CRAY you might find that adding 2 or 3 separate physical hard drives and configure them as the SWAP file for Windows to use.

Also I am going to suggest that you NOT use the default weeding factors, and that you even use MORE of the data than those numbers are currently using.  The reason for this is that as you WEED the data you are introducing ERROR into the model, that in the end will adversely affect any earthwork numbers you attempt to create.

If I get the data to play with I might come up with a different strategy for you to employ.

(I can't tell from your signature are you on the island, or back on the Mainland?)

Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #2 on: March 21, 2010, 10:39:41 PM »
Mike
    Thanks for your help. I sent a message for the method of delivery you would like for the file to work on. The area
shown is my attempt at adding to an area that was already sampled and made a surface from.
So, the area cut out was already done. The State from which is was done was northern New Mexico.
If you do have a Cray to give me, I'll take it.
MJP

Mike,

From where I'm sitting, I would like to have the data and run a few experiments.

Absent that my best suggestion would be that you GRID the project area building the surfaces per grid region of a reasonable size, and then create Data Shortcuts for each Grid area, that you can then pull into the final design file.  Short of purchasing a CRAY you might find that adding 2 or 3 separate physical hard drives and configure them as the SWAP file for Windows to use.

Also I am going to suggest that you NOT use the default weeding factors, and that you even use MORE of the data than those numbers are currently using.  The reason for this is that as you WEED the data you are introducing ERROR into the model, that in the end will adversely affect any earthwork numbers you attempt to create.

If I get the data to play with I might come up with a different strategy for you to employ.

(I can't tell from your signature are you on the island, or back on the Mainland?)


« Last Edit: March 21, 2010, 10:47:34 PM by sourdoug »
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #3 on: March 21, 2010, 10:47:26 PM »
To be clear about what the weeding is doing one would do well to study the formulae, and diagrams shown in help.
However the short form follows:

Weeding reduces the number of points generated along the contours. The weeding factors ignore both vertices that are closer together than the distance factor and vertices that deflect less than the angle factor. A larger distance and deflection angle weeds a greater number of points. The distance factor is measured in linear units, and the angle factor is measured in angular units. The weeding factors must be less than the supplementing factors.

The mid-ordinate distance is the distance from the midway point of an arc to the chord of the arc. The mid-ordinate distance is used to add vertices to a polyline curve, creating an approximation of the curve using straight line segments. The length of these segments depends on the value of the mid-ordinate distance.

This reduces or increases the fidelity the contours on the new surface in relation to the original data you were provided.

For preliminary planning; weeding ones data isn't much of an issue.

For final earthwork numbers ones choice in weeding could result in error larger than 10%.

WEED with respect to how you will use the data.


I amplified the description of WEEDING Factors; as I realized that far too often when I ask my students if they really know what they do most answer that their last instructor simply told them to use the defaults and moved on with NO Explanation of what they really do.  And I do not want to be guilty of the same omission.

(I will send FTP information in the AM, so you can park that data)
« Last Edit: March 21, 2010, 10:52:17 PM by mjfarrell »
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #4 on: March 21, 2010, 10:55:29 PM »
Mike
    I have another question dealing with the limits of machines to work on large data dwg files. What is your guideline
for when to do the griding to make more managealbe?
MJP

To be clear about what the weeding is doing one would do well to study the formulae, and diagrams shown in help.
However the short form follows:

Weeding reduces the number of points generated along the contours. The weeding factors ignore both vertices that are closer together than the distance factor and vertices that deflect less than the angle factor. A larger distance and deflection angle weeds a greater number of points. The distance factor is measured in linear units, and the angle factor is measured in angular units. The weeding factors must be less than the supplementing factors.

The mid-ordinate distance is the distance from the midway point of an arc to the chord of the arc. The mid-ordinate distance is used to add vertices to a polyline curve, creating an approximation of the curve using straight line segments. The length of these segments depends on the value of the mid-ordinate distance.

This reduces or increases the fidelity the contours on the new surface in relation to the original data you were provided.

For preliminary planning; weeding ones data isn't much of an issue.

For final earthwork numbers ones choice in weeding could result in error larger than 10%.

WEED with respect to how you will use the data.


I amplified the description of WEEDING Factors; as I realized that far too often when I ask my students if they really know what they do most answer that their last instructor simply told them to use the defaults and moved on with NO Explanation of what they really do.  And I do not want to be guilty of the same omission.

(I will send FTP information in the AM, so you can park that data)
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #5 on: March 21, 2010, 10:57:17 PM »
Another reason I would not WEED this data too heavily is that if that dataset is or was originally from LIDAR most likely it has already been reduced to a smaller number of points than originally existed.
However IF it is from an unreduced dataset, I would suggest the we get a copy of that original data, so that we might be more selective in what we keep, and what we throw away to get your EG surface.

Now I can sleep easy.  ;-)
Be your Best


Michael Farrell
http://primeservicesglobal.com/

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #6 on: March 21, 2010, 11:00:56 PM »
The grid size, will be established based on several factors.  Age and experience being two of them, size of proposed development areas (phasing) being another.  You know one variable, and for the other two there is little substitute.   
I guess this is one time where being OLD experienced is a good thing.  :lmao:


Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #7 on: March 21, 2010, 11:15:19 PM »
Mike
   This data set is unreduced. I'll send you a copy of it once you send me where to send it, it is about 53 meg and made of 3D polylines.
MJP


Another reason I would not WEED this data too heavily is that if that dataset is or was originally from LIDAR most likely it has already been reduced to a smaller number of points than originally existed.
However IF it is from an unreduced dataset, I would suggest the we get a copy of that original data, so that we might be more selective in what we keep, and what we throw away to get your EG surface.

Now I can sleep easy.  ;-)
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

reno

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #8 on: March 22, 2010, 01:32:51 AM »
I disagree with Michael.  I say that weeding is just fine.  You are using very imprecise data to create your surface.  What is the elevation interval of your contours?  Most likely they are 1' or 2' intervals.  What is the average slope of the site?  Let's assume it's 10%.  At a 10% slope, the contours with a 2' interval are 20' apart.  So in this situation, you have absolutely no data between the contours.  Having a vertex every 1' (or less) along the straight sections of the contours will not add any accuracy to the surface.  Think of it along the lines of significant digits.  What's 5,654,284.056 multiplied by 2.1?  Well, it's 12,000,000.  All that extra precision of the first number (10 significant digits) does not add any accuracy to the final answer because we are multiplying it by a very inaccurate number (2 significant digits).

I do however agree with Michael that you should not just apply the default values unless you understand what they are.  Use your engineering judgment to determine what will give an acceptable error for the data you are provided with.

pkohut

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #9 on: March 22, 2010, 02:15:10 AM »
If possible try and get the original data to build your surfaces from.

Don't know if these options will help, I have a program to create LDD breakline surfaces from polyline contour data. If the size of the original data doesn't choke LDD, then it could import those plines in a couple seconds. Then just import the surface(s) into C3D. (see edit below)

Another program will create LDD surfaces from grid data. Used it with large lidar dataset, and processing takes just a couple minutes.

Edit: just checked and it creates surface data for LDD 04-06, if I saw a flt.bin file for an LDD surface from 07-09 I'd be able to determind what needs to updated in the program to make it work for those as well)

Basically the way it works is you select a drawing file that has just contour polylines in it, then select a directory to store the flt.bin data.  So, saving the data file to directory c:\TestBr\Testing Stuff\DTM\Surface1 will create a Surface1flt.bin in that directory.  It also has the same options for weeding that LDD provides.  After you've blinked and its done, open the LDD project and build the surface, from here you could import to C3D.  The program is fast enough that you can easily play around with the amount of data thrown at it to determine what will choke LDD (IIRC about 1.5GB total memory usage).
« Last Edit: March 22, 2010, 04:13:21 AM by pkohut »

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #10 on: March 22, 2010, 09:52:42 AM »
To those that doubt the impact of Contour Weeding Factors, I offer this Exercise such that you may prove to yourself the negative impacts of same.

Build a Surface using ALL original survey data, and Breaklines.

Export the Contours as Polylines.

Now build TWO surfaces from the polyline data created in step 1 above.

In the First of these use SMALL weeding factors and add the contours to a surface, compare to original, not the differences.

In the Second use LARGE weeding factors, compare to Surface one, and Two; again note the differences.

Perhaps even Compute Volume Surfaces between 1 and 2, 1, and3, and finally 2, and 3.

If you still believe that ones choices in weeding factors have NO IMPACT on the accuracy of the surfaces being created there is little else I can do to alter your mindset.  However the data should convince a rational individual about the impact Weeding factors one ones terrain model.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #11 on: March 22, 2010, 10:54:40 AM »
I agree with Reno.

Contours that were created from Lidar data usually have an awful lot of squiggles.  Without good weeding, these squiggles cause the number of TIN triangles to soar astronomically.

Remember, if we're creating a surface from contours, we're already using our least-reliable possible source for surface information.  We can increase the weeding factors significantly before we impact the reliability of the data any further.

So when creating surfaces from contours like this, I always have to use weeding factors that are significantly more-aggressive than the default settings.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #12 on: March 22, 2010, 11:04:57 AM »
Sounds like we need to use that LIDAR data a little more discreetly to avoid the problems you describe...

The reason I suggest NOT weeding the contour data too aggressively is included in your statement SINC, as it is already an interpolation of the original data, I am leery of introducing additional error by weeding out the data I have to begin with.  My comments are related to contour data in general and not LIDAR data in particular.  As there are other processes one should consider using to reduce the 'point cloud' from LIDAR data prior to building the surface that would reduce those squiggles in the first place.

Have you run the test as suggested in previous post?
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #13 on: March 22, 2010, 11:27:22 AM »
Your test is actually rather irrelevant, since it is comparing the results of weeding with a surface built from real survey data.

By contrast, when we build data from contours, we are already accepting the fact that at any point on our site, we may have vertical error that is as great as our contour interval.    For the most part, error is within half the contour interval, but at any one point, the error can actually shoot all the way up to the entire contour interval.

So given that we are already starting at that point, reasonable weeding the data does not introduce significant additional error.  Obviously, too much weeding can create results that exceed our acceptable tolerance.  But the fact that the contours were created from Lidar data (which may have already been weeded) is irrelevant.  We actually don't CARE how the original contours were created.  No matter how they were created, we know we are creating a surface that introduces a lot of error, simply because of the fact that we are creating surfaces from contours.

All we need to concern ourselves with is that, after we build our new surface, the resulting contours should be "close enough" to the contours we started from, for whatever purposes we intend.  The original source of the contours is irrelevant, except for the fact that it might help us assign a relative level of "reliability" to our starting contours.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #14 on: March 22, 2010, 11:29:41 AM »
 My comments are related to contour data in general and not LIDAR data in particular.  

The test is relevant to show that weeding can indeed introduce error in the resultant surface built from contours.
LIDAR data is another animal, and one will in most instances need to use judgement in reducing that data to create a usable surface model from.
« Last Edit: March 22, 2010, 11:33:17 AM by mjfarrell »
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #15 on: March 22, 2010, 12:47:24 PM »
The test is relevant to show that weeding can indeed introduce error in the resultant surface built from contours.
LIDAR data is another animal, and one will in most instances need to use judgement in reducing that data to create a usable surface model from.

Well, of course weeding can introduce error.  That's a given, and we don't need a test to prove it.

The goal in this sort of task is simply to reduce our Civil 3D surface to a point where we have an acceptable trade-off between performance and accuracy.

And the only thing that's important for that is, after we try some weeding factors, to compare the resulting contours with the starting contours.  If the results are "close enough" for whatever purposes we intend, then we can proceed.  We do not need to worry at all about the fact that the contours originally came from Lidar data.

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #16 on: March 22, 2010, 01:05:42 PM »
Hi Sinc
    I have tried some variations of the weeding and the results of what the contours look like or don't seem hard to submit
in a plat that I would want to turn in.  My example was taken from Lidar and made into a 3d Poly. So, wanting to make a surface
with it is where it got to be challenge. I tried weeding and the contours on the top of mesa's started to make nonsense contours.
So, this is a working from a 1/2 section of data. I am hoping to use data shortcuts to work with the surface and that seems to work.
What was happening in the experience over this weekend of experimenting was that I had to switch from Win 7 32bit to Win 7 64bit
because of memory limitations. Now after playing with the large file of contours it died after 4.7 gig of memory used. I have plenty of memory
and set page file to 12 gig max and wondered why it died so quickly. I thought Civil 3d 2010 could handle it. Am I the only one finding a
another limitation with filesize and Civil 3D?  This has to be answered first before I can start to understand the procedures to use for
working on large projects and phasing. Your thoughts?

MJP


The test is relevant to show that weeding can indeed introduce error in the resultant surface built from contours.
LIDAR data is another animal, and one will in most instances need to use judgement in reducing that data to create a usable surface model from.

Well, of course weeding can introduce error.  That's a given, and we don't need a test to prove it.

The goal in this sort of task is simply to reduce our Civil 3D surface to a point where we have an acceptable trade-off between performance and accuracy.

And the only thing that's important for that is, after we try some weeding factors, to compare the resulting contours with the starting contours.  If the results are "close enough" for whatever purposes we intend, then we can proceed.  We do not need to worry at all about the fact that the contours originally came from Lidar data.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #17 on: March 22, 2010, 01:27:26 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

pkohut

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #18 on: March 22, 2010, 01:50:52 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.

Don't know how fast C3D is for doing this, but LDD is extreamly slow. I've a standalone routine to create LDD surface from grid data that could be imported to C3D.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #19 on: March 22, 2010, 01:56:07 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.

Don't know how fast C3D is for doing this, but LDD is extreamly slow. I've a standalone routine to create LDD surface from grid data that could be imported to C3D.

All well and good however, the user is NOT running LDD (to the best of my knowledge).
Further I have not noticed any issues within C3D, and using external point database(s) for surfaces.
Additionally, being this is LIDAR data, or the origins of the polylines he is attempting to use was, there is NO GRID of data.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

pkohut

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #20 on: March 22, 2010, 02:06:08 PM »
It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.

Don't know how fast C3D is for doing this, but LDD is extreamly slow. I've a standalone routine to create LDD surface from grid data that could be imported to C3D.

All well and good however, the user is NOT running LDD (to the best of my knowledge).
Further I have not noticed any issues within C3D, and using external point database(s) for surfaces.
Additionally, being this is LIDAR data, or the origins of the polylines he is attempting to use was, there is NO GRID of data.


Fair enough, just providing alternatives to import large data sets.  I use Global Mapper to convert lidar data to ESRI GRD data, then import that to LDD. Takes 5 to 10 minutes total.  The routines are custom and if SourDog can use them for his purpose, they're available.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #21 on: March 22, 2010, 02:08:15 PM »
Global Mapper is always in my tool kit....so powerful, so effective....for such a small investment!
However; I like to keep the 'solutions' I offer to the public within the abilities of the application they are trying to use.  As I am sure that you are aware of reluctance to employ third party solutions by some.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #22 on: March 22, 2010, 02:09:11 PM »
Hi Mike
   The file given wasn't in a point format to do that. I was told it was done by Lidar, and put into 3D poly format in a dwg.
Sorry, that I didn't make that more clear.
Mike


It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #23 on: March 22, 2010, 02:12:20 PM »
Hi Mike
   The file given wasn't in a point format to do that. I was told it was done by Lidar, and put into 3D poly format in a dwg.
Sorry, that I didn't make that more clear.
Mike


It just occurred to me that IF you have the original LIDAR data file, one would be best served to put that data into an MDB file, and use the Add Point File Option, NOT the contour data to build your surface.
That part was clear...however I was ever hopeful that the source data could be at hand.
Could be a good time to contact the source and request the original data IF possible.  Just state your challenges, and ask for the original information as a bridge to the solution.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #24 on: March 22, 2010, 02:23:42 PM »
Now after playing with the large file of contours it died after 4.7 gig of memory used. I have plenty of memory
and set page file to 12 gig max and wondered why it died so quickly. I thought Civil 3d 2010 could handle it. Am I the only one finding a
another limitation with filesize and Civil 3D?  This has to be answered first before I can start to understand the procedures to use for
working on large projects and phasing. Your thoughts?

Not sure what you mean by "4.7 gig of memory used".  C3D 2010 is a 32-bit app, and as such, it can only access a max of 4GB of RAM.  However, due to other constraints, you won't see it go that high.  On a 64-bit OS, I generally seem to hit an "out of memory" error once C3D is using about 3GB of memory.

If you have at least 6GB of RAM in your system, the size of your pagefile is relatively immaterial, since pretty much everything C3D uses will be able to stay in RAM (unless you have other programs running simultaneously, and those other programs are also using lots of RAM).

As far as your problems, I think a lot of it is simply because you are trying to generate a surface from contours.  That's simply the worst way to go about it.  If you have no other choice but to use the contours you have, then I'm not sure what you need to do.  I can think of all kinds of nasty solutions, such as building C3D surfaces in chunks and using them to generate points, which you then put into a Point File and use to generate a surface from an external Point File, and things like that.  But they're all kind of nasty solutions, of the "last resort" kind.

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #25 on: March 22, 2010, 02:30:58 PM »
As another possibility, you might get something useful from running the Map Cleanup tools on your contours, to weed down the polylines before trying to add them to the surface.  Possibly, using a combination of Map Cleanup and the surface weeding options will get you something usable.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #26 on: March 22, 2010, 02:43:05 PM »
going the MAP path....one could query the data, extracting coordinates from the polylines, to external file...then ADD that point file to the surface...
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #27 on: March 22, 2010, 02:48:09 PM »
Sinc
    Thanks for that info. I am using Win 7 64 and I really did get it to use just for that program 4.something worth of memory according
to my resource manager. I turned off most all my other programs to do it. So, it probably did max out. I thought even though the
program is 32bit that in a 64 bit os it would go further. Tell me that it might be fixed in 2011 C3D, please. I'll try your ideas to. The info
originally came from city and for preliminary design concept, NOT Final, is supposed to be accurate to within 1/2'. But I can't get the original
point data, so stuck until they actually put a survey crew on the ground. The idea behind the original question and follow up and reaching
limits of Civil 3D 2010 are becoming clearer though. One day maybe, someday and could be would be really nice today, though.

MJP

As another possibility, you might get something useful from running the Map Cleanup tools on your contours, to weed down the polylines before trying to add them to the surface.  Possibly, using a combination of Map Cleanup and the surface weeding options will get you something usable.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #28 on: March 22, 2010, 02:58:28 PM »
Tell me that it might be fixed in 2011 C3D, please.

OK...  "It might be fixed in 2011 C3D."    :-)

We're still in the "those who know can't say, those who say don't know" phase.  All I can tell you is that on the other forums recently, Peter Funk strongly encouraged people who are buying new machines right now to go with Win 7 x64 as their OS, because it will position them for "future versions of Civil 3D".  Read that however you will.

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #29 on: March 22, 2010, 05:13:17 PM »
Sinc,

Well, that is good to know. Thanks for the hint. (On the upgrade path though I'm going to go with
Xeon Processor next time out. I did some research on the latest and for the money (not 6 core) the
Xeon X5520 is going to cost less then $350 and have 5.8 GT/s output, where as lesser including the I7 core less
than 960 series only do 4.8 GT/s. So, on the upward path I'm sure everyone is going to do even more
research to make C3D 20** can do. Just a passing thought.

Mike
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #30 on: March 22, 2010, 05:48:33 PM »
I have yet to see a Xeon chip that outperforms other options, as far as price/performance for Civil 3D.  But all this stuff changes pretty fast...

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #31 on: March 22, 2010, 06:11:56 PM »
SINC, you might be on to something in regards MAP tools...see attached image...


in my book that is a whole lotta nadda
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sinc

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #32 on: March 22, 2010, 07:20:03 PM »
Oooh...  I wonder if C3D is smart enough to weed those points first, or if it is trying to process all those zero-length segments as additional TIN edges...?

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #33 on: March 22, 2010, 07:28:13 PM »
Oooh...  I wonder if C3D is smart enough to weed those points first, or if it is trying to process all those zero-length segments as additional TIN edges...?

I think, what is happening is that it attempts to sort all those points, determining the duplicates, and then ignores them...
exccept it probably crashes BEFORE it gets to the ignore part....

Be your Best


Michael Farrell
http://primeservicesglobal.com/

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #34 on: March 22, 2010, 08:02:35 PM »
OK,

I have confirmation here, that once the data is pushed against the wall with Map Cleanup tools, and the MOB of zero length polyline segments are remove, C3D has NO trouble building that surface, even on this cheesy Laptop that I use for my training classes.
I built it without the inside boundary as shown in the original image posted.  Final Size 108,965KB.

Along the way I also discovered that the MAP Query to Report Function is STILL broken and introduces invisible HEX data into the TXT file that it writes on output.  


For the Record, I was already headed towards pushing this data through MAP, as it is a practice I adopted a long time ago whenever working with any data from any outside source.

Why; you might ask do I do this.  The answer is by using MAP to examine others data, I am absolutely certain that I am not altering the original source files accidentally.  Now if only Adesk would allow us to use MAP on C3D objects.
« Last Edit: March 22, 2010, 08:07:32 PM by mjfarrell »
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #35 on: March 22, 2010, 11:55:51 PM »
Mike
   I went to my Map and used dwg clean up and I don't have a zero length polyline segment option to remove.
Could you show a snapshot of where I can pick that, please.

MJP

OK,

I have confirmation here, that once the data is pushed against the wall with Map Cleanup tools, and the MOB of zero length polyline segments are remove, C3D has NO trouble building that surface, even on this cheesy Laptop that I use for my training classes.
I built it without the inside boundary as shown in the original image posted.  Final Size 108,965KB.

Along the way I also discovered that the MAP Query to Report Function is STILL broken and introduces invisible HEX data into the TXT file that it writes on output.  


For the Record, I was already headed towards pushing this data through MAP, as it is a practice I adopted a long time ago whenever working with any data from any outside source.

Why; you might ask do I do this.  The answer is by using MAP to examine others data, I am absolutely certain that I am not altering the original source files accidentally.  Now if only Adesk would allow us to use MAP on C3D objects.
LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #36 on: March 23, 2010, 12:08:26 AM »
It is found on the Select Actions Page of the wizard...

Note one should NOT select all of those options in one fell swoop..there is a proscribed order for using them.  I normally cover that sequence as part of my normal class.

To select cleanup actions and set options

On the Select Actions page, select the cleanup actions to perform by adding them to the Selected Actions list.
To add a cleanup action to the Selected Actions list, select the action in the Cleanup Actions list, and click Add.


In the Selected Actions list, specify the order of cleanup actions. The order can affect results. The first action in the list is performed first. To change the order, highlight an action and click the up or down arrow.
In the Selected Actions list, select the action for which you want to specify settings.
Under Cleanup Parameters, specify the settings to use for the selected action. For more information about each cleanup action, click one of the following links:
Delete Duplicates
Erase Short Objects
Break Crossing Objects
Extend Undershoots
Apparent Intersection
Snap Clustered Nodes
Dissolve Pseudo Nodes
Erase Dangling Objects
Simplify Objects
Zero-Length Objects
Weed Polylines
To review detected errors before correcting them, under Options, select Interactive. To have AutoCAD Map 3D correct all detected errors without further input from you, select Automatic.
Note Simplify Objects and Weed Polylines are not interactive operations. AutoCAD Map 3D makes these changes automatically during cleanup.
Click Next.
« Last Edit: March 23, 2010, 09:58:09 AM by mjfarrell »
Be your Best


Michael Farrell
http://primeservicesglobal.com/

sourdough

  • Bull Frog
  • Posts: 362
Re: Reducing the file size to make large contour set... help
« Reply #37 on: March 25, 2010, 01:37:21 AM »
Mike
    That did the trick. Thanks
Mike

It is found on the Select Actions Page of the wizard...

Note one should NOT select all of those options in one fell swoop..there is a proscribed order for using them.  I normally cover that sequence as part of my normal class.

To select cleanup actions and set options

On the Select Actions page, select the cleanup actions to perform by adding them to the Selected Actions list.
To add a cleanup action to the Selected Actions list, select the action in the Cleanup Actions list, and click Add.


In the Selected Actions list, specify the order of cleanup actions. The order can affect results. The first action in the list is performed first. To change the order, highlight an action and click the up or down arrow.
In the Selected Actions list, select the action for which you want to specify settings.
Under Cleanup Parameters, specify the settings to use for the selected action. For more information about each cleanup action, click one of the following links:
Delete Duplicates
Erase Short Objects
Break Crossing Objects
Extend Undershoots
Apparent Intersection
Snap Clustered Nodes
Dissolve Pseudo Nodes
Erase Dangling Objects
Simplify Objects
Zero-Length Objects
Weed Polylines
To review detected errors before correcting them, under Options, select Interactive. To have AutoCAD Map 3D correct all detected errors without further input from you, select Automatic.
Note Simplify Objects and Weed Polylines are not interactive operations. AutoCAD Map 3D makes these changes automatically during cleanup.
Click Next.

LDC 2009/C3D 2010/C3D 2011/C3D 2016

Win 10 64bit

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #38 on: March 25, 2010, 10:07:34 AM »
No thank you!


If this project is going to span the entire section...you still might consider using 1/4 sections lines as normal locations to grid the data, and offer up as Data References.
Be your Best


Michael Farrell
http://primeservicesglobal.com/

reno

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #39 on: March 25, 2010, 03:34:03 PM »
or perhaps (if you are using 2010) you could use the cropped surface tools that came with the subscription advantage pack.  haven't used them myself but they might work.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #40 on: March 25, 2010, 06:01:10 PM »
or perhaps (if you are using 2010) you could use the cropped surface tools that came with the subscription advantage pack.  haven't used them myself but they might work.

I'm not sure that 'cropping' the surface reduces the file size....Do you have a README on that feature handy to post RENO?
Be your Best


Michael Farrell
http://primeservicesglobal.com/

reno

  • Guest
Re: Reducing the file size to make large contour set... help
« Reply #41 on: March 25, 2010, 10:16:04 PM »
or perhaps (if you are using 2010) you could use the cropped surface tools that came with the subscription advantage pack.  haven't used them myself but they might work.

I'm not sure that 'cropping' the surface reduces the file size....Do you have a README on that feature handy to post RENO?
'nope'

Jeff_M

  • King Gator
  • Posts: 3942
  • C3D user & customizer
Re: Reducing the file size to make large contour set... help
« Reply #42 on: March 25, 2010, 11:02:07 PM »
This is from the help for it:
Quote
Use the Create Cropped Surface command to create a surface that is a copy of a polygonal area of a source surface. Each cropped area becomes a separate surface object (a new surface) that you can manage and manipulate.

When you create a surface by cropping the source surface, or when you rebuild the new surface, AutoCAD Civil 3D creates a copy from the saved version of the source surface. If you modify the source surface and want the changes to be reflected in the new surface, save the drawing containing the source surface before you rebuild the new surface.

Note When a new surface is created, AutoCAD Civil 3D takes a snapshot of the new surface. To rebuild the new surface, you must first perform the Rebuild Snapshot command and then the Rebuild command.

mjfarrell

  • Seagull
  • Posts: 14442
  • Every Student their own Lesson
Re: Reducing the file size to make large contour set... help
« Reply #43 on: March 25, 2010, 11:35:42 PM »
OK, from that I get that 'cropping' does not reduce file size, just the area.  And whatever data it is that makes up that area is what it is, no bigger or smaller.  Just a more convenient, or efficient method to both get to that smaller area, of the whole.
Be your Best


Michael Farrell
http://primeservicesglobal.com/