Topic: Reducing the file size to make large contour set... help

[sourdough]

Hi All
    I am working on really large files of 2d Polyline (were 3D poly) that are contours. My guess is that they were
part of a Lidar for topo. The file I'm working with is about 42 meg (2d poly) of data. When I try to make this into
a Surface in Civil 3D 2010 I have been killing my machine. I'm using Win 7 64bit with 8 Gig of memory with a
Intel I7 860 and I'm finding after creating contours for the surface that it turns into
about 200 meg file and is running out of memory, even though it only reaches 4.7gig used. What sort of weeding
factors should I use. I used the default 15 and 100 main numbers. I don't want to lose to much of the detail. So,
if anyone has some suggestions. This is just part of a really big project so the future is that I'm going to have to
link all these surface pieces today over time. I am going to use data shortcuts for sure, but just making the surfaces
with this large amount of data is just crazy. This sample is over 150 meg in snapshot.

MJP

[mjfarrell]

Mike,

From where I'm sitting, I would like to have the data and run a few experiments.

Absent that my best suggestion would be that you GRID the project area building the surfaces per grid region of a reasonable size, and then create Data Shortcuts for each Grid area, that you can then pull into the final design file.  Short of purchasing a CRAY you might find that adding 2 or 3 separate physical hard drives and configure them as the SWAP file for Windows to use.

Also I am going to suggest that you NOT use the default weeding factors, and that you even use MORE of the data than those numbers are currently using.  The reason for this is that as you WEED the data you are introducing ERROR into the model, that in the end will adversely affect any earthwork numbers you attempt to create.

If I get the data to play with I might come up with a different strategy for you to employ.

(I can't tell from your signature are you on the island, or back on the Mainland?)

[sourdough]

Mike
    Thanks for your help. I sent a message for the method of delivery you would like for the file to work on. The area
shown is my attempt at adding to an area that was already sampled and made a surface from.
So, the area cut out was already done. The State from which is was done was northern New Mexico.
If you do have a Cray to give me, I'll take it.
MJP

Mike,

From where I'm sitting, I would like to have the data and run a few experiments.

Absent that my best suggestion would be that you GRID the project area building the surfaces per grid region of a reasonable size, and then create Data Shortcuts for each Grid area, that you can then pull into the final design file.  Short of purchasing a CRAY you might find that adding 2 or 3 separate physical hard drives and configure them as the SWAP file for Windows to use.

Also I am going to suggest that you NOT use the default weeding factors, and that you even use MORE of the data than those numbers are currently using.  The reason for this is that as you WEED the data you are introducing ERROR into the model, that in the end will adversely affect any earthwork numbers you attempt to create.

If I get the data to play with I might come up with a different strategy for you to employ.

(I can't tell from your signature are you on the island, or back on the Mainland?)

[mjfarrell]

To be clear about what the weeding is doing one would do well to study the formulae, and diagrams shown in help.
However the short form follows:

Weeding reduces the number of points generated along the contours. The weeding factors ignore both vertices that are closer together than the distance factor and vertices that deflect less than the angle factor. A larger distance and deflection angle weeds a greater number of points. The distance factor is measured in linear units, and the angle factor is measured in angular units. The weeding factors must be less than the supplementing factors.

The mid-ordinate distance is the distance from the midway point of an arc to the chord of the arc. The mid-ordinate distance is used to add vertices to a polyline curve, creating an approximation of the curve using straight line segments. The length of these segments depends on the value of the mid-ordinate distance.

This reduces or increases the fidelity the contours on the new surface in relation to the original data you were provided.

For preliminary planning; weeding ones data isn't much of an issue.

For final earthwork numbers ones choice in weeding could result in error larger than 10%.

WEED with respect to how you will use the data.


I amplified the description of WEEDING Factors; as I realized that far too often when I ask my students if they really know what they do most answer that their last instructor simply told them to use the defaults and moved on with NO Explanation of what they really do.  And I do not want to be guilty of the same omission.

(I will send FTP information in the AM, so you can park that data)

[sourdough]

Mike
    I have another question dealing with the limits of machines to work on large data dwg files. What is your guideline
for when to do the griding to make more managealbe?
MJP

To be clear about what the weeding is doing one would do well to study the formulae, and diagrams shown in help.
However the short form follows:

Weeding reduces the number of points generated along the contours. The weeding factors ignore both vertices that are closer together than the distance factor and vertices that deflect less than the angle factor. A larger distance and deflection angle weeds a greater number of points. The distance factor is measured in linear units, and the angle factor is measured in angular units. The weeding factors must be less than the supplementing factors.

The mid-ordinate distance is the distance from the midway point of an arc to the chord of the arc. The mid-ordinate distance is used to add vertices to a polyline curve, creating an approximation of the curve using straight line segments. The length of these segments depends on the value of the mid-ordinate distance.

This reduces or increases the fidelity the contours on the new surface in relation to the original data you were provided.

For preliminary planning; weeding ones data isn't much of an issue.

For final earthwork numbers ones choice in weeding could result in error larger than 10%.

WEED with respect to how you will use the data.


I amplified the description of WEEDING Factors; as I realized that far too often when I ask my students if they really know what they do most answer that their last instructor simply told them to use the defaults and moved on with NO Explanation of what they really do.  And I do not want to be guilty of the same omission.

(I will send FTP information in the AM, so you can park that data)

[mjfarrell]

Another reason I would not WEED this data too heavily is that if that dataset is or was originally from LIDAR most likely it has already been reduced to a smaller number of points than originally existed.
However IF it is from an unreduced dataset, I would suggest the we get a copy of that original data, so that we might be more selective in what we keep, and what we throw away to get your EG surface.

Now I can sleep easy.  

[mjfarrell]

The grid size, will be established based on several factors.  Age and experience being two of them, size of proposed development areas (phasing) being another.  You know one variable, and for the other two there is little substitute.   
I guess this is one time where being OLD experienced is a good thing. 

[sourdough]

Mike
   This data set is unreduced. I'll send you a copy of it once you send me where to send it, it is about 53 meg and made of 3D polylines.
MJP

Another reason I would not WEED this data too heavily is that if that dataset is or was originally from LIDAR most likely it has already been reduced to a smaller number of points than originally existed.
However IF it is from an unreduced dataset, I would suggest the we get a copy of that original data, so that we might be more selective in what we keep, and what we throw away to get your EG surface.

Now I can sleep easy.  

[reno]

I disagree with Michael.  I say that weeding is just fine.  You are using very imprecise data to create your surface.  What is the elevation interval of your contours?  Most likely they are 1' or 2' intervals.  What is the average slope of the site?  Let's assume it's 10%.  At a 10% slope, the contours with a 2' interval are 20' apart.  So in this situation, you have absolutely no data between the contours.  Having a vertex every 1' (or less) along the straight sections of the contours will not add any accuracy to the surface.  Think of it along the lines of significant digits.  What's 5,654,284.056 multiplied by 2.1?  Well, it's 12,000,000.  All that extra precision of the first number (10 significant digits) does not add any accuracy to the final answer because we are multiplying it by a very inaccurate number (2 significant digits).

I do however agree with Michael that you should not just apply the default values unless you understand what they are.  Use your engineering judgment to determine what will give an acceptable error for the data you are provided with.

[pkohut]

If possible try and get the original data to build your surfaces from.

Don't know if these options will help, I have a program to create LDD breakline surfaces from polyline contour data. If the size of the original data doesn't choke LDD, then it could import those plines in a couple seconds. Then just import the surface(s) into C3D. (see edit below)

Another program will create LDD surfaces from grid data. Used it with large lidar dataset, and processing takes just a couple minutes.

Edit: just checked and it creates surface data for LDD 04-06, if I saw a flt.bin file for an LDD surface from 07-09 I'd be able to determind what needs to updated in the program to make it work for those as well)

Basically the way it works is you select a drawing file that has just contour polylines in it, then select a directory to store the flt.bin data.  So, saving the data file to directory c:\TestBr\Testing Stuff\DTM\Surface1 will create a Surface1flt.bin in that directory.  It also has the same options for weeding that LDD provides.  After you've blinked and its done, open the LDD project and build the surface, from here you could import to C3D.  The program is fast enough that you can easily play around with the amount of data thrown at it to determine what will choke LDD (IIRC about 1.5GB total memory usage).

[mjfarrell]

To those that doubt the impact of Contour Weeding Factors, I offer this Exercise such that you may prove to yourself the negative impacts of same.

Build a Surface using ALL original survey data, and Breaklines.

Export the Contours as Polylines.

Now build TWO surfaces from the polyline data created in step 1 above.

In the First of these use SMALL weeding factors and add the contours to a surface, compare to original, not the differences.

In the Second use LARGE weeding factors, compare to Surface one, and Two; again note the differences.

Perhaps even Compute Volume Surfaces between 1 and 2, 1, and3, and finally 2, and 3.

If you still believe that ones choices in weeding factors have NO IMPACT on the accuracy of the surfaces being created there is little else I can do to alter your mindset.  However the data should convince a rational individual about the impact Weeding factors one ones terrain model.

[sinc]

I agree with Reno.

Contours that were created from Lidar data usually have an awful lot of squiggles.  Without good weeding, these squiggles cause the number of TIN triangles to soar astronomically.

Remember, if we're creating a surface from contours, we're already using our least-reliable possible source for surface information.  We can increase the weeding factors significantly before we impact the reliability of the data any further.

So when creating surfaces from contours like this, I always have to use weeding factors that are significantly more-aggressive than the default settings.

[mjfarrell]

Sounds like we need to use that LIDAR data a little more discreetly to avoid the problems you describe...

The reason I suggest NOT weeding the contour data too aggressively is included in your statement SINC, as it is already an interpolation of the original data, I am leery of introducing additional error by weeding out the data I have to begin with.  My comments are related to contour data in general and not LIDAR data in particular.  As there are other processes one should consider using to reduce the 'point cloud' from LIDAR data prior to building the surface that would reduce those squiggles in the first place.

Have you run the test as suggested in previous post?

[sinc]

Your test is actually rather irrelevant, since it is comparing the results of weeding with a surface built from real survey data.

By contrast, when we build data from contours, we are already accepting the fact that at any point on our site, we may have vertical error that is as great as our contour interval.    For the most part, error is within half the contour interval, but at any one point, the error can actually shoot all the way up to the entire contour interval.

So given that we are already starting at that point, reasonable weeding the data does not introduce significant additional error.  Obviously, too much weeding can create results that exceed our acceptable tolerance.  But the fact that the contours were created from Lidar data (which may have already been weeded) is irrelevant.  We actually don't CARE how the original contours were created.  No matter how they were created, we know we are creating a surface that introduces a lot of error, simply because of the fact that we are creating surfaces from contours.

All we need to concern ourselves with is that, after we build our new surface, the resulting contours should be "close enough" to the contours we started from, for whatever purposes we intend.  The original source of the contours is irrelevant, except for the fact that it might help us assign a relative level of "reliability" to our starting contours.

[mjfarrell]

 My comments are related to contour data in general and not LIDAR data in particular.  

The test is relevant to show that weeding can indeed introduce error in the resultant surface built from contours.
LIDAR data is another animal, and one will in most instances need to use judgement in reducing that data to create a usable surface model from.