Topic: Unit Testing in AutoCAD

[CADbloke]

It would think it would take so much time and effort to build fakes to behave like they would in AutoCAD environment and then with changes to maintain you would never have time to actually use it.
....
Number of fakes required to build, mass number of system variables whose settings will change the behavior of objects, some objects require a certain order for assigning properties, so many different contexts, etc....

I am rapidly coming to the same conclusion. Andrey has already worked this out - the only way to test the behaviour of an AutoCAD object is to run the tests in AutoCAD and the way that works for him is to use Gallio.

As an aside, I tried Microsoft Fakes but that had a  lot of trouble building Fakes. When I eventually got it to build one it only worked on a few things. Chained method calls just didn't work, they didn't call the fakes after they got one layer deep. ObjectId is a struct so faking that is problematic. Properties are relatively easy to fake but chained properties involve behaviour which is not easy to fake. Also amusing was the seven-hundred-thousand line C-sharp file that Microsoft Fakes created.

The managed libraries are only managed wrappers and therefore only part of the AutoCAD object story anyway so faking those just won't work. Even if you manage to fake these you're faking a fake, especially if you fake the SDK DLLs which actually are fakes but dumb ones. That ain't gunna work. It didn't work.

It is necessary to separate independent and integration testing. I did it for my some application. It requires many forces and time, but it gives me big confidence about the quality of my software.

(the bold is my emphasis) Yeah, me too. I recently refactored the core of my app - this improved performance around 5x but it also broke some of my tests because I removed a lot of abstraction and disconnections to the AutoCAD objects I am manipulating. I want my tests back (now!) but I also like the way it runs now. I can see ways to test it but actually doing it is the hard part - I really need the AutoCAD objects because their behaviour and properties are an important part of the process.

The Repo I linked to above is interesting because it has a test runner that runs inside AutoCAD. Here is the explanation I should have added in my original post (sorry) http://civilizeddevelopment.typepad.com/civilized-development/2011/03/tinyrunner-implementing-a-civil-3d-extension.html. Here's a Google search (TypePad's search is pants) for more that Isaac has written more about the Colibra library: https://www.google.com.au/search?q=colibra+site%3Ahttp%3A%2F%2Fcivilizeddevelopment.typepad.com This is the enxt rabbit hole I plan on diving down.

Thinking out loud here ...

Not all tests need to run in AutoCAD, just the ones that concern themselves with the properties and behaviour of AutoCAD objects. These are more integration tests than unit tests but they should still be automated and easy to create and use. My goal is to find a lightweight solution that runs tests in AutoCAD. The solution must be light-weight, flexible and future-proof. It should Just Work^TM.

Creating and managing AutoCAD objects need not be difficult, you should be able to use something like Nunit SetUp and TearDown. Well, that's the theory.

Lightweight
This has two considerations: doing and maintaining. I am not concerned that setting up the initial infrastructure would be a big job if it was easy to use and maintain.
Gallio is lightweight to run but not to maintain. Well, so is AutoCAD really but I digress. What I really want to be able to do is to set the Solution Configuration in Visual Studio and then run the tests in a test runner. It should just work. Unit tests need to be easy and fast or you just won't bother.

I also want to be able to build all the versions and run all the tests at once. With non-AutoCAD unit tests this is relatively simple, batch-build in Visual Studio the run all the tests in the NUnit test runner GUI - it works quite well. I wouldn't even care if the tests to some time to run if it was trivial to run them - a batch file that opens each CAD application, runs the tests and records the results would work for me. This could run on a Virtual Machine with a bunch of CAD applications on it. When it is finished you can reset the VM for the next lot of tests. This sounds do-able to me with a batch file and scripts to run the test runner in each application. Setting it up would be a fair bit of work but once the template was built then repeating the effort would/should be trivial.

Flexible

For me is interesting to write the extensions for NUnit for capabilities of testing AutoCAD, nanoCAD, BricsCAD and any Teigha-based application.

Me too and I want this to work automatically. I want a process that is easy to repeat, something I can create a project template for and just add it to a Visual Studio solution, write some tests and press go.

future-proof
My main concern with Gallio is that it must be maintained and it is HUGE, so that maintenance could be a big effort.

It's kind of liberating to be stumbling around in the dark, in the broad daylight of the Internet. Care to join me? What are your thoughts?

[Andrey Bushman]

Me too and I want this to work automatically. I want a process that is easy to repeat, something I can create a project template for and just add it to a Visual Studio solution, write some tests and press go.

I prefer to use the NUnit instead of the Visual Studio test runner. This case my tests haven't depending of any IDE.

future-proof
My main concern with Gallio is that it must be maintained and it is HUGE, so that maintenance could be a big effort.

Care to join me? What are your thoughts?

Do you mean about writting of NUnit extensions for testing of any CAD?
If "yes" then it is interesting for me. We can discuss it in the Skype or email, for example (I have some ideas about of architecture its implementation).

[MickD]

I remember starting to make my own test runner for Bricscad for integration testing, it wouldn't be hard to hand build your own testing suite.
The thing with projects like Gallio is they have to cater for 'all' types of applications and include a lot of things you don't need or use, by hand coding your own you can build only what you need as you need it.

This maybe a start or food for thought: http://www.theswamp.org/index.php?topic=47549.msg525741#msg525741

I'd also suggest moving to a more functional style of programming, this (mostly) removes the need for mocks and interfaces, by creating class methods that only take simple data types and not 'objects' that hold the data as 'state' they are a lot easier and safer to test and more reusable.

Anyway, for all my CAD/.net dev I'm off to F#

[CADbloke]

I had a play with TinyTest (links are in my previous posts) and I've managed to get it to build and run its own tests inside AutoCAD. It's very simple but that isn't such a bad thing. It does support Setup and Teardown methods which is handy.

A Visual Studio 2013 solution is attached. You will need to replace the project references in the TinyTest project with whatever AutoCAD references you use. Don't forget "Copy Local". You may need to change the .NET framework too (I set it to .NET 4). I changed (ie. probably broke) a few things in it, particularly the Class and method Attributes - I changed those to match NUnit.

Build it, then netload "TinyTest.dll". Run the command RunFrameworkTests and it will (I hope) run its own tests.

I've only just started playing with this but I think it looks promising. I'd like it to have more of NUnit's assertions and a GUI runner and/or more verbose output. A GUI runner is not such a difficult thing to do (famous last words) and probably an interesting WPF exercise.

As an aside, I actually did get the NUnit 2.6.4 GUI runner to run inside AutoCAD but it doesn't play well with the AcMgd etc. DLLs. It also crashes AutoCAD if you try to add it a second time. It was a mess. I didn't look too closely at NUnitLite but I think that has more potential and would be more manageable than the full version of NUnit. I would still use NUnit for my main testing, only deferring to this TinyTest for stuff I couldn't break away from AutoCAD. Yeah, it's a crutch. Meh.

I'd also suggest moving to a more functional style of programming, this (mostly) removes the need for mocks and interfaces, by creating class methods that only take simple data types and not 'objects' that hold the data as 'state' they are a lot easier and safer to test and more reusable.

I agree with that in sentiment but in practise I find it just leads to more layers of indirection, I get a lot of classes talking about other classes and less JFDI. The other thing I don't like about Fakes is that it removes AutoCAD's behaviour and as well all know, that can be a bastard-thing that needs to be monitored. Using Fakes also tests your assumptions, am I actually asking *CAD to do what I think I'm asking it to do?

I prefer to use the NUnit instead of the Visual Studio test runner. This case my tests haven't depending of any IDE.

I don't really mind which test runner I use as long as it works and I get results. I like NUnit's runner because I can run a lot of tests all at once. I like Resharpers runner because it is right in front of me, I can run my tests with a two mouse-clicks (or Ctrl-U, Ctrl-R).

Care to join me? What are your thoughts?

Do you mean about writing of NUnit extensions for testing of any CAD?
If "yes" then it is interesting for me. We can discuss it in the Skype or email, for example (I have some ideas about of architecture its implementation).

Nunit is just as huge as MBUnit / Gallio, NUnit Version 3 even more so. I think MickD is on to something here...

I remember starting to make my own test runner for Bricscad for integration testing, it wouldn't be hard to hand build your own testing suite.
The thing with projects like Gallio is they have to cater for 'all' types of applications and include a lot of things you don't need or use, by hand coding your own you can build only what you need as you need it.

If I develop anything useful I want to make sure that everybody can use it. This The Swamp and other communities have given me tremendous value and help and I really want to be able to give this back. I am happy to discuss things offline but anything I build is for everybody. Here are my license terms from the attached project...

Code - C#: [Select]

1. /*CAD bloke's License terms are simple:  Knock yourself out, whatever, wherever, whenever, however, whyever but if it kills anything it is entirely your fault because you touched it last and you should do backups and source control. If you claim this as your own and extort funds for it then Karma will get you while you are sleeping. If you use it in anger and it works then attribution would be rather nice (and for Isaac too), if it doesn't work then you never heard of CAD bloke and it definitely wasn't Isaac's fault.*/

Isaac Rodriguez is the original writer of TinyText. Thanks Isaac.

Note, I haven't changed much of it as yet, the file headers are a pre-emptive strike so nobody blames Isaac or Autodesk if I broke it.

...edit - added another version of the Visual Studio solution I have fiddled with. I also left the original attached in case I broke it.

[CADbloke]

What I have discovered so far ...

• It works. You can run tests that create objects inside transactions and then use these objects in your test.
• You can't mix NUnit tests with tests based on TinyTest. This is because in the NUnit project you will probably need to add the AutoCAD DLL references and set the copy local to true (this doesn't mean you can use AutoCAD objects in NUnit tests, only that the NUnit assembly will actually load). In the TinyTest Assembly you set the copy local to false, as per normal because it runs inside AutoCAD.
• Running the tests by launching AutoCAD from a .BAT file with a .SCR script to NETLOAD the test DLL and then run the command that fires the tests is much faster than running the tests via Visual Studio's debug.
• You can run the tests in the debugger and it works very well. I included the TinyTest source code by recursively linking it into the Test project so I could debug it (and fix it) but the debugger should step through it anyway if you have built it yourself with PDB etc. symbols and referenced it as a DLL.
• Don't forget to lock the Document if you are messing with the DataBase.
• It's still very much a work in progress. I don't know how much progress I will make in any particular time-frame, presently I am just focussed on getting it running tests so I can run tests, not so I can write the next NUnit. I would like to write a GUI runner for it but I make no promises about the eventuality of that.

...Still going...

[CADbloke]

So, I abandoned TinyTest. Attached is a Visual Studio 2013 Project / Solution that will...
run Nunit 3.0 tests in the NUnitLite console runner inside AutoCAD.
see  attached screengrab for what I am talking about

You will need to edit

• The AutoCAD DLL References
• the paths in the the .bat file that runs the tests
• The debug settings in the project settings in Visual Studio
• the path in the .SCR script fie
• whatever I have forgotten that is hardcoded. It shouldn't be much more.

The Project references are mostly on Nuget, the packages should download the first time you build it.

Yes, NUunit 3 is an Alpha version, I couldn't get NUnitLite 2.x to run. I did get the full version of the NUnit 3.0 Console running too but there seems no point, Lite is easier to set up and use.

If your test project has any AutoCAD references the you can't run it in a test runner outside AutoCAD, ie. not in Visual Studio.

More later, I have literally been left holding the baby

*** If you are using AutoCAD 2013 then you may have problems with Application.DocumentManager.MdiActiveDocument being null. Yes, I checked fiberworld. This change fixed it and the tests passed...

Code - C#: [Select]

1. // Remove
2. Document doc = Application.DocumentManager.MdiActiveDocument;
3. Database db = doc.Database;
4. // Replace with
5. Database db = HostApplicationServices.WorkingDatabase;
6. Document doc = Application.DocumentManager.GetDocument(db);

It ain't pretty but it worked on my machine. Read Owen's reply below the post I linked for why it ain't pretty.

[Mike Leslie]

I have been using the Test Explorer in Visual Studio for testing AutoCAD .NET dlls out of process.

What I mean is that I have three projects:

1)  .NET in-process dll inside AutoCAD
2)  .NET out-of-process dll
3)  .NET test project

The tests can run with either one or two instances of Visual Studio running.

For one instance of VS (when the in-process one is working well), it is the out-of process project and test project that are open.  My tests call my out-of-process functions which then get a reference to AutoCAD, NETLOAD the in-process dll, and either execute a command or (my new favorite) use AcadApplication.GetInterfaceObject to execute a function and return a COM object.  If a COM exception occurs or the returned object failed an assert, the test fails.

If the in-process one needs debugged, then I have two instances of Visual Studio running.  I build and run the in-process one (which starts AutoCAD with my debugger running

   <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x64'">
    <StartProgram>C:\Program Files\Autodesk\AutoCAD 2013\acad.exe</StartProgram>
    <StartAction>Program</StartAction>
  </PropertyGroup>
), I then can run the tests from the other Visual Studio project, but I will get to use the breakpoints and see the variable values from the in-process debugger.


It is a little cumbersome with the two projects, but it has been working well for me since for my work, I usually prefer to drive AutoCAD from the outside as opposed to making new widgets for use in the AutoCAD UI.  It is WAY faster to run a .NET in-process module to do the "real work" than to write the same process using the AutoCAD COM library.  You get access to features that the COM library doesn't support and you can do it with reliable database transactions and less likelihood of "RPC server is telling you to buzz off cuz it's busy." errors.

[CADbloke]

Mike,that sounds effective but, as you said, cumbersome.

You can add the NUnitLite Console runner to an existing NUnit project...
1. In Nuget, add NUnitLite 3 and the NUnit 3 Framework.
2. Get the App.cs file from the zip I posted, it has the code to display the console
3. Copy the Command class and tweak it as you wish, it launches NUnitLite and runs the tests. You can set all sorts of options there.
4. You can set the startup to run a .SCR file to netload the DLL and run the command.

That's about all there is to it, the rest is semantics.

I use a .BAT file to launch AutoCAD to make running the tests a 1-right-click-run operation from Visual Studio's solution explorer, or F5 to debug the tests. Try the Visual studio solution I attached above, it is a very simple example of how I made it work.

EDIT: Duh moment: the .BAT file is totally unnecessary - Ctrl-F5 (Start without Debugging) will run the current build configuration with the debug settings you have so you still need the .SCR file and you need to set the Test project as the startup project.

[CADbloke]

For those of you following along, here is the relevant C# code

App.cs

Code - C#: [Select]

1. using System;
2. using System.Runtime.InteropServices;
3. using Autodesk.AutoCAD.Runtime;
4.  
5. [assembly: ExtensionApplication(typeof (NUnitAutoCADTestRunner.NUnitRunnerApp))]
6.  
7. namespace NUnitAutoCADTestRunner
8. {
9.   /// <summary> AutoCAD extension application implementation. </summary>
10.   /// <seealso cref="T:Autodesk.AutoCAD.Runtime.IExtensionApplication"/>
11.   public class NUnitRunnerApp : IExtensionApplication
12.   {
13.     public void Initialize() { AllocConsole(); }
14.     public void Terminate() { FreeConsole(); }
15.  
16. // http://stackoverflow.com/questions/3917202/how-do-i-include-a-console-in-winforms/3917353#3917353
17. // http://web.archive.org/web/20100815055904/http://www.thereforesystems.com/output-to-console-in-windows-forms-application
18. // http://www.codeproject.com/Questions/229000/Will-Console-writeline-work-in-Windows-Application
19.  
20.     /// <summary> Allocates a new console for current process. </summary>
21.     [DllImport("kernel32.dll")]
22.     public static extern Boolean AllocConsole();
23.  
24.  
25.     /// <summary> Frees the console. </summary>
26.     [DllImport("kernel32.dll")]
27.     public static extern Boolean FreeConsole();
28.   }
29. }

Command.cs

Code - C#: [Select]

1. using Autodesk.AutoCAD.Runtime;
2. using System.Collections.Generic;
3. using NUnitAutoCADTestRunner;
4. using NUnitLite.Runner;
5.  
6. [assembly: CommandClass(typeof (TestRunner))]
7. namespace NUnitAutoCADTestRunner
8. {
9.   public class TestRunner
10.   {
11.      /// <summary> This command runs the NUnit tests in this assembly. </summary>
12.     [CommandMethod("RunNUnitTests")]
13.     public void RunNUnitTests()
14.      {
15.        string[] NunitArgs = new List<string>
16.        {
17.          // for details of options see  http://www.nunit.com/index.php?p=nunitliteOptions&r=3.0
18.          "--verbose"  
19.        }.ToArray();
20.        
21.        new NUnitLite.Runner.TextUI().Execute(NunitArgs);
22.      }
23.   }
24. }

...update - you'll want to decorate your NUnit Test classes with the Single-Threaded Apartment State attribute (new to NUnnit 3, NUnit 2 uses [RequiresSTA])

Code - C#: [Select]

1. [TestFixture, Apartment(ApartmentState.STA)]
2. public class No_cause_for_alarm_This_is_just_a_Test
3. { ...

...I had trouble with AutoCAD 2015 in debug. Probably something about not getting enough FIBER.

[CADbloke]

http://www.pluralsight.com/courses/outside-in-tdd is a very interesting look into different types of automated code testing, in particular Unit Testing, Acceptance Testing and testing the behaviour of your code. There are quite a few other courses there on Unit testing and many other topics so it you lay out the $$ for a 1-month subscription you could get plenty of value from that site.

Note that I still have plenty more tests in the unit test project than I have in the test project that runs inside AutoCAD which are very much integration and behaviour tests, they don't replace unit tests, they add to the confidence-building test collection.

[CADbloke]

I posted my latest version of CADtest to Github. There's a VB version as well. I started a thread at  http://www.theswamp.org/index.php?topic=49035 for any comments you may have.