TheSwamp

Code Red => .NET => Topic started by: Bill Tillman on March 11, 2016, 05:25:35 PM

Title: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 11, 2016, 05:25:35 PM
A project I'm involved with has been running really stable for several months but now suddenly it's faulty with an intermittent problem...and don't you love the sound of that. Anyway, the C# code takes input from a database, does some things for the sales and fabrication departments and then launches AutoCAD and a LISP program to create a drawing. It's been creating anywhere from 50 to 200 drawings each day. So far so good. This thing is totally automated so there's no need or allowance for human interaction.

What's happening now is that around 3-4 runs per day, AutoCAD stalls just before it starts the drawing process. In debugging I've checked. When I check the error messages I get this:

Code: [Select]
ERROR: System.Runtime.InteropServices.COMException (0x8001010A): The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))
   at AutoCAD.IAcadDocument.SendCommand(String Command)
   at BOM_CALCS.LaunchAutoCAD.StartAutoCAD() in C:\Engineering\Calculations\BOM\BOM_CALCS\LaunchAutoCAD.cs:line 55
   at BOM_CALCS.Program.Main(String[] args) in C:\Engineering\Calculations\BOM\BOM_CALCS\Program.cs:line 102
Unfortunately the 3-4 times per day has now grown to 10-12 times per day. There's not much in the way of help for this because it's intermittent. And the same code that runs perfect all other times hits this enough for it to be of concern.


Title: Re: .NET Code Suddenly Takes a Dive
Post by: MickD on March 11, 2016, 05:58:53 PM
this message:

The message filter indicated that the application is busy

could be a clue that the application is getting smashed by more than one request at a time, this link may be of some help

https://msdn.microsoft.com/en-us/library/ms228772.aspx

hth
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 11, 2016, 08:59:23 PM
Thanks, but I'm pretty sure that's not the issue. There are times I've seen this run three (3) runs at a time. It almost always runs one at a time about every 5-10 minutes. Sometimes it's heavier but usually it's quiet enough for it to not have simultaneous runs. And the times I see it fail is when it's running only one instance. Wish it were that easy. This is a hard one to crack.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: kdub_nz on March 11, 2016, 11:04:07 PM
< .. >

Code: [Select]
ERROR: System.Runtime.InteropServices.COMException (0x8001010A): The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))
   at AutoCAD.IAcadDocument.SendCommand(String Command)
   at BOM_CALCS.LaunchAutoCAD.StartAutoCAD() in
        C:\Engineering\Calculations\BOM\BOM_CALCS\LaunchAutoCAD.cs:line 55

   at BOM_CALCS.Program.Main(String[] args) in
        C:\Engineering\Calculations\BOM\BOM_CALCS\Program.cs:line 102

Hi Bill
I don't play with Com, but :
What is the client version
What is the development API version

What is the Command you are trying to send.
What is the code at line 55 and prior that is failing.

Did you develop this app on an earlier version of the client.

Is the App running on the same machine as the client.

Single User or multiple ?
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 13, 2016, 11:50:59 AM
Thanks kdub. I will be able to reply with definite points tomorrow when I'm back in the office. The tricky thing about this is that it has seen days, weeks, on end when it works perfectly. A very experienced developer was hired recently and he introduced an RPC service into the mix. This is when I started noticing the problem. However, in his defense all I can say is it's an intermittent problem. That is out of 100 runs each day only about 10 of them will have trouble. And he has extensive experience in development, I mean this guy practically invented OOP and he thinks the problem is a COM object being left un-closed but so far I'm not seeing that. I will be working on this with him more starting tomorrow morning.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: It's Alive! on March 13, 2016, 07:56:02 PM
... he thinks the problem is a COM object being left un-closed.....

would be my first guess too, but, I haven't invented anything
Title: Re: .NET Code Suddenly Takes a Dive
Post by: kdub_nz on March 13, 2016, 10:41:51 PM


Quote
< .. >
The tricky thing about this is that it has seen days, weeks, on end when it works perfectly. A very experienced developer was hired recently and he introduced an RPC service into the mix. This is when I started noticing the problem.
< .. >

This is one of the issues with peer to peer forums.
The critical information doesn't get offered up in the first post.

My questions shouldn't need to be asked.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 16, 2016, 11:33:07 AM
Thanks again for everyone's patience. Let me see now if I can spell out the problem because this has become a major bug that I must resolve.

This is a completely automated project, no users are allowed to interface with it. The only exception being me the main developer and I only mess with it when it stalls. Basically Apache+php (XAMPP) listens for requests from the network. Upon receiving one the main .exe of my project is launched from a php program. All that is working well. Eventually, the call is made to AutoCAD to produce a drawing.  The code which I use to launch AutoCAD is thus:
Code: [Select]
using System;
using AutoCAD;

namespace AHD_BOM_CALCS
{
    class LaunchAutoCAD
    {
        private static IAcadApplication vAcadApp = null;
        private static IAcadDocument vAcadDoc = null;

        // CHANGE THIS TO AutoCAD.Application.20 FOR ACAD 2015/16
        private static string vAcadID = "AutoCAD.Application.20";

        private static string LispPath = @"C:/Engineering/04VLISP/ANGLE/Bomber/";
        private static string DwgPath = @"C:\Engineering\09Automated Drawings\";
       
        public static void StartAutoCAD()
        {
            try
            {
                // Check for instance of AutoCAD
                vAcadApp = (IAcadApplication)System.Runtime.InteropServices.Marshal.GetActiveObject(vAcadID);
            }
            catch (Exception) // If none found start a new instance of AutoCAD
            {
                System.Type AcadProg = System.Type.GetTypeFromProgID(vAcadID);
                vAcadApp = (IAcadApplication)System.Activator.CreateInstance(AcadProg);
            }

            string titleblock = "Template1(Automated).dwg";

            if (vAcadApp != null)
            {
                vAcadApp.Visible = true;
                vAcadDoc = vAcadApp.Documents.Open(DwgPath + titleblock, true);
                string LispName = "Bomber.lsp";
                vAcadDoc.SendCommand("(load \"" + LispPath + LispName + "\" \"The load failed\") ");
                vAcadDoc.SendCommand(LISPVariables.PrepareAcadCmd() + " ");
                vAcadDoc.SendCommand("(setvar \"osmode\" 703) ");
                vAcadDoc.SendCommand("(setvar \"cmdecho\" 1) ");
                vAcadDoc.SendCommand("(setvar \"pickbox\" 5) ");
           
              vAcadDoc.SendCommand("(setq svPath \"C:/0/Misc/BomberDevDwgs/\") ");
              vAcadDoc.SendCommand("(setq *erp 1) ");
              vAcadDoc.SendCommand("DrawitNow ");
            }
            else
            {
                Console.WriteLine("ERROR: The drawing file is missing.");
                Console.ReadKey();
            }
        }
    }
}

Prior to running this class, the C# project does a bunch of calculations and writes those results to an Oracle database. All of that part is running fine, never an issue. It's only when the LaunchAutoCAD class gets to the SendCommand lines where things go south. And it never seems to fail at the same line. Sometimes none of the commands go and sometimes 2 or 3 of them go but then it stalls and never gets the last command which is "Drawit" which starts the LISP code. The drawing is produced and saved to the server. It runs all day and night long listening for requests. Out of over 100 runs each day about 5-10 times it stalls. I have checked the workstation where this project runs and there does not seem to be anything like a stuck acad.exe process in the Task Manager or anything else. I have cold booted it to see if that will clear the cobwebs but that did not help. As the old saying goes, I'm ready to urinate on a spark plug if that will work.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: ChrisCarlson on March 16, 2016, 12:07:28 PM
Since it's tripping up at the SendCommand() area, you can try removing some of the commands. Do these three really need to be set within the C# section of code? It would be quite easy and clean to run them in the beginning of your lisp routine.

Code - C#: [Select]
  1. vAcadDoc.SendCommand("(setvar \"osmode\" 703) ");
  2. vAcadDoc.SendCommand("(setvar \"cmdecho\" 1) ");
  3. vAcadDoc.SendCommand("(setvar \"pickbox\" 5) ");

Code - Auto/Visual Lisp: [Select]
  1. (if (not (= (getvar "osmode") 703)) (setvar "osmode" 703))
  2. (if (not (= (getvar "cmdecho") 1)) (setvar "cmdecho" 1))
  3. (if (not (= (getvar "pickbox") 5)) (setvar "pickbox" 5))
 

Title: Re: .NET Code Suddenly Takes a Dive
Post by: Jeff H on March 16, 2016, 12:29:03 PM
https://msdn.microsoft.com/en-us/library/ms228772.aspx
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Tharwat on March 16, 2016, 12:53:21 PM
Why using sendCommand since you can call or change the system variable via the application?

e.g;

Code - C#: [Select]
  1. Application.SetSystemVariable("OSMODE", 0);
  2.  
Title: Re: .NET Code Suddenly Takes a Dive
Post by: ChrisCarlson on March 16, 2016, 01:03:01 PM
Why using sendCommand since you can call or change the system variable via the application?

e.g;

Code - C#: [Select]
  1. Application.SetSystemVariable("OSMODE", 0);
  2.  

Don't think that would work from outside of an AutoCAD executed program.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Tharwat on March 16, 2016, 01:08:38 PM
Why using sendCommand since you can call or change the system variable via the application?

e.g;

Code - C#: [Select]
  1. Application.SetSystemVariable("OSMODE", 0);
  2.  

Don't think that would work from outside of an AutoCAD executed program.
Opps, Sorry I did not notice that.
Thanks Chris
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 16, 2016, 01:35:03 PM
Since it's tripping up at the SendCommand() area, you can try removing some of the commands. Do these three really need to be set within the C# section of code? It would be quite easy and clean to run them in the beginning of your lisp routine.

Code - C#: [Select]
  1. vAcadDoc.SendCommand("(setvar \"osmode\" 703) ");
  2. vAcadDoc.SendCommand("(setvar \"cmdecho\" 1) ");
  3. vAcadDoc.SendCommand("(setvar \"pickbox\" 5) ");

Code - Auto/Visual Lisp: [Select]
  1. (if (not (= (getvar "osmode") 703)) (setvar "osmode" 703))
  2. (if (not (= (getvar "cmdecho") 1)) (setvar "cmdecho" 1))
  3. (if (not (= (getvar "pickbox") 5)) (setvar "pickbox" 5))


Thanks Chris, that is one of the ideas I've been toying with but with the randomness of where it fails I don't know it that's the ticket. However, I am cold booting the machine right now and will start trying out some more of these suggestions. You guys are the best.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: dgorsman on March 16, 2016, 01:47:16 PM
I'm wondering if there is potentially a regen involved in the cycle, as the application is made visible and a drawing is opened.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: n.yuan on March 16, 2016, 01:49:23 PM
Why using sendCommand since you can call or change the system variable via the application?

e.g;

Code - C#: [Select]
  1. Application.SetSystemVariable("OSMODE", 0);
  2.  

Don't think that would work from outside of an AutoCAD executed program.

Well, the OP's code has already obtained the AcadApplication object, it can certainly be done (and should be done):

vAcadApp.SetSystemVariable(...)

instead of SendCommand(), which is likely the cause of his problem.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: n.yuan on March 16, 2016, 02:05:09 PM
Using full desktop AutoCAD as a non-user-interactive server app is problematic, to say the least. Especially your code try to use the same instance of AutoCAD session (the code first tries to find existing AutoCAd session, and only start a new session of no existing one is obtained), which is particular bad practice when using AutoCAD as "server" (plus, multiple SendCommand() calls that could be executed out of sync makes thing even worse). Even you can claim it had worked for quite a while, it was mostly out of good luck (low work load), IMO.

If you have to use AutoCAD as server, it would be better to start a new AutoCAD session, build one drawing and then close the AutoCAD session. Just imagine, when drawing building is in progress, and since the process need to get data from somewhere (Oracle DB, as you mentioned) outside the process, and you never know what could happen and the process has to wait the data come back. So, the drawing building process could be long or short. If then a nother request comes in and you code luckily find the existing AutoCAD session and tries to hold it for new request while the existing AutoCAd session is still working on previous request... Your app now may be in a "dead lock" state.

Depending on your workload, running multiple AutoCAD session in the same computer may also be problematic, of course, because AutoCAd desktop app is such a "resouces" hangry beast.

You should consider to use AutoCAD console for this type of work instead of full AutoCAD.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: ChrisCarlson on March 16, 2016, 02:47:34 PM
Why using sendCommand since you can call or change the system variable via the application?

e.g;

Code - C#: [Select]
  1. Application.SetSystemVariable("OSMODE", 0);
  2.  

Don't think that would work from outside of an AutoCAD executed program.

Well, the OP's code has already obtained the AcadApplication object, it can certainly be done (and should be done):

vAcadApp.SetSystemVariable(...)

instead of SendCommand(), which is likely the cause of his problem.

Interesting, stepping back and coming back that makes sense. So based on dropping the SendCommand() is it a moot point between vAcadApp.SetSystemVariable within C# and (setvar ...) within LISP?
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 16, 2016, 03:13:35 PM
https://msdn.microsoft.com/en-us/library/ms228772.aspx
By George, I think you've got something there. However, after the reboot, the project has just made 15 successful consecutive runs. Turns out the tech who told me he rebooted it last week was confusing it with another machine. So to avoid confusion I went into the cold room and rebooted it myself. Now, I'm not saying that a reboot fixed it but it's running better than it was. And looking into that article you referenced our .NET guru and myself are going to look into that further. In any case, I think now the overseers are a little more understanding that it's not the C# code itself, and it's especially not the LISP code. Rather it does appear to be a conflict between Visual Studio and a multi-threaded app like AutoCAD...

   translation =  it's all Bill Gate's and Steve Balmer's fault .... wink, wink, nudge, nudge!

However, there's no proof in the pudding yet, just some reassurance that debugging the code it not the answer. Another tech here just told me that he actually has a script to reboot Windows 7 machines for this very reason. They run automated tasks and similar problems occur unless they reboot it every night.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: dgorsman on March 16, 2016, 03:23:44 PM
AutoCAD is NOT multi-threaded (in fact, you can't do that with AutoCAD customization at all) except for a couple of parts which don't/shouldn't have any relevance here.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 16, 2016, 03:26:59 PM
Using full desktop AutoCAD as a non-user-interactive server app is problematic, to say the least. Especially your code try to use the same instance of AutoCAD session (the code first tries to find existing AutoCAd session, and only start a new session of no existing one is obtained), which is particular bad practice when using AutoCAD as "server" (plus, multiple SendCommand() calls that could be executed out of sync makes thing even worse). Even you can claim it had worked for quite a while, it was mostly out of good luck (low work load), IMO.

If you have to use AutoCAD as server, it would be better to start a new AutoCAD session, build one drawing and then close the AutoCAD session. Just imagine, when drawing building is in progress, and since the process need to get data from somewhere (Oracle DB, as you mentioned) outside the process, and you never know what could happen and the process has to wait the data come back. So, the drawing building process could be long or short. If then a nother request comes in and you code luckily find the existing AutoCAD session and tries to hold it for new request while the existing AutoCAd session is still working on previous request... Your app now may be in a "dead lock" state.

Depending on your workload, running multiple AutoCAD session in the same computer may also be problematic, of course, because AutoCAd desktop app is such a "resouces" hangry beast.

You should consider to use AutoCAD console for this type of work instead of full AutoCAD.
Yes, I agree with all that you said. However, the nature of this project means that several requests may hit simultaneously. Let me elaborate on that if I may. In this setup we use a Load Balancing server which sends requests to one of two identical Windows 7 machines running Apache+php and AutoCAD 2016 Mechanical. AutoCAD is loaded and sitting there with no drawing open all the time. Presently, and due to all the code changes happening at this stage, we let one of them sleep and the other works all day but eventually we will turn both of them on and it can handle the load, even with 200-300 requests per 8 hour shift. I have seen just one of these boxes handle up to three instances successfully without a problem. But when a 4th one hits, it's a no-go and AutoCAD just chokes on it all. And it's not foreseen that even under the heaviest loads that we would put on it, two machines will be more than adequate.

I tried it with your suggestion of launching AutoCAD on each run long ago but since AutoCAD can take up to 3 minutes to load, that just wasn't an option. So leaving it open with no drawing open works fine. There are times when a drawing file is open but again, these computers are not accessible at their consoles. I've also witnessed times when one of the hung drawings is sitting there on the monitor and another run comes in and completes just fine.

There are of course lots of options we will look at over the next few weeks. There is even talk of getting rid of AutoCAD all together and use some other kind of rendering engine, but I've warned them that how do you send a client an AutoCAD file if there is no AutoCAD to produce it. The techs counter with anything running on Microsoft is too error prone and we have to eliminate it if possible. They get no argument from me on that point.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 16, 2016, 03:50:34 PM
AutoCAD is NOT multi-threaded (in fact, you can't do that with AutoCAD customization at all) except for a couple of parts which don't/shouldn't have any relevance here.
Aha, I was not aware of that. I thought it was. Shows how much I know.
https://knowledge.autodesk.com/support/autocad/troubleshooting/caas/sfdcarticles/sfdcarticles/Support-for-multi-core-processors-with-AutoCAD.html (https://knowledge.autodesk.com/support/autocad/troubleshooting/caas/sfdcarticles/sfdcarticles/Support-for-multi-core-processors-with-AutoCAD.html)
I used AutoCAD for loads of renderings a few years ago and would watch the renderings progress with each one of the little squares which indicated that one of the cores was being used. When I saw 12 little boxes prepare the rendering I assumed (wrongly I might add) that AutoCAD was an entirely multi-threaded app.

In any case, after the reboot, the box has now made 32 runs completely without incident. That's much better than it was doing this morning and yesterday. So until we figure out some other ideas, I'm just going to reboot this thing each evening.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: ChrisCarlson on March 16, 2016, 03:54:46 PM
You can bypass AutoCAD loading, kind of

http://www.techsoft3d.com/products/autodesk-platforms/realdwg/

For the price it's pretty haphazardly put together but an option.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Bill Tillman on March 16, 2016, 04:28:53 PM
Thanks again everyone. After rebooting the machine has now run 42 instances, or about a 1/2 day's normal usage without errors. Long ago when I worked in IT the users would always complain that my resolution to almost everything was just reboot. Hey it works doesn't it. I'm not saying this is the final and ultimate resolution but it's working for now and allowing me to get on with other tasks. And I can finally tell some skeptics around here: "Left open Com objects? I ain't got no left open Com Objects. I don't need no stinking left open Com Objects!"
Title: Re: .NET Code Suddenly Takes a Dive
Post by: Jeff H on March 17, 2016, 08:31:15 PM
There is even talk of getting rid of AutoCAD all together and use some other kind of rendering engine, but I've warned them that how do you send a client an AutoCAD file if there is no AutoCAD to produce it.

Maybe also look at https://developer.autodesk.com/api/autocadio/
Title: Re: .NET Code Suddenly Takes a Dive
Post by: It's Alive! on March 18, 2016, 10:51:25 AM
Wow, you all need something like teigha, or realdwg
Title: Re: .NET Code Suddenly Takes a Dive
Post by: CADbloke on March 20, 2016, 06:23:15 PM
If you wanted to scale up you could just rent extra copies of AutoCAD by the month and run them on spare computers.

Teigha .NET Classic (https://www.opendesign.com/the_oda_platform/teigha_net_classic) is definitely an option, it's free for in-house use (https://www.opendesign.com/NonCommercial) (apart from the annual membership) for the first 2 years, then is gets more expensive: https://www.opendesign.com/Commercial

You can bypass AutoCAD loading, kind of
http://www.techsoft3d.com/products/autodesk-platforms/realdwg/
For the price it's pretty haphazardly put together but an option.
Confirmed. If you think the Docs on AutoCAD .NET are ... well, you know ... you should see the docs for their lesser-used platform. Building the installer is a nightmare, there's no pre-fab solution as such. I hope you know and love Wix. I assume building an installer for Teigha stuff is just as tricky.

AutoCAD is NOT multi-threaded (in fact, you can't do that with AutoCAD customization at all) except for a couple of parts which don't/shouldn't have any relevance here.
Aha, I was not aware of that. I thought it was. Shows how much I know.
https://knowledge.autodesk.com/support/autocad/troubleshooting/caas/sfdcarticles/sfdcarticles/Support-for-multi-core-processors-with-AutoCAD.html (https://knowledge.autodesk.com/support/autocad/troubleshooting/caas/sfdcarticles/sfdcarticles/Support-for-multi-core-processors-with-AutoCAD.html)
I used AutoCAD for loads of renderings a few years ago and would watch the renderings progress with each one of the little squares which indicated that one of the cores was being used. When I saw 12 little boxes prepare the rendering I assumed (wrongly I might add) that AutoCAD was an entirely multi-threaded app.
...
I believe parts of the rendering engines are multi-threaded but the Database engine definitely isn't. You can run background threads for things unrelated to Database access but everything for Database access must run on the UI thread.
Title: Re: .NET Code Suddenly Takes a Dive
Post by: It's Alive! on March 20, 2016, 06:49:01 PM
If you wanted to scale up you could just rent extra copies of AutoCAD by the month and run them on spare computers.

The cost of using something like Teigha isn’t per-seat.
It would probably be an order of magnitude faster, and much cleaner option than using send command
Title: Re: .NET Code Suddenly Takes a Dive
Post by: CADbloke on March 20, 2016, 11:22:32 PM
If you wanted to scale up you could just rent extra copies of AutoCAD by the month and run them on spare computers.

The cost of using something like Teigha isn’t per-seat.
It would probably be an order of magnitude faster, and much cleaner option than using send command
Agreed. My suggestion was a way to scale with no more development effort. Doing it properly would definitely work better, as you would expect.