Topic: [Community Project Idea] Document Management Framework

[TR]

I've been toying with the idea of creating a document management framework in C# for engineering type documents. As a base it would handle customizable revision control, document storage, custom properties (searchable) and customizable tree structures (not the same as document storage).

If anything this can be used as base for some of us to get out there and learn C# by creating something and if something fruitful comes of it then that is great.

If anyone is interested in this drop a reply and I'll start working on a draft specification that can then be reviewed.

Note: I am far from a good programmer, especially in C#, so don't let lack of knowledge frighten you from participating. We'll just have some fun and see what happens.

[Troy Williams]

The company I work for has been wrestling with some document management issues. I had been toying with the same idea that you have presented. Except I was thinking about using subversion as the backend seeing as it already does version control really well  . The problem is storing the information in a searchable database (I was thinking about MySQL). Subversion has an answer for this as well, hook scripts. These are scripts that are executed when specific events occur like committing document changes.

What I was going to do was send however added a new document an email with a link back to a webpage where they would fill out a description and keywords that would be used for searching later on. If the user was really keen they could use a program like tortoiseSVN they could use tags like [description][/description] and [keywords][/keywords] to pre fill out any information that they wanted committed. The commit messages would also be searchable.

The only problem that I had is that most people have no clue how to use a revision control system properly. That is where a good UI front end would make or break a project like this.

All in all, I would be interested... We could use the svn binaries to communicate with the server through a  c# app...

[MickD]

Sounds like a good idea Tim. If it's going to work in any form it needs to be simple to start with, perhaps a form that has fields for the doc name, a keyword, its revision and storage location/url in a simple searchable db and expand on that. Perhaps you could then bring in a subversion module at a later date??
The point is you don't want to paint yourself into a corner too early, you want to get results quickly to retain interest 

[Troy Williams]

The point is you don't want to paint yourself into a corner too early, you want to get results quickly to retain interest 

I agree! That's why I haven't gotten anywhere other then thinking about it .

[MickD]

A big part of the project would be a reporting engine/UI instead of a commercial one like crystal reports. I've always wanted to have a go at one (a graphical layout editor with gdi etc)
It could be a good exercise in working with the print preview control and the printer, a simple text output would do for a start.

I agree! That's why I haven't gotten anywhere other then thinking about it...

heh, I have a list as long as my arm of those projects

<edit> and I think I just added another one...</edit>

[TR]

I had thought about subversion but I'm not sure how it would work with XREF's. Also I'm not too sure about how subversions revision controls will work with users custom revisions.


I was thinking something along the lines of this:

Each document can be stored to any random location on the physical disk (whatever we find to be most efficient) and the database will handle the mapping of what the user perceives to be the path to the physical file. For example, if a user requests "\\myserver\mystorage\Job\X12334\10001D.dwg" the document management system will return a file located on the server at "c:\storage\asd213123\1233141\qwerty.dwg" based on the mapping defined in the database. This will allow us to develop an efficient filing scheme yet allow the users to use whatever filing scheme they'd like and all will work out. It will also allow the user to change their scheme in the future and all can be remapped.

With that being said I think the first order of business would be to:

1) Develop an efficient method of storing the physical files.
2) Develop a preliminary database layout.
3) Come up with a mapping scheme from the database to the physical files.

Am I making sense?

[TR]

Am I making sense?

I guess not.

[M-dub]

I can't offer much (anything) in the way of code, but my work life has been submerged into the EDMS world for a while now and I've had a lot of ideas on how the package we went with could be improved, etc. and I'm actually keeping a running list of these improvements.

If I can be of any help to you guys, let me know!  I think this is a great idea.

[TR]

Mike:

I've also done quite a bit of work with a document management package which I won't name (*cough* see signature *cough*) and I find it quite lacking, that's one of the reasons I want to start something new.

Any input you can provide would be more than welcome.

[M-dub]

Cool!
Well, I, like you, have been driven psycho by Cyco, but we're now past that, thankfully.

I had some thoughts on creating our own in-house document management package a few years ago... let's see if I can find the 'preliminary screenshots' I came up with...
Back later...

It also appears that I'm going to be VERY busy at work now, so I'll have to pop in when I can.

[Troy Williams]

They make sense.

Item 1 is probably the trickiest. Subversion would be awesome for this as it stores the original file + deltas of the changes. Unfortunately, it cannot glean the xref information automatically as the dwg files are binary. However, your c# front end could get this information before the binary is committed to SVN. Then the information could be posted to a SQL database. Another thing to keep in mind is that Subversion does support locking of files that are checked out.


If the c# front end is done correctly, it shouldn't matter what the back end of the system is: storing the files directly in a share, compressing the files, using subversion or storing them in a database as blobs.

For starters, like Mick suggested, things should be kept simple. So I think the method that you outlined would be good for starters. Another good thing would be to map out how you envision the work flow. How would the system work from a users point of view?

[TR]

Let me see if I can get some stuff together today/tonight to clarify what I think the best way to approach this is. From there we can discuss.

[MP]

Very interesing! I salute you folks, and don't wish to be a stick in the mud, but I personally would be looking to see how I could exploit and extend the abilities of an off the shelf solution like Synergis Adept <which as I understand it deals with linked documents like xrefs> before investing a lot of time into a new solution, as interesting and rewarding as it may be. It's a huge effort to design, never mind code such an application (in other words cheaper to go off the shelf). Just sayin'.

[TR]

Adept is quite costly compared to the $0 a year it will cost a firm to use our software if we succeed. Also it gives anyone interested a chance to flex their gray matter a bit and help either bang out some code or give some comments/suggestions.

[MP]

Adept is quite costly compared to the $0 a year it will cost a firm to use our software if we succeed.

Guess it depends how you look at it Tim. If you develop the application on company time its cost will exceed any off the shelf solution, and by magnitudes unless your enjoying minmum wage. If on the other hand you develop it entirely on your time you are subsidizing the company on your ticket, which is extremely generous given some past accounts, not to mention risky because they may claim ownership after the solution comes to fruition. Before you dispute the latter, and indeed, before you launch into development like this (if entirely on your ticket) I would suggest getting an agreement in writing to protect yourself and your contributors. Same goes for any contributing parties with their employers. IMO of course.

... it gives anyone interested a chance to flex their gray matter a bit and help either bang out some code or give some comments/suggestions.

I fully appreciate that, and had hesitations posting my comments because of this -- but in the end my conscience won out. Wrong or right as all said may be, I'm compelled to respond in a way that is responsible from my perspective, and as such, my conscience is now clear.

[M-dub]

MP, those are some good points, but Tim's also got a point.
If money wasn't an issue, I'd love to take the route MP is suggesting, but since money IS an issue, Tim's idea has legs.
We have a drawing management database that one of the bosses designed, but I really do NOT like it and I never have.  It's been around for about 5 years or so and they've made a few changes along the way, but it isn't very user friendly.  He knows I hate it and when I was informed that I would be taking over the CAD Dept. for a while, that I would be welcome to keep track of improvements that could be made and we'd address them all at a later date.  I wanted to make the first suggestion "Start over", but I don't think that would go over too well. 

[TR]

I can't speak for anyone else but will be working on this on my own time to learn a little bit more and hopefully develop a solid understanding of C#. As I stated before my company currently has a document management system in place that works, I just think it could be better. If something fruitful comes from this then I will certainly look at replacing our current system with it but that is not my ultimate goal.

In regard to my company claiming ownership to software I work on I don't see how that's possible. When I was hired I signed an agreement stating that any patents I developed would be transferred into their name, that is all. Considering my company has nothing to do with software development and I will be developing this on my own time I'm not really concerned.

[MP]

I can't speak for anyone else but will be working on this on my own time to learn a little bit more and hopefully develop a solid understanding of C#.

It's an outstanding target to use as the basis for learning C#, as one will need to learn C#, the BCL, software patterns and design documenting techniques in order to produce an efficient design that can be effectively shared with others participating in the effort.

As I stated before my company currently has a document management system in place that works, I just think it could be better. If something fruitful comes from this then I will certainly look at replacing our current system with it but that is not my ultimate goal.

Acknowledge.

In regard to my company claiming ownership to software I work on I don't see how that's possible. When I was hired I signed an agreement stating that any patents I developed would be transferred into their name, that is all. Considering my company has nothing to do with software development and I will be developing this on my own time I'm not really concerned.

And yet you should be IMO, as it is irrelevant the company you work for is not invovled in software development per se. The software amounts to a tool that may enable the company to be more effective or efficient (read profitable) and was developed whilst in their employ. On that premise and the clause you cite, they could legally assume (or fight for) ownership because the tool's concept was born of their business requirements, the expertise and experience required to produce said tool was aquired and refined whilst you were in their employ, and the tool works with their data.

I hope you understand Tim, I only ante up this opinion because I recognize the project represents a lot of work for all involved and I care for my friends. That includes you. I'm not looking for a heated debate. I'd be more than happy to find out I'm wrong. I hope ya'll care enough just to ensure your respective butts are covered by persuing do diligence to prove me wrong. Take care.



Edit: Attempted to fix grammar; still a mess but alas, I'm going to play the "I'm too tired" card.

[TR]

I appreciate the concern Michael. I'm fairly sure I know what I'm getting myself into and hopefully I can handle it. I've followed my fair share of open source projects and I feel I have a good understanding of how to get things done.

Regarding the issues with my work claiming the rights. Perhaps you are correct; I'll discuss with the Company President on Monday.

[mohnston]

I've always liked the idea of community projects but alas they are difficult.
A project like this would take a great deal of planning. If specifications are not completely and clearly spelled out the project will fail. There should be as much time spent in planning as in coding, maybe more. And the planning part happens up front which means you don't get to code for a while.
If you really want to learn C# I would suggest small apps that are limited in scope rather than any kind of document management system.

[TR]

I was just reading this about specifications. Pretty interesting.

Thoughts?

[MickD]

I agree to a point, you still need to decide what you want to achieve in the final product, from there you can build a very broad/open spec which still gives you some definite targets to hit but doesn't paint you into a corner and doing unnecessary work.

With db systems I personally think it's good idea to do what you want to do manually, that is, using paper forms and lists (tables) and real folders etc. to log and store info, you will probably change it many times before you get a system that works and is simple for everyone to use. You can get input and help from others who don't code as well.
Once you get your system design 'on paper' it's much easier to code it up, you will have forms designed and data tables and queries etc already to go.

I don't know how you would go about this particular type of system, perhaps all involved could start logging stuff down putting what each thinks is required into lists, creating basic forms to be filed per job etc and briefly documenting the process used to accomplish all this.
If you can get the design started like this it would be a snap to code up, even if it's a manual entry db at first then slip sbv in to automate the data logging at a later date, having the structure up and running should also make this much easier.

[mohnston]

I was just reading this about specifications. Pretty interesting.

Thoughts?

This part of it makes sense

So what should you do in place of a spec? Go with a briefer alternative that moves you toward something real. Write a one page story about what the app needs to do. Use plain language and make it quick. If it takes more than a page to explain it, then it's too complex. This process shouldn't take more than one day.

Then begin building the interface — the interface will be the alternative to the functional spec. Draw some quick and simple paper sketches. Then start coding it into html. Unlike paragraphs of text that are open to alternate interpretations, interface designs are common ground that everyone can agree on.

Confusion disappears when everyone starts using the same screens. Build an interface everyone can start looking at, using, clicking through, and "feeling" before you start worrying about back-end code. Get yourself in front of the customer experience as much as possible.

Forget about locked-in specs. They force you to make big, key decisions too early in the process. Bypass the spec phase and you'll keep change cheap and stay flexible.

That describes what I meant by "spec".
There has to be something in between "just start and everything will work out" and "analyze it until you get sleepy and give up".
My experience has been if I don't spend some time defining exactly what I want to accomplish I end up paying big-time.
I was thinking along the lines of answering questions like:
What kind of documents will it track?
How will they be stored?
What data needs to be tracked?
Who will use the program?
How will they use it?
Will the program have absolute control of the documents?
Will this be a web-based program?

[TR]

I planned on a write-up similar to what you described. Forgive me for misunderstanding as when I hear spec I think of the wet dream of engineering specifications I peruse on a daily basis. 1000+ pages trying to define every aspect of a job, most of which won't work in reality.

I also apologize for not getting this document out earlier but I've acquired some sort of throat phlegm illness similar to the croup and have been sleeping or drinking whiskey while not at work. I promise I'll write something up this weekend and will post it Monday.

[MP]

I was just reading this about specifications. Pretty interesting.

Thoughts?

While there is merit to many of the 37 signals mantras keep in mind the scale of applications they author. Relatively speaking, they author small solutions. Their most ambitious solution basecamp, a service I've personally used for over a year now, is small and simple compared to a capable document management system. Indeed, one of the major complaints folks have about the basecamp service is that their document management model is anemic. It was but a month ago that they finally added crude file versioning after being inundated with requests.

If a functional spec "looks nothing like the finished product" and "has nothing to do with reality" it, in my opinion, it reflects the authors' understanding of the application's requirements and/or ability to commit it to paper, not the uselessness of the technique.

It's an art and science choosing what techniques one should employ to help one manage software development chaos, which has some dependencies on the style(s) the programmer(s) works with effectively. Should one blanket abandon specs? In a word, and in my opinion, no. If you have no target what is everyone collectively aiming for?

Should specs be lightweight? Specs should be detailed enough to fully describe (in plain language) the requirements of the software, but no more. Some applications will be lightweight, others will be more substantial. Again, it's a matter of scale.

I think "Getting Real" is a worthwhile read, and there's merit to much of what they say, principally keeping things light weight, lean, agile, responsive etc. However, I think it's important to keep in mind much of it is not new, and appears to be abbreviated regurgitations of well established strategies and techniques (as can be found in discussions by Martin Fowler etc.) re-floated in the context of the "37 Signals" experience. Subtitle: It would be worthwhile to be conversant with all the development strategies and techniques on which the 37 signals "philosophies" are based, derived and distilled from. In my opinion.

What the hell am I doing, participating in discussions like this before my first coffee?

<blink>

As I was typing this up Tim snuck in his post. Sorry to hear you're not feeling well Tim, sure hope you feel, and get better soon.

[TR]

I think my original intentions were not properly represented. I am not looking to create a complete document management system but instead a set of libraries to be used to create a document management system. We create the tools for storage, revision control, XREF management, etc. and allow the end user to piece together what he needs to create a solution that will fit his company. This will allow the user to get exactly what he needs and not what we feel he needs.

I think taking this approach and then provide a ridiculous amount of samples for creating systems in C#, VB.NET, IronPython, Boo, etc. would provide the end user with the best solution. All we need to do is first focus on writing the server libraries (storage, etc.) and then create a nice client library that will allow users to access their document management system from any .NET client (i.e. AutoCAD).

[MP]

I'm really not trying to poo poo this idea, but without a functional and discrete implementation using the framework I think it will be very challenging to design and bring a framework to fruition that will fulfill the intent. Put another way, a better approach, in my opinion, is to design a lightweight, but fully capable document management system, trying as best one can to genericize the libraries upon which it is founded so that others may benefit from the effort. Maybe this is what you mean.

Tim, I'm not picking on you. You're floating this topic to solicit feedback. Wrong or right this is just what comes into my mind so I'm sharing it. Hope you read it in that context.

[TR]

Michael:

I appreciate the feedback.

To fully understand what I'm trying to convey perhaps it would be best to look toward the world of web frameworks for a second. For example take a look at Ruby on Rails, Pylons or Turbogears. These are nothing more than a bunch of libraries designed to work together to create a web application. You can pick and choose the functionality you would like to use and the even bypass some components using either another library or perhaps a home-brewed one. This provides a very flexible solution for developing web based applications.

Like the aforementioned frameworks we would provide a batch script that would allow us to create a skeleton application using command line switches to define which .NET language you want to create your Document Management System in. This would allow the user to then edit whatever files they need to, compile it and have a fully customized document management system.

[MP]

Conceptually it sounds great Tim and I salute your goals and intent, as well as those who will be working with you to collectively bring the vision to fruition. I still maintain that without at least one concrete application to serve as a context for the use and development of the various libraries, be it public or private, it will be impractical to develop said libraries.

To use one of your examples (as I understand it) -- Ruby on Rails was born, distilled and refined from the programming work David Hansson did on the various 37 Signals products (which illuminated the need for a standard Ruby MVC/Model2 framework and provided the initial context for testing / development) -- not the other way round.

[TR]

I hear what you're saying and I agree to a certain extent. However I feel that functional tests and sample applications using the framework should be suffice to ensure that our libraries are functional.

[TR]

For anyone interested I just threw together a little diagram of what the layout of the system might look like. It's a little tough to follow but it gives a general idea of what I had in mind.

View it at:
http://www.theswamp.org/screens/tjr/DocumentManagementFramework.png



Writeup to follow on Monday.

[TR]

I realize I had promised to post a specification for this project several weeks back and have failed to deliver. Perhaps I was getting in a little over my head with this whole idea as it will end up being more work than I think I'm willing to take on at this point in time. Over the past several weeks my head has been wrapped around web.py and several javascript libraries trying to get a better understanding and also banging out some code on a couple of web site ideas I have. While working on these and my day job I just don't see how I could focus on a project like this.

To those that were truly interested I apologize for not pulling through.

Michael, I think I owe you a beer.

[Troy Williams]

I realize I had promised to post a specification for this project several weeks back and have failed to deliver. Perhaps I was getting in a little over my head with this whole idea as it will end up being more work than I think I'm willing to take on at this point in time. Over the past several weeks my head has been wrapped around web.py and several javascript libraries trying to get a better understanding and also banging out some code on a couple of web site ideas I have. While working on these and my day job I just don't see how I could focus on a project like this.

To those that were truly interested I apologize for not pulling through.

Michael, I think I owe you a beer.

Don't worry to much about that Tim. I do that all the time 

What I have found to work for me is to figure out what I want to do and then break it up into much smaller pieces. For example I want to build a simple, flexible image management system that even my parents would want to use (notice I say want and not can). That is a big job and conceptually I have a lot of questions to clear up. For example, it will have a sqlite backend as well as a flexible tagging system. So what I am doing is creating a code vault that will have these features except that it is much smaller and easier for one person to manage. This program will be useful to other people (how many, I can't say  ) which helps to keep the motivation levels up. The key when you are building these smaller programs is to keep in mind the big picture and make sure that you can reuse as much as possible.

When My Code Vault is completed, I will have some of the key concepts for an image management system in place. As well as a fully functional program that is useful, not just a toy.

Cheers and good luck with your endeavors.