Will software engineers get to board Noah's Ark?

I recently watched the movie "2012" and that was the question that popped up in my head. For those who hasn't watched the movie yet, I recommend it.

Ok, since I am neither a very influential person in my field of work and I don't have the skills to navigate myself through massive earthquake in a car, nor a boss who happens to own a jet, my chance of being selected or forcing my way into the ark seems small, but what about the others?

In the scenario described in the movie (or the bible, whichever you prefer), there will be a massive destruction of the world and the only "thing" that is meant to survive is Noah's Ark along with whatever it manages to carry onboard. In the bible version, Noah took along all possible animal species (one male and one female) and plant seeds, and whoever who believed in his doom talk. Well, the world was a lot simpler back then. In a world as complicated as today, what else do we need?

Instead of just having Noah's family we may have to have representatives from all countries and races in the world, there are also far more species and books to carry, along with more tools and machines (Solar power I hope) that will speed up the rebuilding process. Of course there will now be books (or maybe ebook as they are far easier to carry) and art. There will also be a number of professionals each with expertise in their own field to either maintain the Ark during the waiting period and to direct the rebuilding process (Luckily the Ark in 2012 is far bigger than the one of Noah's). So what kinds of professionals do we need?

To name a few, we need farmers who can sustain the food supply, doctors to care for the sick, probably pharmacists to go along. Police to enforce order, but not much need for lawyer at this point. A bunch of renewable energy specialists would be good to quickly bring the power grid back up. Along with some telecom workers to bring up the corresponding telecom system. Without any of these there will be no need for IT specialists.

Though I think some people in the field of IT will need to be around in order to recreate the world, but not all field of IT is needed. If we are to look at all the softwares, hardwares that are designed and implemented today, 99.99% of them can easily be let go and we won't ever have to feel bad about it. It may even be a better idea to start from scratch. The rest can be stored in a revision control system, so specialists in that field would be good, to go along with some DBA. Now, to recreate all the bad hardwares and softwares that was let go, we need architects (the good ones of course, but that's not easy to define) who can direct the recreation once the rebuilding process starts. It would be nice to bring some business analysts to help defining the new batch of software so that they will be meeting the demands of the new world.

On the other hand, it is most likely that the general programmers will be waving the ark goodbye in tears as it is much lower cost to bring programming books. I am also not sure if we will need project managers as the space in the ark is precious (There can be an endless debate on that). As for the "luxury" systems that are good to have, how about all the BI, CRM, Sharepoint, Mobile App developers? Though I am a BI/CRM developer myself I really can't see the world to prioritize us.
So for us BI/CRM consultants who believe in the end of the world and would like to join the Ark, time to switch :)

Documentations in a typical project

As mentioned in my previous post, one can learn a lot from creating documentations for a software projects, it challenges the writer to approach and view the software project from different perspective and thus allowing the writer to have a better understanding overall.

The following are some of the common documentations one would encounter in a software project:
Software Architecture Document (SAD):

The software architecture documents (SAD) gives the reader a complete overview of the entire software architecture, you may not know how each individual component works, or what it does, but at least you know what components exist and forms the whole architecture. This is the best way to understand the whole enterprise architecture.

Functional Specification:

The functional specification is often written by a business analyst who has been to a meeting with the customer and note down all the functional requirements by the customer. There are usually a function specification for each individual "boxes" in the SAD with each describing the functionality in full detail. A good guidance of writing one would be thinking in terms of user cases to outline what the customer is trying to achieve and thus arriving to the list of functionality requirements of the boxes.

Technical Specification:

The functional specification is then passed down to the lead developer who have several years of development experience along with some domain knowledge to translate into the technical specification. The finished product would then be the "bible" where the developers implement the solution strictly based on this document. In the document, usually the data objects, the manipulation of such objects and the presentation of the objects would be listed in detail. In a test driven design approach, one can write out the unit tests based on the functional specification as a staring point.

Test Cases:

As the implementation phase is coming to the end and the test phase approaches, the test team (a true luxury) will be getting ready to test the application. Together with the business analyst, lead developer a plan of how to test the application will be made and the test case document will be written. The test cases are usually based on the functional specification and need to cover all the use cases. Then the boundry cases need to be tested to make sure they are handled in the desired manner.

Deployment Guide/User Manual:

Once the implementation is completed and the project is entering the test phase awaiting delivery. It is time to start preparing the deployment guide, and the user manual. Some of the non specified behaviors would be described in this document to guide the user. Deployment guide would be required should the customer ever need to redeploy the system to a new server in the future. In the ideal case, the system works exactly the same as expected and no one will ever need to pull out the user manual. Well, when did the world ever function in the ideal way?

Writing Documentations - Walk in when everyone walks out

In one of the projects that I have worked in, the list of the document requirements were quite long and *un*fortunately I was assigned to do them. On one hand it was boring and tedious, and it doesn't boost my morale when I am almost sure that no one will be reading them. On the other hand, this was a golden opportunity for me to disect a project and analyse it inside out. Here is the document list that was passed down to me and expect a list of filenames (with content of course) in return:

• Software Architecture Document (SAD)
• Functional Specification
• Technical Specification
• Reusability Analysis (of some of the components from previous projects)
• Test Procedure
• Test Cases
• User Manual

It was a long list and I didn't know where to begin. In theory this is a list that needs to be completed by several people, but because the project was shorthanded I volunteered to complete as many and as much as possible to save some time for the others. As I have only written some of the documents above only once in my entire life, I am not about to start teach you how those are written and every company have their own guidelines regarding them. I will just tell you the benefits by actually take some time and complete them in a detail manner. In the next post I will mention of how these documents can give you a better idea of what you do affects the whole picture. In conclusion, documentations are important for leaving behind information for the others but at the same time by writing them it will also challenge you to think throroughly of the design and if it in fact is done correctly and serve the true purpose. So don't run away from the task, instead walk in when everyone walks out.

Documentations - The priceless element in a software project

Documentations, when you have it you don't see the value of it (Price-less), when you don't have it you would be giving anything for it (Priceless).

This is especially true for companies who not just develop the software for their customer, but would also be responsible for maintenance as well. Everything seems to be working fine at delivery, acceptance tests passed, acceptance letters signed and the final payment received. At that moment, no one pays attention to the documentations that come with the delivery. Someone may even be agonized over the wasted hours on them.

Fast forward to 2 years later, the customer wants to add a couple of new features. The lead architect along with a couple of developers have since delivery resigned. No big deal right? They are programmers and can find their way around the code if enough time were given. Well, as soon as the solutions opened in Visual Studio, tears/sweat starts dropping down their cheeks as if they have just opened Pandora's box. There are hundreds of files that hang together in a manner that cannot be understood. To make the matter worse, there aren't even any comments written by the code.

This is probably a scenario we have came across at some point in our career. So why do we continue to ignore the importance of having documentations to go along with a software project and fail to dedicate enough resources to get it done in a comprehensive manner? A common answer I have heard was that the documentations were never read. So because the documentations were believed to be useless, we stop writing them and so the negative spiral goes on.

So for all of you who are designing a solution or doing implementations at the moment. Please take a few moments to document what you are doing. A few moment spent by you today will save a lot of moments for others in the future.

This post will follow by another on what kind of documentations are crucial in a software project and some of the key elements that should be included. Stay tuned.

CozyRoc, CRM's key to the outside world

For CRM development, Microsoft has set a few ground rules of how things should be done. One of them is about the CRM database. Developers cannot access the database directly or else the almighty Microsoft will punish you for all eternity (ok, not as bad as that, but you do lose the microsoft CRM support). This particular rule feels like a handcuff on me everytime I work on an ETL project where CRM is the destination. Now, by setting up this rule, then Microsoft does give you a chance of doing things their way, and that is through their data objects. Users can create the data object classes with their crmsvcutil application after they have downloaded the CRM SDK. After that, it is plain and simple EF4 and LINQ. So with that in mind, our way to do the projects (still with the handcuffs on) are to make a class that abstracts this by having a bunch of InsertXXX(List<XXX> items), UpdateXXX(List<XXX> items), DeleteXXX(List<XXX> items) methods. Ok, it is not the end of the world, except there still are quite a few ways to do this on, code maintenance after project completion is a challenge, integrating a bunch of C# code in the SSIS package doesn't exactly sound like a best practice either. So as I am getting fed up with that, I thought to myself that there must be someone else who are equally frustrated as I am regarding this CRM data integration. Afterall, as I have mentioned in my earlier post, BI and CRM fits very well together, so it is only reasonable that they have some sort of tool somewhere in the market that will make life easier for us. After some research I found a couple of candidate:

• Kingswaysoft
• CozyRoc

I had to run a presentation to my client regarding the pros and cons of these tool, plus the pros and cons for not using any of them, I am sharing my findings with you here. Both Kingswaysoft and CozyRoc offered SSIS tools for CRM data integration. There is a new Data Flow component called CRM Source and CRM Destination (As the classical OLE Db Source/Destination). Like the classical OLE Db, in order to utilize it you will also need to set up a connection manager. So we will have to set up a new CRM connection manager. However, we decided to go with CozyRoc for this project because it also include other SSIS tools that would be useful at the project as well. Otherwise, I consider them rather equal. (Kingswaysoft charges a tiny bit more, but the $100 a year per deployment is consider minimal in most companies) The CozyRoc CRM connection manager set up looks something like this: You can set up the server, type of CRM Deployment, credentials, these are quite similar to other connection managers and was quite easy to set up. Once you have that set up you can use the Dynamics CRM Source and Destination at will: The finishing product includes no more messy C# code and is plain boxes with arrows. My client is happy and it saved me a couple of days of implementation time. Win win situations. To read more about them: CozyRoc: www.CozyRoc.com Kingswaysoft: www.kingswaysoft.com  

BI + CRM , 1 + 1 > 2

As you all have read from my mini bio on the right panel of this blog, I am in love with Business Intelligence. However, for the last few days in Malmö, I am having an affair with CRM. It all started with the current project with my client who wants to implement a CRM solution for their clients. Initially hired to handle the migration of data, I volunteer to handle some of the CRM development tasks to expand my technological arsenal. CRM didn't catch my attention until its most recent version, Dynamics CRM 2011, when the demand exploded during 2011, as a side effect the demand for ETL developer exploded as well. As Jukka Niiranen has pointed out in his blog, there are many exciting features that attracts corporates and companies of all sizes. So why would a BI consultant like me wants to learn CRM? From what I have learned from the CRM course, CRM development itself is not difficult for developers of some programming background, the customization is even easier. The key is not about the technical, but rather the understanding of the user habits and the nature of the business. This is very similar to BI. So if you are a good BI developer with good understanding of the business, you path to be a successful CRM developer is halfed. Besides the ETL for data migration to CRM system, SSAS analysis and SSRS reports will create endless of combinations that makes CRM much more powerful for all business areas. Let's say you have a decent CRM solution for a bike importer (reminds you of Advanture works huh?). It logs all the customer and potential sales leads. Some of the standard chart even tells the CEO the sales breakdown by region. Let's say a CRM developer with BI knowledge comes in, and run a data mining on the CRM data to find out what products sells best with product A, B, C etc. and presents the data in a custom SSRS report chart? All of a sudden you have a CRM solution that can do cross selling. How about a CRM system for stock brokers who can monitor and analyze the portfolio of its clients? VaR, Exposures, even some basic analysis analysis of the companies in the portfolio can be done. All of these BI tricks can be done and it is only a matter of hours to include them into the CRM, but it adds a lot more values to the solution. A BI consultant learning CRM is definitely a 1 + 1 > 2, one gets to utilize his business knowledge in another dimension to create more exciting solutions for clients which only imagination (and budget) is the limit. For those who have read my post about the choice of technology, CRM is a technology on the rise, require little overhead to get going and with those extra bonuses of knowing BI. It adds up to be time well spent for me in Malmö, the sunset by the pier was very beautiful and I am enjoying my affair.

*Un*expected traps in an ETL Project

I am writing this post as a supplement to this post by Bjørn Eilertsen. Once we get comfortable with SSIS, we tend to assume we can handle any ETL project with ease. Especially when we have already developed quite a few of the reusable modules that would save us significant amount of time. Thus we would go lower on our estimates in order to win the contract and impress the client. Though it is true that with good experience with SSIS it would lower the development time of an ETL project, but there are some factors that has nothing to do with SSIS but are the main reason to delays and development hours shot off the chart. Besides listing the issues I would also point out a few ways to minimize if not avoid the delay.   Data/Environment access: As a consultant when we comes in to our client and work on a project, their data (or sometimes their client's data) can sometimes be sensitive and may not be available to us until we have completed a long list of task in order to gain access to it. Now, there is no guarantee of how long this may take so start early! This is the first of the many traps in an ETL project so the earlier this is taken care of, the more time there is to deal with the other potential issues.   Data Schema: When the sample data is not immediately available the client may try to appease us by giving us the data schema and claiming that the sample data will strictly follow the schema. It would be nice if it does, but it definitely would avoid a lot of time wasting if there is a plan B should the sample data deviates from the received schema.   Data Quality: In a perfect world, all data is clean. Just like in a perfect society everyone is living happily and there will be no crime. We all know that is impossible. Be prepared to spend some quality time to assure the quality of data. If the SSIS packages have implemented data control and spits out the data rows that contains unexpected data, you are half way done. It may take from a few communication mails with the person responsible of the source data to a month of sitting down to figure out how the data entities are connected. Connection between data entities are tricky especially when they come from different sources and they should not be taken lightly.   Data Model: The destination data model may seem perfectly under control of the SSIS developer and is definitely possible to define prior to the start of the project. However, should there be any changes to this, then there is a risk of facing all the challenges listed above again. The risk of this is high as the data model is usually used to serve as the base of data analysis/mining or an application that utilizes the data. So a glitch in the specification process will likely cause a change in the data model.   Most of these potential traps are difficult to avoid, but it is no point to feel discouraged. I believe it is because of these traps that make the existence of an SSIS developer worthwhile. With careful planning, accurate estimate and a good plan B, the traps would be able overcome with relatively little or no pain.