Welcome to the Jungle (Redux)

Previous: Grails, is it really worth it?

I think there is a general misunderstanding of what kind of value can give Grails to a development shop. It seems that there are people out there that believe Grails, RoR and other breeds mean the end of Software Architectures in the web layer. The reasoning is: if the workload to develop a web application is shorter probably it means we need less skilled developers to code our applications.

Wrong.

This is a bizarre way of thinking from my point of view. I have heard of this before. Lowering the workload of development does not mean poorer skills in your development team. The workload is lower because these frameworks follow the Don't Repeat Yourself (DRY) principle. This means that all developers (seniors and juniors) can deliver faster because they don't have to perform repetitive tasks. That's all. But developers still have to face problems, and have to take decisions about their design to fix them. And experienced developers can identify better solutions.

Back to the original question: Do you need a software architect in a project with Grails? Yes, you do. But may be you don't need a full time architect. I have identified several scenarios:

• Cowboy developer: Grails is a great framework if you are cowboy developer. The framework can enhance your productivity if you are a freelance developer working alone. If you don't need to work with colleagues it means you probably don't need to integrate with third parties (services, legacy databases, existing code). In this case, the cowboy developer can play the role of the architect, which makes all the sense. But, I'm not sure if Grails can be successful in this scenario: Ruby on Rails can fit better from my point of view (cheaper hosting, more accepted...).
• Development shop: Your customers accept Grails as a valid option (this is a prerequisite, don't' forget to confirm first if they are happy with Groovy on Grails. Running inside a JVM and an application server is not enough!). You have a trained team of Grails developers. You probably have requirements regarding integration with third parties. In this scenario you need an architect (or a senior developer that can take the role) to take decisions about how to do the things and draft a design. Due to the fact that Grails is a very constrained development environment, you can save time of your architects. They will focus on giving the best solution, but they don't have to care about small details that Grails takes care of (remember, convention over configuration).
• IT Departments: Congratulations, you have persuaded your management about the benefits of Grails. You have now a team in your staff of Grails developers plus some contractors. Normally, development teams inside the companies focus on the core business.The projects are evolution of others, there is a lot of service integration and maintenance. So your architect(s) will focus on smooth integrations with existing legacy code and services. In this scenario a strong architect is a must because you need somebody with good skills to design the bridge between your existing legacy systems and your shiny new Grails applications. This architect has to be a Grails believer: otherwise he will lose the faith when the firsts unexpected problems arrive, something common when starting with new frameworks and technologies (do you remember your first experiences with Struts, Maven or Hibernate ?)

An architect can save time with Grails because it's a constrained development environment and web developers can be very productive, but forget about saving time in the service layer if you have to integrate with a legacy middleware system, a complex database or web services. The best scenario for the service layer can be compared to a classic development using Spring and Hibernate. Still, I have some doubts about how is the development process and the architecture of a Grails application using a rich middle layer with EJBs and Web Services. I feel like a void under my feet when there is no database and the domain classes are not mapped to this database.

Sometimes you only need to wait to understand why things happen. When Steve Jobs said 'Java is not worth building in. Nobody uses Java anymore. It's this big heavyweight ball and chain', we were surprised and disappointed because like me a lot of people think that there is still room for Java in the mobile platforms. Of course we have J2ME, but we can do very few things with it. What we need is a full J2SE stack for mobiles. And the iPhone could do it... if Mr. Jobs wants. But he does not want because Java has become the flag of Google to do developers to follow them. He chose to keep Java out of the iPhone to keep a strict control of the platform, to keep Google out of his business.
I have just reviewed what Google Android is, it's architecture and the SDK and I think it's much more than a smart mobile platform: it's a philosophy. Think of what SWT is to the Java User Interfaces and apply it to the Android Architecture. The access to the physical layer has been developed in C/C++ on top the Linux OS, and on top of it there is a pseudo-Java Virtual Machine called Dalvik virtual machine. As they say, "Dalvik has been written so that a device can run multiple VMs efficiently. The Dalvik VM executes files in the Dalvik Executable (.dex) format which is optimized for minimal memory footprint. The VM is register-based, and runs classes compiled by a Java language compiler that have been transformed into the .dex format by the included "dx" tool."
J2ME is very similar in the approach, but what looks new is the tight integration with the OS and the use of Java 5 extensions of the language. J2ME is almost JDK 1.0, but if you read the source code of the samples, you can recognize some coding practices of Java 5. I still don't know if the Dalvik VM can do reflection or annotations, I will need some time to test it.
The access to the physical elements of the mobile device are performed thanks to the set of android.* packages. So, it will be a task of the device manufacturer to migrate the C/C++ libraries layer for their devices. I have tried to find some information about how to certificate that the libraries has been migrated correctly. This was one of the key elements for the success of J2ME on the mobile phones, and should be done the same way.
The Java applications are what they call 'Third party applications'. It's not possible yet to develop or modify the underlying Linux OS and the C/C++ libraries. Again, Java runs in a sandbox -sounds familiar, eh?-.
I will continue playing with it. Just some random thoughts, will Sun Microsystems get some money for the licensing of Dalvik VM? How much? Will J2ME be replaced by the Dalvik VM?

Werner Vogels -the Amazon CTO- has just announced in his blog the availability of the S3 storage for Amazon European servers. This is good news because it means that Amazon thinks globally. The customer experience can improve if the latency is very low because of the fastest load of pages and static content.

Last years have been a wonderful time in Amplía Soluciones S.L., but nothing is forever and I think it's time for a change. I will continue as a partner of the company but I will look for new challenges and adventures out there. You can read my public linkedin profile if you want to know a bit about my skills and experience.

There was a time when DBAs dictate how developers should use Their databases. It was early and mid-nineties and Their Word was The Truth. Those poor guys building client-server applications had to bow down before Him/Her and implement the Business Logic inside The Database Manager. Database hardware was expensive, but His/Her Highness could size the system easily because the number of clients connected was predictable. And that's what the IT Manager wanted; predictable figures.

Late nineties came and Web Application started to rule. Suddenly, the number of clients were unpredictable, the behaviour of the applications was radically different and for the first time in years, the DBMS was the bottleneck and The Master of the Data did not realize on time. So He/She asked to the IT Manager for better and bigger hardware.

IT Manager: Will it fix the performance problems?
DBA: We don't know, it depends on the number of clients on peak, the number of users in the critical path, the TV ads campaigns with the URL of the company...
IT Manager: So the bottleneck is the database?
DBA: Yes, the Web servers are almost idle...
IT Manager: Idle? My god! Can they do anything to relief the database?
DBA: Well, we can try move some business logic out to the application layer. If those lazy and hairy web developers could write good SQL...

And then a new trend started: move out of the database as much business logic as possible. The web developers started to code the SQL inside the application to relief the database.

The DBAs were upset, because these web developers coded really poor SQL and they don't even know about triggers, functions or views that could reduce the size or complexity of some of the statements. And they had to get involved in development almost like a Quality Assurance Team, rejecting poor SQL or aggressive tasks against the RDBMS. That was really disgusting!

By nature Good Developers are lazy. Writing SQL was like a pain in the ass, and wrapping the results in object models took a lot of time. The laziest of them all started to write little applications to create automatically the SQL and the code to return the results as object models. These little tools became in Object-Relational Mapping libraries. The Technical Leads explained to the IT Managers how they could save time using them, and the IT Manager was happy because he could show to the CFO and CEO that they were doing something to make that lazy web developers productive.

Then the nightmare of DBA started.

The DBA realized that the number of queries to the databases were increasing, and the complexity of the query was lower. A lot of simple queries... What's going on here? Sounds like a new intern writing crappy code...

DBA: Hi, is there a new intern in the team writing crappy SQL?
Development Lead: No, we don't have new people. What is going on?
DBA: Then somebody of your team is writing really poor code. I can see tons of queries as simple as a line of SQL!
Development Lead: Ah yes! That's the new ORM library we are using!
DBA: Well, that library is rubbish. It generates tons of shitty SQL. Ask the reseller to give your money back ;-)
Development Lead: No, it's opensource and free. And I see a lot of advantages using it. I think we should discuss it with the IT Manager.

IT Manager: Why this tool writes crappy SQL?
Development Lead: It does not write poor SQL. It creates simple queries, that's all. We can configure it to create more complex SQL, but sometimes the number of objects explodes and the application run out of memory.
DBAs: Why don't you use the already created Views?
Dev Lead: We cannot map Views to objects.
DBAs: So you are not going to use views anymore?
Dev Lead: If we can avoid them, yes.
DBAs: No views!? How am I going to optimize your complex queries?
Dev Lead: Well, we are not going to write complex SQL anymore. The ORM will do.
DBAs: And what about triggers and stored procedures?
Dev Lead: Out. Triggers and ORM does not match very well because it's hard to keep under control the changes performed in the DBMS. And Stored Procedures sucks, we have Java.
IT Manager: So, if there is no views, triggers and stored procedures, the DBAs can set your focus on keeping the system healthy and optimized, but not coding processes. Right?
DBAs: Errr... that's not exactly right...
IT Manager: And we can also transfer all these development tasks to the development team. Right?
DBAs: Correct, but...
IT Manager: And if the DBMS has no heavy processes inside, we can save some bucks in hardware, isn't?
DBAs: Probably yes, but if I don't track what is going on in the database system the system can die!
IT Manager: Of course! And that's what you are suppossed to do from now on. Help the development team to optimize how the ORM tools and libraries performs before going live.

Then the DBA role changed and became a slave of the ORM tools.

She could not recommend any more how to code the SQL code because it was an automatic process. And she saw how there was no gain in the performance because of the use of these ORM tools, even worse, she saw how the tuning of the database now was radically different due to the different characteristics of the SQL code. He thought that may be these new problems in the performance could make IT Manager to roll back to the old way, but it never happened. Actually, the IT Manager asked if it was necessary to have that very expensive Enterprise licenses if triggers, views and stored procedures were declining. Moreover, the application developers were implementing some smart caching strategies that really reduce the overhead in the database, saving millions to the company in hardware and licenses.

IT Manager: Should we try an opensource database?
DBA: Damn Gaving King...

Probably the only common thing among all developers I have worked with is the hate to bug-fixing. No matter if bugs belong to them or not. As a developer and as a manager I have quite contradictory feelings.

As a developer there was nothing more irritating than the hateful Bugzilla emails. When I was working as Technical Lead of a development team in Madrid with a QA team in the West Coast of USA they usually started to work at our 18:00 and we usually left at 19:00 (Madrid is GMT+2 summer time). So we counted the number of bugs per hour submitted in Bugzilla and extrapolated to their 8 working hours to get the number of new bugs in the system. We left the office betting on the number of new bugs for the next day. We felt the QA team was our enemy and the relationship between both teams was difficult. We felt like being in 'Groundhog day' movie! We were listening constantly how bug-fixing is important for the customer, how new features were delayed because stupid bugs and so on...

As a manager I knew that bug-fixing was the most irritating part of development for developers, but you cannot avoid it. Bug fixing is a must. I'm very happy when I found new bugs in our products or projects and I tried to make developers to understand that it's great to find bugs before our customers find them -wrong!-. This is true for managers, if you find bugs in advance then the management team does not need to loose the face with the client due to problems with the quality of the product. But it is not true for developers, and I will try to explain why.

• Good developers are creators. They feel their work as part of them. Most of the programmers are people very motivated because they like what they do. As a creator their work is his/her creation and telling them 'your creation has a defect' it's a bit like saying 'your kid is ugly'. We react like feeling attacked by the management, the customer or QA team.
• Programmers don't work like Managers do. Programmers need to find 'the flow'. We need time to get the focus on the problem, and 'the flow' must be durable. Managers work in 'constantly interrupted' mode. So emails reporting defects every X minutes, phone calls or 'mini-meetings' really disturb. Developers mentally link bugs to interruptions.
• Bug-fixing is normally underestimated in the project plans.They normally include the raw time to fix the defects, but not the time taken due to the 'context change' of the developer: from new feature mode to bug-fixing mode and back to new feature mode. These means that developers link again bug fixing to overtime, because it delays the development of the new features with due date.
• Programmers hate to hear from Management: 'It's great to find bugs before the customer finds them'. Why? Because we are not stupid. We know it. It's great to finds bug before the customer finds them and report to you, Bastard Manager. I will have to fix the bug anyway, no matter if the QA team or the Customer finds them. I have no tangible gain for fixing the bug before the delivery. But the Evil Manager does, he won't have to justify bugs to the customer, he will live better.

The summary is: Developers hate to fix bugs because they feel like being insulted (your kid is ugly), annoyed (emails, phone calls, minimeetings), poorly managed (overtime), cheated (finding bugs in advance is great!).

What can we do -as a Manager- to minimize the impact of bug-fixing in your team?:

• Developers will feel attacked every time a bug is found. Remove all adjectives when referring to bugs. Example: This is a stupid bug, this is a trivial bug... Just use the adjectives to categorize them. Use the terms used by your tools.
• Never ever tell a developer how to fix a bug, except if he/she asks for it. Example: This is just an 'if...then...else'... 'Use a try...catch...'.
  
• Do not allow customers to call or send emails directly to the developers. Customer does not know how to communicate the defects. The defects reported by the customer must be handled by the Managers.
• Set up your bug-fixing reporting tools to send the bugs once everyday or better, once per iteration. Only categorized and approved bugs must reach the developers. Avoid unnecessary stress.
  
• Don't forget to size your projects with enough time for bug-fixing.
  
• Dedicate one or two days per iteration to 'bug-fixing days'. Stop the development of new features and get the commitment of the team to work hard that couple of days to 'clean the bug list'. If the environment of your organization allows it, bring drinks, food and have some beers afterwards. Give the same relevance to the bug-fixing days than delivery days.
• It's great to find bugs before customer finds them, that's true. But don't use the excuse of the customer anymore, or at least not so often. For programmers customers are irrelevant, they/we are not customer oriented. Change to 'We will find bugs for sure, and every time we find a bug we should congratulate because the QA team is doing an excellent job. If we don't find bugs then we are not doing good QA'.
  

What do you think? What do you recommend to help developers to fix bugs?

And I must be the biggest sissy of all. There is a lot of buzz about designing your databases thinking in data normalized or denormalized. Now that everybody is building the next Google or YouTube the database design has to support trillions of transactions per second and millions of terabytes, of course running on a multidimensional computing grid of zillions of nodes of databases written in Erlang... <ironic/>
Data normalization is common sense because helps you to avoid the Data Inconsistency Hell when data is repeated across different tables. Only a small fraction of us will have the chance to work in an environment with more than a few thousands concurrent users. And in such an environment the running cost of a standard database architecture is lower than the cost of restricting how your Software Architects must design and your programmers must code the applications. That's what is all about, economics. It's cheaper to have a fully normalized design with no chance of repeating data across different tables and scale the hardware when you need it. If you are not lucky you will probably will have to denormalize part of your datamodels due to performance reasons. This is very common and does not mean to design your core database model denormalized. You can denormalize the database model and create it in another schema, running some batch process to populate it when the impact on the users is lower. This is a typical solution for reporting tools.
But, if you are the lucky (or unlucky, who knows...) enough to work in environments like YouTube or Google with zillions of users then it make sense to use denormalized database models. The reason is again economics: It's cheaper to restrict how your Software Architects must design and your programmers must code the applications than scaling a non-standard database architecture (being a non-standard database architecture a massive cluster of database servers keeping data consistent among them). And maybe it's not possible to scale databases without strong constraints in the application development.
I think this is a simple axiom coming from my experience, the more exigent demand of database resources, the more restrictions on the application development. As a server side developer, caching techniques and in-memory smart data hierarchies (Web Sessions or Stateful EJBs) are the first step to improve the performance of an application. Very soon the Architects start to design keeping in mind that someday they will have to include caching or cluster data replication, for example. I know this is not a very Agile approach, but depends on the Architects' experience to provide a good design thinking in performance or not.
Finally, design fully normalized and move to a denormalized model only when you need it (this is a Agile approach).

Marenostrum is installed inside Barcelona Supercomputing Center. The building used to be an old church! The pictures here.

I have just read the news in Slashdot (Yes, I still read Slashdot) and I found this very interesting article about a new technique to detect GPL violations in proprietary obfuscated code. The technique is called Birthmarking and basically it 'observes short sequences of method calls received by individual objects from the Java Platform Standard API, which is part of the Java Runtime Environment. By aggregating sets of short call sequences the otherwise overwhelming volume of trace data becomes manageable.'

In other words, every piece of java code (and a library is not an exception) moves at bytecode level the information coming and going from the stack following a pattern. So, they identify and classify the patterns of some key objects of a specific GPL piece of code and then they check how the proprietary code uses the stack. If it matches... gotcha!

But I will tell you something... I have seen this concept implemented 15 years ago!!! Not for Java, obviously. Where? At the Faculty of Computer Science of the Polytechnic University of Madrid. There was two terms called Computers Architecture and Computers Fundamentals. We had practices to learn how to code in assembler and microcode. It seems that students copied practices from each other, but students where so smart that they just copied piece of code of other students, or renamed the variables (or labels, it was low level programing). But the core functionality was copied intact. So, the department in charge developed a program called 'El Corrector'. This program checked the input and outputs of each practice automatically (just like today we do with automatic unit testing) and they checked how the program move data to and from the stack. All the practices were compared each other (the patterns were the other students). If they detected similarities in the behaviour of the applications with the stack, they examined the source code and then visually tried to identify a possible copy.
A friend of mine was a bit desperate with the practice, so he asked another friend to have a look to his practice 'to get some inspiration'. He gave him his source code, and he copied parts of it. Both of them were caught by 'El Corrector', and both of them had to repeat the term.

So the ideas behind this paper are not so new, but they are applying it for a real world problem.

The day Steve Jobs said 'Java’s not worth building in. Nobody uses Java anymore. It’s this big heavyweight ball and chain' . Being a Java Zealot as I am, I should find easily a fist full of arguments, but may be he is right.

I have been working in the embedded and mobile world for almost four years in Amplia, and I can tell you that Java is far from being the best option for mobile and embedded development. Even for complex mobile applications I think it's not even an option. There is not a decent implementation of the J2SE for ARM microprocessors, on Linux or Windows CE. There are some implementations out there, but believe me, they are far from the quality standards that Java developers expect. Sun Microsystems realized long time ago that building a J2SE for ARM and embedded OS was hard and expensive and they gave up, abandoning the project.

Of course J2ME it's there. But J2ME was designed for devices with a lot of limitations in term of resources. Devices like mobile phones and embedded hardware. But not for PDAs and the new Smartphones with big screens. Try to build an application with a rich UI with J2ME. It's a nightmare compared to CF.NET for example. J2ME has been very successful for games and simple apps for mobile phones which can gain access to the hardware layer, but nothing else.

But Steve Jobs was not talking about the heavyweight ball and chain of Java because his big memory footprint, slow start up and all that techie stuffs. It's a ball and chain because means a marriage with Sun Microsystems. Steve Jobs would love to have a J2ME virtual machine for the iPhone... if it was for free. But it's not. All devices running J2ME must pay a fee to Sun, no matter if they use it or not. And that's a lot of money. For example, the IBM J9 J2ME implementation for ARM costs $6 per license. Why should Apple pay for it? They have the OS for free -a Symbian license costs between $2.5 and $5-, they have the browser for free -I don't know the license costs of Opera-. So probably Apple can save around $15 per device just using their own dog food. So they said:
1) is it possible to build web applications with HTML and Javascript on Safari? Yes, we just need to allow developers to test easily on Safari. Voila! Safari for Windows!
2) Can we migrate Flash to the iPhone? Yes, and it is easier to port it than a full J2ME stack. Games, Video Streaming and Rich UI apps can be done in Flash too.
3) How many applications today need offline behaviour? Not too much, but less than a few years ago with a rise in the trend. And the device have been designed to be 'always on and connected' to the GSM network and WiFi. The 'offline' behaviour is not so important.

And they decided not to include Java because gave nothing to the business model behind iPhone.

But Mr. Jobs forgot something:
1) Mobile devices (and iPhone too) needs to have access to the hardware to control the low level resources like Bluetooth, GPRS modem, SMS messaging, Wifi connection, Serial ports. A web application cannot access the hardware layer.
2) May be the offline behaviour is not so important, but wireless networks are not as reliable as fixed networks. Offline behaviour from the browser is not mature at all.

So Mr. Jobs and the iPhone needs a piece of software to complete his Masterplan. They need an Embedded Application Server. What? Are you crazy? An application server? In a device? I will explain myself. When we think of an application server, we imagine a J2EE implementation with a plethora of functionality. But this app server is different. Here goes a list of what I think it should implement:
- A small web server to deliver content to the local Safari web browser.
- A scripting language to write server side code. Of course this dynamic content should only be served to the local Safari web browser.
- Massive Multithreading is not a must (it's all local).
- A permanent storage in the device for offline operation.
- Management of the low level resources of the device.
- Some kind of queuing system for asynchronous operations between the device and the remote elements.
- A synchronization service between local storage and remote storage.

This embedded application server should implement a scripting language. Why scripting? Because the logic to implement in a device must be simple by definition. The really complex stuff is the management of the hardware layer, and that's should be implemented in C/C++ and be very restrictive regarding the access of applications to this layer.

There are already some solutions out there in the embedded world but most of them focus on the Synchronization service when the really hard stuff is to control the hardware layer.

Being a Project Manager, Architect, Analyst, Developer or anybody involved in the development of software, you have probably suffered the 95 percent syndrome.
Imagine a software development project where things are going reasonably well: things are on schedule, Quality Assurance says things delivered are OK, the customer feel the progress, people are happy because they don't have to work overtime and they architect and developers can apply the Best Practises. You are very close to the end of the development cycle (if you are working with an iterative and incremental methodology) and confident about the result. It's time to review the overall status of the project, and then you check with the development team what is done and what is pending.
Project Manager: 'Requirement X is implemented'
Architect: 'Yes, it has been developed'
Quality Assurance: 'We have tested it and it has some bugs'
PM: 'How many bugs?'
QA: 'About 50'.
PM: '50???'
Architect: 'Yes, but they are trivial.'
PM: 'OK, great. I need an estimation for these bugs'.
Architect: 'OK. We have first to finish the pending tasks.'

And iterating over each requirement, we find out that most of the requirements are not fully implemented. Your feelings of confidence on the schedule are starting to fade away...
PM: 'What about the status of the Integration Environment?'
Architect: 'Well, we are having some trouble putting together all the stuff developed by the team'.
PM: 'What can of problems?'
Architect: 'Stupid things like different naming conventions in the URLs and connection strings, corrupt data in the database, etc...'
PM: 'Who is in charge of fixing this things?'
...<Silence>...
PM: 'I know, everybody is busy finishing their tasks... But somebody must work in the Integration environment'.
Architect: 'I can do it...'
PM: 'No, I need you to work with the developers to fix the bugs.'
Architect: 'Me fixing bugs?'
PM: 'Yes, what's wrong?'
Architect: 'Nothing... I will work in the bugs'.
PM: 'I know nobody wants to fix bugs. OK. Can developer X work on the Integration stuff?
Architect: 'He has pending tasks'.
PM: 'I know... We can assign his pending tasks to developer Y.'
Architect: 'But the context change can kill his productivity. Can we contract another developer?'
PM: 'No, budget is very tight and the a contractor will need to be in context anyway?'.

So now your feelings of confidence have fully faded away. Now it's time to calculate the delays and reschedule the project. Things are not going so well, so you decided to meet more often to track the status of the project. After a couple of Status Meetings, you realise that the bugs are being fixed, but new bugs arise. The integration environment does not work properly yet, and now the Business Analysts come with requirement changes from the client. Meanwhile, the client is getting nervous because he cannot see the progress like he saw in the beginning of the project. Developers are now working overtime to fix bugs and complete pending tasks and some of them are starting to complain about your poor management skills in the coffee breaks.

So what have happened here?

1. May be things were not going so well and you did not realise of it.
2. May be these problems are part of software development and you cannot avoid it.
   

I think this is part of the Software Development Game. The 95 percent syndrome comes as result of our education as Software Engineers. At the university they told us that everything has a solution, that there is only a way to build software. Wrong. They did not tell us that the biggest problem of software development is uncertain. Software Engineering has a very big problem as a Science: It has been created by Mathematicians and not by Engineers. For a Mathematician, uncertain is Hell. So they avoided uncertainty. I still can remember how Software Development was seen as a mathematical function,


Where x where the requirements, and Y was the software build. Terrible mistake. The waterfall model was defeated and iterative-incremental models came. And things were better. Now we had the following polynomial representation of software development.

Where each monomial can be an iteration. The good thing of this approach is that it assumes is not possible to build software as an image of the requirements, but it's possible to build software that can get closer and closer to the requirements. Why? let's put both formula together:


The only way to get P(x) = F(x) is when n is infinite. This means that we need infinite iterations to build Software that covers 100% the requirements.

Going back with the mere mortals, it's not strange to have a beautifully crafted Project Plan and a Gantt chart with a workload assigned for each tasks. We can create easily a chart like this:


The Project Manager thinks that the tangent of the function can vary, and with the changes he can control the delays from the schedule, and recalculate the schedule. But he is wrong. He is wrong because he is assuming a function like the one above, a function that assumes that the workload to build a requirement is the same at the beginning of the project and at the end.

The truth is that the function that describes the progress of a development is like this:


And this chart says a lot of things:

1. The workload to build a requirement is dependant of when is being developed. There are a lot of factors (integration tasks, QA team performance, requirement changes...).
   
2. Requirements being developed at the end of the cycle are more "expensive" than requirements at the beginning. The requirement can be simple in itself, but now it's part of a very complex situation.
3. It's almost unavoidable to leave requirements out of the iteration. If you want to keep the schedule, you will have to move requirements to the next iteration.
4. And 5% of the project can cost like 95% of the project. This is the 95 percent syndrome.

What can we do to manage the 95 percent syndrome? I'm afraid this is part of the Software Development game and we can't avoid it. So we will have to live with it. How? My personal recipes:

1. Ask the client to classify the requirements by priority before starting the iteration.
2. Ask the architect to classify the requirements by complexity before starting the iteration.
3. Build highest priority and complexity first.
4. Schedule several 'bug fixing days' in between the iteration. Stop the development progress and work on the bugs these days.
5. Keep in mind that Integration tasks depends directly on the amount of elements to integrate and it's not a linear function. Probably it can a be logarithmic.
6. Keep a buffer in the Project Plan, and do not say a word to anybody.
7. Do not accept changes in the current iteration. If you don't have any choice, then reschedule the project as if you were punishing the client.
   
8. Explain to the client on every meeting that most of the times some requirements are moved to the next iteration. Since most important requirements are built first in the iteration, only low priority requirements are moved.
   
9. Do not reschedule the iteration eternally. Try to close it, even with less of the 95% of the requirements.

These are my thoughts about this special syndrome, but I would like to hear from you. Please post your tips and recipes.

I have read this post: 'Architects: If you can't code, don't apply for senior dev jobs' and I can't wait to share something happened to me not so long time ago.

I was interviewed for a position of Chief of Development in a start up. I passed two interviews with Human Resources and I only had left a face to face interview with the CEO of the company. He was(is) a very well known IT professional with an impressive technical background, but he moved to the dark side: management. Seems he got fists full of dollars from stock ops in the late nine tees from a company located in Mordor (sorry, Redmon).
The face to face interview was strange: first, he asked me random technical questions but never looked at my face, he continued looking at the screen of his laptop... 'This is not a good start', I was thinking... Then he looked at me, took some sheets of paper that handed to me and said,

-'Well, now it's time for the written coding test!'.
What? A written coding test for a management position? I was astonished, and he realized of it. - 'Is there any problem with the coding test?', he asked me.
- 'No, just a bit bizarre, this is a management position, right?'
- 'Yes, but there are a lot of development managers out that have never developed a single line of code, and my managers MUST know to develop', and at the same time he handed me the papers.
- 'I agree, but obviously nowadays as a programmer I cannot score very well compared to a senior developer'.
- 'I know, you will be compared to other managers'. And then he gave me another sheet of paper with a question about threading. I identified it as one of the threading problems to apply for Google.
- 'The problem says to write the solution in C', I said, 'I can't code C directly without googling, can I write it in Java?'.
- 'No, I need it in C'. And continued looking at the screen of this laptop for several minutes.

I tried to write the solution and I must admit that I was not so blocked in my life. Even when I was a student at the university never ever I had that feeling. When my adrenaline faded way of my veins my mind became clearer and I could code the solution, pretty much the right one.

I gave the solution to the CEO and asked me how I felt. 'Bad' I said. 'I guess I'm not interested in this position anymore'.
- 'Why?', he looked surprised.
- 'If you want a Development Manager to test their knowledge about coding, you should explain that in advance'.
We shook our hands and then I left the building. One day later I called the Human Resources responsible of the recruitment and I told her I was not interested in the recruitment process any more.

- 'Why?', she was very surprised.
- 'It's quite unusual to ask a Manager to perform a written coding test without notice, I felt like falling into a trap, I did not feel comfortable'.
- 'But you did right, that's the CEO told me!', she sounded even more confused.
- 'No, I did not react very well; I was shocked and I almost lost my nerves. I could barely control my temper'.
- 'I'm sorry for that; my CEO is quite unusual and I think you could make a good team with him. Please, if you think it twice, let me know and call me back asap'.

I thought it twice, three, four and N times and never called back. I think it's important to know from the early beginning what are your roles and duties. I believe that Managers have to manage, developers have to develop and only under very special circumstances managers have to go deep into the code. Even in small companies and start ups it's important to understand this. Otherwise we can find some bizarre situations, like CEO submitting changes to CVS after an attack of 'I will teach these lazy programmers how real men code!', or Marketing VP changing CSSs or HTML files directly in the FTP server of the corporate website 'because IT guys need days to do it!', or CFO creating email accounts for his girlfriend, friends and family in the corporate MS Exchange Server (what does 'run out of space on server disk' means?). Yes, all of them are true.

One of the most interesting technologies to come are mesh networks. In Amplia we are making some prospects on these technology applied to the M2M (machine to machine) communication. Zigbee fits perfectly for small amount of data, but it's not good when dealing with mobile communications (PDAs, laptops, MP3 players... anything interactive and portable).
The only real ubiquitous networks today are the mobile operators networks. GPRS has an acceptable deployment, closer to 100% of the population in developed countries, but 3G technologies are still far from being everywhere. And probably they will never replace the GSM/GPRS in some areas. 3G networks let the users enjoy of the "mobile experience", providing connectivity anywhere. The problem? The concept of flat fee is out of the scope of the mobile operators, and they charge per use (ok, they give flat fee with a traffic cap). This means that mobile operators cannot attract to the mass of users that use the Internet to share media content, 80% of the Internet traffic in Spain, for example. They only attract business users.
A couple of years ago, Wimax looked like the perfect technology for mobile operators (or alternative operators) to reach out the average Internet user. The deployment was cheaper because of the operation range of the stations and the speed of communication was higher than 802.11G. But we are in 2007 and Wimax has not exploded. Why? There are several reasons:
1) it seems that the deployment of the network is not so cheap,
2) Wimax was sold as a replacement of Wifi, but actually is a replacement for Cable and DSL, very mature technologies.
3) Wimax hardware is not very popular, and it's expensive compared to DSL+Wifi solutions, for example.
4) The spectrum limits the amount of data guaranteed per users. So, depending the density of the area, the size of the cell can be like the size of standard mobile networks.

So basically if you want to deploy a Wimax network in metropolitan areas, you will have to compete with Mobile Operators and Internet Service Providers. The niche where Wimax can beat other technologies are:

1. non very populated flat areas, here Wimax is a winner.
   
2. metropolitan areas with flat-fee of traffic data, for mobile users.
   

I think that cheap mesh networks can really beat Wimax, because of the cost of the deployment. We can read here how the deployment of a single base station can grow over 120000$. With this base station, 1200 users in a path length below 1km could share a 384kbits connection. A path length of more than 1km needs an outdoor terminal to get a good download ratio.
I read in Linux devices that Meraki was selling a Wifi mesh router for less than 50$ (37€). There are also discounts per volume. We have tested mesh networking with OpenWRT and it works, the problem was the cost of the devices. The cheapest one was three times the cost of the Meraki's gadget. If we take the budget of the Wimax base station, we can buy 2000 routers, and we still have 20000$ to pay the installation of the devices (10$ per device... a bit low). If we deploy the network in a grid of 5Km x 4Km, we can cover 20 sq-Km, assuming a distance between nodes of 100meters. That's 4-5 times the area covered of by the Wimax base station!
May be I'm making some naïve assumptions, but the point is: may be Wimax is dead because Mesh Networks are cheaper to deploy. Wimax follows the 'mobile operator' approach: complex centralized infrastructure. Mesh networks follows the opposite approach: simple distributed infrastructure. In the wired world, internet has proven that the distributed approach is the right one. Google is the perfect example of the success based upon the concept of distributed computing. And do you know who is backing up Meraki? Google, of course...
Mobile operators and ISPs, Google is going to take over your business soon!

Building a project for an start-up has nothing to do with building a project for a big company: people in the start-ups normally are betting their pocket money in the project, and this means that delays, unexpected issues and bad performance of the development team have direct impact in the life of the entrepreneurs. This is the reason why I don't want to build software for start-ups: if you have a little of empathy with them, you will carry the heavy load of their personal problems, besides of your own company troubles. Big companies normally can maneuver better when problems arise in the project, and everything is strictly professional.
SOUKI is a beautiful piece of software we built for dutch investors. We decided to go with JEE 5 because our good vibrations in another projects, and because the client wanted to extend the system to integrate different "races" of systems, becoming an 'enabler' of services. One of the requirements of the client was to be as 'Ajaxie' as possible, and so we did. If you check the source of the pages, you can see the references to DWR, the cornerstone of the AJAX behaviour.
As they say in their blog "Souki.com lets you find and explore people & places based on mutual reviews". The site uses a matching engine developed by WCC that makes a 'bidirectional matching': the user offers their likes, and the site offers the places to visit. The likes of the user is based upon their visits and reviews of places, and the scoring of the places is based on the reviews of the users. A pure Web 2.0 concept.
If you are from Netherlands, you can enjoy it now. If you come from US, you will have to wait a bit longer. Anyway, you can browse it here.

Since I started developing enterprise applications professionally (I developed video games when I was a teenager, but that's another story...), I usually design the first version of the data models implemented in the applications developed (may be that's the reason some people say my applications sucks...). I have worked with DBAs and Business Analysts, but it seems that the responsibility of putting in a E/R model the concepts it's mine (well, except when I was the Technical Lead of Xpert Online in Adecco that I had the chance to recruit an excellent professional to perform this task).
Developers ask me if there are some kind of design patterns to follow, and they are very surprised to know that there are a book series that shows how some very well known data structures of the enterprise applications are already designed. These series of books are The Data Model Resource Book, Vol. 1: A Library of Universal Data Models for All Enterprises published by Wiley.

This book helps developers to bypass the 'White Paper syndrome' and start with a design that can cover 30% or 50% of your needs. More than design patterns, the designs are already built valid data models. The CDs come with DDLs files to create the designs in Oracle (and SQL Server? I can't remember). Some of the designs are very easy to understand, but some others are complex and very specific to the target business, so the long explanations that the author dedicates to each model really helps.

A must for any Enterprise Architect.