Composite Data Warehouse – Part One

A database design trap

For many organizations the database maturity becomes a pivot point when entrepreneurial agile team culture seamlessly morphs in to a waterfall think tank. Loss of flexibility and development freedom creeps seamlessly from the policy, security, design, change, maintenance, support and other standpoints. Team in such a position might have a couple of database development servers that are rarely synchronized with production. After a few months of coding the development and production environments drifts apart so much that it becomes very hard to manage successful releases. The DB Administrators, who are in many cases are overloaded with multitude of administrative tasks, will have no time to merge production changes back in to development, neither would they have time for setting continuous development integration environments.

When database design solidifies enough, one of the primary goals on the team becomes avoiding the risk of impact to existing functionality: the simplest way to avoid impact is by minimizing database changes, and… After few deployment disasters (that everyone has), management will create a set of new deployment policies (with good intentions of course), that quite quickly can result in is a risk avoidance at expense of innovation. The application is aging.

Foundation that embraces innovation

It is a breeze to talk about an outstanding setup here at Fortigent, developers don’t even realize how lucky they are. Code production is blooming, developers cracking releases to production on regular basis. Years go by and there is no glimpse of hesitation to a change, no signs of fear, and no comments about outdated DBs. So, what is it do we have here at Fortigent that fuels an innovation?

At the core of continuous integration, a seamless and almost unnoticed process, brings latest production database and hot “out of the oven” development code together:

• Automated process scripts latest production database on regular intervals and keeps all DDLs and metadata under code name “Base”;
• New or changed database code is stored in the separate location called “Delta”;
• Deltas from many projects are applied to the Base and compiled into a “Latest” DB version;
• The common latest DB version is loaded individually for each project at compile time, either on the build server or on the development PCs
• Numerous unit testing frameworks hit the application on all levels to verify functionality, identify impact, and flag integration deficiencies.

Code from a different teams compiled together with production database changes that flow in to development world as early as possible, giving developers a chance to adapt and produce high quality solutions, while operation can tune any aspect of database.

This is an environment that allows us to embark on the new database project “Composite Data Warehouse” that is paving the path for greater scalability, improved performance and growth.

To be continued…