Getting There from Here

20 Jul

by Brett W Green

How do we get there from here?  That’s a question engineers ask themselves a lot. The sequence of questions that leads us to this important inquiry usually goes like this:

  1. What do we not like about our current solution?
  2. If we could do it all again, what would it look like?
  3. How do we get there from here?

In the process of getting to the answer, we usually find ourselves having to make a lot of compromises.  The “perfect” solution, if such a thing exists, is typically very difficult to achieve either due to cost, risk, hardware limitations, resources, or a host of other impediments.

A practical engineer is a good engineer, however, and the good ones will try to (a) do something that feels at least somewhat “elegant” and/or (b) do something that is along the path towards to so-called perfect solution.

Often, in a time crunch, we are nonetheless forced into doing something we’ve taken to calling “hackalicious”… a completely inelegant change to code that solves the problem and lets you move onto something more important at the time.  Any engineer that says they’ve never hacked something is lying.  I wager that every software system on Earth (and in orbit, for that matter) has at least one piece of code that the engineers would describe as a hack.  Hacks work, or we wouldn’t use them, but they are typically very difficult to work around or change and, if a system has too many, can render the entire system a “blower upper” rather than a “fixer upper”.

We’re constantly in the process of evaluating our software systems here at Fortigent to see what kinds of changes we’d like to make in both the short and long term across all of our various systems both internal and external.

Some of the changes are made to ensure that our systems can scale both because of a growing client base and because the Earth revolves around the Sun.  Every month we add nearly 750,000 transactions into our system .  A system designed for the volume we had 3 years ago may not be adequate today.

Other changes are made to improve performance of the system both for end-users and internal processes. This can involve smaller tactical changes to improve the speed of one part of the system,  simple hardware upgrades, and sometimes larger architectural paradigm shifts.

Finally, we make changes to make it easier for us to make changes! An accumulation of small changes, lack of automated testing and poorly designed architecture can lead to parts of the system that are very resistant to change and therefore cost us more money to maintain and make it slower for us to deliver new features to market.  These types of changes are typically tough decisions to make.  You’re often forgoing some short-term gain that will ultimately pay off in the long term.  A team that never makes this trade will arrive eventually at the “blower upper” phase.  A team that does this too often is probably misaligned with the business.

Here’s a “hit list” of the larger changes we are looking at through the end of this year and into 2013:

Infrastructure Changes:

  • Moving targeted disks onto faster RAID 10 Fibre Channel SAN disk
    • RAID 5 offers data protection and most of our disks use this.
    • RAID 10 maintains this protection while adding “striping” which can improve performance, although this is a complicated subject (http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1.2B0)
    • Some of our peripheral disks are on SATA storage which just doesn’t have adequate I/O throughput.
  • Moving targeted disks to SSD storage
    • SSD compared to traditional “spinning media” hard disks is like iPods vs. Vinyl.  Anyone who thinks SSD isn’t the future still shops for old VHS tapes at the thrift store.  Although prices have fallen, it’s still relatively expensive so this will probably be targeted at critical areas.
  • SQL Server Enterprise Edition Features
    • Partioning will allow us to cut large tables into virtual physical “chunks” which make it far more efficient to access the data along the typical business lines.
    • Compression allows us to squeeze more data onto disk “pages”, trading CPU time for I/O time which, with those spinning vinyl disks, is usually an excellent trade-off.

Architectural Changes

  • Data Warehousing
    • If we can be said to be doing warehousing, it’s somewhat incomplete.
    • Our only true warehouse table stored data at Account, Bucket (aka investment) and Date level.
    • This is not “granular” enough to do things like unrealized/realized gain reports and, since most of our performance reports are portfolio-based, is not aggregated enough to completely satisfy the report needs.
    • Although we’ve made a lot of progress, we’re still doing too much I/O at runtime.
    • Still very much in the design and thinking phase here, but hoping to make some changes in 2013.
  • File Storage redesign
    • Currently, all of our report and proposal files are stored in a large SQL Server database.  This is very difficult to maintain and causes I/O problems with our other database mechanisms
    • We are looking to move all of our file storage off to a file-system based solution by end of Q3.
  • Persisted Positions storage and usage
    • Many of our systems calculate security-based positions ‘on the fly’ by aggregating the transactional changes on that security since its inception.  That is a lot of I/O and a lot of unnecessary work.  We are working on a system for storing and maintaining calculated positions that should improve performance and consistency within the system.  This is targeted for release at the end of Q3.
  • Segmented database servers
    • True scalability comes from being able to add hardware and servers as volume increases.
    • We’d like to be able to segment larger advisors into their own servers, or several medium-sized advisors onto one server, improving overall system performance.
    • This allows us infinite growth and the ability to take on an advisor of any size without worry.
    • It may also be an advantage for closing on potential clients with more strict security needs who want their data housed independently.

We hope these and many other exciting features we’re adding will allow us to grow through 2013 and beyond!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: