Skip to content

A Dark Art

Hai Scelto Me.

Capped in snow, the Alps lay glistening before him, in all their majesty. Ski slopes, vast lakes and even the funiculars were clearly visible. The crew of the early morning flight afforded him this rare treat. A grandstand view from the cockpit.

The brief was meant to be a short assignment. The computer application had, inter alia, hit a brick wall. Trouble-shoot, Fix and Report was the succinct demand.

“Che diavolo sta succedendo?!” And lots more colourful language was to follow at the Level 1 Crisis meeting. A production outage demanded an eviction from the CIO’s office. From then on it contained a ménage of Managers, DBAs and Developers. Much of that first morning was wasted simply waiting for people to arrive. There was much speculation on the cause of the seemingly meaningless error message. Eventually, only the useful remained.

She was standing by his bedside. He awoke to find her there. Dressed in white and statuesque in appearance. He watched her turn and walk away, through the far wall.

His first step in the diagnosis of the problem was to ask “What exactly is the problem?” What is the actual error message? Make sense of it. Verify it. Concentrate on the actual error message because the rest are usually noise, caused by the knock-on effect of the first error. At what precise point does the problem occur? Can it be re-produced on demand? Can you see the line of code that is actually causing the problem? In short, do not speculate! Remember, there may be several different errors, so eliminating the first (or last) error may give rise to the next one.

What is causing it? What has changed? If you do not know then you need to be granted “privileges” – to be able to logon to the Production servers, execute code in de-bug mode, and to invoke Profiler and PerfMon. Having obtained the metrics, then determine what they actually mean! Which ones are important? Which ones can actually be trusted? Actually making sense of such data is a dark art.

What can you actually do about it? What kind of short-term fix (“hack”) are you allowed to make? Can you fix it without re-factoring, by simply scaling-up or possibly scaling out. It is usually cheaper to scale up than to re-write code, because a re-write means time and money. Afterwards, implement a Tactical solution, and then the correct long-term solution. Don’t forget about it. Add it to the backlog. Deal with Technical Debt.

In the foothills of the mountains there lies a medieval city. Destroyed by Attila, rebuilt by the Venetians. Upon his arrival for the week-end, something drew him to the site of Lotto’s altar masterpiece. Benefactors masquerading as saints? The gathering was somewhat amused by the Guide’s interpretation of the master’s message.  Solemnly, the master is saying unto us, Salvation can be obtained through Religion and Politics!

The steep uphill ride in the funicular was brief. Just a short time to reflect on the Missing Panels; and the altarpiece’s original monastic destination. A destination worthy of the complete work.

In the cool of the evening he made his way around the Citta Alta, starting out from the grand Piazza. To the city walls with those panoramic views of the plains, and then back towards the Rocca. He never quite made it to the top of the hill. Something made him stop at the Porta dei Morti. A stranger enlightened him. “By removing the deceased via a secret door, the Devil shouldn’t know she had gone.”

The ringing of the bells drew the crowds into the Basilica. He joined the throng for the morning Service. Difficult to follow, for it was not said in the vernacular. The faithful amidst the sea of the curious. One uplifted, the other in awe of their surroundings, not least the Cupola. Gazing at Fantoni’s wooden carving, he felt her presence most strongly during the Confiteor. Hast thy chosen me?

The Daily Shout was a quick conference call catch-up with all the interested parties. It was usually kept short, to 30 minutes, but much more than a Daily Stand-Up. Daily Objectives and Deliverables. Attended by experts and decision makers. To remove blockers, showstoppers and meddlers. Think Big Picture, then Small Picture.

Having solved the immediate problem, his advice, to those who were prepared to listen, was to look at the bigger picture. A wider audit would ask more questions, such as, “Is the system designed to perform”?  Also, always establish a Baseline and then proactively monitor system performance to that baseline. But he struggled with grasping the other big picture.

The boys choir was bringing the mid-day service to an end. The queues were already forming in the side aisles. At the head of the aisle there was a wooden structure, with two entrances. Two seats separated by an impassable divide. A structure that facilitated a dialogue between two strangers in the presence of another. “Lei ti scelto. Perché lei sapeva te sarebbe d’aiuto. Preghiamo.”

The final two Gatherings were diametrically opposite in nature and purpose. The first was the review of the Audit Report, finally created after endless drafts caused by endless reviews. Most importantly such a report is not a blame game, but should focus on Lessons Learned. Recommendations should be prioritized. The final meeting had a lot less colourful language. The initial brief had been accomplished.

He drove north in the pouring rain. Along the winding road by the vast inland sea. “At the top of the mountain there lies a meadow. In the midst of it there stands a fortress, without guards. Strangers cannot enter but they will know of your coming.” The skies were clearing when he made the ferry crossing.

The shadows were lengthening as he began the final treacherous ascent on foot. Urged on by the final words of the instruction.  “Stare con loro. Pregate con loro. E il vostro lavoro sarà fatto.” Guided by her presence for the last time.

La Maison de chasse du Roi

The approach to the house was from a narrow lane, off the beaten track. Without any fanfare. The casual passer-by would have missed it completely.

It was a long walk along the meandering drive. Past the old stable blocks. Over an old brick bridge and up to the spacious forecourt. The front door opened without fuss or hindrance; and the first sound that was heard were the six chimes of the grandfather clock in the grand Reception. After the usual exchange of pleasantries, some thought was devoted to the choice of room. Much was available, but some were rarely, if ever, occupied. Finally, Number Nine was chosen. A suite, on the uppermost floor of the Tudor house.

The first place of rest was a welcoming guard’s armchair, with a high hooded back, covered in buttoned down leather, and resplendent in a warm shade of burgundy. From deep within it, the outlook was of the grand curving staircase, the high wooden ceiling and the mighty clock. At the top of the staircase was a narrow door, and then another flight of stairs to the uppermost landing. A final long and narrow corridor led to the chosen place of rest from a gathering storm.

The suite comprised a study, bedroom and bathroom, which were linked together by a brightly lit corridor. In the Study stood a captains desk, a royal swivel recliner and a standard lamp with a strawberry coloured shade. Oddly, the chair was lazily revolving to a standstill, coming to rest when it faced the onlooker.  At the firm tug of a cord, the motor started and at the same time a bright light illuminated the bathroom, which was austere but immaculate, with white floor to ceiling tiles and a cream tin bath tub.  In the bedroom was a Queen’s four-poster dressed with satin linen, close to a large sash window with embroidered curtains expertly drawn together.

The bedroom was already lit by a small but exquisitely ornate bedside table lamp, oriental in origin, across which strutted a majestic turquoise peacock. It seemed that the suite was ready and waiting, exuding a warm and somewhat eager welcome, almost as if the arrival had long been desired.

In the warm confines of the suite, there was time enough before supper to reflect upon the day’s challenge. Expediency versus Best Practice! Systems Development can pose questions which involve the complex interaction between technical and human factors. Although providing a sound technical foundation for evolving business requirements is key, consideration must be given to the human resources tasked with delivering the technical solution. The Architect ought to demand the delivery of a technical solution that conforms to Best Practice.  So, purchase the necessary skills to make up any shortfall. Overcoming opposition to new techniques has always been the challenge facing IT project teams.

With the clock chiming eight times, the evening meal was served. Seated in the conservatory at a table by the window, the meal commenced with a decent measure of Chateaux Margaux. The ambience of the room was quietly warm, due in large part to the conversations having a fairly educated air about them. Each table was properly laid, with fine china crockery and Sheffield steel cutlery. Smoked salmon roulade was served with warm toast and a hint of freshly ground pepper. The shoulder of lamb was boneless, perfect in texture from having been slowly roasted since noon. To finish, the crème brulee was warm, with a sharp coating. Supper was a triumph. A meal befitting the tradition of a Hunting Lodge of the King.

The constitutional walk started slowly, taking the route over the footbridge, around the stable block, with a brief sojourn in the rose garden. Gathering speed, and with long strides across the broad lawn, the walk continued with some exertion along the footpath, coming to an abrupt halt by the water. By now, the menacing dark gigantic clouds, borne by strengthening and swirling winds, were beginning to shed their heavy load. From the distant shelter of the boathouse, through the pouring rain, the lights of the conservatory shone brightly from an otherwise unlit house. Peering through the mist, the suite on the uppermost floor of the grand house was barely discernible, for it lay in complete darkness.

Upon the return, and without any effort, the suite was once again a warmly lit place of refuge. A full measure of Malmsey Rich Madeira stood on the red leather desk. A suitable time and place to reflect upon an often experienced purely technical debate, invariably the scene of a vigorous exchange of opinion. Debating on whether to use SSIS or T-SQL is a typical example of the frequent struggle between Expediency and Best Practice.

T-SQL:- Common knowledge from having been tried & tested for over two decades; comprising familiar programming constructs (e.g. IF/ELSE, WHILE); code that can be generic and meta-data driven (using Dynamic SQL); with a degree of parallelism (using a Batch Scheduler).

SSIS:- Relatively new software, with rich functionality (e.g. variety of data sources); Better at row-by-row (in-memory) processing; with a Graphical User Interface (making it easier to understand and de-bug); which lends itself to Parallel processing (if only for the orchestration of numerous Stored Procedures).

The twist is that the two are not mutually exclusive! The database engine is still best at Set manipulation. So, it has become common practice to embed “Best SQL” within SSIS Packages. A good example is the use of the T-SQL MERGE TYPE II  statement to perform “SCD” processing:-

INSERT INTO dbo.DimTarget (columns) SELECT (columns, ToDate = NULL) FROM
……..MERGE dbo.DimTarget as t
……..USING dbo.StgSource as s
……..ON t.BusKeyId = s.BusKeyId
…………….INSERT (columns, ToDate) VALUES (columns,NULL)
 …………….UPDATE SET t.ToDate = GETDATE()
 …………….Output $ACTION ActionOut, columns — for the second insert
 ….) as m  WHERE
 ….m.ActionOut = ‘UPDATE’

Sleep began to creep in, slowly but surely, leading to a gentle doze for a short while, aided in part by the warm glow of the lamp, until the first awakening.

The phone was ringing. The nostalgic ring of bygone days. The voice at the end of the line was at first exclaiming then immediately questioning. The subject was a Face that had appeared for a few moments. A familiar one. It had happened so quickly that the reason for its appearance had been missed. Why, oh Why, had it been shown?!

The laughter in the conservatory had started to die away. The friendly chatter in the kitchen and the footsteps to and from the scullery and pantry continued for a while longer. Tables were laid for the last time in the day. Slowly and almost methodically, the lights in the conservatory, corridors, and finally the reception areas went out. Darkness had crept in all over to slowly occupy the house.

The phone was ringing once more. The voice at the end of the line was ecstatic: “The answers are out there! Just google the right questions. I know why now!” And with that, deep sleep soon set in.

A single chime was sounding in the distance.  The sudden awakening was caused by an urgent, seemingly desperate knocking on the sash window. Sitting stiffly upright, the next sound that could be heard was the motor whirring away in a dark bathroom. Moving to the window, swiftly drawing back the curtains, peering through the rain beating fast against the thick glass, there was nothing to be seen. Then, a brief quiet descended by the window. After returning to bed the knocking came again, growing louder in desperation. To no avail. The window would remain securely fastened for the third and final fruitless attempt. Eventually, sleep returned, again aided in some way by the relief and comfort that was obtained from the warm glow of the lamp.

Three Chimes. The Final Awakening. In a cold room, the sash window was painfully rolling back up by itself; entirely of its own accord. Somehow, the curtains had come away from their tie-backs. In the eye of the storm, a broad shaft of moonlight had outshone the darkness. And so it took a few moments to realise, in the light that there was to discover, that the lamp had been switched off.

A small town in England

They came together in a small town in the north-west.  Their brief was to assemble a Methodology.

They were experienced professionals who had spent a life time in developing information systems. But, much time can be spent on developing a full-blown systems development methodology. And such time costs much money.

If that were not enough, the sponsors had their own ideas on the content of their work. Whilst not completely at odds with the professionals, they did have their own expectations. It was a struggle initially.

He chose to stay in a nearby village. It was a pleasant evening stroll on Queen Anne Street, along the river bank for most of the way, to the Riverside Inn. It was a quiet Monday evening. In the Drawing Room, the grand fireplace was blazing. He stood in the doorway and admired the rich wall-paper, leather sofas and, in particular, the grand piano.

After supper, he retired to the grand piano, where, after some finger-warming exercises, he played Beethoven’s Fur Elise, repeatedly, albeit with his own variations, to himself.

They resumed their lengthy discussions in the morning. Over the years, he had come to encapsulating systems development in the acronym: “CADBUDS”:-

  1. Concept – a business sponsor has a simple concept – the overarching purpose;
  2. Analysis – Much analysis of the sponsor’s requirement, followed by technical analysis;
  3. Design – translating the business requirement into a technical solution – always the most difficult step; so, reduce risk or later re-work by insisting on “Design Reviews”;
  4. Build – coding and testing the solution; insist on Code Reviews.
  5. User Acceptance – ensuring that the solution is what the users actually had in mind
  6. Deployment – deploying the solution into the Production environment
  7. Support – Supporting (maintaining) and enhancing the system

In addition, there may be separate additional stages which deal with providing Detailed Requirements and, most importantly, Reconciliation – intra-system (largely technical) as well as inter-system (to verify Data Quality) etc. In which case, the acronym becomes “CRADBURDS”, especially if one likes rolling the “R”s.

His argument was that whatever piece of work you did, however big or small it might be, whether it was part of Agile or Waterfall – if you simply remembered (and actually implemented)  CRADBURDS then you would not go far wrong.

He returned to the Riverside Inn and resumed his renditions of Fur Elise. This time he was not alone. A child’s face appeared around the door, watching and listening intently. When he left, the child took his place, retrieved his Grade 1 book from within the piano stool, and began to practise.

The sun rose on that small town in England and he enjoyed the walk along the river bank and through the old town, along the cobbled streets to the place of work; where the arguments continued.

The problem with Waterfall is that it does not work in fast-paced environments, such as investment banks. Paralysis by Analysis. Conversely, the problem with Agile is that it descends into JFDI. Someone has a concept and demands a quick turnaround; so the developers proceed to coding and testing without any proper analysis and design. In short, CRADBURDS too often becomes CBDS!

And so it became almost routine – he would practice first, and then the child would follow. It may be that the child was inspired by his finger work. The child was patient too – happy to hear what was being played. When he left and the child started to play, the child was not alone either.

Eventually, they agreed on a set of templates. It is necessary to have a written definition of, for example, what constitutes a “Technical Architecture”. It is important to clearly differentiate between a Conceptual Data Model, a Logical Data Model and a Physical Data Model. These are 3 different artefacts. One is not the same as the other. In the heat of Agile systems development, it is necessary to be able to turn to the templates to resolve any misconceptions – especially if some members of the project team have never had to deliver any of these artefacts before.

The child was making remarkable progress; and getting a bigger audience. When they had first gathered around to hear the little one play, their faces were sombre, almost expressionless, mainly unkempt, and  some with deep reddish-brown marks around their necks. But, their presence did not bother the little one. As the days went by their expressions changed; as the child got better, they became happier.  It was time to sit the Grade 1 exam.

They were all agreed upon the importance of having an Estimating Model. A methodology must include an Estimating model. Especially for large Fixed-Price projects. But, it cannot be a black-box. There must be transparency. It must be possible to drill-down into the detail. How did you arrive at that number of man-days?

They waited for the child in tense anticipation. And eventually, the child came through the door beaming and waving his Grade 1 certificate. They gathered around clapping and slapping the child on the back in adulation. They demanded the child played for them again. For the encore, the child played His favourite piece – Fur Elise.

Alone in the Dining Room, he thought he heard a small commotion across the hallway and the faint but unmistakable strains of his favourite piece of music. He finished his meal, pushed his chair back beneath the table and walked across the hallway and stood in the doorway of the Drawing Room.

He spent his last day tidying up the VISIO diagrams.  There was much to do before he caught the train back to the Big Smoke. As he went about applying the finishing touches to the diagrams (“works of art”, in his opinion), he could not help but reflect upon his time at the Riverside Inn; and the last night, in particular.

He was sure he had heard a commotion in the Drawing Room. But, when he had stood in the doorway and looked in, the room was empty.

The Red File

They made the journey up the motorway together. Mostly, they made casual conversation about the other staff, projects and clients of the consulting division. There was no need to go over the contents of the Red File again.

The management accountant was a dour northerner. The meeting began with reference to the latest correspondence, which left the reader in no doubt about the urgency of the matter. The problem was not that the numbers did not add up. They did, to the nth decimal place. The problem was the staff. The assistant management accountants had made their views crystal clear – they were not going to use the system because it was, in their view, “not fit for purpose”, although their choice of phrase was somewhat more colourful.

The purpose of the system was to enable the staff to prepare annual budgets. The budgeting cycle was a very short one. Senior management required numerous iterations. Serious number crunching was involved. The problem was that somehow the staff’s expectations went beyond just number crunching and accuracy. Someone had either raised their expectations;  or may be nobody had actually managed their expectations.

Together, they (the staff) had christened the system “BASIL”. It is common practice to give a computer system a name, which is invariably an acronym. In this case, the letter “B” stood for Budgeting; the rest was probably some combination of the words “System”, “Information” etc. It did not matter. What did matter was that the staff of the fairer sex intentionally (and rather successfully) pronounced “BASIL” in the same way that Sybil Fawlty did. And, like Sybil, they were not amused either!

In short, each of the assistants anticipated being able to prepare their budgets at the same time as the other, if need be. Further, they required and expected an “immediate” response from the system after making a few adjustments. Seconds and not Minutes.  Thus it was that UAT began and then came to a halt rather abruptly. Thus began much serious correspondence, all of which had been gathered together in an appropriately coloured file, for Alex’s eventual perusal.

Management expected (possibly commanded, in the nicest possible way though) him to resolve the problem quickly.  This gave him two problems. He was already committed to completing an accounting qualification in two-thirds of the allotted time, which involved lengthy evening classes far from home. He also realised BASIL had to be re-architected and this would involve close co-operation with the staff.

At first, driving between the three places, several times in the week, was easy. Then, for a split second, he fell asleep at the wheel on the motorway. That single moment did a great deal to make him aware of his physical limitations. He never forgot the importance of being “fed, watered and rested”; as well as, whenever possible, letting “the train take the strain”.  

And so he came to divide his time adequately between staying in different parts of the country. He looked forward to his days “up north”, when he would stay overnight in a comfortable inn and enjoy English puddings for desert! Occasionally, he joined the staff in sampling the local ales, most of which definitely needed to be sampled locally because “they did not travel well”! Gradually, he became, effectively, one of the staff. It always worked best for him this way anyway – the best results are to be achieved when there is no divide between the consultant and the consulted.

The problem with the design was that the underlying software was essentially multi-read but single-write. In those days, multi-dimensional database technology was just made that way. A single physical database file could only be updated by a single user. (Many years were to pass before such database technologies were designed to comprise separate physical partitions, where each partition could contain different sets of data, at different levels of granularity and with different periodicity.)

So, he proposed an alternative design that would involve separate physical databases (one for each assistant accountant), with an additional database which would contain the “common data”.  This design enabled each accountant to work completely independently of the other. And, because each database was much smaller now, the response times for re-calculations was almost instantaneous. He was getting there.

Re-architecting the existing single multi-dimensional database to comprise several smaller databases was relatively simple. But, there was some scope for error, so he made sure that he involved his new “colleagues” from the outset, that is, in the unit testing. (Ten years later, this kind of collaborative systems development technique between the developer and the end-user came to be known as “Agile”.)  It is not that a written specification is not necessary. It is. The key to success lies in involving end-users whose “Day Job” is running the business in the analysis, design and unit-testing. Because it is their system!

Finally, there was the problem of the consolidated numbers. The specification had been written for the management accountant, not his assistants. From the main man’s perspective, a single database certainly lent itself to producing the consolidated number, after applying complex percentage allocations. But, a single database was not the answer. The final solution simply required a separate database which simply aggregated the base numbers. Some simple controls were all that was needed to ensure that each set of the base data could not be used until it was ready. Everybody’s requirements had now been met.

Initially, his Management were not happy, though. The “re-work” was taking longer than expected and his overnight stays in posh countryside inns (with nice dinners) did not help the consulting division’s monthly “bottom-line” either.  But, the client was completely satisfied and that was all that mattered to him. The sales force made sure that they obtained an excellent Customer Reference – “It was an excellent example of the company’s commitment to its clients!”

In the board rooms, senior management (on both sides) breathed a collective sigh of relief. They had closed the Red File.

Capacity Planning – an Introduction

Provide sufficient but not excessive capacity. No budget is too big anymore!

Serious consideration needs to be given to the probable final size of an application from the outset.

Sizing is always subject to continuous change, but the key issue is that it should be controlled change – to prevent unexpected system failure or degradation.  

1. Identify The Data Requirements

This task should be driven by the Users of the system, not IT.

The users of the system need to broadly define the boundaries. Only the actual users of the system can do so. Only they can include and exclude data. They cannot be expected to get it right first time though.

For example: The initial business requirement is:-

50 Actual & Forecast Measures for the last 10 and next 10 years on a Daily basis, for 200 products, for 50 major Customers in 500 locations – Roughly: 100 Measures x 20 Years x 365 days x 200 products x 50 accounts x 500 outlets = 3,650,000,000,000 data values.  = 3.65 trillion

If you used the default DECIMAL datatype:  Precision=38, which uses 17bytes, then 3.65 trillion cells equates to 3.65 x 17 = 62.05 TB. That is too much data for your average information system, implemented to an average budget!

Get the data boundaries correct. Manage expectations.

For example:- Explore whether it is acceptable to store data monthly (instead of daily):- 3.65 trillion then becomes 0.12 trillion. Next, is it possible to store data using the DECIMAL(19,6) datatype, which uses 9 bytes? This would give: 0.12 x 9 =  1.08 TB. Not every Account would have 500 outlets:- if the average was only 50, then this would give: 100GB. Still a lot of space, but a long way down from 62TB.

Keep narrowing the boundaries. Look at each item of data – what is its actual datatype?

Integer? Date? Every less byte helps.

Because it only gets worse from now on. Seriously!

2. Architecture – the broad components in an Information System

Data needs to be captured from source systems – the original copy should be stored in a Staging area.

Data needs to be validated, cleansed and transformed – the normalised copy should be stored in a separate area aka “Operational Data Store” (ODS).

Data may need to be “simplified” for reporting and analysis – the de-normalised copy should be stored in a separate area aka “Presentation layer”. This “simplification” may just be a collection of database Views, but it may need to be physically stored as well.

Data may need to be stored in a separate OLAP database, so that it can be accessed quickly and to reduce the load on the ODS. This depends on the end-users reporting requirement.

In short, that is (at least) 4 versions of the data! (You could also add another version for the reports.)

3. Infrastructure – the hardware/software required for an Information System

It is “best practice” to have a separate Server for the Relational Database, Integration Services and Analysis Services Database components. 64-bit v 32-bit?

This has a direct impact on licensing costs. Which Servers need to have the Enterprise Edition of SQL Server? Will the Standard Edition suffice?

Which is the most appropriate licensing model? CPU or CAL? The rules change again with SQL Server 2012, when it is done on a CPU-Core basis.

What should be the spec for the CPUs and RAM for each of the above Servers? Make sure that major changes to a Cube are fully stress-tested during UAT to ensure that the correct amount of RAM is made available in Production during the deployment of major changes.

Also, what impact might there be on the network infrastucture? Is there sufficient band-width to cope with regular “Cube Synchronisations”?

4. The Environments

For all of the above – how many “environments” are necessary? Ideally, you need separate:-

“Production”, “UAT”, “Development” and “DR” environments.

So, at least 4 then. That is a lot of Infrastructure! For small organisations, compromises may be necessary.

In summary, Capacity Planning needs to be done from the outset and re-assessed at each major deployment to ensure that the optimum infrastructure is made available. Technical Design techniques, such as Table Compression, exist which may help to reduce the amount of physical capacity actually required.

Sizing does matter. Especially for small to medium-sized companies. Sizing from the outset does make a difference. Make continuous efforts to reduce (“prune”) the size of an application. For example, it can make the difference between buying an expensive SAN or not. And money does matter.

Data, Information, Knowledge, Wisdom

In the context of Information Systems. Nothing more.

Such systems contain Data. Such data is usually stored in tabular format, in relational databases. A query language is needed to extract this data e.g. SQL.

This data is gathered from a variety of places. Actuals from Point Of Sales or Invoices, Budgets from spread-sheets, and Forecasts from even smaller spread-sheets. Some of these source systems can be completely trusted, others not. 

Each piece of Data e.g. Price=2.99 is of no use on its own. It needs to be used in conjunction with something else. But, it needs to be valid. Thoughts which might lead to Discussions which might then lead to Decisions (and hence Actions) are not much use when based on, effectively, rubbish (that is, inaccurate) data. Data that is even a few decimal points out of place.

Consider the effect of storing exchange rates using the MONEY datatype instead of DECIMAL (19,6). 4 decimal places do not provide enough accuracy, even though it uses less space.  “Validation” plays a key part in the information gathering exercise. In other words, the absence of validation processes should be questioned. This includes “in-built validation”, such as, Referential Integrity constraints. 

Whenever you write a piece of code that gets data from somewhere else – consider whether it should be validated in some way. Place “unknown” data in clearly labelled buckets, which can then be correctly cleansed in due course.

Such systems grow to contain Information.  They combine several pieces of data (e.g. volume, price of a product in a location) to generate useful information (e.g. sales, margins, net profit). Taken together, such systems provide “big picture” information i.e. total sales of all products in all locations in each month over the past n years. Data that is analysed by using many dimensions requires a special kind of data storage – a multi-dimensional (aka “OLAP”) database.

Where the data volumes are small, it becomes possible to apply modelling functionality. “What-if” questions can be asked, with an almost immediate response. Modelling adds a lot of value to the data but it is still only information.

But, OLAP databases have their limitations. Once you get lots of dimensions and data it becomes more difficult to make good use of all of it. You need a lot of analysts constantly looking in different parts of your vast database. Databases that are many terabytes in size are very useful, but only with the appropriate tools. 

Data Mining tools could be said to be “knowledgeable“. They can be used in the automated extraction of relevant information from multi-terabyte databases. Book purchasing is a good example.

You buy a book. You are then informed on a regular basis about which similar books to buy. Up to a point. When you stop buying books after a time then you stop getting this unsolicited, though useful, advice. Presumably, if you value this advice then you need to buy another (similar) book. Was this intended?! Probably not.

In simple terms, Wisdom can be defined as making the best use of knowledge. I suppose Data Mining applications may be considered to be making best use of the knowledge available, albeit in a one-dimensional space.

The problem with “making best use knowledge” is that knowledge consists of discrete and disparate parts and it is not always possible to make the connections. The human brain usually makes a good fist of making the connections but its success largely depends on whether it possesses the knowledge in the first place. So, the key (but not sole) issue becomes making the best knowledge available.

Consider your medical history. But, not just your medical history – also your immediate family, your cousins and even the medical history of your ancestors. It is likely that some of the medical problems you encounter as you go through life would have been experienced by the others. You (and your physicians) could save a lot of time, expense and possibly suffering if you had easy access to that information.

Illnesses which are rare are more difficult to diagnose. But even these illnesses will have been researched in the past. Knowledge about them will exist in various medical journals as well as deep in the bowels of the great schools of medicine. Alongside this, almost in a parallel world, the giant pharmaceutical companies will also possess raw data, facts and information about these rare illnesses and the possible life-improving treatment thereof. And, finally, nutrition – that also plays an important part in our good health. Your 5-A-Day can be met by decent portions of veg and good smoothies!

Obviously, the Internet allows you to bring a lot of the above together. Providing you ask the right questions via the input of the correct keywords. The problem is that even if you did do that, some of that information may not be visible to you.

Organisations value and protect their data. And so they should. Their information systems contain valuable information, which has been gathered in vast quantities; then cleansed, transformed and summarised; safely stored; all of which has been achieved at great cost. But which gives them a competitive edge. (Needless to say, information systems on their own are useless without good people, ideas, products etc.) In short, they are unlikely to share it, even if they are “public bodies”. Unless it is in their interests to do so.

Vast “databases” already exist – data warehouses, social networking sites and medical records. Technology is not the constraint (limitation) anymore.  If permitted, these databases can be “searched constructively” with the Search Engines of today.

Wisdom is needed to create a “network” of “useful” databases, which can be “searched”, to make best use of knowledge. For each organisation’s own “selfish” interests. For the greater common good.

The Definitive Agreement

NorthStar entered into a legally binding agreement with Valley Corp. The Definitive Agreement.

Fresh out of college with a First in one of the new engineering disciplines, Alex was keen to face the challenges posed in systems development. It was the mid 80s and the country was awash with confidence and money. Money which needed to be invested. The lure of the North Sea oil fields was irresistible and so NorthStar bought a stake in a concession comprising several part blocks. Once the oil would start to flow, Opex would be low and Abandonment costs a distant irrelevance. 

CapEx was the main problem. An Oil Rig is a big beast. Alex, who had travelled much after graduating, wondered if he would ever get to see one. (BTW – He never did.) Whether NorthStar needed additional capital to survive or not was not clear to Alex. It did not matter. Suffice to say that Valley Corp provided NorthStar with a Facility. The repayment of which largely depended on the price of a barrel of oil, now and in the future; never mind that pesky trio of taxes – CT and PRT and Royalty.

Eoin ran the financial affairs of NorthStar. As appearances went, Eoin reminded Alex of Trigger, especially the Trigger in that blue suit, stood at the bar. Eoin was reputed to be a big fan of Covent Garden (the Opera House, that is), a subject in which Alex had no interest whatsoever at the time.

It was through the NorthStar project that Alex met his mentor. Together, his Mentor, Senior and Alex delivered a computer program which modelled Eoin’s working interpretation of the Definitive Agreement. It was not perfect, there were a few teething problems but soon Eoin signed it off. Job Done! To celebrate, all four went for a celebratory lunch in one of those posh wine bars. This was the occasion that Alex learned that there was “only one way” to have a steak cooked. That only way was Medium-Rare.

Somehow, although in the sales-driven company that Alex worked for it was not hard to deduce how, Valley Corp got wind of the good work that Alex & Co had done for NorthStar. The Board of Valley Corp wanted Marcus to demonstrate unequivocally that their investment in those part blocks was sound. This meant a proper computer modelling program, not a mere VisiCalc spread-sheet, run on a mainframe computer. Surely this could be delivered quickly by the team who already done it. Right?!

Management convinced themselves that Alex was capable of delivering the solution with no help from his Senior and not much more from his Mentor. This was the first but thankfully only mistake. After his initial meeting with Marcus et al.,  Alex reported back to his Mentor informing him that he (Alex) had no idea what Marcus was on about! After examining the “Business Requirement” document (prepared by Rocky), it became clear to Alex’s Mentor that Marcus had a different but equally interesting interpretation of that Damn Agreement. Except that the document was Rocky’s interpretation of Marcus’s own definition of the DA. Not Good.

A few more meetings later, it became clear that Rocky was struggling with the detail. Things were becoming decidedly fraught. Then Alex and his Mentor discovered Gemma. Cool, calm and collected, Gemma also worked for Marcus but, unlike Rocky, Gemma was on the same wavelength as Marcus. Arguably, she was the only person who was. At which point Alex (supported by his Mentor) made the suggestion that saved the project.

It was proposed, discussed and agreed that Marcus and Alex would meet at 07:30 every day for an hour – that’s all Alex was allowed – and that Marcus would take Alex through a Worked Example of the DA. Later in the day, Alex and Gemma would pore over what had been discussed earlier; and Gemma was permitted to seek clarification on the odd detail from Marcus. So that by the next meeting everybody (especially Alex) was on the same page.

And so Alex came to learn the basics of Facilities. He came to understand what was meant by Drawdowns, Repayments (of Capital as well as Interest), Opening and Closing Balances. He could see the importance of fluctuations in the price of a barrel of oil. And, he was somewhat surprised to discover just how much was to be “lost” through those pesky taxes, especially Royalty.

Once the W/E was completed, it took Alex just one more day to complete the computer modelling program. Much to everybody’s amazement.  This assignment taught Alex a few lessons:-

The best Business Analyst is the person whose “Day Job” is actually running the business in some way or another;

A Worked Example is an essential artefact in the process of specifying the Business Requirement; and

A computer program does not take a long time to code and test if one has a thorough Worked Example from the outset.

In particular, it is essential for the Developer to have actually written the W/E himself. In that way, the Developer acquires a sound understanding of the requirement and this helps him to achieve that most difficult of tasks – translating the business requirement into a detailed technical design. 

As soon as the computer program was complete, Marcus asked Alex to run a number of “Scenarios”. This turned out to be a relatively short exercise. It seemed to Alex that Marcus was most pleased with the outcome of the Scenarios. And then, Marcus lost interest in the project altogether. All further communication was via Gemma and then only to “tidy-up” the documentation. Alex was never to see Marcus (or even Eoin) again.

Sometime later, rumour had it, both parties agreed to end the Definitive Agreement. Alex was not told which party had sought the termination. It could have been either NorthStar or Valley Corp. Presumably, one of them had realised that they would be better off without it. Not sure.

Alex was convinced (rightly or wrongly) that Marcus had long since determined that the investment was not sound. He did not need a computer modelling program, especially one which needed to be run on an expensive timesharing mainframe computer. Being able to run lots of different scenarios at the press of a button certainly helped though. And it was definitely needed (using persuasive graphical material) to convince the Board.

Alex was convinced that Marcus had arrived at his conclusion by simply using mental arithmetic supported by some calculations on the back of a fag packet.