Last month, I was delighted to be able to give a presentation to the Enterprise IT Strategy Forum, an event at Heythrop Park in Oxfordshire attended by IT decision makers from many large organisations. My presentation, on how to leverage existing IT assets to support the delivery of digital transformation was well received, and I promised to post a transcript of sorts here.

As a founder of an IT consulting business I spend a lot of time talking to IT leaders and decision makers.  When it comes to digital transformation, I have found that by-and-large everybody is wrestling with the same problems. The problem statement is raised in a myriad of different ways, but fundamentally, it is this: How can an organisation support the rapid change required by the modern business, but still draw on the functionality and data of legacy systems?

The monkeys and the sloths

I think perhaps this can be best illustrated by retelling a conversation I had a few weeks ago with the Chief Architect of a large, well known UK company (the identity of which I will not divulge for reasons that will become clear), about their early forays into the world of digital transformation.

He put it like this:

“The problem I have, Jon, is that the front end of our organisation appears to be run by monkeys. They run around with wrenches breaking stuff and causing havoc. They buy SaaS business software on the internet with their credit cards, hack into whatever data sources they can find to get data, use the wrong data, integrate with bits of string and sticky tape, and throw websites and apps together in a completely uncontrolled way, leaving a trail of destruction in their wake, creating technical debt that IT has to mop up. They have complete disregard for our enterprise architecture and wreak havoc with our strategic plans, but, they make stuff happen. The future of this company relies on them supporting the business needs in this way.”

“On the other hand, corporate systems land is populated by sloths, for whom change occurs on more of a glacial timescale. This is the kind of organisation where it takes weeks to approve a configuration change and three months to provision a server; if you want to deliver a project of any consequence in under six months then you might as well forget it.  By the time these guys have got out of bed in the morning, the monkeys have already whipped out their credit cards, purchased something on the internet and are well on their way to monetising it. Of course, these guys follow strict process and procedure, as they have been taught; they keep the lights on and we’d fall apart without them. After all, they look after the strategic assets of this company – the legacy systems and data.”

“How can I bridge this chasm between these two species?”

It’s time to call “time” on the phrase “Shadow IT”

Let’s have a closer look at this and bring it into real-world terms. In a traditional “us-and-them”, “IT versus the business” set-up, the monkeys and the sloths are never going to see eye-to-eye.  They are different species and may as well live on different planets. They have completely different drivers.

The monkeys are your digital natives, driven by real business problems. Those problems are real, urgent, customer-driven business needs to get things done. Real concerns that they need to address quickly. Real opportunities that they need to jump on before the competition do. Real, real life. They build their stuff – and central systems teams call it Shadow IT – because they need to respond quickly and they are not supported appropriately by the sloths that run those corporate systems.

The sloths, on the other hand, have standards! They are the custodians of valuable assets: company data and functionality. They work with complex systems that have evolved over decades; they – and nobody else –  understand how to coax new functionality out of these beasts whilst keeping the lights on. However, these systems also come with complex and convoluted processes and procedures, and many of them were simply not built with speed and agility in mind. The role of the sloths is to protect these assets and ensure they are maintained so that the business can continue to run on them.

Front-end agility is not matched by legacy tooling

Over the past 15 years, there has been a revolution in front-end development. It is now super-easy to create consumer digital experiences using modern tools. However, that revolution has not occurred back in the enterprise systems landscape.  Granted, cloud applications are transforming the speed with which new corporate systems can be implemented and modified, but in most enterprises today, much of the core resides in legacy systems and that situation isn’t going to change rapidly.

In summary, most organisations today are capabile of building experiences rapidly at the edge, but their central systems cannot support this, either architecturally, or organisationally.  It is such a common challenge in the enterprise that Gartner coined a name for it a couple of years ago: “Bi-Modal” or “Two-Speed IT”.

Breaking the Digital Deadlock

The good news is that there are ways to break this stalemate and the rest of this post will illustrate strategies and approaches that companies are using to solve this problem.

Build a New Platform and Change your Ways of Working

To deliver these experiences at pace, we need to enable rapid and simple access to existing enterprise IT assets – data and functionality – in a way that does not rely on old-school, slow-moving delivery mechanisms and processes.

Furthermore, many organisations find that, regardless of how they attack this problem, they inevitably become bogged down in existing dogma; months to provision a server; endless negotiations over access to key resources; layers and layers of process that make molehills into mountains.

The solution is to build something new – something outside of these constraints; something that can get into those legacy platforms and expose functionality and data in a way that can be used by projects on the edge of the organisation to deliver business value.

There are two very good reasons for building something new; and it’s not just about technology.  Firstly, it needs to be something new so that it can be deployed quickly, using modern techniques, and making use of the latest technology (more on that later in this post). Secondly, and equally importantly, new technology enables new ways of working. This platform needs to be used in a way closer to that run by the monkeys, but maintaining some enterprise thinking and control like the Sloths. It needs to be set free from existing enterprise dogma and enabled to take risks (more about this later, too).

Technical Silver Bullets

There are three technology elements, above all others, that we have seen turbo-charge any digital initiative, as follows:

Middleware

We believe that the single most important piece of software that organisations should invest in to support a digital transformation is a middleware platform. Middleware platforms enable systems to talk to each other. Used cleverly, they can enable your legacy applications to talk in the new world with relatively limited ability to engage in legacy development.

Unfortunately, many organisations have become somewhat allergic to the concept. This is understandable as there are many stories of ineffective middleware implementations costing millions and doing nothing but complicate the IT landscape. However, this reaction, whilst understandable, is increasingly misinformed. Firstly, most of these middleware initiatives have failed because the organisation failed to address the operating model consequences of implementing the new approach to integration. Secondly, middleware platforms (or, more accurately, the suite of software tools that they have become), have grown up rapidly in the past five years. It’s no longer simply about passing data between enterprise applications and, with a bit of luck, doing this in a standardised and reusable way; now, middleware platforms come in many different shapes and sizes and offer services that revolutionise the way data is exchanced across your business.

The middleware marketplace is complex and evolving – but the core elements that are relevant to your digital platform needs are:

• API Management
  • Expose APIs for other systems to consume. This is the primary model for integration today, because it is effective and simple.
  • API Management platforms allow you to keep an inventory of your APIs, publish them for outside usage; control access to them; control throughput; control versioning etc.
• Modern ESB to enable rapid exposure of existing assets
  • Connectivity, via multiple protocols including legacy adapters
  • Transformation, translation and mapping of data
  • Queuing capability to deal with real-world offline situations
  • Monitoring to enable rapid response to issues
  • Flexible, often visual, development environments (IDEs) to promote rapid implementation of new integration flows

By adding these technologies to your estate, you give your organisation the capability not only to connect your internal systems together quickly and easily, but also to expose functionality and data – with suitable access control – to the world – customers; suppliers; intermediaries etc.

Data Virtualisation

My second must-have technology is Data Virtualisation. Data virtualisation tools enable you to reach into your legacy systems, pull out the data and combine it with new data sources, surfacing up via APIs that can be consumed directly by your front-end applications.  Add to that staging and cacheing capabilities, which when used wisely can protect back-end systems from unplanned surges in demand, and this technology enbles teams to rapidly expose enterprise data – in a controlled way – for new use cases.

We have seen organisations use this technology to great effect; the virtual data repositories that they are creating – perhaps curating is a better word – often come from a wide variety of legacy systems and other data sources.  Exposing this data to front-end applications enables far more rapid delivery using data that would otherwise take months to get at through existing legacy system development.

Cloud, Infrastructure-as-Code and Push-Button Deployment

Gaining access to your enterprise assets and supporting rapid development against those assets is not of use if the infrastructure you use cannot be provisioned rapidly and scaled to support peaks and growth in demand. Businesses that have gone down this road find very rapidly that the demand for the services they publish mushrooms. The bottom line is that when you publish services to the big, bad world, you lose a certain degree of control over the usage profile of that data.

To deliver ultimate scalability and rapid deployment, companies must turn to cloud infrastructure and scripted deployments. Effective deployment to the cloud enables almost instant scaling to meet demand, and the latest incarnation of middleware platforms can be built to scale dynamically to support the load.  Likewise, supporting the scaling out development capability using scripted deployments enable further increases in speed of delivery. I remember many times the bottleneck to delivery being lack of a suitable testing environment. With modern techniques, these should be problems of the past.

The Recipe for Success Must Extend Beyond the Technology

Operting Model Change

It is well agreed that to do “Digital” you need to fundamentally change the way your organisation works. What that means for any one particular organisation will differ, of course – your mileage may vary – but one thing I can say for certain is that this is absolutely true for the IT department.

The IT department must move from being a centralised gatekeeper of all technology, and start to embrace technical decisions at the edge, close to the business needs.  You must consider how much of those technology decisions get implemented at the edge. IT must move to take an enabling role, guiding and supporting local decisions, giving the wider business the remit to build the systems that work for its needs, rather than simply conforming to IT’s view of what that should be. After all, who knows the business requirements better – the manager out in the field, or the Unix admin in the data centre?

Embrace Agile

In the real world, requirements change all the time. In the cut-and-thrust of the front line, your staff should be and will be focused on what’s right for your customers, not constrained by what IT can deliver. Yes, of course, down that road be demons – that’s where you end up in Shadow IT land, you say.  But no. The answer here is to make your response to change more flexible.

It is very challenging to make a wholesale change in the way all your IT implementation teams work. There will be huge resistance and ineffective implementation of agile practices can turn any organisation into anarchy. However, if you pair this approach with your digital platform implementation – i.e. support the integration and data implementation work with agile methodologies, you can turn this layer in your architecture into the gears that smooth out the dynamic demands of the front office and hide the glacial timescales of the back office.

Consider DevOps

In the context of this article, DevOps is all about smoothing out the route to the deployment of new functionality to your live environments.  It’s about moving from old-style monthly, quarterly or even half-yearly, large, complex deployments to a continuous stream of small updates to deliver functionality frequently and often.

It is possible to use old-style mechanisms to deliver frequent updates, but there comes a point when the overheads involved are prohibitive, and the DevOps approach is the only viable one to take. As with Agile, DevOps takes time and care to implement, and it is best to start off with a small, focused team and build out from there.

A Sprinkling of SOA

Service Oriented Architecture (SOA) seems to have received a particularly bad press in the past few years. This is understandable, as pretty much everyone can relate a story of how a big SOA project failed to deliver anything. The problem is, that we are in danger of throwing the proverbial baby out with the bathwater when it comes to SOA. SOA has a bad name for two reasons: (1) every vendor imaginable jumped on the bandwagon and it became a buzzword with no substance behind it, and (2) many architecture departments embarked upon boil-the-ocean projects to architect their entire landscapes. Unsurprisingly, these projects took years (and lots of cash) to deliver and were no longer relevant to their businesses when they were delivered – a complete waste of time and money.

However, a little bit of SOA thinking is hugely beneficial in a world where you have federated responsibility for design and delivery of services. A SOA “map” of the landscape, showing what services are needed where, what systems they relate to and who is delivering them, is massively useful as it guides the activities of the design and implementation teams, avoiding costly overlaps and awkward gaps in the service landscapes.

The key to getting this right, however, is to do “just enough” of everything so that the detailed design work can be done by others. This takes a special sort of architect – one that is confident and happy to guide the “doers” without feeling that they need to own and control the entire landscape. Without this guidance, your API landscape will mushroom into a beast that is hard to work with and very inefficient.

Legalised Skunkworks

Delivery of a digital capability such as we are describing here should be treated as an innovation project and incubated rather than tightly managed. You should put your best people on it and then allow them the freedom to experiment and build the capability free from the preconceptions and dogma of the past. The best way to do this is to give them guiding principles but few process constraints. Try to avoid the temptation to populate this team with stars from the old world – no matter how good they are, they will bring with them preconceptions and constraints that would hinder the new team and impact its success. This will be an unpopular decision with some, but it in time, as the new team establishes and expands, more of your existing team members can be brought into the fold.

You will find that the fresh unconstrainted thinking of this team will allow them to focus on the task of building a capability that can meet the new demands of your digital business by finding solutions that existing teams would not imagine.

Start small; one sure way to fail is to announce to the world that from now on everything will be delivered in a new, digital way. Pick some discrete, business-led projects with clearly-defined benefit cases and use the new team to deliver them quickly and effectively.  You will soon find others knocking at your door for a piece of the action and then you can scale appropriately.

Putting it all together

Delivering digital is not easy. Even teams that have a strong vision and strategy still struggle with execution, especially when it comes to delivering experiences that are backed by existing legacy functionality. The strategies and approaches outlined above present a way to drive a digital future making use, where necessary, of existing IT assets. There will be huge challenges to overcome, you cannot completely rewrite your processes / rewire your business overnight, but if you take it a step at a time, it will be surprising how quickly you can develop the nucleus of a new, digital-first way of working.  Over time, you can move your entire IT organisation into this new world, but start small and build from there.

Collectively the team at Wheeve have decades of experience in digital transformation projects that have integration at the centre. If you are now thinking about how to kick-start this process, contact us today for an initial discussion.