Translate

December 19, 2016

API Management in Azure using Microsoft's solution - API's and Architecture, match made in heaven

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

The architect is arguably the one most concerned with API's as it is determined by the architecture where the API's reside, what should be API'afied. This has nothing to do with application architecture or integration architecture. The architectural structure of software has no relevance as to when and what to expose through an API. It is the business or domain architect that decides what will be exposed through an API, when and only when the Product Owner wants to reap the benefits of an API to his product. API development may not be controlled by rules, instead it should be governed by principles and guidelines. Although why something should be exposed as an API is a business (architect) call, the how it is exposed is a technical (architect) call.
API's need empowered teams to develop them and more importantly to maintain them. This is not an small thing. API's being extremely hard to develop, they are also extremely strainuous to an organisation because they typically upset the whole structure of the organisation. Even when that organisation is already a product organisation with product teams and product owners.

API's and Architecture

API's are among the hardest things to develop. Why this is, you can read in my previous post here. I think that the role of the architect is extremely important when it comes to API's because they provide access to functionality and information without compromising the underlying systems. And it's not access to just some information, but to information that has meaning on a business level. It's about disclosing your business to the unknown consumer. Well, it's the consumer of your API. So this means that your API's need to fit into the grand scheme of all things IT in your organisation. This is where the architect comes in, because the architect is the city-planner of your IT landscape. See my post on the role of the architect in your organisation, and how that architect may fail miserably here
Basically it would be the architect that has a strong say in what the boundaries of your IT landscape are and therefore where the API's are to be found. Mind that this is the IT component of your business architecture.
Say what? Well, it's not that difficult to understand. Consider your business architecture, in a world where IT is considered in terms of products that the organisation is making money off. (As opposite to seeing IT as an enabler or unique selling point.) So basically the organisation defines its business in terms of products. These products and the relationships between them are a way to see the business architecture. It's a very complex exercise to do this and on top of that, many organisations don't even want to make money perse. GO's and NGO's are examples. Still, consider that you do stuff that is of benefit to others, than what you do and what you'll do it with are your products.
These products are at times realised using IT. Maybe the full product is IT or parts of it are. These IT products and their relationships are the IT components of the business architecture I was referring to.
Every connection between any two products is a prime candidate of becoming an API, or becoming API'afied, if that were a word.
Did you notice that I am referring to the business architecture of the organisation and not to an application architecture that is adopted by an organisation. For example, I'm not referring to the 4-tier architecture by Forrester, or the traditional 3-tier (presentation, business, data) architecture, or an SOA or anything like that. Like I discussed in the API-first design post, these architectures have no relationship to your decision what should and what should not be API's. Your business architecture is. Sometimes, especially in larger organisations, that business architecture is decomposed into multiple domain specific architectures, again on a business level. Here you might argue that the relations between domains are even more prime candidates to become API's. And again, how your application architecture is defined has no effect on that.

The Platform

So this is where the architect has value in the API driven universe. Consider this; API's thrive in a platform economy, who decides whether or not there will be a platform and how the platform is shaped? It's the architect and no one else. And this is because the architect is the only one that views things beyond product boundaries, that's the architect's scope. The Product Owner and his Product Team are only concerned with the product. A true Product Owner will also understand the value of API's, because it means more users of his product and more users means more value, but it's the architect that decides a platform is to be developed, raising the value of API's several levels upwards. You should take a look at 'The Age of the Platform' by Phil Simon to understand the value of a platform and why it is such a key to success for many fast growing organisations. Mind that platforms are not only valuable in relation to external customers but also for internal customers.

So basically, the architect is the key role when it comes to API's. The architect defines the business architectures, models the IT products in it, considers where interfacing between domains and products is done through API's. The architect also decides whether or not to work on a platform and that kind of implies going for API's as well.

Organisational Impact

It should also be the architect to determine whether or not the organisation is fit for developing API's. Like I stated in API-first design, developing API's is one of the hardest things to do in software development. Which not only implies, but makes it rather explicit that your organisation needs to be ready in various areas to be able to develop and maintain API's. Continuous Delivery, and preferably Continuous Deployment needs to be in place. You would like to strive for a DevOps IT organisation and you need a very capable Product Owner. On top of that the developers working on API's need to be able to understand the relevant business concepts and need to have extensive experience in the pitfalls of software development. The architect's responsibility is to inform management, including HR management, the impact of adopting a platform strategy, or an API strategy on the organisation. Since architects need to be able to quantify the impact of a change to the IT landscape across all aspects of the enterprise, the architect will need to be able to quantify the impact on staff, staffing and processes alike.

Think about the opportunity to start small, have a single team mature enough to develop and maintain API's. That team should be mandated and thus empowered to develop API's and put them in production, with full autonomy. If the team needs to deploy in production, it should be their call. They should be held accountable and they're responsible for the success of the API. Once that works take all lessons learned and scale your organisation to more teams to work on API's for their products. The architect will know what domain and what product will be best to provide API's to interface with in order to create the most business value.

Governance vs. Control

And then there's the other thing, which is governance. There's no room for control in an API economy. A Product Team that develops API's to access the product's information and functionality will need to have full autonomy. And that means that it can't be controlled. Control means that there's a committee that decides whether or not the team has been abiding by the restrictive rules defined by some other committee. This assumes that it is possible to define rules that API's need to follow without knowing the API and therefore the context within the API is being used. Utterly impossible because when you start developing an API you have no clue about your customer and by monitoring very closely you get an understanding of who your users are and why they want to use your API.
So should there be no control at all? Of course there needs to be control, but it's of the scalable kind. Scalable control is called governance. Governance is done by first define a series of principles. A principle is a rule or belief that governs ones personal behaviour. It's intention is that because one believes that the principle makes sense and one is better off by applying the principle, life is better.

A principle is a rule or belief that governs ones personal behaviour.

By their very nature, principles are used for governance and not for control. It's is one of the most formidable ways to get self organising and autonomous teams work in concert and by the same playbook. The interesting part here is also that you can choose not to stick with a principle in a certain situation. A clear motivation is available implicitly because one abides by the principles, unless there's a good reason. It also allows in your governance structure room for classifying principles in those that can be dropped when the team chooses so and which need an explicit approval from some authoritative body, like an Architecture Board.
Besides the principles, which are set by the architect, the architect needs to define a set of guidelines that inform the Product Team the best course of action to develop an API. These can be related to processes, like how to publish an API. But also technical, for example naming conventions, versioning strategies, etc. Good guidelines are no-brainers but converge various alternative solutions to a problem into a single solution that is the default selectable solution to the problem. It's different from a principle because it's not appealing to a developer's common belief system but to the developer's common sense.
Governance is an architect's dominion. It's not related to a specific team, but it's there to ensure the integrity of the complete IT landscape. This is where the architect shines. The Product Team is responsible for its own principles and guidelines and they should never conflict with the overall, architect driven principles and guidelines. One could argue that the team only makes architecture decisions, based on the principles and guidelines.

Mind that principles are hard to define, almost as hard as API's. Always think of an API as something that sounds like a rule, that has a clear rationale and has an impact which on the one hand details the impact on the existing landscape, the external impact. But also has an internal impact, which details limitations, prerequisites etc on how the software subjected to the principle is developed.
Quite often I see principles that hint towards solutions. For example: High available systems are deployed on a cluster. The rationale is clearly a technical one and the impact restricts the technical aspects of the solution. But an architecture principle's rationale should be a business rationale instead. The impact should allow for as much freedom to the team developing the software without compromising the principle. For API's one needs to be extra careful when defining principles. Stating that API's are always exposed as SOAP webservices seems to be a good principle. SOAP is a great way to do b-2-b integration and it allows for a standardised approach to security, validation, etc. But what if the better way to expose the API is as a RESTful service using JSON? Well in that case the Product Team can motivate why the principle is dropped. But a principle like API's are always exposed based on internet standards and web-based protocols achieves the same, it also allows the product team to select the best solution for their situation and still hold on to the principle. Furthermore, the architect could define a guideline, or guiding principle, stating that for b-2-b connections SOAP is preferred and for b-2-c REST/JSON is preferred.

Empowerment

I've stated before, the Product Team should have a high degree of autonomy when it comes to API development. You want governance in place to make sure that anarchy won't reign. This means that the Product Team should be empowered to decide for itself when principles are selling them short, when a new not yet anticipated situation surfaces or the API's 'market' changes over night due to new insights. In addition, you want the team to be empowered to decide how they develop API's in the most efficient way. Meaning that thinking of an API and having consumers calling the API should be an efficient as possible process. This is an important statement, because it means that the team can't just say 'screw the process or the organisation for that matter' because they have autonomy. Because doing so, would not benefit the complete process from idea to used product.



Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

December 16, 2016

API Management in Azure using Microsoft's solution - API-first design

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

API's are about the hardest thing to create in software engineering. Designing software to act as an API before anything else is even harder.

There's a new buzz going around in IT; API Management, and everything associated with it. I think it's even more buzzy than Microservices. Especially with shout-outs like "API First", compared to "Cloud First", "Mobile First" and "Web First" of yesteryear, we need to be cautious about what this means. This post is not about what API's are, why you should develop API's, or why API development is probably the hardest area in software development in this day and age. This post is about how you should look at Microsoft's offering on API Management and how to use it. Not the definitive guide, but a guideline instead.

This post is focusing on the API-first buzzzzzz...

Microsoft Azure

Skip to the next section when you're not interested in a view of mine on Azure, otherwise keep reading.
Whatever you think about Amazon AWS and the incredible pace by which Dr. Werner Vogel and his team is telling the world what The Cloud is and will be, you have to agree that the more amazing player in the Cloud arena at this point in time is arguably Microsoft.
Amazon has a track record of providing an amazing experience to their customers, both at Amazon.com but also on AWS. Clearly considering what the customer wants and doing everything to make the customer want to spend more money with them then anywhere else. They're doing this for over a decade now. I would argue that AWS is for those that understand the Cloud. Microsoft is arguably only since about 5 or 6 years a Cloud player to consider. Although announced in 2008, their implementation of the Cloud was in early 2010 and called Microsoft Azure in 2014. It really matured in 2016 with proper RBAC (Role Based Access Control) and proper infrastructure-in-code capabilities through the Azure Resource Manager. Other than AWS, which is a true 'API first' and 'Automate everything' platform, Azure is only since recently truly catching up in allowing everything to be done through API's and automation. Which is too bad since one of the Cloud's most awesome traits is the virtually infinite amount of resources allowing to provision and deploy a virtually infinite amount of versions of an application. Anyway, Microsoft's lack of API's and automation capabilities in the Cloud is consistent with its desktop platform Windows, which is a point-and-click affair instead of a scripting affair. Even administrators on Windows will use the point-and-click interface instead of the CLI (Command Line Interface) which is so accustomed to their Unix and Linux colleagues. It makes the platform less intimidating and many of the hard work is hidden behind a button and some checkboxes. This is how the Cloud is managed as well, through a Portal. It's a "point-and-click first" philosophy it seems. And many of the customers of Azure are happy with this. Who wants to be intimidated by a computer after seeing 2001 and Terminator anyway?

Setting AWS and Azure side-by-side one might see the following key difference: AWS allows you to do IT like playing with Lego. Amazon started with the basic building blocks. The Lego of 30 years ago and they allow you to build pretty much everything, if you're willing to spend the time. As time passes they come out with more advanced blocks, like the roofs in Lego. And just like Lego came with the little Lego men, gears, axles and so on, Amazon introduced more and more advanced services on AWS. But the idea, it seems, has always been to allow the customer to build pretty much everything she wants. Microsoft on the other hand seems to have followed the PlayMobil path. The fun is in the playing part and not so much in the building part. So open the Azure box and unlike AWS, you can start playing from day one. And as time passes by, there are more and more lower level components available, so you can start playing new games by building new toys.
Neither approach is better than the other, but you should carefully look at where you are, what you can do and what you want in order to be able to select either AWS or Azure. And IMHO, if you're just selected to go with Azure, don't even think about also doing AWS. And the other way around, because you would be most likely dealt lemons and you won't be able to make lemonade out of them.

API Management

First of all, API Management is a concept. I'm taking it you understand what an API is and I won't go into it any further than necessary for understanding this post.
API Management is done by people and some of them are using tools and technologies we call API Managers. Just like an enterprise, it has Management, and Management is made up of Managers.

Second of all, API-first design is not a fluke and it's not a hype either. Instead it is calling doom and gloom over your IT organization, or quite the opposite when you really know what you're doing.

Third of all, there's nothing MicroServices about API's although you see both walking around hand-in-hand more often than not.

Let's start with number 2...

API-first design
With API-first design, what you do is, you think about the functionality or information you want to expose and design it such that you don't have a clue about who will need or want to use that functionality or information. What you need to do next is worry about this. Start worrying a lot, because if you don't have a clue about who will access your functionality or information, let's call it a service, you don't know their intent either. And you don't know how often they'll access your service. You don't know whether or not you can trust them either. Are they taking security as serious as you do? Are they complying with all regulations you need to comply with? etc. Worries, worries, worries.

By not making any assumptions about the consumer of your service, you will need to make sure that it can stand the assault of all kinds of consumers. I do see your mind fluttering into the direction of non-functionals. Things like resilience and availability maybe? And encryption and access management possibly as well. Drop the mother in the cloud and you've got your scalability straight out of the Cloud's elasticity. (Of course you're very intimate with my post on this topic). And when you're in the Cloud, you know pretty much for sure that your Cloud provider will have all kinds of network protection in place without you even knowing it. Making sure that you, nor any other customer of theirs, will suffer from a DDOS attack and the likes.
But when we talk about API-first, we also need to think about the granularity of the service's exposed functionality or information. We need to think about it very, very carefully. In general we can say that less is more when it comes to this. But in fact, you want to hit that sweet-spot, because only then you're API is build to last.

That sweet-spot is something really important and almost impossible to find, let alone to hit. It's like the mythical G-spot. We know it should be there, but once you find it, you don't have the guarantee you will be able to find it all the times you're looking for it.
Granularity of your service defines it's usability and it is extremely business-bound. So first of all, you will need to know exactly the business context of your API and, and this is important, understand the business context of your consumers. And you should, no you must, accept that you'll have this latter one wrong. The most unlikely consumer will be your most valuable customer, so unless you're capable of reacting to her needs very quickly. Able to adapt to a massive change in positioning of your API in the grand scheme of the universe due to a pivot of your business, don't even try to go for API-first design.

By getting down to business, understand that an API is to be considered a product. It's not just access to your productised software solution, no it is your product. Nobody will care and shouldn't care how you do the magic behind the API as long as you're doing it. So you need to treat your API as a product. And that means that your API should have business value and as such, you should be very much aware that your API is valuable to your customers. In short, think about your revenue generated by your API. This is where your business context is showing it's ugly face again, because initially you can only assume how your API can be valuable to your customers. Hence you can only guess, albeit an educated guess, how you can get revenue from your API. Here again you'll need to be able to adapt to changing views and accept that hindsight is 20-20.
What this means is that you'll have to build (business) metrics into your API. Think upfront how you can verify your hypothesis and then do the validation of it as well. And don't be shy to change your product. A good product is one that is used, a great product is one that is valuable.
A key aspect of your API's granularity is about how elementary it is. You don't want to have an API that is only providing basic information, one that is only spewing data. Instead you want an API that is providing information in context. How much context is part of finding that sweet-spot. For example an Address by itself might mean nothing unless you know whose address it is, so a service that spews out addresses is of limited value, but one that provides access to a person and her address might be invaluable. Or maybe access to information is not that much of a thing to consider and instead you provide access to a function, for example to put some random text on API Management on the internet.

A good product is one that is used, a great product is one that is valuable.

The point here is that there's no way to determine what makes sense as an API and what not, unless you understand the context of your API at your end. And this only because it provides you with a starting point to make educated guess about the context of your API's consumer.
And at all times, the quality of your product needs to be such that you won't need to enter dialogue with the customer about the product, explaining why it's not working.

By the way, your architecture, blue-prints or design-patterns are no indication of what should be an API in the sense of API-first. In fact, I would argue that they are not an indication of what should be API's and managed as such and what not.
Considering you're developing according to service oriented concepts, and quite possibly adopting the principles of Forrester's 4-tier architecture. Which is in fact not a bad thing at all and probably a very beneficial view on things in general. So considering this, you might be inclined to consider very service in a specific architectural layer to be an API. For example the Aggregation Tier in Forrester's model or the Systems Tier in that model, could be considered to be an API. Which is total rubbish!
In software engineering we don't just do, we think about it and decide. So calling something a service in a specific part of our architecture an API is nonsense, should be cause of being fired in fact. And in fact, after thinking about it and deciding you have a service that is bound to be reused doesn't mean that will be an API either. So you'll have to do some more thinking... think... think... think. And then worry about calling something an API when you know in what situation the service is going to be used. Worry about calling something an API when you know who's going to call you to yell at you because the service is not abiding the service contract. Because if you know this, you know who your service's consumer is, and most likely there's no need to do any API'afyication on your service. Doesn't mean that you shouldn't treat it as an API, it means you don't have to treat it as an API if that doesn't warrant the effort.

In software engineering we don't just do, we think about it and decide.

API's have all kinds of stuff in them one way or the other, that allows them to be unleashed into the world. It's all the stuff that ensures that they survive the onslaught of calls by initially unknown subjects, the unsubs.
It's the part of API-first design that ensures that none of the wonderful promises of pots with gold by exposing the service's splendour, is compromised. It's the resilience, availability, scalability, security as well as metrics and other business requirements we too often consider IT stuff.
Think about the idea of implementing it over and over again for every service interface. If that means you're starting to sigh, and very deep at that, than only do it for the real API's. The service interfaces for which you don't know who'll call them. But when you get a smile on your face, eyes start to gleam, then you might as well take it from the other end; think about why you shouldn't treat the service as an API. Could be because of money (using standard tooling to API'afying service interfaces cost money) or maybe the operational overhead is not worth it.

So, in order to do API-first design, you need to understand your own business, meaning you need to understand what you want to expose and how you can create business value with your information or capabilities. This is hard enough for most software engineers, so they'll need the support of a business expert. Than you need to think about why others, and you don't know who that might be, would like to use your API, your product, and why it's valuable for them. This requires business knowledge and some understanding of how to see business value in using information flows. This is not the average business person you need to work with.
Than you need a thorough understanding of how to develop a resilient, performant and secure piece of software that can run in the Cloud, i.e. and elastic environment that is to be considered to be crashing all the time. And if you have all this in place, you need to be able to be roll-out new versions of your product on a very regular basis, with an extremely high level of quality. And on top of that, never get frustrated because your business pivots over and over again, because they're not really sure for whom the product is intended.

API's are about the hardest thing to create in software engineering. Designing software to act as an API before anything else is even harder.




Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

December 6, 2016

The Demise of the Project Leader in the DevOps world (Google Translated version)

Disclaimer: This is a translation by Google of the post "De Teloorgang van de Projectleider in de DevOps wereld", which is in Dutch. So bear with me when stumbling over tricky grammar and in case you can read and understand Dutch, head over to the original post by clicking here.

They are still, last week, I have seen one. A real project[manager]. The kind that, when you [ask] him or her, can show a certificate that clearly states that Prince2 course is successfully completed.

The project[manager] is a dying profession. At least, in Software Development. Fortunately, because they have absolutely no business. At least, in Software Development.

The question is why there still are so many project managers in the IT sector. A simple answer is no, but probably it's because we have so many project managers in other sectors. In the construction example, we have developers who realize projects which are then led by project leaders. And just when we compare make the fundamental error in the IT architecture by the IT architect with a 'buildings' architect [1], we do so with project leaders.

We think we need project managers in IT and that is mainly due to that damned waterfall method we used to see in our IT projects since time immemorial. Kolder course, waterfall method, there has never completed a project within the planned budget and completed in the scheduled time period something useful. Can not, because there is no single user that actually knows what he or she wants. Let alone that the user knows what he or she needs.

No wonder that we have our sights now focus on agile methods, the sweetest as they do in Spotify. In brief:


You define an idea, you develop software, you let the user whether it was a good idea and you go on to the next idea.


And so here it happens. This is what comes to light that the project does not belong in IT. Because if you think this cycle is a project, you'll miss it a lot. Also by the way if you think that some of these cycles form a project.

The fundamental problem of the project is that a project is finite. You start on a project and at one time the project is finished. We're ready when the project is finished. And that of course is nonsense for IT projects for IT projects are never finished. Haha, you think now, which is the well-known IT joke that projects always on time and on budget go. But it's not a joke, because when an IT project is completed, it all begins actually.

When the project is in fact finished, the software is used and only then we can work to make it usable it. we look at the project, his or her job done when the project is completed. The project will therefore only give the software to a user. Within time, within budget. Irrespective of the fact whether the software turns out to be usable.

The project thus focuses on what the user wants and not what the user needs. And that is the fundamental reason why IT projects are absurd, and why project managers are disastrous.

We happily watching for a while, so we have a few years ago started with agile. Apart from the agile manifesto [2] and the hype around Scrum, we're in IT for over 20 years, very active in the iterative development with the shortest development cycle. And therefore we are since then, until about 5 years ago, been busy with projects delivering software iteratively. Instead of long journeys waterfall project managers have divided their projects into small waterfalls. We see that iterative projects anticipate less in terms of time and money. But still the user gets what he wants and not what he needs. Or they naturally.

It should be clear; As long as software is developed by project-led projects, then that results in desired unusable software.

And then there was agile and Scrum, and since then the user gets what they need, not necessarily what he wants. By not from the outset to define what functionality the software should provide, but gradually adjust the scope and adapt existing functionality to something useful, we need agile projects that provide something useful.

The really good project managers know to think to this new way of working. It is the type of project that is not immediately shoot the stress is not clear what should lead his team. The scope of the project, but also the duration is thus determined by the budget. The project is over when the money runs out. The Project Manager will, if he or she has heart of the matter, therefore deliver the greatest possible functionality for that budget.

And therefore the project is no longer necessary. Because it makes no sense to scopes projects based on how much money is available. That should be clear.

Then there is another small, tiny, point of focus to work with new agile. Early on, namely as soon as possible, there, in a real agile project, something in production and is used by real users. That may have to be adjusted because it is not usable. And suddenly, the project manager is also responsible for solving production problems.

All, or at least too much, organizations have a galvanic separation between the project and operational management. Change-run organizations with clear, often indefinitely elaborate, hand-over procedures. Different responsibilities and often different budgets. [3] These organizations are full of silos that do the best work by itself, but together are far from optimal.

The Change-Run model does not work in practice. This is often denied. Especially by people who think the waterfall method works. And many of those people are project leaders who deny their demise and otherwise not recognize yet.

They be lost because organizations have projects in IT are extremely absurd. Of course there is a whole group of organizations that is just hip and do not understand why they are. A growing group understands the ups and focuses what is desirable and what is no longer needed.

IT departments transform themselves more and more from project organizations to product organizations. Organizations abandon Change Run divorce and move to DevOps [4]. Specialist teams are dissolved and multifunctional teams are appointed. No software projects to be done, but products developed software. Because there are no projects to be done there are no project managers needed. In any form. However, there are product requires owners. Unfortunately for all those without work-come-to-sitting project, a product owner is no such thing as a project manager. namely the product owner is the owner of a product that understands the product, and above all understand what is and is not usable. He or she is responsible for the product, cradle to grave, from the cradle to the grave so.

He, or she, has the mandate to the functionality offered by any means. Is also another budget responsible for the product. The product owner is not judged by the amount of features that will be hoisted into production in the time available and the available budget, but the value of his or her product for the organization. And that value can only be measured when the product is actually used.

But there is hope for the project. Indeed, there are quite a few product owners who do not understand it yet. They develop their products in a project that has quite a few puffs of a waterfall process. Or at least iteratively. And as long as product owners do not yet fully understand what an MVP [5], the project still work. The current crop of product owners are not sufficiently experienced in the delivery of a thing. The project leader, and certainly one with iterative processes is known, is therefore a big setback for. But for how long?

The tide turns namely, product owners understand more and more what is their role and how they can fill it the best. Organizations are finding that it is not wise to their project to schools to product owners. Apart from the odious Prince2, it also be that the project has sufficient knowledge and often no affinity with the product. That 'one thing' that an organization makes money with it. Product Owners then get out of the business and not like the project leaders from the IT. The project is unnecessary.

Thus the demise of the project is therefore inevitable. Not a matter of if but of when.

As always, thanks for reading this post and I hope that the post has put your thinking. I realize that I quite bluntly go and drop it all pretty black / white. And as with anything on the internet you should this article with a dose of common sense reading. Should you make the nuance. I want to invite you to do that last through the comments of this post. Likely that others also find it interesting to know how you think, and of course there is no doubt that I find it interesting.

PS: There are projects in IT that drop something then never come to more, or evolve. These are the projects that you might be comparing projects from the building. IT infrastructure projects that will be delivered something physical. In the best case will be completed in phases, so that it can be enjoyed quickly yields, but when completed can not really speak of further development. In these cases, a project still certainly useful. But in a world in which more and more infrastructure code is expressed is it rather than product development.


[1] But this article is not about, you want to know what's wrong with that, you can read my blog post about the architect. Which can be found here: https://itarchitectureandstrategy.blogspot.nl/2015/02/how-youll-fail-as-it-architect-when-you.html.
[2] Look at http://agilemanifesto.org/ the manifesto. Read it carefully before you draw your conclusions.
[3] Change-run organizations are the result of not understanding Taylorism. https://en.wikipedia.org/wiki/Scientific_management, like the agile manifesto.
[4] In a DevOps organization working Developers (Dev) and managers (Ops) closely. See also: https://en.wikipedia.org/wiki/DevOps
[5] A MVP Minimum Viable Product, is a version of a product with the minimum set of functionality to still create value for the user. See also: https://en.wikipedia.org/wiki/Minimum_viable_product

De Teloorgang van de Projectleider in de DevOps wereld

Disclaimer: This post is in Dutch, I will translate it into English at some point, but by clicking here you can read a translation by Google.

Update: Lees ook het artikel "Agile Projecten: De veranderende rol van de Project Manager" voor een bredere uiteenzetting.

Ze zijn er nog, vorige week heb ik er nog eentje gezien. Een echte projectleider. Zo eentje die, wanneer je het hem, of haar, vraagt, een certificaat kan laten zien waarop duidelijk vermeld staat dat een Prince2 cursus met goed gevolg is afgerond.

De projectleider is een uitstervend beroep. Althans, in de Software ontwikkeling. Gelukkig maar, want ze hebben er helemaal niets te zoeken. Althans, in de Software ontwikkeling.

De vraag is echter waarom er toch zoveel projectleiders in de IT-sector te vinden zijn. Een simpel antwoord is er niet, maar waarschijnlijk komt het omdat we in zoveel andere sectoren ook projectleiders hebben. In de bouw bijvoorbeeld, daar hebben we projectontwikkelaars en die realiseren bouwprojecten en die worden dan weer geleid door projectleiders. En net als dat we de kapitale fout maken in de IT-architectuur door de IT-architect te vergelijken met een ‘gebouwen’ architect[1], doen we dat ook met projectleiders.

We denken dat we projectleiders nodig hebben in de IT en dat komt vooral door die vermaledijde waterval methode die we sinds mensenheugenis in onze IT-projecten toegepast zien. Kolder natuurlijk, die waterval methode, er is nog nooit een project opgeleverd dat binnen het geplande budget en in de geplande tijdspanne iets bruikbaars heeft opgeleverd. Kan ook niet, want er is geen enkele gebruiker die ook daadwerkelijk weet wat hij, of zij, wil. Laat staan dat de gebruiker weet wat hij, of zij, nodig heeft.

Geen wonder dus dat we al onze pijlen tegenwoordig op agile methoden richten, het liefste zoals ze dat bij Spotify doen. In het kort:
Je hebt een idee, je ontwikkelt de software, je laat de gebruiker bepalen of het een goed idee was en je gaat verder met het volgende idee.

En hier gebeurt het dus. Dit is waar aan het licht komt dat de projectleider niet in de IT thuishoort. Want als je denkt dat deze cyclus een project is, dan heb je het flink mis. Ook trouwens als je denkt dat een aantal van die cycli samen een project vormen.

Het fundamentele probleem van een project is dat een project eindig is. Je begint aan een project en op een gegeven moment is het project klaar. We zijn klaar als het project af is. En dat is natuurlijk kolder voor IT-projecten want IT-projecten zijn nooit af. Haha, denk je nu, dat is de bekende IT-grap, dat projecten altijd over tijd en over budget gaan. Maar het is geen grap, want wanneer een IT-project klaar is, dan begint het allemaal pas eigenlijk.

Wanneer het project namelijk klaar is, wordt de software gebruikt en pas dan kunnen we eraan werken het bruikbaar te maken. Kijken we nu naar de projectleider, dan is zijn, of haar, werk gedaan als het project is opgeleverd. De projectleider wil dus alleen maar de software aan een gebruiker geven. Binnen de tijd, binnen het budget. Los van het feit of de software wel bruikbaar is.

De projectleider richt zich dus op wat de gebruiker wil hebben en niet wat de gebruiker nodig heeft. En dat is de fundamentele reden waarom IT-projecten onzinnig zijn, en waarom projectleiders desastreus zijn.

Dat hebben we gelukkig al een tijdje in de gaten en daarom zijn we met agile begonnen een paar jaar geleden. Los van het agile manifesto[2] en de hype rondom Scrum, zijn we in de IT al meer dan 20 jaar heel actief in het iteratief ontwikkelen met een zo kort mogelijke development cyclus. En daarom zijn we sinds die tijd, tot ongeveer een jaar of 5 geleden, druk geweest met projecten die software iteratief opleveren. In plaats van lange waterval trajecten hebben de projectleiders hun projecten in kleine watervalletjes verdeeld. We zien dat iteratieve projecten minder uitlopen voor wat betreft tijd en geld. Maar nog steeds krijgt de gebruiker wat hij wil en niet wat hij nodig heeft. Of zij natuurlijk.

Het moge duidelijk zijn; Zolang software ontwikkeld wordt in door projectleiders geleide projecten, dan resulteert dat in gewenste, onbruikbare software.

En toen was er agile en Scrum en sindsdien krijgt de gebruiker wat hij nodig heeft en niet per se wat hij wil hebben. Door niet van meet af aan vast te leggen welke functionaliteit de software moet opleveren, maar gaandeweg de scope bij te stellen en bestaande functionaliteit aan te passen tot iets bruikbaars, hebben we agile projecten die iets bruikbaars opleveren.

De echt goede projectleiders weten om te denken naar deze nieuwe manier van werken. Het is het type projectleider dat niet meteen in de stress schiet als niet duidelijk is wat zijn team moet opleveren. De scope van het project, maar ook de duur, wordt daarmee bepaald door het budget. Het project is afgelopen als het geld op is. De projectleider wil, als hij, of zij, hart voor de zaak heeft, dus zoveel mogelijk functionaliteit voor dat budget opleveren.

En daarom is de projectleider niet meer nodig. Want het snijdt geen hout om projecten te scopen op basis van hoeveel geld er beschikbaar is. Dat moge duidelijk zijn.

Dan is er nog een ander klein, piepklein, puntje van aandacht met dat nieuwe agile werken. Al vroeg, namelijk zo snel mogelijk, staat er, in een echt agile project, iets in productie en wordt het gebruikt door echte gebruikers. Dat moet misschien worden aangepast omdat het niet bruikbaar is. En plotseling is de projectleider ook verantwoordelijk voor het oplossen van productieproblemen.

Alle, of in ieder geval nog teveel, organisaties hebben nog een galvanische scheiding tussen de projectorganisatie en het operationele beheer. Change-Run organisaties met duidelijke, vaak tot in den treure uitgewerkte, hand-over procedures. Verschillende verantwoordelijkheden en vaak ook verschillende budgetten.[3] Deze organisaties staan bol van de silo’s die wel op zichzelf optimaal werken, maar gezamenlijk verre van optimaal zijn.

Het Change-Run model blijkt in de praktijk niet te werken. Dat wordt vaak ontkent. Vooral door mensen die denken dat de waterval methode werkt. En veel van die mensen zijn projectleiders die hun teloorgang ontkennen en anders nog niet onderkennen.

Ze gaan teloor omdat organisaties door hebben dat projecten in de IT uitermate onzinnig zijn. Er is natuurlijk een hele groep organisaties die gewoon hip is en niet snapt waarom ze het zijn. Een steeds grotere groep snapt het wel en richt zich wat wenselijk is en niet meer op wat gewenst wordt.

IT-afdelingen transformeren zich meer en meer van projectorganisaties naar productorganisaties. Organisaties stappen af van Change-Run scheidingen en stappen over op DevOps[4]. Gespecialiseerde teams worden ontbonden en multifunctionele teams worden ingericht. Er worden geen softwareprojecten gedaan, maar producten ontwikkeld in software. Omdat er geen projecten meer worden gedaan zijn er geen projectleiders meer nodig. In geen enkele vorm. Wel zijn er product eigenaren nodig. Helaas voor al die zonder-werk-komen-te-zittende projectleiders, is een producteigenaar niet zoiets als een projectleider. De producteigenaar is namelijk de eigenaar van een product, die snapt het product, en begrijpt vooral wat wel en wat niet bruikbaar is. Hij, of zij, is verantwoordelijk voor het product, cradle-to-grave, van wieg tot in het graf dus.

Hij, of zij, heeft het mandaat om te welke functionaliteit op welke wijze geboden wordt. Is ook nog eens budgetverantwoordelijk voor het product. De producteigenaar is wordt niet beoordeeld op de hoeveelheid functies die in de beschikbare tijd en met het beschikbare budget in productie gehesen wordt, maar wat de waarde van zijn, of haar, product voor de organisatie is. En die waarde kan alleen gemeten worden wanneer het product daadwerkelijk gebruikt wordt.

Maar er is hoop voor de projectleider. Er zijn namelijk aardig wat product eigenaren die het ook nog niet snappen. Ze ontwikkelen hun product in een project, dat nog behoorlijk wat trekjes heeft van een waterval traject. Of in ieder geval iteratief. En zo lang product eigenaren nog niet helemaal snappen wat een MVP[5] is heeft de projectleider nog werk. De huidige lichting producteigenaren zijn nog onvoldoende ervaren in het opleveren van een iets. De projectleider, en zeker een die bekend is met iteratieve trajecten, heeft hier dus een flinke streep voor. Maar voor hoelang?

Het tij keert namelijk, product eigenaren begrijpen meer en meer wat hun rol is en hoe zij deze het beste in kunnen vullen. Organisaties komen erachter dat het niet verstandig is om hun projectleiders om te scholen tot product eigenaren. Los van het verfoeilijke Prince2, is het ook nog eens zo dat de projectleider onvoldoende kennis en vaak geen affiniteit heeft met het product. Dat ‘ene iets’ waar een organisatie geld mee verdient. Producteigenaren komen dus uit de business en niet zoals de projectleiders, uit de IT. De projectleider is dus overbodig.

Daarmee is de teloorgang van de projectleider dus onvermijdelijk. Niet een kwestie van of maar van wanneer.

Zoals altijd, bedankt voor het lezen van deze post en ik hoop dat de post je aan het denken heeft gezet. Ik realiseer me dat ik nogal kort door de bocht ga en het allemaal redelijk zwart/wit neerzet. En zoals met alles op het internet moet je ook dit artikel met een dosis gezond verstand lezen. Moet je zelf de nuancering aanbrengen. Ik wil je uitnodigen om dat laatste te doen via de comments van deze post. Waarschijnlijk dat anderen het ook interessant vinden te weten hoe jij erover denkt, en uiteraard staat het buiten kijf, dat ik het interessant vind.

PS: Er zijn projecten in de IT die iets neerzetten om er vervolgens nooit meer aan te komen, of door te evolueren. Dit zijn de projecten die je wel zou kunnen vergelijken met projecten uit de bouw. IT-infrastructuur projecten waarbij iets fysieks wordt opgeleverd. In het gunstige geval wordt er gefaseerd opgeleverd, zodat er snel genoten kan worden van de opbrengsten, maar wanneer opgeleverd kan er niet echt worden gesproken van doorontwikkeling. In deze gevallen is een projectleider nog zeker wel nuttig. Maar in een wereld waarin ook infrastructuur meer en meer in code wordt uitgedrukt is ook hier eerder sprake van productontwikkeling dan van projectontwikkeling.

[1] Maar daar gaat dit artikel niet over, wil je weten wat daar mis mee is, dan kan je mijn blog post over die architect lezen. Die vind je hier: https://itarchitectureandstrategy.blogspot.nl/2015/02/how-youll-fail-as-it-architect-when-you.html.
[2] Kijk op http://agilemanifesto.org/ voor het manifesto. Lees het aandachtig voor je je conclusies gaat trekken.
[3] Change-Run organisaties zijn het gevolg van het niet begrijpen van Taylorism. https://en.wikipedia.org/wiki/Scientific_management, net als het agile manifesto.
[4] In een DevOps organisatie werken ontwikkelaars (Dev) en beheerders (Ops) nauw samen. Zie ook: https://en.wikipedia.org/wiki/DevOps
[5] Een MVP, Minimum Viable Product, is een versie van een product met de minimale set aan functionaliteit om toch nog waarde te scheppen voor de gebruiker. Zie ook: https://en.wikipedia.org/wiki/Minimum_viable_product

November 10, 2016

The Bimodal misconception, why Gartner has to realize the ignorance of its customers

Okay, first of all, I'm talking about Gartner's Bimodal IT thing here which is defined by them, that is Gartner as follows: "Bimodal is the practice of managing two separate but coherent styles of work: one focused on predictability; the other on exploration.", which is taken from here. And it makes no sense, because Gartner omitted to mention that this is an application architecture thing.

The problem I have with Gartner's definition of Bimodal IT is that it implies rather explicitly that predictability and exploration don't go together, which is of course totally correct. You explore because you can't predict. But the reason why Bimodal IT was introduced is because we face a stand-off between waterfall and agile, between Silo'd IT and DevOps, between client/server and service oriented systems, between legacy and the new world.

Reading the rest of Gartner's definition (Mode 1 is optimized for areas that are more predictable and well-understood. It focuses on exploiting what is known, while renovating the legacy environment into a state that is fit for a digital world. Mode 2 is exploratory, experimenting to solve new problems and  optimized for areas of uncertainty. These initiatives often begin with a hypothesis that is tested and adapted during a process involving short iterations, potentially adopting a minimum viable product (MVP)  approach. Both modes are essential to create substantial value and drive significant organizational change, and neither is static. Marrying a more predictable evolution of products and technologies (Mode 1) with the new and innovative (Mode 2) is the essence of an enterprise bimodal capability.) you understand the flaw in their reasoning. The flaw is in that the definition assumes that an organization can be divided in two ways of addressing its business, or rather that it can adopt a way of working in that it addresses its business simultaneously in a 'we know what the customer wants' and a 'we want to know what a customer wants' way of working. Which is bogus, of course. Totally bogus.

At one of my current clients, Bimodal IT is a hot topic. Here it is used such that the back-end systems are considered Mode 1 systems and the front-end systems are considered Mode 2 systems. Furthermore, the epic mistake is made to consider everything running in the cloud to be a front-end system and everything in the on-premise environment, which is a traditional data-centre, to be a back-end system. Makes no sense to me, and I can guarantee that within the next 6 months there will be back-end systems running in the cloud.

To understand Bimodal IT, we need to understand why the term was coined. Basically, you need to re-read the definition and understand what is being said. You need to put it in the context of time, and address it from an architectural perspective. Because that is what Bimodal IT is, well that is its raison d'etre.

So, let's start with 'one focused on predictability' and forget about this. It's an ambiguous term. Ambiguous because for one we want our software delivery cycle to be predictable, we want the quality of our solutions to be predictable and we want the costs of our solutions to be predictable. At all times, no difference between front-end and back-end, no difference between cloud and data centre and no difference between old and new. At the same time, we can't predict what the best feature set of our solution is. We can't ever predict it, no difference between fron
t-end and back-end, no difference between cloud and data centre and no difference between old and new.
So predictability only applies to our solution delivery process and not to our solution.
And then there's 'the other on exploration' which we should forget as well. It's an ambiguous term as well. It's ambiguous because we need to always explore to find the right feature set for our solution. This is because we can't predict the right set. It has nothing to do with where we're hosting, whether it's front-end or back-end or anything else.
So Bimodal IT by Gartner's definition makes no sense, because from the outset it's wrong.

But here's the catch, at the time Gartner coined the term, it made perfect sense. It's just that Gartner forgot to mention that Bimodal IT actually hinges on application architecture.

Back to this customer. Bimodal IT is used as an excuse not to comply with the ancient and really seriously out dated processes in IT that are (to be) followed within this organization and I assume unwillingness to try to modernise the processes. By just simply referring to Gartner and their definition of Bimodal IT and stating that this is the exact situation at their organisation, some of my clients architects and project managers are implying that the processes only apply to what's running in the back-end and what they're working on, which is the front-end.(*)

Let's see why thinking in terms of Bimodal IT is making sense. It makes sense because at the time Gartner coined the term, we developed our systems predominantly being Waterfally and the hipsters were being Agile. There was nothing exploratory about it, and read the Agile manifest and it says nothing about exploration, but the more about predictability.
Mode 1 and 2 are all about delivery cycles. Mode 1 is long, Mode 2 is short. One could say Mode 1 is about being ignorant and Mode 2 about being smart.

So, ignorant vs smart, what's that about?

Well, in Mode 1, we tend to think that it makes sense to first define the complete solution, design it and build it before we deliver it to our users. And because for some reason, release into production always results into (often major) headaches, we also tend to think it makes sense to not release that often, so we limit the occurrences of headaches. Hey, if you only bang your head against a wall twice a year, you only run the risk to crack your skull twice a year. So makes perfect sense, doesn't it?
Then there's Mode 2, where we think that is makes sense to just grab the most important, or valuable features and develop these and take them to our users, and then do it again with the next most valuable solutions and put them out to our users. And we tend to think that because we only release a tiny thingy, it's very unlikely that it'll go wrong and will only cause (if at all) a dull feeling in the back of our head. Hey, if you nudge your head against a wall every few days, at the most you know that the wall is still there, but that's about all to be felt.

So Bimodal IT is where one group of people thinks that doing something really hard is less risky when you make it even more complex but do it fewer times and the other group of people thinks quite the opposite, that doing something really hard is less risky when you simplify it and do it more often. Kind of practice makes perfect.

And yes, years ago when Gartner came up with Bimodal IT, we really had this Mode 1 and Mode 2 division as described above. But in this day and age, the Mode 1 people are a rare breed. Mind that we're discussing this in an era where major banks (ING) are firing pretty much everybody that is Mode 1, where major airlines (KLM) are spending millions of €'s to transform in to a full Mode 2 organisation and where it's hip to be Mode 2 at IT departments of local and national governments.


But Bimodal IT is there to stay for a while. Not because of predictability, nor because of exploration. Not even because of Waterfallians (Mode 1 people). Nope, it's because of Monoliths. Those ancient systems that were delivered in a day and age when we thought Client/Server was awesome and everything should be consolidated into one system so we could guarantee data consistency and we could really vouch for abiding to the DRY principle (Don't Repeat Yourself).

These systems can't be changed, they are architected to be unchangeable. We don't want them to be that way, but they are. It's because our application architectures of a couple of years ago resulted in unchangeable systems, we can only change them by changing the complete system. The monoliths are preventing us from becoming post-Bimodal IT organisations.

Back to the misconception. Bimodal IT is something that has to do with the monoliths in our IT landscape and nothing with how we perceive our IT or the systems in our IT landscape. The misconception is that we need to adopt a Bimodal IT approach to how we do IT because we don't have to. We have to get rid of the monoliths and we need to start by isolating them behind a layer of changeable software. Hide them like a skeleton in a closet of services.

By adopting Bimodal IT today, or continue with Bimodal IT today, means you either have a totally wrong understanding about how to manage your IT department, or you are still in the previous millennium when it comes to IT architecture, or you're just using Gartner's reputation as an excuse for not having to modernise your IT processes.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan


Although this post is my personal opinion, there're quite a few articles to be found on the internet about this topic that I've read. So here're some interesting reads:

Gartner Glossary: Bimodal IT
- Gartner's bimodal IT mistake: DevOps can deliver velocity and quality
- Bimodal IT: Gartner's Recipe For Disaster
- Beware the Dangers of Bimodal IT
- Gartner’s Evolution of Bimodal IT
- The Flaw at the Heart of Bimodal IT




*: Interestingly enough, they forget to mention that the back-end is running on-premise and they're in the cloud and yes, the front-ends hosted on-premise are just as much under the same regime as the back-ends.