In most software based businesses it is quite common to think about platforms. Building an own internal platform that implements the common assets of the business so that new products and solutions can be built faster and cheaper or leverage the data that was collected from previous projects – for internal use or externalized as a platform-as-a-service business. While now many of the classic platform functions like authorization, authentication, logging, metering, configuration… are mostly commoditized and solved, there is an unbroken need for platforms that implement higher level business functions. In addition to this functionality focused approach, the new thing in platform development is around data storage, data sharing and data distribution.

Preface

When organizations build individual software solutions and products over the years, at a certain point in time it seems reasonable to consolidate the existing solutions and extract their commonalities into a shared common asset – called software platform. Or – at an earlier stage, there is so much opportunity in a certain business that the organization believes it needs a central platform to enable a set of solutions and business on the horizon. In this case, there may be only a few or even zero existing solutions yet. I have seen multiple approaches that started with a single solution / product; this product was very successful and the business opportunity to do more in that area was significant, so it was decided to build a platform from the single successful product/solution to boost the development of “more like this” products and solutions. A similar situation can be observed when a project or product collected a valuable set of data that now generates the opportunity to build several new services and business around the re-use of that data.

The reasoning is often based on the assumption that the platform – as shared asset that is reused in the existing and to-be-developed systems – must be maintained only once and allows reduction of cost and shorter time to market for new solutions. However, the challenges within this approach are often underestimated which may lead to exactly the opposite than the expected cost and time savings. The following paragraphs outline typical pitfalls and challenges and provides best practices of how to approach a change to a platform based approach successfully. Here, the article follows the base assumption that the platform shall be either exposed internally within and organization or externally as commercial service.

The idea of cost-optimization through shared software components and shared data by going into a platform approach is also often combined with an approach to reduce the operational cost for the target platform and focus the engineering efforts towards differentiating areas of the platform, away from common base platform functionality. Todays cloud offerings play a significant role in this kind of approach, because they allow an organization to simply make use of infrastructure and platform services instead of developing them as part of their own platform in-house. This applies to very basic functionality like metering, logging or authentication up to complex services like machine learning and artificial intelligence based services.

First of all, I like to point out what this article is not: it is not an article that explains how to build a software product well from a SW engineering point of view. The art of creating successful software solutions is covered in many books and articles, to which shall be referred at this point. This article is also not a re-definition of Product Line Engineering (PLE) for software products that run in the cloud. This topic is also covered in literature and discussed widely in conferences and expert groups. However, this article tries to highlight the best practices of both disciplines in the area of platform scoping and cloud based platform development in addition with the aspect of human centric issues (social and organizational elements) in order to provide a more holistic view on the challenge at hand.

So, I will first provide a general overview on the challenge and the complexity of a platform approach, both: consolidating on existing solutions and as a base for new solutions. Here, the technical complexity and requirements diversity is one central aspect but also non-technical aspects typically lead to severe problems during such an undertaking. Missing acceptance of the approach, unexpected costs, technical problems on the way and a striking resistance of people to abandon their old habits all lead to the fact that many such approaches remain unsuccessful. Secondly, this article will highlight a few best practices to start a platform approach, from the existing PLE frameworks, from software architecture but also from a peoples-cooperation point of view, that help to deal with the complexity and handle the risks. Even for projects and people that are either not aware of PLE or simply do not want to follow a product line approach, some of the PLE methods can be used to strive for a less risky implementation of the platform approach.

As a result of reading this article, you should develop the idea that building a platform should be considered well. When thinking of all consequences and total cost of ownership, sometimes individual solutions are the better variant. As any technology, platforms are no silver bullets. I will provide some hints that help to make the judgment.

What is a Platform?

There are many definitions for this term in computer sciences. In this paper, the term is about software platforms and is used according to a definition that can be found in several papers and books:

“A software platform is defined by a well specified interface that allows other software to be built on top of it.”

So, it is about functionality that is provided by a set of software which can be used by other software (or better, those who write the software). When building a platform that is designed to make data reusable, the same applies, because you will require functionality that manages the access to the data, you may want to build services for meta data management or monitor and audit who uses which data to which extend. All functionality that shall be used by other people’s code.

A SW platform can be provided in multiple ways: as a set of binary libraries, as a collection of pure code that can be compiled into a product, as a complete product that is delivered as an installable medium, as software as a service in a cloud environment or in any other provisioning. The point is that the functionality is accessible by other software through a well-defined interface and that the software is managed by the creating department or company. This interface in turn also can be provided in different ways: as code level API, as a web-service, via a messaging interface, REST API, you name it. Such a software platform so provides the underlying layer(s) in a system that is more specialized to serve a more concrete business. The difference between just shared libraries or components (see OSGI or SDC for examples) and a platform is, that the platform has a business scope and aims to support all base functionality within this scope, so it is managed and an actively governed asset that delivers a set of services under one umbrella.

As an example, modern cloud platforms services provide platform functionality in exactly this definition. Services expose APIs which are used to interact with the service, the service implementation is hidden from the user and every service does exactly one thing very well. The key is that all services are crafted to be used by developers to build solutions and higher level services. As you can see in the for instance in the AWS portfolio, platform services can be on different levels in a technology stack: Base infrastructure like compute and network, middleware for system integration, databases and data storage, data streaming and messaging, data analytics and many more. The same applies for all platforms that are in scope for this article.

A product or solution of one organization may serve as a platform for the next organization. As long as the layers provide public interfaces to re-use their functionality and they are fully managed by their providers, they can be defined as a platform. Of course, there might be internal platforms (used within a company) and platforms that are open to the public with a business model around the use of shared services.

In the larger context of product line engineering, the platform will provide the common assets that are going to be reused in the products that are developed in the scope of the PLE approach. So, the platform becomes the technical enabler on which the products will be set up.

Platform Business Models

Platforms, like all software, can be delivered in several ways to serve their users and so, they may be used in different ways to generate revenue. We can basically differentiate the following fundamental business models for platforms:

• Company internal solution business – shared cost model: Internal re-use of SW that should speed up development for solutions in solution business. Platform cost is shared across projects and solutions. The platform itself is seen as a pure cost factor, revenue is generated only by solutions on top of it.
• Company internal product business – shared cost model: Internal re-use in product line approach where the platform provides core elements to all products within the product line. Platform cost is shared between products. The platform itself is seen as a pure cost factor, revenue is generated only by solutions on top of it.
• Company internal product or service – platform has own loss and profit responsibility. Here the platform is managed as a product with an internal price which may be pay-per-use or fixed-price oriented. Only works when used in large enterprises across many departments.
• Platform as a licensed (at a price or free to use) product offering, providing a set of tools and modules that build the basis for further development. You will find many vendors who call their products a platform because you can develop more sophisticated products based on their product. Typically, this requires local installation / provisioning through download or media.
• External, public, managed service offering, typically based on pay-per-use consumption models. Target here is to maximize amount of users of the platform. This is where AWS and other cloud providers are at home.

Please note that the business model differs from the deployment model. You can host internal platforms as well as external, commercialized services.

What all business models share in common is that it shall be used by 2 or more solutions or products, that the platform cost is somehow distributed over all users and so price typically lowers with more adoption. And because the fundamental model is based on maximizing the re-use, a failure of platform service may lead to massive impact at all users. So, failing to meet basic operational quality will have a killing effect on any platform business. The platform needs special attention in all aspects during the planning, the development and the operations processes. Operational qualities like security, scalability, multi-tenancy, robustness, cost efficiency and long term developer support become the key elements that drive success or failure.

For the sake of completeness, I also want to mention another concept of platform business model. The model outlined here, focuses on platforms that aim to provide a convenient way to connect providers of a good / resource / service to those who want to consume and pay for such a good / resource / service. Examples are the following:

• Uber: Uber does not own cars, taxies or busses but provides a platform for people who want to offer a car and a driver and those who want to order a transportation ride.
• AirBnB: This company does not own hotels or rooms to rent but provides a platform for those who offer space and those who need it.
• eBay: Also here, eBay focuses on making it easy to sell and buy things, while eBay is only the intermediate enabler, not seller or buyer itself.

There are more examples for these platforms which all have in common that they provide functionality that enables other companies or people to make business – in a specific domain – via the platform. Those platforms are different from the ones I outlined above, because their main users are not solution builders and software developers but actual end users on two sides: The provider / producer side and the consumer side. Like platforms in other domains, e.g. Facebook as social platform, these platforms are out of scope for the remainder of this paper.

What is Product Line Engineering?

Before diving into the topics, a very brief introduction into the part of PLE which is relevant for this article shall be provided. It is intended for those readers who did not yet have contact with PLE before. Because several methods of PLE will be mentioned in this paper, the context shall be outlined beforehand. I want to highlight this, because PLE is a very structured approach to platform development and can be used in other contexts as well.

A general definition of PLE may be:

“A software product line is a set of software intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment and that are developed from a common set of core assets in a prescribed way.”

The main characteristics of a product line so focus on three aspects:

• Business driven: All products and so also the shared assets (platform) are shaped and defined because of market demands and business considerations. A global optimum over all platform based products is in focus, not the individual optimum for any single product.
• Managed variability: There is a clear and explicit process of identifying and managing variability and commonality between products on all organizational levels; and over all phases of the product lifecycle.
• Strategic reuse: Reuse is planned and part of the product strategy and the product portfolio. This also spreads over all products of a product line and so all involved organizations. So, for example, the reuse and its impact is not necessary limited to a single department within a company.

From these very basic aspects it becomes clear that PLE does focus on a broad and global optimization of product development. It so automatically dives into processes and organizational topics and provides tools for managing and modeling the artifacts that are required to execute such an approach. At its core, the PLE approach relies on a selection of target products and a clean analysis and modeling of the commonalities and the variabilities among the planned (or existing) products in order to determine the scope of the shared common assets.

Why do we want a Platform anyway?

Some typical Examples how Platforms emerge

As a starting point, it shall be discussed how a business owner or R&D manager might come up with the idea of wanting a platform. In many, but not all, cases this starts with the idea of extracting the common features or data of existing solution or products into a platform. The examples are simplified but hopefully help to get a grip on the topic:

Multiple solutions or products in the same domain

Organizations that build and own multiple products within the same business domain, consider platforms as a mean to optimize development cost, operational cost and reduce time to market for new products. Here, the common functionality of all products in that product line (see PLE) becomes scope of the platform. Think of a series of cars which share the same chassis, motors and other parts. The same applies for solution business, where several solutions within the same business domain require a common set of functionality. Business drivers here are cost and time reduction for custom solutions, to gain a competitive advantage during bid phases and save cost during development.

Multiple solutions or products in similar domains

Organizations that serve multiple business domains that share some common functionality expand the concept from one product line to many product lines and from one set of solutions to multiple. For example, in logistics you need to transport goods from A to B. More specialized, transporting food may be a bit different than transporting steel coils. More abstract or generic functionality with even broader adoption.

Carving in or buying products that complete your portfolio

In larger organizations it can happen that a software solution or product is bought along with the department which created it to complete a product portfolio. The same situation applies when two departments which were formerly independent are merged into a single department. The two department’s products will now be under the responsibility of a single head – with the need to optimize cost over all developments. Both parties might have common functionality in their portfolio where the organization may consider consolidation into a shared platform, to reduce development and maintenance cost. Another approach is integration, but that shall not be in focus now.

Leveraging USPs of existing Products

Looking at the history of Amazon Web Services, the platform was born when the architecture for the online web shop of Amazon.com was changed massively from a more monolithic solution to a more micro-service oriented solution. Service integration was required at large scale, which resulted in a messaging infrastructure known as Amazon Simple Queue Service (SQS). Also provisioning of scalable compute, storage and network was required which allowed to rent out free capacities in datacenters to 3^rd parties over the year. This is now the Elastic Compute Cloud (EC2). So, Amazon managed to standardize and externalize core elements of their super-high scale and effective web-shop solution into a platform business. The USPs of these services were and still are their high operational qualities which are valued by millions of users every day. There are more examples from the open source world; e.g. Kafka emerged as an own messaging system product out of development at LinkedIn because they solved the problem of messaging at massive scale.

Leveraging data that was collected in a successful Project

Let’s assume a company the builds control and monitoring systems for larger industrial plants. Within this control and monitoring systems (also called SCADA systems; Supervisory, Control and Data Acquisition), which runs on premises of the plant, there is a huge amount of data that is basically not automatically leveraged yet. Now this company realized that there is a business opportunity to analyze the data and build recommendations from it how to run the plant more efficiently, or with less errors. They build a simple product that collects the data into a central place from the plants (e.g. a cloud based storage like AWS S3), analyze the data for a specific problem to solve and successfully created actionable insights. The customer like it and a new business was born; a great success. Now, the business owners want more, the data that was collected promises more businesses that leverage data analytics so the data shall be re-used. Starting small this company then realizes over time that the data collection, cleansing and storage has a cost which should be shared between the new businesses. Also, data access needs to be controlled and the ever growing data catalog needs to be maintained and curated for effective use. So they start to implement functionality – services around the data pool data does the management and provides common implementations to recurring task that pop up in all the new solutions. They also think of re-using some of the features from the first product in the new solutions. So, they actually started to build a data provider / data distribution platform around their data collection.

Assumed Benefits

The assumed benefits for creating a platform are often the same as for any platform development project. The platform development tries to foster reuse of functionality that is needed more than once in order to reduce maintenance and development cost or build new revenue channels around already developed services and existing data. Maintenance is the costliest part in a software lifecycle and so any duplication of work has direct impact to the economic profit of an organization. Studies show, that 70% to 90% of the cost for operating a software product in the market are generated after the first release, mainly driven by the staff that is required to operate, service, maintain and extend the product. Also, if features are provided as shared components in a platform, new products and solutions are expected to be ready more quickly when built on top of it which also reduces time-to-market. Especially time-to-market is a crucial aspect for many business strategies and so is often highly prioritized. Today, speed and ability to adopt to a changing market is considered as key capabilities of successful companies.

In another direction businesses focus on the technological innovation and leveraging of USPs to build a new business. Such a business decision then is not directly influenced by a comparison on existing functionality over multiple solutions but is driven by the idea of leveraging the company’s USPs from the existing solutions. So the business case is more directed by strengthening the market positioning or entering new market segments, rather than reducing development cost and time-to-market in the first place. Typical examples are platforms that implement a new innovative technology or follow technological trends (e.g. virtualization, cloud computing, IOT, big data, multi core, mobile computing and similar). Nevertheless, reuse is in scope, because also the new technology needs to be used often, in order to compensate the research, development and operational cost. But here externalized, commercial reuse is the primary element.

Intuitively, people find it easy to believe that re-use can lead to substantial cost reductions. As early as 1988 Gaffney et al. set out to study the relationship between the relative costs of re-use and tried to determine the effects of amortizing the costs and benefits across many re-users (in this article, the re-users are called clients or users of the platform). To explain this relationship, they created the Payoff Threshold. The Payoff Threshold tells us how many times we have to re-use a component before we recover the investment we made in order to develop the shared component. Today, it is well understood that the number of re-uses has substantial impact on the question if it was worth to build a reusable component that can be used by developers to build products and solutions. But what not shall be forgotten is that not only creating a reusable component is costly but also to make people re-use it.

Building shared components and services is far more complex that building the same functionality for just one product or solution. This fact has its foundation in the increased efforts in requirements engineering, stakeholder management, testing and addressing a set of far more complex operational requirements. Specifically, the fact that you need to make sure that no one single platform user can cause a negative impact to any other platform users just by (miss-) using it, generates a lot of complex requirements. The exact number of course is dependent on the concrete situation, but you can assume that it takes at least 3-5 times more effort to build such shared services in relation to a single-use of the same feature in a single product.

I will now discuss several aspects that will lead to the idea what makes the platform approach for existing solutions complex and costly and how we can come closer to a realistic Payoff Threshold as mentioned above.

What Platform Development and Operations mean

… or: Being aware of the cost drivers in platform development.

Managing platform scope – Start small with a bold vision

It is essential to be clear about the scope of the target platform, or better: what the start scope is (we call this the initial scope). If used within one organization only, the larger the platform becomes and the more organizations and people are impacted, the more it becomes a product line approach. Beside the pure technical challenges, also the organizational and human-centric challenges become more and more substantial to the success of such a platform development project.

The answer to this question can be derived from a sound business planning and the business strategy and/or a bold vision. Business owners, product owners, R&D managers and project leads should agree on the business scope of the platform in order build the foundation for the technical scoping. This step is essential also to early identify the involved stakeholders and the monetary impact to different departments in an internal re-use scenario.

It gets easier though when thinking about a service business and start small. Sometimes it is easier to serve external customers then internal departments, due to a much clearer contractual relationship. When you consider how AWS grew, it was a platform with only a hand full of services in the beginning. In 2011, AWS released over 80 significant services and features, the year after in 2012 nearly 160 – and in 2017 AWS already launched 1,430 new services and features. Think about how successful AWS would have been if all those many features and service were considered into a large concept at day one. It would have failed.

In the favor of agility and avoiding to much complexity in the beginning, it should be considered to also start small in your platform approach. Discussing all possible features and variations upfront will lead to a fairly large scope which is hard to fund, hard to implement and hard to manage. Starting small, grow over time – with a bold vision in mind – should also apply for platform development. Which does not mean that you should not have a bold vision and be clear about the key architectural requirements that drive your decision making.

Separation of Infrastructure vs. Business Features

Most systems implement a layered architecture. From the bottom upward the features get more and more domain specific and finally provide the actual business services that hold the customer value. The ultimate goal of any software project is to provide these high value business features which contain the differentiators and market segment relevant assets.

Nevertheless, every system needs supporting functions and some base infrastructure to function properly. The pyramid on the right illustrates a typical setup. Beginning with the basic operating system, or compute capacity to be more generic, we need infrastructure for data persistence and data exchange, runtime containers, service integration, configuration and other tools to finally create our business relevant features.

The potential of a software component or feature that is in the lower part of the pyramid to be re-used is quite high. The lower parts of the pyramid are of general purpose while features get more specialized the higher we move up in the pyramid. Highly specialized features are typically rarely re-used by other applications unless they can be configured for related and similar use cases. Here is an example of feature clustering from the logistics domain that would map to such a layering:

As defined above, every layer in this schema could be designed as a platform for the next layer. And so every layer needs a dedicated scope and a set of features it provides.

When planning for a platform that is distilled from existing solutions, it is crucial to separate infrastructure features from business features in the discussions. It appears logical that it could be a goal to realize as much business features as possible into the platform, to gain a maximum benefit over multiple applications. But this might be paid off by the flexibility that is required in these features. As a general rule, it is much harder to re-use specialized features than more generic features, but at the same time those more generic features may be less valuable because there are many options available to replace them and they do not solve customer problems.

It will be more and more difficult to provide features in the upper layers of this pyramid that fit the requirements of multiple applications or solutions. This can manifest either in the diversity of functional requirements or, and this is the more difficult case, in the diversity of non-functional – or better: operational and developmental – requirements. This is also somehow natural; otherwise it would not make sense to build different products or solutions at all. They differ in their business features; it is just the question how much. This is why really successful platforms invest a lot into providing very high standards in operational and developmental requirements like security, resilience, performance, cost position, scalability, compatibility, ease of use and extensibility to increase the user base and so the amount of client solution/products in a sustainable way.

So, a clean decision will be required how much of the existing or future business features can be generalized to become part of the platform. The generalization efforts require configurability and flexibility as new qualities in the existing features which typically increase the solution complexity and so the cost for the feature provisioning. In the most cases it is advisable to start with a harmonization of the infrastructure and work up the pyramid step by step. In the simplest case, the platform provides pure infrastructure services and some general tools. On the other end one can think of a fully configurable product base platform that covers 90%+ of the application features. Thinking of the car manufacturing domain, there you find examples of highly configurable product platforms that allow a portfolio of products with little engineering overhead per new product – as long as it is in scope of the platform’s configurable feature set.

In a cloud based migration effort, with the usage of AWS, the basic first step around base infrastructure and middleware can be done in very short time, by standardizing your application portfolio on the base of AWS infrastructure, databases, storage, messaging and more, your entry level in the pyramid above can be substantially higher. This generates more value out of your engineering hours immediately and reduces the amount of heavily lifting significantly.

Major efforts around the obvious feature development

When transforming the applications and solutions in an organization from individuals to solutions based on a shared platform, not only technical work is required in writing and changing software.

There is set of aspects and issues that need to be taken care of beside the pure technical development of platform features. The establishment of a platform that has multiple clients (internal and external users of the platform) comes along with a new set of task, liabilities and responsibilities that all lead to increased efforts in the platform project. Also the organization and the processes around the platform development will need special attention and proper adjustments. In product line engineering, these topics are also discussed intensively and PLE also provides several best practices in the area. An example is the maturity of an organization to drive global optimizations over platform and application / solution projects.

Here is a short summary of the non-technical aspects:

• Ensuring that the platform development is business driven & strategy aligned
  • Make sure that the key persons (e.g. architects, key developers) understand the client’s domains and their cost drivers and the platforms strategic rational
  • Do not spend effort on things that do not contribute to the business use cases (explicit exclusion)

• Creating proper awareness of sponsors expectations and political forces
  • Every sponsor or client may have other expectations, be aware that you cannot serve all at the same time. Specifically, if you serve internal customers only, politics can be a significant issue.
  • Expect that the clients will generate conflicting requirements for the platform
  • Balance the different client’s demand and the development capacity of the platform team – make sure that client’s deadlines to do not impact the platform roadmap (and pressure on the team) too much
  • Incremental approach, continuous delivery is advised – foster early feedback

• Prioritizing platform feature quality over platform feature completeness
  • Prioritize, prioritize, prioritize together with all clients (they have all different ideas of what is important)
  • Embrace controlled change, priorities and scope will change due to internal or external forces
  • Operational quality is more important than a broad feature set
  • Be able to always (!) deliver a product-quality release

• Enforcing real architectural governance
  • Install a change control board which decides on prioritization (roadmap) for all requested changes, in small independent teams this can be as small as the Product Owner and the Key Developer/Architect.
  • Even under high pressure, do not allow quick changes before clarification of the impact to other clients is done – rollbacks are very expensive

• Putting a focus on the usability of the Platform on developer level (developer habitability)
  • Provide developer level information for effective and efficient work with the platform
  • Provide tools and concepts, focus on client’s productivity
  • Once an interface is released and used, you will not be able to change it easily
  • Most platforms are designed to allow building product/solutions, so the developers and builders are the most important stakeholder! Act accordingly.

• Hardening of platform against malfunctioning plug-ins and clients
  • When providing a framework to build and plug-in additional functionality, ensure the platform stability
  • Expect that plug ins, apps, solutions and on-top products will degrade the platform quality (e.g. performance, stability) for other users and clients and prevent this
  • Implement measures to securely handle plug ins and API usage, which includes a checking and release process for externally developed modules
  • Focus on multi-tenancy and tenant isolation, not only from security standpoint but also from a performance isolation standpoint, e.g. reducing performance degradation blast radius of resource intensive clients.

• Providing Training, Service and Support for developers and users
  • Provide a variety of options to build platform knowledge on client side and do ongoing consulting
  • Install a variety of options for reactive support
  • Install architectural guidance during early client project phases, e.g. solution architects
  • Include features in the platform that support the support activities

• Balancing the coupling of platform team and Product / Solution teams
  • The platform team should keep constant contact with the clients to understand needs and problems and to provide updates on roadmaps and news
  • At the same time the platform teams should be exposed to operational issues to build sustainable quality improvements
  • But you need to be able to also develop further, so support people are required to help customers on standard problems and architectural guidance
  • Defend the platform goals and make sure that platform developers are not firefighting problems in the client’s releases, unless it concerns platform issues and flaws that prevent client releases.

Efforts due to more complex development processes and managing dependencies

The following applies when the platform is created with a set of known client and products/solutions are designed on top of a platform within the same organization.

PLE defines two different development processes: The domain engineering (creating the platform) and the application engineering (creating the solutions). With the introduction of the platform into existing and new solutions, the solutions’ development will now depend on the releases from the platform team. It sounds logic and simple, but this fact shall not be underestimated. Specifically, this becomes a problem, when a (set of) product/solution is explicitly designed to make use of planned platform features and both, platform and product/solution are developed at the same time.

First of all, a development team along with responsibilities for the platform needs to be created. The team needs to establish a platform development process which also takes into consideration what the solutions require in terms of release cycles, testing periods and roadmaps. The linkage needs to be synchronized and they will have dependencies. An explicit management of these issues is vital to the success of the platform and the solutions that depend on it.

One solution to it, is to drive the platform as a product. Optimally, your platform organization consists of a set of teams, each developing and operating platform services independently. They release when they release and users of the platform just work on what’s available after every release. No over promising on roadmaps. However, ever solution / product builder team will have to make decisions around all the features your platform does not provide (yet). Do I implement this myself or do I wait for the platform to solve this problem? This question raises the “when will the platform deliver” issue and can generate a massive pressure on both sides when not handled right. Be prepared.

The idea of managing the platform as an internal product comes along with a few principles:

• When discussing with users and clients about platform features, you mainly discuss based on what the platform can do today. Not what it might be able to do next year
• When discussing the future, do not promise anything that is beyond a 3-month horizon and only discuss features that are being implement and you have a good maturity in. Note – your priorities will most properly change often and quickly, so do not over promise
• The platform usage has a price, and your clients have full clarity what they get for it. The judgment if a features should be used from platform side vs. implementing it into the solution must be backed by actual to be expected operational cost for the solution team
• Platform / solution builders should not plan their releases on features that are only planned to be delivered from the platform.

Of course, in an internal software platform, where the platform team shares the same lunch table with the solution team, it is reasonable to discuss about future platform features and consider to not implement features into the solution that will be delivered by the platform short after. Still, the amount of influence into the platform team increases with the amount of clients they have, so changes in priorities and direction may come quickly. As solution team, you should always plan only what the platform provides today, and receive future features of the platform as a gift, which you may or may not accept in incorporate into your solution.

Data Platforms – more than just large data pools

Data platforms are not a new phenomenon but it seems that the need to build re-usable assets around collected data is growing in a rate we have not seen before. Specifically, IOT, based on the data collection from field devices, is a driving factor. Once data is collected from devices and numerous other systems, the data should be used in the most effective way so that new business features and even completely new business models may prosper. The data collections quickly grow into hundreds of Gigabytes or even Petabytes and, with the right tools and proper research, this data contains massive valuable information. Data Lake is an often used term in this context, depicting a place where all this mass data, structured and unstructured, is stored in cost effective fashion, waiting for more and more use cases that generate valuable insights from them.

We will not discuss Big Data or Data Lake concepts here since this is a problems space for its own. However, under the assumption that data is not just sitting in one place and people are just randomly using it somehow, data platforms also trend to fall under the same challenges as traditional platform development. With one additional complexity: mass data access.

Why is that the case? Small and large data pools that shall be used among a set of users slip into a growing space of functional requirements. Storing and curating data is not free of cost, so a demand rises to control data access or even bill users for the use. Also, there are recurring problems when products / solutions want to make use of the data. Here is a list of functional building blocks which can be found in many data centric platforms:

• Data access management, defining who is allowed to read / write which data. This can be on every possible level, e.g. data collection, object, file, row, column, … or a combination of such. With ever growing data and more and more users that should use it, the management of the access can be a cumbersome and error prone operational task. Depending on the restrictions and security requirements around the data, a substantial problem domain. The solution to this problem often directly depends on the underlying data storage technologies, since not all technologies support all desired authentication & authorization concepts. The automation plays a significant role here, specifically when a diverse set of technology is used to store mass data; e.g. a mixture of files, databases and data warehouses.
• Data access control, enforcing the access rights on data storage technology level. This includes access control logging. In some environments it might be necessary to be able to proof who accessed which data (e.g. GDPR relevant data).
• Data at rest and transit security, e.g. encryption
• Data loss protection, e.g. disaster recovery implementations
• Meta data management and data catalogs. If use of data in multiple use cases is a goal, then it is often required to ease the finding of relevant data and to be able to navigate the data pool. Meta data management deals with the semantic content and the structure of the data in the data collection, storing this information in a central place and allow users to browse this information. Which relations exist in my data, where do it find data of devices XYZ, how is data reflecting device hierarchies, do mean “temp” and “temperature” the same in these 2 data sets … Such and many more different questions may arise. This is specifically important when new use cases are in an exploration phase. Several products exist that specifically try to address this problem with standard data models and highly automated meta data extraction and catalog features.
• Data ingest, transformation, cleansing and loading. Many organizations what to solve the problem of ingesting and integrating data from multiple sources in a single place. Not all products should cope with this problem. Data should be provided in the central data pool in a way that shows high quality with well managed meta data around it. Also here, we see a large list of companies that provide software product around this specific problem.
• Cost effective mass storage. One very common feature is, that data that is only regularly used should be stored in cheaper (and less performant) data stores to reduce operational cost. This concept is often called “tiered storage”, which multiple storage tiers that provide different availability, performance and costs combinations. Also “temperature models” are often considered here, differentiating data between hot, warm and cold states which relate to how often the data is queried per time frame.
• Use case optimized storage. In contrast to cost optimization, data platforms also want to provide data query performance optimized data storage. So, depending on the data query behavior of clients, different storage technologies may be used and data may be existing multiple times in multiple optimized forms. To achieve this, data pipelines need to be maintained that transform data from its raw format to one or more target formats and store the consumable data into a dedicated storage technology (e.g. an object store, database or other). The variability of the client’s data query behavior over all clients of the platform is a central complexity and cost driver. Optimizing for performance often times also consume considerable resources on infrastructure level (e.g. CPU power, network bandwidth, drive IO and RAM). Also, the different storage technologies may require to make use of their specific performance optimization features which requires highly specialized people.
• Data analytics and machine learning tools and frameworks. In order to make data useable for a concrete use case it is first required to analyze the data and find ways to generate actionable insights from it. Machine learning is also often utilized here and custom model training is used to calculate valuable inference from data sets and streams. It occurs that setting up such environments can be complex and requires specialized knowledge, so embedding such features into the platform can shorten the required time to build new products and solutions dramatically. The AWS platform provides a service called Amazon Sagemaker that does exactly this, it encapsulates the machine learning environments into an easy to use service. Building, training and deploying machine learning models so becomes quite easy.
• Data visualization tools and frameworks. When every of your products wants to explore and show data graphically to end-users, then it is considered to embed a generalized visualization solution into the platform.

Of course there is more to mention. All this functionality is designed to shorten the time for commercial use of the data, manage the data and its security, reduce the cost to store it, provide the data in a most optimal way to the users – the solution / product builders. So, the fundamental challenges are the same as in the classic platform development because you also need to consider which of all this functionality should be part of the platform, and which not. Where should I build myself vs. using a platform feature vs. using a 3^rd party product or service? How to deal with variability specifically in the area of operational requirements?

Data platforms however have one more complexity driver: the mass data interface. While traditional platforms, with a focus on services that expose functionality along with some smaller (ok, that’s a rather blurry definition, agreed) data, data platforms also need to provide access to large amounts of data. Thinking of API calls and the amount of data that you can return in a single response – maybe some Megabytes – you need to define ways around that problem if you want to make Gigabytes and Petabytes available. So, data platforms typically provide two classes of interfaces: 1) functional, service interfaces and 2) mass data interfaces.

Dealing with large amounts of data is a complex problem to solve, and it needs good consideration how data volume, data velocity and data variety is managed. The more different use case you want to work on top of your data, the more complex data query optimization and mass data access requirements you will encounter. Here are just some keywords as examples for different data query requirements:

• Query volume and velocity
• Use data behind a service vs. raw data access
• Predefined vs. custom queries
• Structured vs. unstructured data
• Type and volume of required data calculations at query time
• Type of traversable data relations
• Compression, Aggregation
• Explorative analysis vs. static
• Batch vs. real-time data access
• Required data formats and types
• On-demand data integration
• Protocol support, e.g. available and supported drivers in a given technology

Summary

Determining realistic economic benefits and risks for a platform approach out of existing solutions, products or unique technology is quite complex and requires a sound analysis of the exiting software. Even if the functional view indicates a high potential of reuse, the operational and developmental qualities that are required plus the additional efforts to manage dependencies and stakeholders may still be contradicting. At the same time, platforms promise highly relevant business cases when you manage to maximize the re-use of the platform. Scaling to thousands and millions of users is where the goals should be.

When we plan to change the rules for solution and product development within one organization, we are well advised in not forgetting the people which are involved. Considerable efforts should be planned to involve all stakeholders early, to make them part of the decision making and to motivate the approach based on a solid business case for the platform. Developer habitability shall not be underestimated when it is about the acceptance of the platform in the development teams that shall use the platform.

As a rule of thumb, the diverse PLE literature claims that a platform feature will cost about 3 to 5 times more as the very same feature when it was developed as direct part of a concrete solution or product. This is due to the increased quality demands and the implemented variability and flexibility. However, this number aims at a green field platform approach where no existing applications have to be touched. Extracting existing features from existing solutions may be quicker and less effort might be required if the platform follows the same implementation concepts and uses the same infrastructure. However, this is rarely the case. Typically, a new platform uses innovative or at least more modern infrastructure and concepts so that existing solutions and platforms become less similar on implementation level. With this, the migration efforts on solution side can increase significantly. It is hard to provide a similar rule of thumb for the Payoff Threshold in the case that we want to migrate existing solutions. It all depends on how much structural change is required. The cost for providing the platform will properly not as high as in a green field approach due to the existing code and experience, but the cost to use the platform in existing solutions will out weight the provisioning cost-savings easily. Still, the approach might form a valid business case and is worth to follow – but the complexities shall not be underestimated.