These are the news items I've curated in my monitoring of the API space that are related to deploying APIs and thought worth enough to include in my research. I'm using all of these links to better understand how APIs are being deployed across a diverse range of implementations.16 Jun 2017
I’ve been watching the conversation around how APIs are discovered since 2010 and I ave been working to understand where things might be going beyond ProgrammableWeb, to the Mashape Marketplace, and even investing in my own API discovery format APIs.json. It is a layer of the API space that feels very bipolar to me, with highs and lows, and a lot of meh in the middle. I do not claim to have “the solution” when it comes to API discovery and prefer just watching what is happening, and contributing where I can.
A number interesting signals for API deployment, as well as API discovery, are coming out of Amazon Marketplace lately. I find myself keeping a closer eye on the almost 350 API related solutions in the marketplace, and today I’m specifically taking notice of the Box API availability in the AWS Marketplace. I find this marketplace approach to not just API discovery via an API marketplace, but also API deployment very interesting. AWS isn’t just a marketplace of APIs, where you find what you need and integrate directly with that provider. It is where you find your API(s) and then spin up an instance within your AWS infrastructure that facilitates that API integration–a significant shift.
I’m interested in the coupling between API providers and AWS. AWS and Box have entered into a partnership, but their approach provides a possible blueprint for how this approach to API integration and deployment can scale. How tightly coupled each API provider chooses to be, looser (proxy calling the API), or tighter (deploying API as AMI), will vary from implementation to implementation, but the model is there. The Box AWS Marketplace instance dependencies on the Box platform aren’t evident to me, but I’m sure they can easily be quantified, and something I can get other API providers to make sure and articulate when publishing their API solutions to AWS Marketplace.
AWS is moving towards earlier visions I’ve had of selling wholesale editions of an API, helping you manage the on-premise and private label API contracts for your platform, and helping you explore the economics of providing wholesale editions of your platforms, either tightly or loosely coupled with AWS infrastructure. Decompiling your API platform into small deployable units of value that can be deployed within a customer’s existing AWS infrastructure, seamlessly integrating with existing AWS services.
I like where Box is going with their AWS partnership. I like how it is pushing forward the API conversation when it comes to using AWS infrastructure, and specifically the marketplace. I’ll keep an eye on where things are going. Box seems to be making all the right moves lately by going all in on the OpenAPI Spec, and decompiling their API platform making it deployable and manageable from the cloud, but also much more modular and usable in a serverless way. Providing us all with one possible blueprint for how we handle the technology and business of our API operations in the clouds.
Twilio launched their Twilio Function this last week, localizing serverless infrastructure for Twilio API consumers, when it comes to powering key functionality that Twilio brings to the table. This seems like a logical move for mature API providers, keeping in tune with shifts in how developers are integrating with APIs, and deploying their applications in a DevOps, continuous integration world.
I could see other API providers following Twilio’s lead, jumping on the serverless bandwagon, and localizing compute within their API ecosystems. I can see this approach converging with other movements in the SDK space where service providers like APIMATIC are enabling the continuous deployment of SDKs, samples, and other scripts for API integration. Allowing developers to quickly deploy integration scripts, in the programming language of choice–all baked into their existing API platform developer arrangement.
It makes sense that some of these common approaches that are emerging across the API space like containerization, webhooks, serverless, evented and other real-time technologies make their way to being baked in, or at least augmenting existing API operations. I don’t think that every API provider should be following Twilio’s lead in every area, but they do provide a pretty interesting example consider when we think about where the API space might be headed–I find the most mature API providers are just as important to keep an eye on as much as each wave of startups.
I’ll keep an eye on serverless being localized like this with other API providers. It seems like an opportunity for some provider, to develop a white label solution to help API providers deliver scripting, events, webhooks, and other emerging ways to orchestrate and integrate with APIs like Twilio is doing.
I am working on an API definition and design guide for my human services API work, helping establish a framework for approaching API design as part of the human services data and API specification, but also for implementers to follow in their own individual deployments. Every time I work on the subject of API design, I’m reminded of how far behind the API sector is when it comes to standardizing what it is we do.
Every month or so I see a new company publicly share their API design guide. When they do my friend Arnaud always adds to his API Stylebook, adding it to the wealth of information available in his work. I’m happy to see each API design guide release, but in reality, ALL API providers should have an API design guide, and they should also be open to publishing it publicly, showing their consumers they have their act together, and sharing with the wider API community the best practices in play.
The lack of companies sharing their API design practices and their API definitions is why we have such a deficiency when it comes to common API patterns in use. It is why we have so many variations of web APIs, as well as the underlying schema. We have an API industry because early practitioners like SalesForce, Amazon, eBay, Flickr, Delicious, Twitter, Youtube, and others were open with their API operations. People emulate what they see and know. Each wave of the API sector depends on the previous wave sharing what they do publicly–it is how this all works.
To demonstrate even further about how deficient we are, I do not find companies sharing their guides for API deployment, management, testing, monitoring, clients, and other stops along the API lifecycle. I’m glad we are seeing an uptick in the number of API design guides, but we need this practice to spread to every other stop. We need successful providers to share how they deploy their APIs, and when any company hires a new developer, you should ALWAYS be given a standard guide for deploying, managing, testing, as well as designing APIs.
It’s not rocket science, and honestly, it’s not even technical. It just means pausing for a moment, thinking about how we approach each stop in the API lifecycle, writing up an overview, publishing, and sharing it with API stakeholders, and even the wider API community. Every company doing APIs in 2017 should be crafting an API design guide so you can get to work on guides for the other areas of your lifecycle, thinking through and standardizing your approach, and making it known to every person involved–ideally, you are also being very public about all of this, and sharing your work with me and Arnaud, so we can get the word out about the good stuff you are up to! ;-)
I am in the process of eliminating the MySQL backend behind much of my research, eliminating a business expense, as well as an unnecessary complexity in my architecture. There really is no reason for the data I use in my business to be in a database. Nothing I track on tends to go beyond 10K rows, with most of the tables actually being less than 100 rows–perfect for spreadsheets, and my new static approach to delivering APIs, and websites for my research.
The time had come to update some of the products on my website, and I thought my product page was a perfect candidate for this approach, providing me with the following elements:
This simple approach to publishing static APIs using Google Sheets and Github is working well for little data like this–I am all about the little data, while everyone else is excited about big data. ;-) I even have the beginning of some documentation and an updated APIs.json for my website.
Next, I’ll work through the rest of my projects, organizations, tools, and other data I track on as part of my API research. I’ll be publishing a complete snapshot of this data at API Evangelist, as well as subsets of it at each of the individual research projects. When I’m done I’ll have a nice static stack of APIs for all of my research, easily managed via Google Sheets, and YAML on Github.
I saw the news that Google’s Spanner Database is ready for prime time, and I wanted to connect it with a note I took at the Google Analyst Summit a few months back–that gRPC is the heart of the database solution. I’m not intimate with the Spanner architecture, approach, or codebase yet, but the API focus, both gRPC core, and REST APIs for a database platform are very interesting.
My first programming job was in 1987, developing COBOL databases. I’ve watched the database world evolve, contributing to my interest in APIs, and I have to say Google Spanner isn’t something I anticipated. Databases have always been where you start deploying an API, but Spanner feels like something new, where the database and the API are one, and the way the database does everything internally and externally is done via APIs (gRPC).
Now that Spanner Database is ready for prime time, I will invest some more time in standing up an instance of it and get to work playing with what is possible with the REST APIs. I also want to push forward my grPC education by hacking on this side of the database’s interface. Spanner feels like a pretty seismic shift in how we do APIs, and how we do them at scale–when you combine this with the elasticity of the cloud, and the simplicity of RESTful interfaces I think there is a lot of potential.
I have been profiling the Microsoft Azure platform over the last couple of weeks, and I found their approach to talking about the regions that were available was worth taking note of. I haven't actually assessed who has more regions, but Azure's approach seems to be pretty advanced, even if AWS might possess more regions (gut feeling). By profiling these cloud services and their available APIs using OpenAPI I am hoping to eventually develop a machine-readable approach to comparing which providers are available within which regions.
Google has a regions page, but it doesn't feel as forward leaning as AWS and Azures. It is interesting to watch how each of these providers is handling the availability of API services in a variety of regions across North and South America, Europe, Asia, Africa, and the Middle East. I've been watching how providers are thinking about the availability of API resources in different geographic regions for a while, but after seeing Azure evolve in this area, it is something I'll keep a closer eye on it moving forward.
Increasing the number of available regions is definitely the biggest concern for providers, something that small providers will be able to piggyback on and expand using as the top cloud providers grow and expand their regions. API providers and API service providers should be expanding the number of regions available, but everyone involved needs to also get more organized about how they communicate with customers about which regions available--region availability should be communicated at the highest level, like we see with the AWS, Google, and Azure deployment pages, but should also work to articulate which regions are available at the individual API level.
As data and algorithmic nationalism continue to grow, we are going to see more focus from providers when it comes to enabling their customer's deployment and operation of APIs into exactly the region they need. I"m guessing with the evolution of software-defined networking (SDN), we are going to see more control over the transport and routing of the data, content, and other resources we are making available via our regionally deployed APIs. Along with other channels and building blocks that I tune into I will start working to define a schema for tracking on regions, allowing me to index which APIs are available in specific regions using APIs.json.
I have been talking about selling wholesale APIs for some time now, allowing your potential customers to pick and choose exactly the API infrastructure they need, and develop their own virtualized API stacks. I'm not talking about publishing your retail API into marketplaces like Mashpe, I'm talking about making your API deployable and manageable on all the major cloud providers.
You see this shift in business with a recent AWS email I got telling me about multi-year contracs for SaaS and APIs. Right now there are 70 SaaS products on AWS Marketplace, but from the email I can tell that Amazon is really trying to expand it's API offerings as well. When you deploy an API solution using the AWS Marketplace, and a customer signs up for a one, two, or three year contract, they don't pay for the underlying AWS infrastructure, just for the SaaS, or API solution. I will have to expore more to see if this is just absorbed by the API operator, or AWS working to incentivize this type of wholesale API deployment in their marketplace, and locking in providers and consumers.
I'm still learning about how Amazon is shifting the landscpe for deploying and managing APIs in this wholesale, almost API broker type of way. I recently came across the AWS Serverless API Portal, which is meant to augment the delivery of SaaS or API solutions in this way. With this model you could be in the business of deploying API developer portals for companies, and fill ingthe catalog with a variety of wholesale API resources, from a varietiy of providers--opening up a pretty interesting opportunity for white label APIs, and API brokers.
As I'm studying this new approach to deploying and managing APIs using marketplaces like this, I'm also noticing a shift towards deliving more algorithmic APIs, with machine learning, artificial intelligence, and other voodoo as the engine--resulting in a shift towards machine learning API marketplaces. I really need to carve off time to think about API deployment and management in this way. I've already begun looking at what it takes to deploy bare bones, wholesale APIs using AWS, Google, Heroku, or Azure clouds, but I really haven't invested much in the business side of all of this, soewhere Amazon seems to be slightly ahead of the curve in.
I have several volunteers available to do work on Open Referral's Human Services Data Specification (API). I have three developers who are ready to work on some projects, as well as an ongoing stream of potential developers I would like to keep busy working on a variety of implementations. I am focusing attention on the top four cloud platforms that companies are using today: AWS, Azure, Google, and Heroku.
I am looking to develop a rolling wave of projects that will run on any cloud platform, as well as taking advantage of the unique features that each provider offers. I've setup Github projects for managing the brainstorming and development of solutions for each of the four cloud platforms:
- AWS - A project site outlining the services, tooling, projects, and communication around HSDS AWS development.
- Azure - A project site outlining the services, tooling, projects, and communication around HSDS Azure development.
- Google - A project site outlining the services, tooling, projects, and communication around HSDS Google development.
- Heroku - A project site outlining the services, tooling, projects, and communication around HSDS Heroku development.
I want to incentivize the develop of APIs, that follow v1.1 of the HSDS OpenAPI. I'm encouraging PHP, Python, Ruby, and Node.js implementations, but open to other suggestions. I would like to have very simple API implementations in each language, running on all four of the cloud platforms, with push button (or at least easy) installation from Github for each implementation.
Ideally, we focus on single API implementations, until there is a complete toolbox that helps providers of all shapes and sizes. Then I'd love to see administrative, web search, and other applications that can be implemented on top of any HSDS API. I can imagine the following elements:
- API - Server-side implementations, or API implementation using specialized services available via any of the providers like Lambda, or Google Endpoints.
- Validator - A JSON Schema, andany other suggested validotr for the API definition, helping implementations validate their APIs.
- Admin - Develop an administrative system for managing all of the data, content, and media that is stored as part of an HSDS API implementation.
- Website - Develop a website or application that allows data, content, and media within an HSDS API implementation to be searched, browsed and engaged with by end-users.
- Mobile App - Develop a mobile application that allows data, content, and media within an HSDS API implementation to be searched, browsed and engaged with by end-users via common mobile devices.
- Developer Portal - Develop an API portal for managing and providing access to an HSDS API Implementation, allowing developers to sign up, and integrate with an API in their web, mobile, or another type of application.
- Push Button Deployment - The ability to deploy any of the server side API implementations to the desired cloud platform of your choice with minimum configuration.
I'm looking to incentivize a buffet of simple API-driven implementations that can be easily deployment by cities, states, and other organizations that help deliver human services. They shouldn't be too complicated or be trying to do everything for everyone. Ideally, they are simple, easily deployed infrastructure that can provide a seed for organizations looking to get started with their API efforts.
Additionally, I am looking understand the realities of running a single API design across multiple cloud platforms. It seems like a realistic vision, but I know it is one that will be much more difficult than my geek brain thinks it will be. Along the way, I'm hoping to learn a lot more about each cloud platform, as well as the nuance of keeping my API design simple, even if the underlying platform varies from provider to provider.
I'm always on the hunt for new ways to define, design, deploy, and manage API infrastructure, and thought the AWS Cloud Formation Designer provides a nice look at where things might be headed. AWS CloudFormation Designer (Designer) is a graphic tool for creating, viewing, and modifying AWS CloudFormation templates, which translates pretty nicely to managing your API infrastructure as well.
While the AWS Cloud Formation Designer spans all AWS services, all the elements are there for managing all the core stops along the API life cycle liked definition, design, DNS, deployment, management, monitoring, and others. Each of the Amazon services is available with a listing of each element available for the service, complete with all the inputs and outputs as connectors on the icons. Since all the AWS services are APIs, it's basically a drag and drop interface for mapping out how you use these APIs to define, design, deploy and manage your API infrastructure.
Using the design tool you can create templates for governing the deployment and management of API infrastructure by your team, partners, and other customers. This approach to defining the API life cycle is the closest I've seen to what stimulated my API subway map work, which became the subject of my keynotes at APIStrat in Austin, TX. It allows API architects and providers to templatize their approaches to delivering API infrastructure, in a way that is plug and play, and evolvable using the underlying JSON or YAML templates--right alongside the OpenAPI templates, we are crafting for each individual API.
The AWS Cloud Formation Designer is a drag and drop UI for the entire AWS API stack. It is something that could easily be applied to Google's API stack, Microsoft, or any other stack you define--something that could easily be done using APIs.json, developing another layer of templating for which resource types are available in the designer, as well as the formation templates generated by the design tool itself. There should be an open source "API formation designer" available, that could span cloud providers, allowing architects to define which resources are available in their toolbox--that anyone could fork and run in their environment.
I like where AWS is headed with their Cloud Formation Designer. It's another approach to providing full lifecycle tooling for use in the API space. It almost reminds me of Yahoo Pipes for the AWS Cloud, which triggers awkward feels for me. I'm hoping it is a glimpse of what's to come, and someone steps up with an even more attractive drag and drop version, that helps folks work with API-driven infrastructure no matter where it runs--maybe Google will get to work on something. They seem to be real big on supporting infrastructure that runs in any cloud environment. *wink wink*
Establishing a feedback loop with your API community is one of the most valuable aspects of doing APIs, opening up your organization to ideas from outside your firewall. When you are designing new APIs or the next generation of your APIs, make sure you are tapping into the feedback loop you have already created within your community, by providing access to the alpha, beta, and prototype versions of your APIs.
The Oxford Dictionaries API is doing this with their latest additions to their stack of word related APIs, by providing early access for their community with two of their new API prototypes that are currently in development:
- The Oxford English Dictionary (OED) is the definitive authority on the English language containing the meaning, history, and pronunciation of more than 280,000 entries – past and present – from across the English-speaking world. Its historical record of the English language is traced through more than 3.5 million quotations ranging from classic literature and specialist periodicals to film scripts and cookery books.
- bab.la offers quick and easy translations and answers to everyday language questions. As part of the Oxford Dictionaries family, it provides practical support to people using a language that is not their mother tongue.
To get access to the new prototypes, all you have to do is fill out a short questionnaire, and they will consider giving you access to the prototype APIs. It is interesting to review the questions they ask developers, which help qualify users but also asks some questions that could potentially impact the design of the API. The Oxford Dictionaries API team is smart to solicit some external feedback from developers before getting too far down the road developing your API and making it available in any production environment.
I do not think all companies, organizations, and government agencies have it in their DNA to design APIs in this way. There are also some concerns when you are doing this in highly competitive environments, but there are also some competitive advantages in doing this regularly, and developing a strong R&D group within your API ecosystem--even if your competitors get a look at things. I'm going to be flagging API providers who approach API development in this way and start developing a list of best practices to consider when it comes to including your API community in the design and development process, and leveraging their feedback loop in this way.
I'm endlessly fascinated by APIs and enjoy studying their evolution. One of the challenges in helping evangelize APIs that I come across regularly is the many different views of what is or isn't an API amongst people who are API literate, as well as helping bring APIs into focus for the API newcomers because there are so many possibilities. Out of the two, I'd say that dealing with API dogma is by far a bigger challenge, than explaining APIs to newbies--dogma can be very poisonous to productive conversations and end up working against everyone involved in my opinion.
I'm enjoying reading about the evolution in the API space when it comes to GraphQL and gRPC. There are a number of very interesting implementations, services, tooling, and implementations emerging in both these areas. However, I do see similar mistakes being made regarding dogmatic behavior, aggressive marketing tactics, and shaming folks for doing things differently, as I've seen with REST, Hypermedia, and linked data efforts. I know folks are passionate about what they are doing, and truly believe their way is right, but I'm concerned you will all suffer from the same deficiencies in adoption I've seen with previous implementations.
I started API Evangelist with the mission of counteracting the aggressive approach of the RESTafarians. I've spent a great deal of time thinking about how I can turn average developers and even business folks on to the concept of APIs--no not just REST or just hypermedia, but web APIs in general. Something that I now feel includes GraphQL and gRPC. I've seen many hardworking folks invest a lot into their APIs, only to have them torn apart by API techbros (TM) who think they've done it wrong--not giving a rats ass regarding the need to actually help someone understand the pros and cons of each approach.
I'm confident that GraphQL will find its place in the API toolbox, and enjoy significant adoption when it comes to data-intensive API implementations. However, I'd say 75% of the posts I read are pitting GraphQL against REST, stating it is a better solution. Period. No mention of its limitations or use cases where it might not be a good idea. Leaving us to only find out about these from the GraphQL haters--playing out the exact same production we've seen over the last five years with REST v Hypermedia. Hypermedia is finding its place in some very useful API implementations like FoxyCart, and AWS API Gateway (to name just a few), but its growth has definitely suffered from this type of storytelling, and I fear that GraphQL will face a similar fate.
This problem is not a technical challenge. It is a storytelling and communication challenge, bundled with some very narrow incentive models fueled by a male-dominated startup culture, where folks really, really like being right and making others feel bad for not being right. Stop it. You aren't helping your cause. Even if you do get all your techbros thinking like you do, your tactics will fail in the mainstream business world, and you will only give more ammo to your haters, and further confuse your would be consumers, adopters, and practitioners. You will have a lot more success if you are empathetic towards your readers, and produce content that educates, and empowers, over shames and tearing down.
I'm writing this because I want my readers to understand the benefits of GraphQL, and I don't want gRPC evangelists to make the same mistake. It has taken waaaay too long for linked data efforts to recover, and before you say it isn't a thing, it has made a significant comeback in SEO circles, because of Google's adoption of JSON-LD, and a handful of SEO evangelists spreading the gospel in a friendly and accessible way--not because of linked data people (they tend to be dicks in my experience). As I've said before, we should be investing in a robust API toolbox, and we should be helping people understand that benefits of different approaches, and learn about the successful implementations. Please learn from others mistakes in the sector, and help see meaningful growth across all viable approaches to doing API--thanks.
Building on earlier stories about how my API partners are making API deployment more modular and composable, and pushing forward my understanding of what is possible with API deployment, I'm looking into the details of what DreamFactory enables when it comes to API deployment. "DreamFactory is a free, Apache 2 open source project that runs on Linux, Windows, and Mac OS X. DreamFactory is scalable, stateless, and portable" -- making it pretty good candidate for running it wherever you need.
After spending time at Google and hearing about how they want to enable multi-cloud infrastructure deployment, I wanted to see how my API service provider partners are able to actually power these visions of running your APIs anywhere, in any infrastructure. Using DreamFactory you can deploy your APIs using Docker, Kubernetes, or directly from a Github repository, something I'm exploring as standard operating procedure for government agencies, like we see with 18F's US Forest Service ePermit Middlelayer API--in my opinion, all federal, state, and local government should be able to deploy API infrastructure like this.
One of the projects I am working on this week is creating a base blueprint of what it will take to deploy a human services API for any city in Google or Azure. I have a demo working on AWS already, but I need a basic understanding of what it will take to do the same in any cloud environment. I'm not in the business of hosting and operating APIs for anyone, let alone for government agencies--this is why I have partners like DreamFactory, who I can route specific projects as they come in. Obviously, I am looking to support my partners, as they support me, but I'm also looking to help other companies, organizations, institutions, and government agencies better leverage the cloud providers they are already using.
I'll share more stories about how I'm deploying APIs to AWS, as well as Google and Azure, as I do the work over the next couple of weeks. I'm looking to develop a healthy toolbox of solutions for government agencies to use. This weeks project is focused on the human services data specification, but next week I'm going to look replicating the model to allow for other Schema.org vocabulary, providing simple blueprints for deploying other common APIs like products, directories, link listings, and directories. My goal is to provide a robust toolbox of APIs that anyone can launch in AWS, Google, and Azure, with a push of a button--eventually.
I'm keeping an eye on the different approaches to deploying infrastructure coming out of AWS, Google, Microsoft and other providers. In my version of the near future, we should be able to deploy any API we want, in any infrastructure we want with a single push of a button. We are getting there, as I'm seeing more publish to Heroku buttons, AWS and Azure deployment packages, and I recently came across the Google Cloud Launcher, which I think will work well for deploying a variety of API driven solutions--we just need more selection and a button!
All the parts and pieces for this type of push button API deployment exist already, we just need someone to step up and provide a dead simple framework for defining and embedding the buttons, abstracting away the complexities of each cloud platform. I want to be able to take a single manifest for my open source or wholesale API on Github, and allow anyone to deploy it into Heroku, AWS, Google, Azure, or anywhere else they want. I want the technical, business, and legal complexities of deployment abstracted away for me, the API provider.
API management has matured a lot over the last 10 years, and API design and definitions are currently flourishing. We need a lot more investment in helping people easily deploy APIs, wherever they need. I think this layer of interoperability is the responsibility of the emerging API service providers like Restlet, DreamFactory, or maybe even APIMATIC. I will keep tracking on what I'm seeing evolve out of the leading cloud platforms like AWS, Azure, and now Google with their Cloud Launcher. I will also keep pushing on my API service provider partners in the space to enable API deployment like this--I am guessing they will just need a little nudging to see the opportunity around providing API deployment in this seamless, cloud-agnostic way.
There have been many advances in the way that we deploy APIs in the last couple of years, but I still want more of an embeddable, push botton way to deploy generic or even more specialized APIs. This is something I've ranted about before, asking where the deploy to AWS and Google buttons. I'm seeing more AWS solutions emerge, helping deploy from Github using AWS Codeploy, and the regular number of deploy to Heroku buttons, but not the real growth I'd like to see occur--making it a drum I will keep beating until I get what I want.
I was working on my OpenAPI toolbox, cataloging open source tools that put the OpenAPI specification to work, and came across a deploy with Heroku button for the Grape Doorkeeper, which helps you "create an awesome versioned API, secured with OAuth2 and automatically documented". This should be the default for all server-side API deployment frameworks, allowing push button deployment of any open source API framework to the cloud platform of your choosing.
If I have my way, it won't just be API frameworks that will have deployment buttons. Specialized API designs, available in a variety of frameworks will be available for deployment with a single click of a button. We should be able to deploy a product API, or a user API, to AWS, Heroku, Google, or Microsoft, with a single click. There should be a wealth of open source templates for us to choose from on Github, with deploy buttons, and easy to follow wizards that help us set things up properly.
Smells like an opportunity to me. I'll have to think more about where the revenue would come from in such a model, but I'm sure it would be easy enough to upsell deployments to some premium features and services. I understand that both the areas of API design and API deployment are playing catch-up with API management at the moment, but someone needs to get to work on streamlining the API deployment button experience across all major cloud platforms and get to work on crafting some useful API server deployments that people can put to work instantly. #please #thankyou
I was looking through the Github accounts for Amazon Web Services and came across their Serverless API Portal--a pretty functional example of a forkable developer portal for your API, running on a variety of AWS services. It's a pretty interesting implementation because in addition to the tech of your API management it also helps you with the business side of things.
The AWS Serverless Developer Portal "is a reference implementation for a developer portal application that allows users to register, discover, and subscribe to your API Products (API Gateway Usage Plans), manage their API Keys, and view their usage metrics for your APIs....it also supports subscription/unsubscription through a SaaS product offering through the AWS Marketplace."--providing a pretty compelling API portal solution running on AWS.
There are a couple things I think are pretty noteworthy:
- Application Backend (/lambdas/backend) - The application backend is a Lambda function built on the aws-serverless-express library. The backend is responsible for login/registration, API subscription/unsubscription, usage metrics, and handling product subscription redirects from AWS Marketplace.
- Marketplace SaaS Setup Instructions - You can sell your SaaS product through AWS Marketplace and have the developer portal manage the subscription/unsubscription workflows. API Gateway will automatically provide authorization and metering for your product and subscribers will be automatically billed through AWS Marketplace
- AWS Marketplace SNS Listener Function (Optional) (/listener) - The listener Lambda function will be triggered when customers subscribe or unsubscribe to your product through the AWS Marketplace console. AWS Marketplace will generate a unique SNS Topic where events will be published for your product.
This is the required infrastructure we'll need to get to what I've been talking about for some time with my wholesale API and virtual API stack stories. Amazon is providing you with the infrastructure you need to set up the storefront for your APIs, providing the management layer you will need, including monetization via their marketplace. This is a retail layer, but because your infrastructure is setup in this way, there is no reason you can't sell all or part of your setup to other wholesale customers, using the same AWS marketplace.
I had AWS marketplace on my list of solutions to better understand for some time now, but the AWS Serverless Developer Portal really begins to connect the dots for me. If you can sell access to your API infrastructure using this model, you can also sell your API infrastructure to others using this model. I will have to set up some infrastructure using this approach to better flush out how AWS infrastructure and open templates like this serverless developer portal can help facilitate a more versatile, virtualized, and wholesale API lifecycle.
There is a more detailed walkthrough of how to get going with the AWS Serverless Developer Portal, helping you think through the details. I am a big fan of these types of templates--forkable Github repositories, with a blueprint you can follow to achieve a specific API deployment, management, or any other lifecycle objective.
I was playing with a new API design and deployment solution, from some of my favorite developers out there this weekend called Materia, which bills itself as "a modern development environment to build advanced mobile and web applications"--I would add, "with an API heart".
Materia is slick. it is modern. While very simple, it is also very complete--allowing you to define your underlying data model or entities, design and deploy APIs, and then publish a single page applications (SPA) for use on the web, or mobile devices. Even though I'm one of those back to land, hand-crafted API folks, I could see myself using Materia to quickly design and deploy APIs.
I say this in the most positive light imaginable, but Materia reminds me of the Microsoft Access for APIs. Partly its the diagramming interface for the entities, but it is also the fact that it bridges the backend to the frontend, allowing you to not just design and deploy the database and APIs, but also the resulting user interface that will put them to work.
I know they are just getting going with developing Materia, but I can't help but share a couple things I'd like to see, that would make it continue to be the modern API driven application it is striving to be:
- OpenAPI Specs or Blueprints - Allow users to import, export, and manage my APIs in the popular API definition format my choice.
- Schema.org - Provide users with a wealth of existing entity models to choose from, so they do not reinvent the wheel.
- Github - Allow for the publishing of projects, and importing of them to and from Github, allowing for the sharing of server design and deployment patterns.
There are a number of other things I'd like to see, but I'm sensitive to the fact that they are just getting started. These three areas would significantly widen the initial audience for Materia beyond the developer class, which is who the application solution should be targeting. Like I said, it has the potential to be the Microsoft Access of APIs for small businesses, which isn't quite the Microsoft Excel of APIs, but a close second. ;-)
Nice work guys! It is the another positive advancement in the world of API design alongside Restlet launched their API design studio, and Apiary setting this modern era of API design into motion with Apiary. I'll be tuning into Materia's evolution on Twitter, and play more with the server and designer editions available on Github.
In 2014 we are making significant progress in deploying APIs in support of city operations, but we still have so much more work ahead of us when it comes to making public resources available. You can find a dedicated developer area full of data sets, and APIs, in most major US cities like New York, Chicago, San Francisco, Seattle, Philadelphia, Washington D.C. and many more, but what else can we do to really pick up the momentum and quality?
Standardizing API Design Practices
APIs are not that difficult to design with the right education, and experience. Developers who work on city contracts, or are employed by the city should all be taught common web API design practices, and be exposed to modern API design tooling like Swagger and Apiary. Even with this type of education, there will still be many differences between city deployments based upon needs, and tactics, but a little training could go a long way to make city operations more streamlined.
Open Solutions For API Deployment
There are a lot of common approaches to delivering city services, which means there should also a be a number of ways to provide standardized, open solutions for deploying APIs that support city operations. There should be a wealth of open source, Wordpress like solutions for deploying APIs in support of government operations. Sometimes connecting to legacy systems is just to much work, and deploying a simple, standalone solution, then syncing using data dumps or directly with backbend system might be more fruitful.
Common API Management Vision
I’m pretty impressed with the standard approach to deploying city developer areas, and delivering data sets, and APIs, but in reality this is the result of the hard work of Socrata, one of the API management providers dedicated to the government space. I think Socrata, and the other vendors out there are definitely one piece of the puzzle for managing APIs for city operations, but I also think we need other competing, open solutions similar to API Umbrella which is being used across the federal government.
Open Source Tooling Across Cities
When it comes to helping cities better serve their citizens, and save money along the way, I can’t think of a better place to start than by providing common, open source tools for delivering web, and mobile applications on top of city data and APIs. We have to stop re-inventing the wheel for each city when it comes to developing common apps, city needs are going to be very similar across cities--just take a look at solutions like Open311, and the let’s get to work on delivering similiar solutions for every part of city operations.
There is no reason each city should have to go at it alone when it comes to designing, deploying, managing, evangelizing, and putting APIs to work across city operations. We should have standard data models, API definitions, and a wealth of open source tools for cities to put to work.
I don't see APIs as the solution for all cities problems, but I do think that APIs should be common practice for ALL cities. Every city should be publishing all of their data and content in machine readable way, without causing employees any extra work—it should just be part of normal operations.
In the future, all cities will have standard APIs across all cities, and common open source solutions that can be put to work serving citizens in all aspects of city operations. This is how we are going to empower our cities to more with less, and make governing a more inclusive for everyone.
I have been doing a lot of research into the world of financial APIs, specifically looking at some of the larger companies providing APIs that deliver market news, data, corporate profiles, and other data that make markets go round.
As I consider some of the common building blocks that are common across many financial API--real-time data frameworks, and visualization tools are two of the top items that I think will be part of every financial API stack in the future. Almost every API I looked at had some sort of real-time stream, promising data faster, as well as a way to extract meaning from these streams using template, or custom visualizations.
I’m tracking on real-time API services and tools, and i’ve been seeing some of these frameworks, like Firebase getting baked in by default to some API platforms. I am also tracking on visualization tools, I just don’t have the research published as a Github repository yet, like I do with my real-time research.
I will keep tracking on API providers who are doing interesting things with real-time or visualizations, and hopefully be able to publish more examples. I can’t help but think there are some pretty interesting opportunities for open frameworks, and white label solutions for API providers when it comes to real-time, and visualization layers on top of their existing APIs.
The new Heroku Button gets us one step closer to a new age of API deployment, where anyone can deploy the APIs they need without any developer or IT resources. As I’m working on packaging up API designs for my screen capture API, and image manipulation API, this type of approach is what I’m envisioning for all of my APIs in the future—push button API deployment.
You shouldn't have to wait to deploy the API you need. Just as we are beginning to deploy pre-packaged application stacks like Wordpress, and Drupal, we should be able to deploy common API deployments for images, blogs, videos, and much, much more, with a single click of a button. Once any new API is launched it can be configured, and connected to other systems using the API, allowing it to operate as part of larger stack, or stay as completely independent node that just checks in with the mother ship from time to time.
While there will remain a handful of API leaders like Twilio and SendGrid who will have a big presence, many of the APIs in this next wave of API deployment will be smaller, and more transient in nature, taking advantage of current cloud trends around PaaS, and containerization. This new type of APis will possess a self contained blueprint for the OS, database, server-side code, API definition, and even configuration, integration, and automation tooling for the API.
I’m working on Docker definitions for creen capture andimage manipulation AP, as well as other APIs I develop in the future, but first I think any API I deisn shouldl run as a Heroku app, that anyone can deploy in their own account with a single click of a button. It won't take much work to make this happen for any API I deploy, and since 3Scale API infrastructure, which I use to secure my APIs, already runs on Heroku--making securing, and managing my Heroku deployed APis seamless.
Disclosure: 3Scale is an API Evangelist partner.
In my monitoring of the API space, when I started seeing a large number of blog posts, tweets, companies, and other elements I track on get tagged with the same tag over and over, I take notice. My blogging, CRM, and news curation system all have their own tag cloud interface for the week, showing which tags have been applied--so if a tag gets heavy usage, I know it.
Over the last couple of years, I've spun up new research into other areas within the world of APIs, beyond my core design, deployment, management, evangelism, discovery, and integration research. I created separate buckets beyond just provide and consume to track on these new areas, called trends, opportunities, and priorities.
In 2014 it is beginning to seem like each of my trend research areas are getting baked directly into API platforms, ranging from real-time features with Firebase, to reciprocity by default using Zapier. API providers are learning that having a real-time layer, or a reciprocity layer baked into their platform is a good thing, and why reinvent the wheel when you have kick ass solutions like Firebase and Zapier.
It makes sense that API providers would be looking externally to deliver aggregation, real-time, reciprocity, and even voice layers for their API platforms--this stuff is hard, and why spread yourself too thin. Intuit just bought reciprocity provider itDuzzit, and I think we will more providers integrating Zapier into their platforms by default like Nimble did. We'll also see more API platforms bring in Firebase as a real-time layer like Nest did for their Internet of Things (Iot) thermostat API platform.
Overall it seems like a white label solution that any API provider could put to use when considering solutions for aggregation, real-time, reciprocity, voice or even data solutions including spreadsheet connectors, analysis, and visualization, would do well in the space. At the very least, any company looking to step up and provide solutions in these areas, should definitely have a strong partner program like Zapier and Firebase have brought to the table.
I will have to start considering how to migrate aggregation, real-time, reciprocity, voice out of the trends bucket and into either the provide or consume buckets, or maybe both. It would seem that both API providers and consumers need to be educated in these areas, and made aware of what solutions are available.
I’m not that worried about the overall structure of API Evangelist at the moment. That is one of the beautiful aspects of how I architected the site(s), is that each research area lives as its own node on the network, so I can move around, shift as I need to find the right formula—something that helps me in a very fast moving space, where my understanding is constantly shifting and evolving with the swift currents of the APi space.
Photo Credit: Diego Naive
I was taking another look at the Google Cloud Platform yesterday, and stumbled across Google Cloud Endpoints. It was something I saw come across my feeds, but really didn’t give it the time it needed to see what it was all about. With the new Google Cloud Endpoints, Google is making a strong push to be not just an API deployment provider, but their approach also reflects what I’d consider to be an evolution of backend as a service (BaaS) deployment.
I think Google describes their service better than I can do it justice:
Google Cloud Endpoints consists of tools, libraries and capabilities that allow you to generate APIs and client libraries from an App Engine application, referred to as an API backend, to simplify client access to data from other applications. Endpoints makes it easier to create a web backend for web clients and mobile clients such as Android or Apple's iOS.
While much of the Google Cloud Platform offering looks a lot like the cloud offering over at Amazon Web Services, AWS definitely does not have API deployment as a service, baked into their cloud stack, like Google does with Google Cloud Endpoints.
All Google has to do now, is open up the Google Console as a Google Cloud Endpoints management console, giving developers on the Google Cloud Platform the ability to design, deploy, and manage their APIs. Then if Google baked in Google Discovery services for all Google Cloud Endpoints, developers would have a pretty slick discovery layer on top of their cloud API stacks. Hell, all you need then is to allow generation of APIs.json for each collection, and boom you have a pretty complete API design, deployment, management, and discovery platform in the clouds.
Now that Google is added to my API deployment research, I will be keeping a closer eye on what they are doing in respects to being a cloud API platform.
I’m seeing an increase in the number of API deployment services this year, such as startups like StrongLoop and API Spark. These companies are looking to help all of us deploy APIs from common systems, often without the need for IT or programming resources.
The providers I’m seeing emerge are catering to some of the lowest hanging fruit for deploying APIs. The commonly used, and easiest to access systems, that contain the valuable content, data, and media we need to make accessible via APIs.
The common source for many of these API deploy solutions are:
These common information sources, represent the places where the average person in any size company, organization or government agency will be storing their valuable resources. It makes sense that this new wave of API deployment startups would target these services.
If you consider every system integration option that classic API gateways have delivered, it provides a good reference for finding opportunities for building independent API deployment solutions, that if done right, could be a startup all by itself.
Not all companies can afford a full API gateway solution, and their needs are too small for a gateway. There is an emerging opportunity to help people quickly deploy APIs from common sources like spreadsheet, database, file stores, as well as more unique sources like specialty CMS and CRM systems.
Sometimes I like to look at the expanding API universe as a result of the SOA big bang, where all the tools that were in your SOA toolbox, being liberated, and available as independent services that you can get from an increasing number of new API deployment startups.
While at API Craft in Detroit this week I had the pleasure of hanging with two leads on the WordPress(org) development team, and discuss the API strategy for the blogging platform. Andrew Nacin (@nacin), Lead Developer, and Ryan McCue (@rmccue), WordPress Plugin Developer, facilitated an open circle discussion to work through the challenges that WordPress is facing when developing an API for the open source blogging platform.
At face value, I know a number of API developers who will be less than pleased when they hear about a WordPress API, as both PHP, and WordPress are easy targets for developer’s hatred, for generating less than perfect code. ;-) But, in the end you can’t ignore some of the stats on WordPress usage:
- Used by more than 22.0% of the top 10 million websites as of August 2013
- Most popular blogging system in use on the Web, used in more than 60 million websites
- Over 409 million people view more than 15.5 billion pages each month
- Users produce about 41.7 million new posts and 60.5 million new comments each month
These stats don’t even touch on the number of WordPress plugins and mobile solutions there are developed on the insanely popular website, and blogging platform. I couldn't think of a better platform to start rolling APIs out APIs to the masses, and quickly educating a very large group of tech savvy folks along the way.
Many of these WordPress sites are already integrated with popular API platforms like Twitter, Facebook, Google, and Amazon, so it makes sense to educate WordPress administrators on the APIs they are already using, as well as introduce them to the concept of having their own API around the potentially valuable content they generate.
We Are Talking About Developing 60 M+ Distributed APIs
There is no API deployment or management service provider that is even close to numbers of this magnitude. When ready, the WordPress API will give millions of websites, and the individuals, companies, organizations and government agencies behind them, an Application Programming Interface (API). (weird, I haven't spelled that out in a while) What does this mean for the API sector? I do not know yet, but to me it definitely resembles the long tail of API deployment, that I, and 3Scale have been advocating for the last four years.
Building For The Lowest Common Denominator
One dominant theme in the conversation around the WordPress API at API Craft, was about building an API for the lowest common denominator. WordPress is an open source blogging platform built using PHP and MySQL, something that enables it to be installed on just about any hosting platform, which has contributed to the platform's growth. However, many of these platforms restrict common aspects of the Internet like the ability to use all of your HTTP verbs, like PUT or DELETE, or simply do not offer essentials like SSL, which is needed for oAuth. The bar for developing an API that will be used by 60 M+ providers has to be set pretty low, ensuring it will work across the entire ecosystem—something WordPress is already pretty damn good at.
Ensuring That The WordPress API Is Extensible
One of the hallmarks of WordPress is the ability to create custom themes and plugins, that extend the functionality of the blogging platform, and the API will be no different. WordPress is much more than just a collection of pages, posts, tags, and links. WordPress has been used as the core of just about any time of website, serving up books, photos, videos, maps, and any other content type you can dream up. The WordPress API has to be able to provide a common interface that can be used to create, read, update, and delete core WP content types, like pages and posts, as well as be extended to any other interpretation the WordPress community can dream up—adding another dimension to the already massive scope of delivering an API to 60M+ websites.
Single Place To Educate Everyone About APIs
I can't think of another platform that has introduced the average person to web programming, more than WordPress (maybe Drupal?). When you go back to the concept of lowest common denominator, WordPress employs PHP and MysQL to drive its functionality, and combined with the plugin and theme extensibility, the platform has provided a rich gateway to the world of web, and mobile app development. The opportunities to introduce, and educate the masses of the benefits (and downside) of APIs, is huge! While the already established API developer community may not want the WordPress community developing APIs, it will be the quickest way we from 10K APIs, into the millions of APIs. WordPress drives 22.0% of the top 10 million websites, which will instantly provide access to some potentially high value content and data, that will significantly increase the size of the overall API economy, as well as contribute to API literacy in every business sector, around the globe.
At the time of this post, the WordPress API is only a plugin, which means it is an optional part of the platform, requiring a WordPress site administrator to know about the plugin, and install as part of their individual WordPress installation. However, the goal is to bake the API into WordPress as part of the 4.1 release, which could happen as soon as this fall, or at the latest the first or second quarter of 2015. While not all 60 M+ WordPress sites will immediately update to the 4.1 release, a significant portion of websites around the world will instantly have an API--good or bad.
I’m pretty excited at the idea of so many websites having APIs, then at the same time I'm completely terrified. It is kind of like flooding the market, with some very high value APIs, while also adding millions of very shitty APIs, and I'm sure a lot in the middle. With a lot of the problems we are facing around discovery and integration at 10K, imagine what things will look like with millions of APIs all of a sudden. Do we want every website to have an API this fast?
Beyond just giving a website an API, I'm also very interested in the potential of using WordPress API as purely an API core, allowing it to be used to drive content and data for mobile, and single page applications (SPA). Considering the innovation we've seen around the core WordPress platform, from the community, I imagine we'll see similar when it comes to just raw API deployment, and when you start considering the potential when bundled with the latest containerization movement, led by Docker.io, and being driven by Amazon, Google, Microsoft, and Red Hat—your head will start to spin.
WordPress API + Virtual Containers = API Deployment As Part Of The Fabric Of The Cloud
I will be carving out more time to consider the implications of the introduction of the WordPress API, and hopefully provide more feedback to the WordPress API development team. If you are a seasoned WordPress developer, and experienced API consumer, you might consider getting involved too. You can get access to the source code for WP API over at Github, and you can submit issues via the issue management for the project—they also provide updates via the WP blog tagged as json-api.
I’m looking through the most recent API server release from StrongLoop, and I can’t help but see echoes of what I’ve been researching, and covering across the API Evangelist network. API management has been front and center for years, but API deployment is something that is just now being productized, with a wealth of new service providers emerging to provide API deployment solutions that go beyond DIY frameworks, and enterprise API gateways.
Let start with walking through their announcement of their StrongLoop API Server:
- LoopBack 2.0 - An open source framework for quickly creating APIs with Node, including the client SDKs.
- mobile Backend-as-a-Service - An mBaaS to provide mobile services like push, offline-sync, geopoint and social login either on-premise or in the cloud.
- Connectors - Connectivity for Node apps leveraging over ten supported data sources including Oracle, SQL Server, MongoDB and SOAP.
- Controller - Automated DevOps for Node apps including profiling, clustering, process management and log management capabilities.
- Monitoring - A hosted or on-premise graphical console for monitoring resource utilization, response times and function tracing with the ability to send metrics to existing monitoring tools.
Just as StrongLoop did in their release post, let’s dive deeper into LoopBack 2.0, the open source core of StrongLoop, which they say "acts as a glue between apps or devices and data via APIs written in Node”:
- Studio - A graphical interface to complement the command-line tooling and assist developers in building Loopback models.
- Yeoman and Grunt - The ability to script tasks, scaffold, and template applications and externalize their configurations for multiple environments.
- ExpressJS 4.0 - The latest update, for the well known Node.js package, bringing improvements by removing bundled middleware and refactoring them into maintainable modules, revamped router to remove confusion on HTTP verb usage and decoupling Connect, the HTTP framework of Node from the Express web framework. It is also the E in the MEAN stack (MongoDB, ExpressJS, AngularJS, Node.js).
- Project Structure - An expanded directory structure has been expanded to make it easier to organize apps and add functionality via pre-built LoopBack components and Node modules.
- Workspace API - Internal API making it easier to define, configure, and bootstrap your application at design time and runtime by simply defining metadata in the form of JSON.
This is one of the few sophisticated, next generation, API deployment frameworks I have seen. We have had gateways for a while, and we have a new breed of database and spreadsheet to API providers like APISpark. We also have a new wave of scraping to API solutions from Kimono Labs and Import.io, but I’d say Orchestrate.io gets us closest to the vision I have for StrongLoop, when it comes to API deployment.
I’ve referenced this ability in my stories on virtual API stacks:
- Virtualized API Stacks
- Traffic and Weather - Virtualized API Stacks
- Virtual Containers, Stacks, APIs And Application Management
- Virtual Stack Composition Like The Absolut Drinks Data API
This new approach to API deployment allows us to rapidly define, deploy, and orchestrate stacks of API resources for use in our web, single page, and mobile applications. I really feel like BaaS, as an entire company, was just a short growth phase, that leading us to this point, where anyone can quickly deploy their own BaaS, for any broad, or niche purpose. I also see my research into the world of APIs and Single Page Apps (SPAs) reflected here, in StrongLoops API platform vision.
I feel that StrongLoop has an important take on API deployment, one that reflects where leading API, web, single page, and mobile app developers have been for a while now. The difference is that StrongLoop is providing as a standardized platform, allowing developers to much more elegantly orchestrate their entire lifecycle. You have everything you need to connect to existing resources, generate new API resources, and organize work into reusable parts, to deliver the web, single page, mobile apps you need.
I am closely watching this new generation of API deployment providers, companies like StrongLoop, Orchestrate, Flynn, and Cosmic. I see these players being the next generation API gateway, that goes way beyond just providing an enterprise gateway to internal assets. This newer vision is much more directly aligned with the needs of developers, enabling them to rapidly design, deploy and manage the API services they need to drive the web, single page, and mobile apps that are the vehicles in the API economy.
I’m exploring different ways that APIs can be deployed, with an emphasis on deployment by non-developers. There are numerous cloud services available, that allow non-developers to execute common business tasks like registration forms, surveys, payments, and product sales, and when you combine these business functions with Zapier, Google Docs and APISpark—you can deploy an API, no code skills required.
This story begins with the ability to deploy an API from any Google Spreadsheet using APISpark, putting API deployment within the grasp of the average business user. Next, I want the easiest possible way to get data, from multiple sources, into a Google Spreadsheet? Answer: Zapier (or other reciprocity provider, like IFTTT). To support this, I started looking through the numerous Zapier recipes, that allow my me to publish results to a Google Spreadsheet—there are 167!
The most obvious data source I see is Twitter. Everyone time there is a Tweet from specific user, or from a specific Twitter search, you can have it published to a Google Spreadsheet, and when you have that spreadsheet connected to an APISpark API, the results will be automatically available via API.
The second most common source of data I see, would be cloud based forms. I see providers like Wufoo, Gravity Forms, and JotForm, to name a few, that allow you to submit form submissions to any Google Spreadsheet, and with the APISpark integration, all your form submissions are automatically available via API.
After that, I see numerous commerce, payments, and other key business functions, that Zapier enables publishing of data and content into a Google Spreadsheet from. All of these services have APIs, that is why Zapier is able to do what they do, but that would require a developer to tackle with custom API integration (not for this story). This story is all about enabling non-developers to deploy APIs, from common business functions, no coding necessary--Zapier is our middleman.
Beyond Twitter, forms, payments, and product sales, I will explore other easy to implement, API deployment using Zapier, Google Docs, and APISpark. These represent API deployment scenarios I don't think people are tracking on, and with a little education, I think we can bring problem owners up to speed, and increase the number of APIs available, driven by average person, and the common business problems they face every day.
Disclosure: APISpark is an API Evangelist partner.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.