Today we’re announcing the availability of source code for Fr8, a service we’ve been building for the last year that runs at fr8.co. Fr8 (pronounced “Freight”) is a distributed application integration platform that seeks to combine the usability of consumer-oriented tools like IFTTT and Zapier with the deep power, platform-independence, and extensibility of distributed, open platforms like Docker.
Fr8 seeks to provide an excellent solution for three different audiences:
- Developers who want easy ways to integrate their own technology with the cloud at the business-process level, without having to do any hosting.
- Power Users who want a point and click environment for easily assembling application flows (we call them “Plans”) to handle business processes, along with an easy way to publish the Plans for the benefit of the third group:
- Consumerish End Users who want simple tools they can click on and use without having to see how the sausage is made.
Fr8 was created by The Fr8 Company (fr8.co), a small San Francisco startup. The project has had about 50 contributors to date and 17,000 commits, and maintains a paid staff of about 15 engineers around the world. Fr8 is at its heart platform-independent, but platform SDKs are available to simplify the creation of Fr8 Activities and Terminals in C#,Ruby, Python, and Java.
We’ve deployed and operate a Fr8 Hub at fr8.co. You can register a free account and craft workflows and build apps without code. You can create Fr8 Plans that trigger on various events, or that are manually run on your command. There’s a graphical Plan Builder and dozens of Activities that can be added to Plans.
The focus of today’s announcement, though, is on the developer environment we’ve crafted to encourage the use of Fr8. We’re hoping that engineers, consultants, integrators, IT teams and others will find Fr8 an attractive way to solve their varied integration tasks. While many problems will be solvable using the existing set of Activities, we’ve invested a ton of energy into making it easier to craft new Terminals and Activities, and extend existing Activities.
We’re hoping that people will try both using Fr8 and also developing and extending Fr8. We are interested in feature requests for our internal development roadmap, and also have made some bounties available for any developers interested in doing some early experimentation with Fr8.
To see Fr8 in action, take a look at one of our videos, here.
Things to Try
- Try Fr8 now, as a user.
- Visit the developer site, or check out the Ruby, Python, or Java SDKs
- Read the Blog, which has things like the Weekly Dev Report.
- Read about the Roadmap.
- Join the Fr8 conversation on Slack
- Post a bug or feature, Terminal, or Activity request to the core repo issue board or to one of the Slack channels.
- Contact The Fr8 Company directly.
Fr8 uses a hub-and-spoke model.
Terminals host discrete Activities, and provide metadata on them in response to discovery requests from associated Hubs. At Design-Time, the Hub mediates a conversation between the Client UI and the Terminal, enabling the Terminals to declare the UI they want without having to concern themselves with layout. At Run-time, the Hub traverses the Activities in the Fr8 Plan, dispatching the Plan’s Payload Container (a JSON element) to the Terminal that is responsible for the next Activity. The Terminal then executes the Activity, processing the Container, and likely adding or modifying Crates of JSON data to it. The Terminal then returns the Container to the Hub. As a result of this approach, anyone can be in the Terminal business, and as long as their Terminal responds to a small set of HTTP requests with properly formatted JSON, it can be coded up in any language, on any server, in any physical location.
Hubs are similarly distributed. The Fr8 Company runs a production Hub at fr8.co, and a development hub at dev.fr8.co, but expects and hopes that others will run Hubs as well. Hubs can be run on premises or behind the firewall, and collaborate with other Hubs in the public space.
Open and Portable
A Fr8 Plan is entirely represented as a JSON file that can be downloaded from one Hub and uploaded and run on another Hub. In this regard, we have striven for the comfort this provides to builders of web objects as varied as HTML pages and Docker containers. We certainly hope that The Fr8 Company will someday be able to make some money servicing enterprises who want to use fr8.co to manage and process their Plans. But we think that an important part of becoming a breakout technology for application integration is an architecture that eliminates fears of lock-in. We don’t think that your enterprise or developer workflows should be “Zapier-specific” and useless elsewhere.
We also de-emphasize the concept of a central database. Hubs do have databases and do store Containers and Activities, but in general, all of the information needed to process a Container travels with the Container, including its operational state (stack, index values of loops, etc). This allows a Plan to pause, waiting for something long, like a user email response, and then take up where it left off.
With Fr8, if the functionality you’re looking for isn’t readily at hand, you can extend Fr8.
Our goal for our Terminal infrastructure has been to reduce to a minimum the amount of work that this requires. There’s more than one way to get leverage:
- You can build a new Terminal that exposes new Activities to Fr8. You can host your Terminal yourself or have us host it for you on fr8.co.
- You can take an existing Terminal, and add more Activities. For example, the Google Terminal currently only has the raw basics: Get Google Sheet, Save to Google Sheet, Monitor Google Form. There are at least 100 obvious Activities that can be built to go deep into the existing API functionality exposed by Google.
- You can take an existing Activity, and make it more sophisticated. This can be prosaic, such as adding field validation to an Activity that should have had it in the first place. But it can also be more substantial.
- You can clone an existing Activity to create a specialized variant. This means that you neither have to convince the Fr8 core team to implement your feature, nor to agree to merge your code. You can simply publish your variant Activity and use it yourself, or make it available to others as well. With Fr8, we anticipate, for example,that there might be 5 different flavors of Google Get Sheet. One might be optimized for unsophisticated users, with a lot of investment in popup help via Fr8’s /documentation endpoint. But another might be designed for data analytics pros and offer a bunch of query filters and other advanced tools. Yet both of these activities can live in a single Google Terminal, leveraging common code.
Fr8 provides access to a rich front-end UI experience, and Activity developers specify the UI they want with simple text declarations, similar to how web pages insert form elements . The growing gallery of UI controls is available declaratively to all Activities, and the growing power of the Fr8 SDK’s increasingly make it unnecessary to directly manipulate JSON. The Client duly renders the UI and then passes any resulting user inputs back to the Terminal via the Hub. So at the end of the day, you don’t have to do any front-end layout work to present rich UI to your Activity’s users, nor do you have to leave your preferred dev environment.
Most of the limits of first-generation iPaaS stem, we believe, from the lack of focus on a shared, dynamic data model. It’s not hard to map API parameters to key value pairs, since they’re all just strings, but once you hit the limits on that, you run into problems. It can be difficult or impossible to do even modestly clever things like iterating over the rows of a spreadsheet. Fr8’s approach is to use a shared registry of data schemas we call Manifests that express the inner structure of individual Fr8 Crates, the core unit of data in Fr8. As an example, consider the Standard Table Data manifest. Both the Load Excel File Activity and the Get Google Sheet Activity, at run-time, load a spreadsheet, extract the data, and store it as JSON in a Crate “of Manifest Type: Standard Table Data.” This particular manifest provides a json definition for the rows and columns of a table:
An Activity that knows how to consume tables of data can choose to support this Manifest. If it does so, then the output created by upstream Activities like Get Google Sheet can be automatically sent and procesesd by the downstream consuming Activity.
We don’t in any way claim to be awesome at designing universal data schemas. In fact, the initial Manifests at the fr8.co Manifest Registry are pretty rudimentary (and we probably need to reframe them in something like JSON Schema). We’ve focused on making it easy to publish Manifests, and with a little curation to prevent unnecessary fragmentation, we think the registry can become quite useful.
Fr8 encourages Activity designers to make their output payload available in multiple Manifests. For example, an Activity might output a JIRA issue using a Crate with the Jira Issue Manifest (a relatively complex data structure that enables 1..N comments to be stored with the issue) but also output the basic properties of the issue in a separate Crate of Standard Payload Manifest (Fr8’s “lowest common denominator” Manifest, consisting essentiall of a flat list of key-value pairs). This allows Plan creators to make use of upstream Jira Issue data with a wider range of downstream Activities.
All of that data modeling is useless, though, if there isn’t an easy way to map fields from upstream outputs to downstream inputs at Design-Time. Fr8 calls this process Signaling. Here, the upstream Salesforce Get Data Activity has been configured by the user to select a Salesforce Account. The Activity immediately signals all of the fields associated with this kind of object. The dialog box below has been generated by the downstream Publish to Slack Activity. It requests from the Hub a list of all upstream fields that have been signaled, and the user has an easy way to choose the field to be inserted, in this case, into a Slack message.
When we talk with people who have tried first-generation cloud app integration, what we hear is that there’s a barbell effect. Either the solutions tended to be very powerful and require a lot of coding and hosting chops (Huginn, for example), or they tended to emphasize friendly UI but struggle to provide more complex integration processing.
For example, a classic example of common business integration is the Mail Merge. As of this writing, there’s no way with any of the older solutions to simply Mail Merge from a Google Sheet to, say, DocuSign Envelopes. With Fr8, considerable attention has gone into the data model, and actions like Mail Merge are straightforward because Fr8 supports Looping, Enumeration/Iteration, Branching, Subplans, Call Stacks, Notification services, and other programming concepts, yet exposes them in the Plan Builder with simple point-and-click.
With Fr8, an Activity can decide to get data from an external cloud source, and it can decide to send data, but it can also opt to just focus on processing. Build a Message is a useful Activity that allows upstream data to be synthesized into a new custom message, which can then be accessed by downstream activities that post, publish, or send. Math is another useful example of middleware. Math functions are also on the drawing board.
Fr8 also tries to make reusability easy. With Fr8, you can design a Plan, configure it, publish it, and make it available to others through the Fr8 Plan Directory, a growing database of ready-to-run Plans.
We’ve made sure that the output of one Plan can be easily piped to the input of another Plan. We also have built in a rich subplan mechanism which essentially allows the creation of discrete functions that can be managed in a single Plan. We’re hoping that this will create useful leverage, allowing much more complex logic flows to be built out of component parts.
Good for Integrators
We think that one group that will really like Fr8 are the consultants, integrators, builders, and IT guys that are responsible for much of today’s integration. You can design a plan, test it, and then share it with a client. You can write a custom Terminal and run it in your own environment, while still enabling it to interact with Activities from publicly available Terminals. You can publish your Plans to the Fr8 Plan Directory and (we think, eventually) charge for them, if that makes sense for you.
Other Fr8 Features
Any technology claiming to offer viable iPaaS needs a way to package functionality into simple Apps that can be deployed to non-technical Users. Fr8 Apps can be generated and distributed using simple URL’s. These solutions provide end users with simple straightforward forms that trigger underlying Fr8 Plans of arbitrary complexity.
Fr8 provides a publish/subscribe mechanism that enables Activities to trigger a Plan based on received external events. Any Terminal can register with web services and post events to the Hub.
We’ve been at work on this for about a year, and while we’re proud of what we’ve accomplished we’re the first to admit that Fr8 is still awfully raw, and we’ve got a lot more work to do. Notably:
1) The security architecture is young and will need a lot of work. We encrypt crate data on the database, and use https, but right now we don’t provide tools, let alone sophisticated tools, for making it easy to encrypt a crate of payload in a way that selected downstream activities can decrypt it. We also haven’t added more than a touch of the enterprise management tools Fr8 will need. We’re hoping that we’ll attract the attentions of some of the people in the open source community who really understand security.
2) We haven’t had any open source gurus on the core team, and as a result we’ve probably screwed some stuff up, like licensing or repo structure. We’d be very grateful for corrective suggestions.
3) Our source code and our tests have a lot of inconsistencies. We know we need to make it more unified in look and feel.
4) The overall UI really needs to be more polished, attractive, integrated, and consistent.
We’re really pleased to share Fr8 with you today. I’d like to thank all of the engineers that have worked on Fr8 over the last year. Especially Alexei Avrutin, who handles all of the devops almost single-handedly when he’s not directing other engineers, Vladimir Borisov, who has provided so many architectural improvements to Fr8, Yakov Gnusin, who has built so many complete subsystems from out there deep in central Asia, Bahadir and the great young team he has built in Ankara, and Andrei and Sergey and Tony, Marjan, and Vadim. It has been a great year and I anticipate many more.
August 17, 2016