Warning: session_start() [function.session-start]: open(/home/content/30/7423630/tmp/sess_q3ai0s09tlmf5kui78g1pcq0i6, O_RDWR) failed: No such file or directory (2) in /home/content/30/7423630/html/wp-content/plugins/simple-twitter-connect/stc.php on line 33

Warning: session_start() [function.session-start]: Cannot send session cookie - headers already sent by (output started at /home/content/30/7423630/html/wp-content/plugins/simple-twitter-connect/stc.php:33) in /home/content/30/7423630/html/wp-content/plugins/simple-twitter-connect/stc.php on line 33

Warning: session_start() [function.session-start]: Cannot send session cache limiter - headers already sent (output started at /home/content/30/7423630/html/wp-content/plugins/simple-twitter-connect/stc.php:33) in /home/content/30/7423630/html/wp-content/plugins/simple-twitter-connect/stc.php on line 33
Developers Archive - TransSwipe - Merchant Services and Credit Card Processing

TransSwipe - Merchant Services and Credit Card Processing

Archive for the ‘Developers’ Category

A Dashboard and Admin for ACH Transfer Integrations

Today, we released an intuitive new dashboard for White Label partners to manage customers, view transaction details, and discover business trends. All information within this interface leverages the Dwolla White Label API powering the partner’s ACH transfer integration, but provides the data in a way that’s easy to manage and act upon from a business operations perspective.

By providing a clear view into the integration data, the new dashboard makes it simpler for partners to provide quality customer support to their users, reconcile every payment, and keep tabs on the heart of their payments integration. We’ve built the admin interface so White Label partners don’t have to jeopardize their own time building, maintaining, and scaling their own DIY dashboard.

Check out Product Hunt to see what others are saying!



On the main dashboard, partners will find beautiful charts and graphs built from the data being collected from within their integration with the Dwolla White Label ACH API. Customer and transaction data will be arranged in a way that provides a straightforward look into the health of the business and makes it easy to establish and analyze business trends over time.



Partners can easily look up and edit customer information from within the dashboard, an important feature for providing quality customer support and communication. Customer service is a critical piece to running a successful business—partners like GOAT have been able to reduce cashout related support tickets by 80% after integrating Dwolla’s bank transfer API.



View transactions received and sent across your platform. Search and view transaction details, or simply use this functionality to assist in your reconciliation processes. Run basic accounting and business operations faster.

Save hours of your own developers’ time by eliminating the need to build a custom view of your White Label integration. Utilize a smarter interface for managing payments—your customers and accounting department will thank you.

Interested in learning more? Contact Dwolla

10 best blog posts from the minds of our developers

Every day on Twitter, Slack, GitHub, or one of the many modern forms of communication, our developers and product designers are approached with interesting and insightful questions. This past year, they’ve turned those insights into blog posts and shared them publicly, breaking down complex ideas into articles that everyone can learn from.  

However, content is fleeting. A massive amount of information is published daily. Reflecting on the value of these technical posts we’ve published, we thought rather than let those insights fade into the vastness of the internet, we’ve pulled together the 10 most popular posts from our developers and product designers. Enjoy.

1. Developing for simplicity when cleverness is the enemy

As our team of developers prepared to launch a new product, they needed to take a step back and consider the value they were looking to add. This post explores that process.

2. APIs and the power of collaboration for innovation

A look at how the relationship and power of APIs has changed over the years, and how the right collaboration could fundamentally change an industry.

3. How we did it, inside the developer portal redesign

In one of the most interesting redesigns in Dwolla’s history, our Vice President of Product explains how we pragmatically approached our developer portal update to satisfy even the pickiest of developers.

4. Fake it as you make it: why fake services are awesome for developers

This post explains why using fake services to test products can be more beneficial and effective than the alternatives, including a real step-by-step example.

5. Building with Microservices and Docker

Two members of our engineering team dive into how we used dockerization and microservices hand-in-hand to build a more scalable system.

6. What I’ve learned in designing for developers

Our brilliant Head of UX explains how she discovered the different Dwolla developer personas to design the best possible developer portal based on direct insights.  

7. Arbalest: Open source data ingestion at scale

Through the lens of open source software, get a pros perspective on the value and process of implementing a scalable data solution—great post for fans of Amazon AWS and Tableau.

8. Ask hard questions, build a better product

A rare chance to get inside the mind of a VP of Product at a fintech company, this blog post discusses how our team approached rethinking the product from the lens of leadership.

9. Cutting the busywork for developers with better design

Our developer portal upgrade was a major initiative (which we talk about in a post mentioned above). In this post, get insight into the specifics of what developers love and hate about interacting with APIs, and how we optimized for this.

10. Building a data platform, embracing events as the atomic building blocks of data

From the mind of one of our most innovative data engineers, this blog post looks at how we structured our data platform to break information down to its simplest most useful form.

To learn more about Dwolla or the API, head to our developer documentation now.

Custom transaction limits and next day transactions for your users

Get paid more quickly with Dwolla NextDayWe understand that not all businesses are created equal and that each has different needs. Whether it’s faster processing times or higher transaction limits, we are here to help you design a payment solution tailored to your business.

For approved partners, as part of a paid white label solution, Dwolla can enable those sending you funds to send up to a custom amount for each transaction and/or have those transactions process the next business day.

This allows users sending to an approved partner to bypass Dwolla’s $5,000 per transaction limit for personal accounts and $10,000 limit for business or nonprofit accounts. This also speeds up transfer times for transactions destined for an approved partner’s Dwolla account to  one business day, instead of the standard 3-4 business day bank transfer processing time.

Partners that benefit from raising the per transaction limit on transactions destined for their account are those that need to regularly move large sums of money, such as investment platforms facilitating transfers from investors to development projects.

Partners that benefit from increasing the standard bank transfer processing times of their users are those that frequently receive many payments, such as property management platforms receiving transfers from tenants each month.

If your platform facilitates high dollar or a large volume of bank transfers, you need a payment platform that doesn’t hinder, but supports the way your business operates. Contact an integration specialist today.

Get started with your own integration

We’ll help you design your ideal payments experience.


Thank you

A Dwolla representative will reach out to you within one business day.


There was an error and your the form was not submitted.

Bank account balance as an API endpoint

Posted in API, API help, Blog, Developers, Dwolla developers, endpoints, Product Updates on April 12th, 2016

We’re making a new feature available for white label customers that lets them ask for users’ permission to check the balance in their bank account.

Dealing with returns is one thing and we can appreciate that but there is a great deal of applications where this can be valuable for a business or developer:

  • Mitigating risks when pre-funding. Some businesses make the choice to pre-fund accounts before the transaction clears. If that is done the balance endpoint can provide a view into the risk the business is really taking.
  • Mitigating risks in trading environments. One of the problems in a trading environment is reconciling what someone says they have when they make a trade, and what they actually have. Many times the accounts for trading purposes are segregated and intended to be sacrosanct but programmatically checking the balance of the associated account that provides liquidity for trades has previously been incredibly hard.
  • Other things we haven’t thought of yet. It’s important from our perspective to give developers and businesses a platform to innovate with. Users of white label software platforms can grant this permission if they find it valuable to do so given the features associated with the platform they’re using.
  • Following the trend that the bank account is transforming into the pre-loaded account. This feature gives software developers the ability to check the bank account balance the way pre-funded account balances are checked, similar to how checking a balance in a Dwolla account works in our V1 APIs.

So how does it actually work?

Once a software application gets the permission from the account holder and the funding source is added to a white label application through instant bank verification, the developer gets a GUID that represents the account. It looks like this:


That is utilized in the /funding-sources/ endpoint to request balance for that authorized funding source:


By adding ?balance=latest to the end of the request the application is requesting the latest balance. Additional data is returned to the API call as a result. Here is a sample of what that additional data looks like:

"balance": {
    "value": "107.28",
    "currency": "USD",
    "lastUpdated": "2016-04-09T00:12:43.527Z"

The full response would look something like this:

  "_links": {
    "self": {
      "href": "https://api.dwolla.com/funding-sources/692486f8-29f6-4516-a6a5-c69fd2ce854c"
    "customer": {
      "href": "https://api.dwolla.com/customers/36e9dcb2-889b-4873-8e52-0c9404ea002a"
  "id": "692486f8-29f6-4516-a6a5-c69fd2ce854c",
  "status": "unverified",
  "type": "bank",
  "name": "Test checking account",
  "created": "2015-10-23T20:37:57.137Z",
  "balance": {
    "value": "107.28",
    "currency": "USD",
    "lastUpdated": "2016-04-09T00:12:43.527Z"

This new feature will be made available to white label customers who are in our custom package.

Think you could make use of this new technology? Drop us a line and we’ll help you get started.

Drop us a line

We’ll help you get started with better payments.


Thank you

A Dwolla representative will reach out to you within one business day.


There was an error and your the form was not submitted.

On-demand Bank Transfers, made easy


In 2011 we released an OAuth API that made it easy for developers to request transfer permissions from their customers with our Dwolla-branded platform.

Today we’re making the ability to bill your customer later available for our Dwolla White Label customers in our v2 API.

It’s called On-Demand Bank Transfers.

Developers using our white label APIs can enable their payers to authorize transfers for variable amounts from their bank account using ACH at a later point in time for products or services delivered. It’s one simple additional step—a quick authorization from the customer when they instantly verify a bank account. This is great for companies like:

  • Cloud computing services. Fees can be different every month, requiring ongoing authorization so a customer can easily pay for a service, and the company can easily bill for the service.
  • Utilities. A water company bill is rarely the same each month. Same with electrical, and gas. The amount collected at the end of each month is usage-based, or metered.
  • Ride sharing or asset sharing platforms. The amount a customer is charged for a ride across town depends on a variety of factors. We make it easy for sharing companies to bill their customers, while reducing the hassle for the end customer on each trip.
  • B2B services that bill on a variable basis. Some orders may require a bank transfer on NET terms and others may be fulfilled once the goods are delivered. Either way, both should be possible.

The instant bank verification and on-demand authorization occur with Dwolla.js to make it incredibly easy to add to your software. It adds an extra step to the bank transfer process when verifying the bank to acquire the permission for the account holder.

On-demand payments from Dwolla

Once you have collected all of the authorizations required for a bank transfer, including the additional authorization from the end user for on-demand bank transfers, your software application kicks off a transaction that looks like this whenever the customer needs to be billed:

    "_links": {
        "source": {
            "href": "https://api-uat.dwolla.com/funding-sources/5cfcdc41-10f6-4a45-b11d-7ac89893d985"
        "destination": {
            "href": "https://api-uat.dwolla.com/customers/C7F300C0-F1EF-4151-9BBE-005005AC3747"
    "amount": {
        "currency": "USD",
        "value": "225.00"
    "metadata": {
        "customerId": "8675309",
        "notes": "Payment for January 2016"

As usual, there are no per transaction fees for either party in the transaction, and with our white label services your brand is front and center. If you’re a developer and have more questions, head to our API Support discussion board and post your questions. Our engineering team regularly reads and responds on the board.

The cost and complexity to make bank transfers better isn’t unknown and it affects new fintech companies, large established business, and even big VC’s… We all have the same problem.

We didn’t either and that’s why our team has spent the last 5 years focused on bank transfers. It’s why we feel on demand bank transfers are so incredibly valuable for our customers.

There is a better way and we’re excited to be a part of building the future making bank transfers easier for businesses and developers.

Contact us to enable for your application

Fake it as you make it: why fake services are awesome for developers


This blog post comes from Shea Daniels, a developer here at Dwolla. When Shea isn’t busy building awesome new things, you can find him out for a run. api-blue

It’s often said in life that we “stand on the shoulders of giants.” This rings especially true now that we’re in an era of abundant open source software and SaaS providers. Now, more than ever, we build applications by relying on tools and services that others have made. This may even be standard practice inside your own organization as other teams deliver functionality through a microservices architecture.

Building software by composing services is extremely powerful, but it can still be a rocky road. Several factors can make it difficult to write and test your code:

  • Complex scenarios may not be easy to test with real data
  • Running elaborate business logic may consume resources
  • Sandbox environments may not exist for 3rd party APIs

Just fake it

So what can be done to mitigate these issues? The answer is to fake it while you’re making it!

You can see this in everyday life. Whenever the real thing is too expensive or impractical, we sub it out with something fake as a stand in—think movie props or mannequins for tailors. This is also a fairly common engineering practice; my favorite examples are the boilerplate capsules used to evaluate rockets and other space hardware.

In the software world, if you practice TDD you should be familiar with the use of test doubles (mocks, fakes, and stubs) for dependencies in your unit testing code. Used instead of the real implementations of objects, fake dependencies isolate the code under test by providing predictable results given certain input. This isolation is useful for tracking down issues and fully exercising your code without complicated setup.

The same concept can be applied when developing an integration with a third party service. By building a fake copy of the web service, you gain the same advantages of isolation and repeatability as you test your application. This is especially useful if the service you’re depending on is being developed in tandem and has yet to be fully implemented.

There are some existing tools for quickly standing up your own fake services, such as Nock and Frock. But with Node.js and a few NPM packages, it’s easy enough to build your own from scratch.

In this post we’ll include:

  • An actual example
  • How to get started
  • Possible scenarios
  • Some of the downsides

A real example

Let’s break down a real example that Dwolla has recently open sourced: Nodlee. You can see it in action by checking out our instant bank account verification demo—here it’s used as a backing service.


Getting started

Nodlee is a simple web server written in Javascript and run via Node.js. It depends on the following NPM packages, which you can see in the package.json file:

minimist – argument parser library

If you haven’t used Node or express before, there are a ton of great tutorials, or you can read through the Nodlee source code to get a feel for it. The readme has a lot of great info and the code entry point is app.js.


The first thing to do when building out a fake service is to look at the documentation for the real API and experiment with the service to discover how it works. With that knowledge, you can figure out which endpoints need to be mocked and what the responses should look like.

For a simple example, here’s the Nodlee health check endpoint response: health.js

module.exports = function (req, res) {
  res.json({ healthy: true });

These responses can be as simple or as complicated as needed. If returning the same canned response every time isn’t enough, consider scanning the request for sentinel values that you define. Then you can use those values to decide which data to send back. You can even use a templating language like Handlebars for generating your responses if you want to get swanky.

Complex scenarios

For the instant account verification product we were building, even sentinel values and templating weren’t quite enough. We found that we were constantly editing multiple files in the fake service code to set up complex scenarios.

The first step to making this easier was to consolidate all of the possibilities that determined a particular scenario into a single list of options in the code: scenario.js

module.exports = function Scenario() {

	this.contentServiceId = 0;
	this.isMfa = true;
	this.mfaTimeout = false;
	this.mfaPass = true;
	this.mfaTypes = ['q'];
	this.refreshesSet = 1;
	this.refreshesLeft = 1;
	this.errorType = 0;
	this.accounts = [
			routingNumber: '222222226',
			accountNumber: '5031123001',
			accountType: 'CHECKING',
			accountHolder: 'John Wayne',
			accountName: 'Your Account #1',
			balance: '1000'

This object can then be checked in all of the service endpoints in order to determine the appropriate response. With this in place, developers could set up the flow to behave however they wanted just by editing this single file.

Sentinel values on steroids

We have local development covered now, but what about testing in sandbox environments where we can’t edit the fake service code? Not only that, but what if we wanted coverage of our flow with automated UI tests (e.g. Robot Framework)?

What we need now is a service with a memory longer than a single web request and a way for automated tests to trigger whatever scenario is needed. This is where the minimist and node-cache NPM packages come into play.

With minimist, we are able to take certain inputs in a web request and treat them as if they were command line interface options. Those options can then be translated into order to set the properties of the Scenario object we’ve just discussed: 

var scenarioData = optionsParser.parse(req.body.someInput)


exports.parse = function(options) {

	var data = new Scenario();

	if (options.indexOf("-") < 0) {
		return data;

	var args = parseArgs(options.split(' '));

	if (args['nomfa'])
		data.isMfa = false;


	return data;

Now that we have the options set for the scenario we want, we use node-cache to persist it across web requests: scenarioManager.js

var cache = new NodeCache({ stdTTL: 900, checkperiod: 300 });

exports.set = function(...) {


	cache.set(userSessionToken, scenarioData);
	return scenarioData;

Now we can use the cache to access the scenario that’s currently being tested at any point we need to build a response: getMfaResponse.js:

module.exports = function(req, res) {

    scenarioManager.get(req.body.userSessionToken, function(scenarioData) {

        if (!scenarioData.mfaTypes) {
        } else if (scenarioData.mfaTypes.length < 1) {
        } else {

The downsides

As with anything, fake services are not a silver bullet. There are a few caveats to keep in mind:

  • Are you sure you understand the real service well enough to build a fake version of it?
  • Watch for edge cases where you may not be duplicating the behavior of the real service
  • If you do find edge cases, be sure to cover them with appropriate tests; manual testing with your fake service is not a replacement for good test coverage with unit/integration tests
  • Plan for regular maintenance of your fake to keep up with any changes in the interface or behavior of the API you depend on
  • Using a fake does not relieve you from the task of running your code against the genuine article

The last bullet point is important since there’s a large difference between “should work” and “actually works.” At some point in your workflow you’ll need to test the full production stack!


Here at Dwolla we’re committed to making developers’ lives easier by providing a great API for moving money with ACH transfers. We’ve found the concept of fake services to be invaluable in making this happen. If you found this useful, please share this article and comment with your own experiences. Happy building!

Check out the documentation

Facilitator fees are now free to use

Facilitator Fee

In June 2015 Dwolla rolled out new pricing. We removed transaction fees and started offering value added features and controls to our business clients.

In these new packages we focus on making the banking system, and largely ach transfers, easier for developers and businesses to use.

When our new pricing was introduced we bucketed some older functionality and some new functionality. As a company, we made 1 decision in that process that we’d like to apologize for and make a change to.

The facilitator fee was moved into the $1500+ package as a paid feature and today we’re making it free again. While I could go into the rationale behind the original decision, it doesn’t matter and it just needs to change.

What matters now is making a fix and making sure budding developers, who are getting companies off the ground, can make use of the feature.

The facilitator fee does a few things well:

  • It allows developers to build fees into their applications.
  • It allows developers to charge a fixed fee or a percentage fee for any transaction passing through their application.
  • It allows developers to build software that leverages ACH transfers, Dwolla balance transfers, credit transfers, and anything else someone could use on the Dwolla platform to move money and charge an appropriate fee.

Fees might sound bad but a tremendous number of non-payment companies charge small fees to cover operational costs or as a part of the service they provide. Many times it’s not charged to the end user.

Here are a few places we’ve seen facilitator fees used successfully:

  • Some organizations have small service charges as a part of transactions. $2 for example on each transaction keeps their technical team staffed and kids getting lunch cards loaded on time.
  • Software developers building software for third-party companies can actually build their fee into each transaction rather than charging upfront. This helps more companies get to market without loaded up-front costs.
  • Software marketplaces where the marketplace may take 10% of the original purchase price for a service ordered. This has become more and more common. We help the marketplace split the payment as the money moves.

There are some other key benefits for businesses operationally with this structure:

  • When your software application takes a facilitator fee, your company account only receives the fee and you can route the rest of the money to the person receiving the transaction. This happens regardless of whether or not it’s a bank transfer, balance transfer or credit transfer.
  • When you only receive the fee and the other party gets the rest of the money directly, that means you’re never holding your customer’s money for the purpose of transmitting it to another party. That’s a positive thing!

In closing, we put Facilitator Fees into a package when it shouldn’t have been. This is a feature that wants to be free—you want it to be free and we’ve heard you loud and clear.

Go build something great with Facilitator Fees.

Building tools out of blocks (at Dwolla)

This blog post comes from Devangelist, David Stancu. Check out his work on GitHub or hit him up at Awesome IT

building tools, blocks

Here at Dwolla we are building an innovative way to move money. However, thinking differently poses a challenge: we need to also create unique solutions that are best suited to our mission critical objectives. How can we do this in a time effective manner while still reaching all of our goals? The answer lies in open source software.

API V2 and SDK Support

The API team took a very data-centric approach in their decision to use the Swagger ecosystem. This let our internal engineering team focus on the API’s data and behavior first before forcing them to make considerations with regards to implementation. After they were finished, the Developer Relations team was handed the task of designing end-user SDKs for this version of the API (aka ‘V2’).


As the Developer Relations’ engineering lead, I researched the Swagger ecosystem and quickly came across the swagger-codegen project, which would use templates to generate end user SDKs in over 10 languages for any API modeled in accordance to the Swagger Spec. I immediately cloned the project and attempted to generate some SDKs for a few languages popular with Dwolla developers; namely, Ruby, Python, PHP, and Java.

There were a variety of problems with generating SDKs using swagger-codegen. To highlight a few:

  • Generated code often included syntax errors
  • Data in nested containers could not be or was improperly serialized
  • HTTP 201 requests displayed no data to the user
  • Requests that replied with an HTTP 200 and no data would throw exceptions even if the request would complete
  • No implemented OAuth stubs for Ruby or Python

Problems Shouldn’t Discourage

It is easy to see why it may have been more simple for me to just say, “well, this is going to be hell to fix, I’ll just write the SDKs individually myself.” I thought this same thing, but after taking a step back, there was no good reason as to why I couldn’t use the building blocks available to me. The strategy was to review, fix, and improve more than it was to reinvent.

Developers often feel intimidated when they read another person’s code—and even more so when they have to use that code for their own production purposes.

Even very good changes can be scary at first.

Doing two good things at once

As a developer, your problem may feel unique unto yourself, but that does not mean that the problem doesn’t share components with solutions being attempted by others. Simply put, if something is useful for you it may be useful for many. By submitting fixes to the swagger-codegen project I’ve helped not only Dwolla, but the other organizations using this software.

Swagger-CodeGen Quote

For example, OAuth is one of the most popular authorization standards for RESTful services. By adding support to this project for OAuth to certain languages, I enabled that software to help more people than it did before—it’s about community

It’s an interesting time for programmers in the workplace. The ‘open source movement’ has produced a smarter developer; one that is not afraid to look around, ask questions, and share their knowledge. This “intellectual crowdsourcing” is essential to writing robust, stable and well-audited software—and this is what we do at Dwolla every day.

Excited by what we’re talking about? Head over to our developer portal and play around.

Go to the developer portal now

Benefits of using an event sourced architecture

This is a blog post from Corey Oliver, one of our software engineers at Dwolla. To hear more from Corey, follow him on Twitter at @coreyjonoliver or check out some of the work on his blog

API collaboration gears

Where’s the money coming from and where’s it going? The solution Dwolla implemented to answer this question is an event sourced architecture, which has shown adaptability in many use cases.

This post will walk through why Dwolla chose an event sourced architecture over other solutions.


There are two primary approaches or solutions we can consider for tracking money, each with their own pros and cons.

A current-state based architecture is where the application has access to only the current “state of the world.” A good example is storing the balance for an account, say $20, and then replacing that balance with $30 when the account is sent $10. The account balance recorded after each change is the state of the current world aka current-state.

This is the go-to for typical CRUD apps. For one, this solution is well known among existing tools; it is also straightforward making it easy to retrofit to older solutions used by a business.

Now consider events, which signal something that has happened in the outside world. Each event stores its own payload containing information on what happened and at what point in time it happened. The payload is also immutable—it can’t be changed.

An event sourced architecture is one where all state changes in an application are stored as a sequence of events. Many domains in the real world behave like this. Take balancing your checkbook—each debit or credit to your account corresponds to a line in your checkbook. Summing all lines should yield the current balance of your bank account assuming all information is accurate.

Both of these solutions present their own collection of positives and negatives.

As with many things in business, solutions are highly dependent on the problem at hand. For most problems a business will encounter, a current-state based architecture is a solid foundation for building a solution.

But let me emphasize the caveat most. While a current-state based architecture is arguably the most widely used pattern for solving certain business problems, more effective solutions exist.

An event sourced architecture is powerful

event sourced architecutre graphic

Building an event sourced application is less straight-forward than its counterpart, current-state based architecture. It’s also not an option many software developers or product owners are familiar with.

  1. Applications that are less coupled are not as dependent on one another. This architecture property enables Dwolla to do significant horizontal scalability, and can also provide redundancy and resiliency in cases of system failure or degradation.
  2. Using this system, events compose an audit log of actions that have occurred in our system. No change in the system occurs unless an event was recorded. Also, we not only have the ability to provide the current state of the system, but also know how and by what means it got there.
  3. You can build applications on top of events that synthesize into the event data.  Some examples of this include:
  • reconciling user financial data against funds held at  a financial institution
  • identifying fraudulent activity
  • displaying transaction records and payment activity

Why should you care?

Finance is a crucial part of everyone’s life. This includes account holders at Dwolla. Using an event sourced architecture allows us to track financial events and generate the audit log to confirm them. It maintains the integrity of the information.


So where’s the money? Tracking money in a system is a complex endeavor. At Dwolla, an event sourced architecture has proven beneficial over alternative methods for modeling money ownership—organizations should consider their needs and determine whether an event sourced architecture is an effective and competitive solution for addressing their business problems.

Read more from our engineers on Medium

Arbalest: open source data ingestion at scale

If APIs are the manifestation of products and services that enable innovation and collaboration, open source is the foundation upon which APIs are built.


Open source software development is arguably one of the greatest examples of human collaboration. Since the release of the GNU project in 1983 and Linux in 1991, to today powering billions of mobile devices, open source powers the software around us. From the programming languages we use to how we run software, Dwolla has immensely benefited from open source.

In addition to benefiting from open source software, we are excited to “pay it forward” and contribute back to the open source community. Open source software encapsulates shared experiences and problem solving. We not only want to share what we have learned, but also learn from the community’s feedback and contributions.

Processing event streams

One area we are contributing to is data and analytics. Events are the atomic building blocks of data at Dwolla. We use some great open source tools to process, analyze, and visualize this data. However, we needed a way to query all this data interactively, using the flexibility and scale of Amazon Web Services.

We are excited to release Arbalest, a Python data pipeline orchestration library for Amazon S3 and Amazon Redshift. It automates data import into Redshift and makes data query-able at scale in AWS.

Arbalest is the backbone of our data pipeline architecture and has enabled Dwolla to query and analyze billions of events. In a few lines of code it takes care of:

  • Ingesting data into Amazon Redshift Schema creation and validation
  • Creating highly available and scalable data import strategies
  • Generating and uploading prerequisite artifacts for import
  • Running data import jobs
  • Orchestrating idempotent and fault tolerant multi-step ETL pipelines with SQL


Arbalest is a lightweight library designed to be composed with existing data tools. It is also written in Python, which is arguably a de facto programming language for data science. Arbalest embraces configuration as code. Unlike sometimes unwieldy configuration files, an Arbalest pipeline is only a few lines of code that can be tested, packaged, and reused. Finally, it automates the complicated and potentially brittle process of ETL (extract, transform, and load).

We hope Arbalest enables data analysts and developers to spend less time managing their data and more time answering questions.

Use cases

Why use Arbalest? Arbalest is not a MapReduce framework, but rather designed to make Amazon Redshift (and all its strengths) easy to use with typical data workflows and tools. Here are a few examples:

  • You are already using a MapReduce framework to process data in S3. Arbalest could make the results of an Elastic MapReduce job queryable with SQL in Redshift. You can then hand off to Arbalest to define additional ETL in plain old SQL.
  • You treat S3 as a catch-all data sink, perhaps persisting JSON messages or events from a message system like Kafka or RabbitMQ. Arbalest can expose some or all of this data into a data warehouse using Redshift. The ecosystem of SQL is now available for dashboards, reports, and ad-hoc analysis.
  • You have complex pipelines that could benefit from a fast, SQL query-able data sink. Arbalest has support out of the box (arbalest.contrib) to integrate with tools like Luigi to be part of a multi-dependency, multi-step pipeline topology.

Getting Started

Arablest is not supposed to replace existing data tools, but work with them. In our initial release, we have included some batteries, for example, strategies for ingesting time series or sparse data and support for integration with an existing pipeline topologies. We have described a few use cases, but are excited to see more applications of Arbalest in the community.

For more information on Arbalest, be sure to install the package, check it out and star it on GitHub, read the documentation, or look at our presentation from Tableau Conference 15.

fred galoso

This blog post shares insights from Fredrick Galoso, a software developer and technical lead for the data and analytics team here at Dwolla. Fred has led the creation of our data platform, including its data pipeline, predictive analytics, and business intelligence platform. In his free time he contributes to a number of open source software projects, including Gauss, a statistics and data analytics library.

View Arbalest on GitHub now

©2017 TransSwipe


Warning: Unknown: open(/home/content/30/7423630/tmp/sess_q3ai0s09tlmf5kui78g1pcq0i6, O_RDWR) failed: No such file or directory (2) in Unknown on line 0

Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct () in Unknown on line 0