It’s been a little more than a year since BackerKit went surprise-fully-remote due to the pandemic. In that time, the Product team has learned a lot about taking care of ourselves, each other, and our customers while working remotely. We wanted to share some of the things that have made remote life better for us and a few practices you may want to try with your team.

Some context for you

We are spread across East Coast and West Coast time, but keep West Coast hours, so some of these things will not work if your team is distributed globally and mostly works asynchronously. (Prior to the pandemic, we were all colocated in the Bay Area except for one dev in beautiful Montclair, New Jersey.)

BackerKit had also already been doing remote pairing once a week on our team’s official Work From Home day, so we’d already settled on Tuple for screen sharing and got the team outfitted with comfy gaming headset/mic combos. 

Getting started

One of the best things we did when the shelter-in-place started was explicitly set a goal of doing six experiments to see how we could improve our experience pairing full-time remotely. We love to iterate, and this was no exception. Two of our early wins came from this mandate: the second screen and daily diff.

Second Screen

Every blog post about remote pairing will tell you to have the person’s face visible. Most of them will tell you to do it on a second screen. I was initially very skeptical of having video on while pairing. I was worried that my Zoom fatigue would be even worse from having someone stare at my face literally all day. But it turns out that when you’re pairing in person, you primarily focus on the code on the screen while occasionally glancing at your pair’s face. Setting up a second screen with your pair’s face means you can do the same thing, from the safety of your own homes! I am now 100% Team Second Screen For Video, and whenever I don’t have it (uncharged iPad, etc.), I miss those subtle cues that you get from seeing someone’s expressions. 

Everyone got a refurbished iPad for their second screen, but once I got an external monitor, I started using the big monitor for sharing code and my laptop screen for the hangout. We use Tuple for screen sharing and audio, and use a muted Google Hangout for video.

Daily Diff

After our cross-functional standup ends, the devs stay on and do a live review of the previous day’s commits on GitHub. Some teams do Daily Diff silently, reading through the code themselves, but we’ve found it most effective for our team to have a rotating facilitator share their screen and talk through the code in each commit. The devs who did the work can add any color that they think would be useful, and light debate sometimes ensues about the trade-offs of a given approach. If something merits more than a quick back-and-forth, we move it to our weekly dev team meeting to discuss in more depth. As a pairing shop, most of our programming happens out loud. When we were co-located, we got a lot out of the background chatter of the other pairs’ thought processes. Now that we’re remote, that information is gone, and daily diff helps fill its role pretty well.

Experimentation via goals

When we’re figuring out what our quarterly goals should be, we often are able to identify an area that we’d like to improve, but don’t necessarily know specifically what we want to change. So we’ve embraced the ambiguity and, with much success, have had goals to improve something. We’re currently trying out new ways to incorporate explicit technical learning into our process (so far: mob refactoring sessions, dev hack days, discussing conference talks). Having the goal of simply trying and evaluating a number of experiments allows us to play with new ideas without feeling tied to them for a whole quarter.

Pairing, even for short durations

At BackerKit, we pair nearly all the time and rotate pairs every day. But meetings are still part of our reality, and changes in responsibility and environment (hello children home all day) have also been a reality because of the pandemic. Both of which can result in some days feeling pretty disjointed. Challenging our black and white thinking about whether to start pairing with someone — even just for an hour — was really useful. Although we still sometimes solo for an hour here and there, recognizing that there’s still plenty of value in pairing with someone for an hour or half an hour helped us feel connected and productive.

Embracing hand signals

Tuple, our screen-sharing product of choice, didn’t initially handle two people calling each other simultaneously very elegantly. It was confusing. Since we usually had video going (on those second screens), we developed hand signals to mean “I’m calling you” and “I’m waiting for your call.”

We later realized this wasn’t strictly necessary since we could simply unmute the video before we started the Tuple session, but it is still fun to use silly hand signals sometimes.

As a whole company, we have also adopted a practice of holding up a crossed pointer and middle finger to indicate visually that we have an idea to share. This has been really helpful for cutting down on instances of everyone trying to talk at once or accidentally jumping in before someone is done (since it can even be hard in person to know when someone’s finished speaking). We don’t usually keep a formal stack, though, and also haven’t come up with an actual name for the gesture. We alternate between saying things like “Ian is doing the finger thingy,” “I see Lindsey’s crossed fingers,” and briefly trying to name the gesture “dibs” (i.e., “I see Max has dibs”). 

New company goals

We had a company all-hands shortly after the Bay Area shelter-in-place began, and in it, Maxwell Salzberg, our CEO, laid out our revised company goals. They were:

1. Take care of yourself
2. Take care of each other
3. Take care of our customers

It was incredibly meaningful to know that all of BackerKit, from the top down, was prioritizing our humanity and understood that it was going to be a tough time, while not forgetting that our customers needed our support, too. Throughout this rollercoaster of a year, we’ve tried to prioritize our mental health and have encouraged each other to do what is needed to make it through. (As a parent of two small children, I feel like I hit the lottery, as my team embraced appearances by babies and toddlers in standup and pairing sessions.)

Iteration 4 life

Hopefully, some of these practices could be useful for your team! We don’t know what the future of remote work looks like at BackerKit once it’s safe to be in an office again, but using experiments to improve our process is definitely here to stay, no matter what.

To learn more about what the BackerKit team is working on and get crowdfunding tips, make sure to sign up for our Community Newsletter.

The post How Experimentation Kept Us Shipping Code and Growing Through a Pandemic appeared first on Crowdfunding Blog & Resources | BackerKit.

Working as a distributed team can be tricky, but if you do it right, it can be great. I’ve been a remote developer at BackerKit for three years. It’s been a great ride. In this post, I’m going to share some of the things I’ve learned, and what product teams can do to make it all work.

It’s important to reduce friction wherever possible. Just because a team member is remote, doesn’t mean it has to feel that way. Remote team members can feel like full team-members, and the on-site workers can feel as though they aren’t slowed down by the fact that other people are remote. Being thoughtful of the five modes of operation will go a long way when finding the best ways to communicate. Those modes are:

1. Holding efficient distributed meetings
2. Maintaining a permanent telepresence portal 
3. Effectively remote pairing
4. Leveraging digital organization tools
5. Embracing casual interactions

Each of these modes are important in their own way, and each involve different considerations to optimize interactions between team members. The omnipresent challenge in each of these modes is reducing the friction that the distance creates. With a little thought and a little attention to establishing effective habits and protocols, the telepresence technology can blend into the background, and remote-teams can be incredibly effective.

Holding Efficient Distributed Meetings

Being remote doesn’t mean being absent. Things get better when remote workers are fully present at meetings. Being a remote participant at a meeting is familiar to most professionals, as is the friction that can go with it. Reduce that friction.

For small-ish teams, keep it simple. What works for BackerKit is a simple Google Hangout sessions with a high quality microphone. We use the Jabra Speak 410 for smaller meetings and the Jabra Speak 810 for larger ones.

At BackerKit, we’ve established the convention that the person who is the organizer of the meeting in the calendar makes sure to have a computer pointed at the Hangout before the meeting begins. Google makes it easy to attach Hangout URLs to Google Calendar meeting events, and you can even configure your GoogleApps domain to add Hangouts to new appointments automatically, making it all routine and thoughtless. Each meeting is also linked to a GoogleDoc with a meeting agenda and notes (more on that later).

Once we decided what our meeting telepresence setup looks like, we realize the details aren’t as important as the clarity of the decision. Like so many things, what is actually decided isn’t as important as the fact that it is decided. And that decision has to be known by everyone. Friction has been reduced.

Maintaining A Permanent Telepresence Portal

It’s also important to be accessible outside of meetings. For smaller teams with less structure, and with fewer formal meetings, this is especially important.

First, it is important that remote team members be full team-members, and be as accessible as the on-site folks. It’s natural to choose the path of least resistance, and if there is more friction in speaking with a remote team-member, people in the office will always look for an on-site team member first.

At BackerKit, we have an open office plan, and in our development corner, we have an always-on video chat set up. Remote developers are expected to be accessible there when not pairing, in meetings, or purposefully absent to deep-focus on something. Presenting the remote team members as friendly faces in the office with an always-on, static location lowers friction, and opens doors to a conversation that would otherwise not happen. Reducing the friction can almost make the distance disappear. When people don’t have to fiddle with headphones and microphones, it works wonders for making remote workers feel truly present.

Having an always-on connection provides the remote team members with ears in the room. When remote developers are not pairing, it’s great to be able to not only participate, but also just overhear all of the casual conversation that happens in the developer’s corner. Having just a bit of reference to what the team has been talking casually about gives remote workers the important context they need to tackle assignments, and to effectively participate in more formal conversations at later meetings.

Having an always-on connection also provides the remote team members eyes in the room, which, it turns out, reduces friction further because remote devs don’t have to wonder who is currently pairing, who is at lunch, if the project manager is available, which developers might be available for a quick chat, and so on.

At BackerKit, we use the Pluot/Daily service for our always-on connection. It’s simple and affordable. They provide dedicated little Linux appliance which can be connected to a monitor or TV via HDMI or DisplayPort, and a decent USB speakerphone. We configure the appliance to always join our static conference room and leave the thing on all the time so that remote developers can drop in with no friction, using their own dedicated hardware or via a free browser.

Effectively Remote Pairing

Pair programing is an esoteric art, and remote-pair programming has its own challenges. It’s important to have an established routine to set expectations, synchronize schedules, and help reduce setup friction. At BackerKit, developers are expected to come to stand-up having already breakfasted and caffeinated, and be ready to pair one minute after standup. We rotate pairs, and we don’t close our five minute stand up until we know who is pairing with who for the day.

We have our sharing technology worked out (Screenhero*) and it is fired up and ready to go right after standup. Remote devs don’t drop out of the standup Pluot/Daily room until they are joining a pairing Screenhero connection so there is no time lost in transitioning between portals. Having these expectations settled and routines established helps pairs get up and running quickly.

*Screenhero has merged with Slack. The stand-alone Screenhero software will shut down on Dec 1, 2017. Most of the functionality of Screenhero is now available in paid versions of Slack.

Leveraging Digital Organization Tools

If you want to enable remote-work, consider remaining just a tad more organized than you think you need to be as your team grows. Organized knowledge repositories and well-defined asynchronous workflows become more important as more people are involved. It eventually becomes impossible for a Product team to hold all the acquired knowledge in their heads, and tools become necessary just to understand what everyone is working on. Fairly standard product team organizational tools and best-practices can be leveraged to address these challenges. These tools and practices provide additional value to distributed teams because they foster information transfers into modes that aren’t dependent on co-location and synchrony.

Organization is always a work in progress, but at BackerKit, we try to have a place for everything and everything in its place. Notably, we’ve had some discussion about where this information should live. We’ve considered separate repositories on GitHub, Google Docs, and other wikis. We aren’t in love with the GitHub wiki in particular, but we’ve realized that having a clearly dedicated place is a better approach than delaying organizational decisions. We can always move the information to a different platform if we change our minds. Again, clarity of process is far more important than the process itself.

BackerKit is a fully agile shop, and we’ve always broken our work into stories and tracked them in Pivotal Tracker. As our team has grown, we’ve found the need to include more and more information about stories in Tracker, but having verbose stories really helps the remote developers who may have slightly reduced access to casual conversations with our product manager and designers.

Use a messaging system that isn’t email. It makes communication better and is especially helpful for remote team members who have reduced access to conversations and the sticky-notes you would otherwise be putting up on the wall. We use Slack (and Hipchat before it). If something is important or needs to go to the whole team, it goes in our Dev/Product Slack channel. Unlike conversations, Slack can include links. Unlike conversations, Slack persists and is still there when people who are absent return. And unlike conversations, Slack doesn’t disrupt the paired-development.

We have specified times for certain types of discussions. These include morning standup, weekly retros, weekly 1-1 meetings with team leaders, monthly developer’s technical discussions, and security reviews. Having a time-and-place for these things is generally a great idea because it helps keep designers and developers focused on the task at hand – allowing them to prioritize more easily, and rest-easy knowing that there will be a forum to discuss issues that crop up during work.

Every standing meeting has a living GoogleDoc attached to a calendar appointment. Each one has a living agenda on it and team members are free to dump things into that agenda during the work intervals between meetings.

Corralling conversation this way is not only a great way to organize thoughts for growing teams, but it goes a long way in keeping remote folks in the loop. As much as our always-on Pluot connection provides ears in the room, remote-workers will inevitably miss a non-trivial amount of the casual conversation. By trying to corral non-urgent substantive discussion into formal meetings, remote-workers have the ability to more-fully participate.

Having semi-formal documentation of knowledge-earned and work-to-be-done serves a primary function of allowing larger teams to work together. It also works wonders for bridging the distance and working asynchronously.

Embracing Casual Interactions

Establishing deep relationships takes face-time and downtime – and there are no fully effective shortcuts. Distributed teams need to be especially cognizant of allowing space for casual interactions, and they also need to get together in meatspace sometimes.

Having remote workers feel ‘present’ means carving out spaces to be social and allowing relationships to breathe. This supports team members to be empathetic human beings and helps build trust. This can be the most difficult challenge for distributed teams. Establishing rapport, especially with the non-developers, is hard. Simon Sineck says it well in his “Millennials in the Workforce” talk:

There should be no cellphones in conference rooms…. “When sitting and waiting for a meeting to start … everyone should be focused on building relationships. We ask personal questions, “How’s your dad? I heard he was in the hospital.” “Oh he’s really good thanks for asking. He’s actually at home now.” “Oh I’m glad to hear that.” “That was really amazing.” “I know, it was really scary for a while there.” — That’s how you form relationships. “Hey did you ever get that report done?” “No, I totally forgot.” “Hey, I can help you out. Let me help you.” “Really?” — That’s how trust forms. Trust doesn’t form at an event in a day. Even bad times don’t form trust immediately. It’s the slow, steady consistency and we need to create mechanisms where we allow for those little innocuous interactions to happen.

Remote folks can consider being a few minutes early to meetings or staying online for a few minutes afterward to create opportunities for casual non-work conversation. This is another reason why having the telepresence-tech well established is so important.

But this still won’t be enough for distributed teams. There is absolutely nothing (yet) that can replace full face-to-face interaction. As great as the various telepresence solutions are, they are not high-bandwidth-enough to establish deep relationships.

At BackerKit, our remote developers spend more than four weeks a year on-site. This includes an annual 3-5 day corporate retreat where the entire company is offsite, where the agenda gives a generous amount of downtime. These times are essential. The telepresence in-between keeps the relationships alive, but meatspace is usually where they are built and evolve.

Remote workers should try to visit often, especially with rapidly growing teams with lots of new faces. It’s the moments of down-time and non-work time where real trust and empathy are built. Remote folks need to go to the office often to share in that, break-bread, and share beverages (caffeinated or otherwise).

Notes on Distributed Teams and Company Growth

BackerKit has grown significantly in the past two years, and each stage posed significant but overcome-able challenges. We’ve learned a lot, below are just a few of the highlights.

Small distributed teams don’t have to think too hard about it in the beginning, but once a team reaches a certain size, they will need to decide if they are willing to commit to making the ‘distributed’ thing work. Remote workers can’t make it happen alone. Even with the simplest telepresence setups, someone will need to be in the office to turn them on. But with a little thoughtfulness on both-sides, it can work well.

Three years ago, having a distributed team as a small company was easy. Location wasn’t very important. The conversations were always small enough that remote workers were able to fully participate, either via A/V telepresence or our main HipChat room. Hearing everyone was easy, and microphones weren’t an issue. The lone remote-developer (me) of our six-person company was always either pair programming, or on the always-on Google Hangout in the small office room. We were not incredibly organized, but we didn’t need to be. It worked.

As BackerKit grew larger, distributed work got harder. Being big enough to need a procedure, but not big enough to have it all worked out, was a hard time for us to be distributed. Conversations became more fragmented. Everyone was doing the start-up scramble, meetings were sometimes ad-hoc, and we often didn’t have the patience to work out the telepresence technology. Not everyone on the company needed to be on every thread, and it took a lot more thought and attention to make sure that remote workers were part of the conversations they needed to be in.  At these early growth stages, it was essential to have a guardian angel for the remote-workers in the office– someone who was always willing to restart the Google Hangout and own the telepresence technology. (Thanks Kaz!)

As we grew larger and decided that we were committing to being organized and being thoughtful, being distributed got much easier. Once we had two or three pairs of developers working in our code-base, we quickly developed the infrastructure and procedure we needed to keep our work organized. We solidified our organizational procedures and culture, started using google-calendar religiously, and started to cross-reference everything. It became easier for a remote team member to plug into our process, see the landscape for the week/month, and contribute. We also took on an office manager (Thanks Annie!) who worked wonders to get our technology needs worked out and make the rooms work for remote team-members.

Making it all work

Be thoughtful. Define your different types of team interactions and optimize them. Reduce the friction wherever possible, and don’t forget to be human. The BackerKit Product Team is proof that having remote developers can work well.

The post Developer’s Guide to Successful Distributed Teams appeared first on Crowdfunding Blog & Resources | BackerKit.

Digging the BackerKit Vibes

In my decade after college graduation, I held a wide variety of positions. I specialized in hardware configurations, drafted software contracts, and became a full-stack dev at a startup focusing on diversity and inclusion. Each team had a unique culture that brought along a diverse set of opportunities pushing me to grow.

I joined the BackerKit team as a Developer earlier this summer. I was excited to see what was unique about BackerKit, but was blown away with all the team’s positive characteristics. If you are curious about what sets the BackerKit team apart, here are a few facets that stood out revolving around the interview and on-boarding process along with all the support I’ve received.

Technical Interviews Don’t Have to be Painful?!

If you happen to be in the tech scene, this is not going to be a shock…technical interviews tend to suck. If you are not, you can get the gist from Sarah Mei’s tweet stream and Sam Phippen’s Medium post.

Real Experience
The first thing that really hit me about BackerKit was the difference I felt when leaving the interview. Typically I leave an interview feeling exhausted and uncertain. Not this time! While interviewing with BackerKit, it felt more like a collaboration than a test. I felt relaxed and comfortable, and was focused on getting to know the team and seeing if I felt like a good fit.

Real Pairing
Most developer interviews have some sort of pair programming that is included. This usually involves the interviewee solving a problem while the interviewer observes and gives feedback. It was no surprise that BackerKit wanted me to come in and pair, but I couldn’t believe it when we actually pair programmed! No standing alone at the whiteboard, and no solving problems in Java. We worked on a real story that we were seeing for the first time together as a pair. We were able to brainstorm together, shooting out ideas, and discuss the best way for us to move forward. It was so nice to see how a day in the life would really be working through a problem and collaborating together.

Real People
Part of the reason why full-day interviews can be so exhausting is you need to be on 100% of the time even discussing technical topics over lunch. At BackerKit, we went to lunch together as a team, but instead of discussing what side projects everyone was working on, we spent our lunch just talking about life outside of work. Everyone had a passion outside of tech and would spend their nights and weekends doing something else they loved. When I met with the co-founders, Rosanna and Maxwell, they also brought up outside passions. It was apparent that everyone on the team loved working at BackerKit, and had something unique they were interested in when not at the office.

On-boarding For the Win!
For being a company on the smaller side, I was not expecting there to be any sort of official on-boarding, but BackerKit gets the importance of starting employees off on the right foot. I arrived to the office the first day with a welcome packet full of goodies and a clear syllabus that accounted for everything!

Meeting the Team
My first week was dedicated to on-boarding with teams across the entire company. We spent our time digging deep into crowdfunding, learned the history of BackerKit, and discovered what led the company to be where it is today. We explored what BackerKit does to support creators and their backers. My favorite part of the week was spending time sitting in with the support team diving into the various features and discovering potential roadblocks.

Getting up to Speed Through Pairing
In my previous position, I spent only a few hours a week pairing. This was not the case at BackerKit. I was a bit nervous about the prospect of pairing for an entire day. Turns out, full-on pairing is an awesome way to get up to speed with a new code base! We had to slow down a bit for the first couple weeks, spending a little extra time mapping out relationships and allowing me to get used to the editor-of-choice’s (RubyMine) features and shortcuts. Having only been here for 3 months, I’m completely comfortable navigating my way through the app and the team is now approaching me with questions on how things work!

Support From All Angles

BackerKit has a core value of learning through failing. An environment that removes the fear of setting challenging goals allows a path forward to experiment and find the simplest solution. Feedback can then be assessed to see what lessons have been learned and where to apply the knowledge gained.

Even though this approach builds strength, learning through failing brings along a natural course of ups and downs. What has stood out most was the proactive efforts to acknowledge and provide guidance throughout the learning process.

Weekly One-on-One
Everyone on the team has a weekly one-on-one with their department manager. Here you can set goals for yourself, evaluate what has been working, and address any struggles. I appreciate having a manager that can help me recognize areas of growth. In the past, I’ve tended to burn out from taking on too much too fast. We worked together to reduce the load on my plate reprioritizing top goals to focus on. This removed a huge weight off my shoulders that I hadn’t realized I was carrying.

Peer Support
At BackerKit, my peers are not just on the developer team, but the company team at large. It’s been awesome getting to work daily with teammates across all departments. I’ve set some personal goals for myself, including writing this blog post, and it’s been so comforting to have so many of my peers share the excitement as I make progress.

Stay Tuned

“When you learn, teach, when you get, give” ~ Maya Angelou

I have learned more than I could imagine in my first 90 days here at BackerKit. Finding the right opportunity and joining a new team is not an easy task, but BackerKit has gone above and beyond to do their best to make me feel welcome and set me up with the tools to succeed. I’m excited to grow and share what I’ve learned along the way here on the BackerKit blog.

Does BackerKit sound like a fit for you? Check out our current job openings. And if you would like to chat with the development team, drop us a line: engineering@backerkit.com.

The post Finding My Fit At BackerKit appeared first on Crowdfunding Blog & Resources | BackerKit.

Many web applications have the need to send links to users that allow the user to perform an action without logging in. Examples include password reset and confirming an email address. These links need to embed some kind of identifying information so the server can discern which user is performing the action. If the action is sensitive, the link should also include some amount of obfuscation so the URLs cannot be guessed or generated by nefarious actors.

We have a few ways to generate links that serve this purpose. Many applications start with something like Rails’ has_secure_token. This helper wraps up the logic to generate and persist tokens in a datastore. This approach is well understood and easy to reason about.

What are some of the downsides of storing tokens on the server? Storing the tokens in plaintext means if an attacker gets access to the application’s database, all of the stored tokens are exposed. Hashing (or digesting) the tokens asymmetrically fixes this issue but we still have to persist and protect this data.

What if we don’t need to persist the token at all? What if we could generate a token, send it out in a link, and then verify it when it gets sent back to us? In this way, we don’t have to store the tokens, which means there is less to protect.

We can achieve this stateless approach with encoded and signed tokens. Rails provides a mechanism to generate and verify tokens via ActiveSupport::MessageEncryptor and ActiveSupport::MessageVerifier.

We recently implemented stateless tokens in our app so users could confirm subscriptions to mailing lists. Below we’ll go through some code to get us there.

Let’s look at some code

We have a controller with a create action that allows a user to submit an email address and a confirmations action that verifies a submitted token.

class SubscriptionsController < ApplicationController
  def create
    subscription = Subscription.build(email: subscription_params[:email])

    if subscription.save
      SubscriptionsMailer.confirm_subscription(subscription.id).deliver_later  
      redirect_to subscriptions_path
    else
      redirect_to subscriptions_path
    end
  end

  def confirmations
    subscription = Subscription.verify_token_and_find(token: params[:token])

    if subscription.present?
      subscription.touch(:confirmed_at)
      flash[:notice] = "Thanks for subscribing!"
    else
      flash[:error] = "Oh no! Your token is not valid."
    end

    redirect_to subscriptions_path
  end

  private

  def subscription_params
    params.require(:subscription).permit(:email)
  end
end

And our model looks like this:

class Subscription < ApplicationRecord
  CONFIRMATION_IN_DAYS = 7
  scope :confirmed, lambda { where.not(confirmed_at: nil) }
  validates :email, uniqueness: true

  def generate_token(expiration_in_days: CONFIRMATION_IN_DAYS)
    expiration = Time.current + expiration_in_days.days
    token_values = { id: id, expiration: expiration }
    self.class.encryptor.encrypt_and_sign(token_values)
  end

  def self.verify_token_and_find(token:)
    begin
      data = encryptor.decrypt_and_verify(token)
      given_expiration = data[:expiration]

      if given_expiration < Time.current
        return nil
      end

      find(data[:id])
    rescue ActiveSupport::MessageVerifier::InvalidSignature
      nil
    end
  end

  def self.encryptor
    key_password_for_mailing_list = ENV.fetch('KEY_PASSWORD_FOR_SUBSCRIPTION')
    SUBSCRIPTION_ENCRYPTOR_KEY = ActiveSupport::KeyGenerator
      .new(key_password_for_mailing_list)
      .generate_key(salt, 32)

    ActiveSupport::MessageEncryptor.new(SUBSCRIPTION_ENCRYPTOR_KEY)
  end
end

Our mailer is relatively straightforward except for the line where we call subscription.generate_token, which does the heavy lifting for generating and signing our token:

class SubscriptionsMailer < ActionMailer::Base
  def confirm_subscription(subscription_id)
    subscription = Subscription.find(subscription_id)
    @link = confirmation_url(subscription: subscription)

    mail(
      from: "support@yourwebsite.com",
      subject: "Please Confirm Your Subscription",
      to: subscription.email
    ) do |format|
      format.text
      format.html
    end
  end

  private

  def confirmation_url(subscription:)
    subscriptions_url(token: subscription.generate_token)
  end
end

What’s the catch?

One downside of this approach is the need to manage another key, which involves protecting it (likely via using environment variables) and being able to rotate the key in an operationally simple manner. There is not a great option out of the box to manage these workflows but it’s worth considering the costs and benefits of not storing this data.

Conclusion

We wanted to find a way to avoid storing keys in our database for security and storage reasons and this approach serves our purpose. Expiring, rotating, and protecting keys still require forethought, but that’s nothing new.

Want to learn more?

• This RailsConf talk has a nice overview of thinking in stateless terms
• Check out the source code
• Read these blog posts for a more in depth overview of using these tools
• If this stuff is interesting to you and you’d like to chat, drop us a line: engineering@backerkit.com

The post Using Rails’ MessageVerifier for Stateless Token Management appeared first on Crowdfunding Blog & Resources | BackerKit.

In a previous post we explored how our application tracks sent emails. We set things up to update or insert a record for each email with the current state (delivered, dropped, etc). In this post we’ll look at an alternative data model where we’ll store each event as its own record and then roll up the events to get the current state of a particular email.

With this different modeling we’ll be able to understand the full lifecycle of an email allowing us to debug issues better as well as have a better understanding of how our users interact with our emails. In our original setup we could only know about the most recent email event. We’ll use the concept of event sourcing to maintain this history.

Our Setup

Here’s what our table will look like:

CREATE TABLE emails (
  id SERIAL PRIMARY KEY,
  message_id varchar NOT NULL,
  status varchar NOT NULL,
  created_at timestamp DEFAULT current_timestamp,
  updated_at timestamp DEFAULT current_timestamp
);

CREATE INDEX ON emails (message_id);

Here is some data to get us started.

INSERT INTO emails VALUES (
  DEFAULT, 'first-message', 'sent', now(), now()
);
INSERT INTO emails VALUES (
  DEFAULT, 'first-message', 'delivered', now(), now()
);
INSERT INTO emails VALUES (
  DEFAULT, 'first-message', 'opened', now(), now()
);
INSERT INTO emails VALUES (
  DEFAULT, 'second-message', 'sent', now(), now()
);

We have two emails being tracked. One has been sent, delivered, and opened; the other has only been sent, but not yet delivered or opened.

Check out this SQLFiddle if you’d like to explore the data or queries yourself.

Query for a Single Message

Given we have a single message_id ('first-message', for example) and we want to find its current state, we could write a query like this one:

SELECT
  message_id,
  status
FROM emails
WHERE message_id = 'first-message'
ORDER BY created_at DESC
LIMIT 1;

Our output looks like:

message_id     | status
-----------------------
first-message  | opened

Query Across All Messages

If we wanted to get the current state of all the emails, we would need to write a different query than we’re used to. We still want the most recent event but now we want it across a list of emails. As a result we want to get a single event (the most recent one) for each message. We can use Postgres’ DISTINCT ON notation to do just that.

SELECT DISTINCT ON (message_id)
  message_id,
  status
FROM emails
ORDER BY message_id, created_at DESC;

Note: this query has been updated from the original post to be simpler as a result of feedback on Hacker News.

Our output looks like:

message_id     | status
-----------------------
first-message  | opened
second-message | sent

With this kind of data model and queries, we can now understand the current state of the world as well as interrogate our system to understand all the statuses through which an email went.

Further Reading

• Martin Fowler’s event sourcing post
• Explore refactoring this query into an ORM layer
• Learn about using EXPLAIN ANALYZE to understand performance of this kind of query
• If this stuff is interesting to you and you’d like to chat, drop us a line: engineering@backerkit.com

The post Getting Started with Event Sourcing in Postgres appeared first on Crowdfunding Blog & Resources | BackerKit.

Heroku makes it dead simple to deploy and operate apps; Docker makes it even easier! Heroku has a Docker runtime in beta.

This article will review how to setup a simple Ruby app to build and deploy as a Docker container on Heroku using CircleCI.

The app we’ll be using

We run a dashboard for internal use with a range of widgets that give us data about various events in the company and application. We use Smashing, which is a Sinatra-esque Ruby framework for building dashboards.

Setting up the Dockerfile

Heroku recently released their new Heroku-16 stack. We’ll base our image off it. We’ll then install required system dependencies so we can install gems. Next we’ll install bundler and our gem dependencies. From there we’ll give the container a default CMD that will boot up the dashboard.

FROM heroku/heroku:16

RUN apt-get update -qq && apt-get install -y \
  build-essential \
  libpq-dev \
  nodejs \
  ruby-dev
RUN mkdir /my_dashboard
WORKDIR /my_dashboard
ADD Gemfile /my_dashboard/Gemfile
ADD Gemfile.lock /my_dashboard/Gemfile.lock
RUN gem install bundler
RUN bundle install
ADD . /my_dashboard
CMD rackup --port=$PORT

In order to build this image, we’ll need to install Docker. Assuming we’re on Mac, we’ll head over here to install Docker. Once we’ve installed Docker, we’ll cd into the project directory and run the following commands:

docker build . -t my_dashboard
docker run -p 9292:9292 my_dashboard rackup

Now we should have the app up and running!

If everything looks good, deploy the image manually to Heroku using the following commands.

heroku plugins:install heroku-container-registry
heroku container:login
heroku container:push
heroku open

Our dashboard is alive on Docker!

Building and deploying via CI

We want to be able to deploy our app from our continuous integration system instead of pushing manually. We’ll use CircleCI.

Add a circle.yml file to our project with the following config.

machine:
  services:
    - docker

dependencies:
  override:
    - echo "no deps"

test:
  post:
    - echo "yay - no tests"

deployment:
  production:
    branch: master
    commands:
      - docker build -t my_dashboard .
      - docker login --email=_ --username=_ --password=$HEROKU_TOKEN registry.heroku.com
      - docker tag my_dashboard:latest registry.heroku.com/my_dashboard/web
      - docker push registry.heroku.com/my_dashboard/web

We’ll need to configure an environment variable for HEROKU_TOKEN (taken from our Heroku dashboard) so Circle can authenticate against the Heroku container registry.

Upon the next successful master build, we should see our image be built and then deployed to Heroku all from CI!

Open questions

Working with Docker raises some open questions around developer workflow, especially for more complicated applications (e.g. those with a separate worker process).

Here are some of the issues:

• How do we run tests? Do we build an image (including test gems) and run tests against it?
• How would we manage compiling assets if we needed to?
• How do we ergonomically deploy multiple processes (assuming one process per container)?

Further reading

This post is just a taste of working with Docker. If you’re interested, check out the following resources to learn more.

• Docker video intro
• Written tutorial on getting started
• Introduction to Docker Compose if you want to have multiple containers coordinating with each other
• Overview of using registries

The post Building and Deploying Ruby Docker Images with CircleCI and Heroku appeared first on Crowdfunding Blog & Resources | BackerKit.

Our application is driven by calls to action delivered to project backers via email. We send ~2 million emails a month with 98%+ deliverability.

Given the important role email plays in our business, we wanted to have better technical and behavioral insights into what was going on with our sent emails. We wanted to know more about our email at the project and backer level as well as what kind of impacts our emails have on conversion rates. Here are some of the steps we took to get a better picture:

Webhooks vs on-the-fly querying

We use Mailgun for sending email. Their platform provides two main ways to understand what your email is doing:

1. HTTP APIs with rolled up statistics
2. Event-based webhooks

We weren’t sure of our use cases for the data yet, so we decided that storing the data in our database would be the most flexible path forward.

Sending headers via ActionMailer callbacks

We instrumented the mailers of interest to include additional SMTP headers including the backer_id and project_id. ActionMailer callbacks allowed us a convenient hook into the email-sending lifecycle that looks like the below wrapper:

class BackerMailer < ActionMailer::Base
  after_action :add_metadata_headers
  ...

  def sample_email(backer_id, project_id)
    @backer = Backer.find(backer_id)
    @project = Project.find(project_id)
    ...
  end

  def add_metadata_headers
    headers['X-Mailgun-Variables'] = {
      "backer_id": @backer.id,
      "project_id": @project.id,
      "subject": mail.subject
    }.to_json
  end
end

Upon processing the event webhooks, we read out these headers again and persisted the following data:

• backer_id
• project_id
• email subject
• message_id (unique across Mailgun)
• event status (delivered, dropped, complained, bounced)

We set up an endpoint in our app to receive webhooks from Mailgun:

class Emails::SentEmailsController < ApplicationController
  before_action :verify_signature

  def create
    if missing_required_params?
      # tell Mailgun not to try again
      head :ok
      return
    end

    process_event!
    head :ok
  end

  private

  def missing_required_params?
    # assert the required headers are present so we process
    # only the emails for which we've passed in our custom headers
    ...
  end

  def process_event!
    sent_email = SentEmail.find_or_initialize_by(
      message_id: params["Message-Id"]
    )

    sent_email.backer_id = backer_id
    sent_email.status = event_status
    sent_email.subject = subject.squish

    sent_email.save!
  end

  def verify_signature
    # verify the signature of the request
    ...
  end
end

Storing and updating events

The underlying model and table looked like this:

class SentEmail < ApplicationRecord
  belongs_to :backer
end

# == Schema Information
#
# Table name: sent_emails
#
#  id         :integer          not null, primary key
#  message_id :string           not null
#  status     :string           not null
#  backer_id  :integer          not null
#  subject    :string           not null
#  created_at :datetime         not null
#  updated_at :datetime         not null
#
# Indexes
#
#  index_sent_emails_on_backer_id   (backer_id)
#  index_sent_emails_on_message_id  (message_id) UNIQUE
#

With this setup, we could query for emails sent to a particular backer or roll up statistics for a given project. This setup has given us much better insight into the emails we send to each user and the higher-level effect our emails are having on each particular project.

Questions we can now answer

With the above setup we can interrogate our system about email deliverability. Here are some specific queries we’re running:

• Backer level deliverability
  • backer.sent_emails.group(:status).count
• Project level deliverability
  • project.sent_emails.group(:status).count
• Project and email type deliverability
  • project.sent_emails.group(:subject, :status).count
• Email deliverability across our system
  • SentEmail.group(:status).count

Having this data in our system means we can now surface relevant statistics to our project creators about what our application is doing and how emails affect their business.

A couple of optimization improvements

With the above implementation we are seeing response times averaging around ~30ms with reasonable P99 outlier response times. Although there are a handful of improvements we could make (see below), we are cautious not to pre-optimize and have been happy with this setup.

We are continuing to monitor things. In the event of performance issues, here are a couple ideas we are considering for the future:

• Enqueueing background jobs for later asynchronous processing in the webhook instead of doing Postgres queries in-band. We would be trading a Postgres query for a Redis query here.
• Extracting the controller action into middleware and reducing the amount of the Rails stack required to process the action.

Further reading

• Mailgun webhooks and stats API documentation
• Explore event sourcing to maintain an immutable list of events; not just the most recent status of each email
• If this stuff is interesting to you and you’d like to chat, drop us a line: engineering@backerkit.com

The post Tracking and Analyzing Emails Using Webhooks appeared first on Crowdfunding Blog & Resources | BackerKit.

At BackerKit, we occasionally see high volumes of traffic from malicious clients. (Kickstarter has faced a similar problem.) These DDoS attacks result in degraded performance and frustrate our customers. Not cool!

We implemented Kickstarter’s Rack::Attack and configured constraints on the number of requests allowed in a time period based on IP address on our troublesome endpoints. Yay, problem solved!

Like most tools, Rack::Attack requires tuning; our initial stab at configuration led to customers being blocked. We needed a way to clear out blocked IPs that were friendly.

Unblocking Friendlies

We started with tooling developers could use to manually clear a key.  Keys of blocked IPs look like rack::attack:allow2ban:ban:104.62.144.205. Given a target IP address we could go find that key in our cache (we use Redis) and delete it.

Our first manual step was to filter the Rails’ cache’s keys and find ones used by Rack::Attack that includes our target IP to unblock.

Rails.cache.data.keys
.find_all{ |k| k.include?(':ban:') }
.find_all{ |x| x.include?('104.62.144.205') }

We could then delete the found Redis key. Note this approach will work on only cache backends supporting the data method. Notably, ActiveSupport::Cache::FileStore (frequently used in development) does not support it. In addition, this operation is O(N) so beware of performance issues on large caches.

We wanted to type less and make fewer mistakes, so we wrapped some of that logic into the following helper:

# config/initializers/rack_attack.rb
module BannedIpGetters
def banned_ips
with do |conn|
return conn.keys.find_all { |x| x.include?(':ban:') }
end
end
end

ActiveSupport::Cache::RedisStore.include(BannedIpGetters)

the above helper lets us write:

ip_key_to_unblock = Rails.cache.banned_ips.find_all{|x| x.include?('104.62.144.205')}
Rails.cache.delete(ip_key_to_unblock)

Empowering our Coworkers

Fixing these issues became time-consuming for our dev team, so we created a simple dashboard for our coworkers to use. The dashboard lets them see which IPs have been blacklisted and provides a button to unblock a given IP.

The controller presents an index page with a list of banned IPs and an endpoint that allows for deletion of a blocked key:

class Staff::RackAttacksController < Staff::BaseController
def index
ips = Rails.cache.data.keys.find_all { |x| x.include?('rack::attack') }.sort
@banned_ips, @other_ips = ips.partition { |x| x.include?(':ban:') }
end

def destroy
if params[:ip].starts_with?('rack::attack')
Rails.cache.delete(params[:ip])
end
redirect_to :back, notice: "#{params[:ip]} unblocked"
end
end

Our view looks like:

<% @banned_ips.each do |ip| %>
<%= ip %>
<%= link_to 'Lookup', "http://ip-lookup.net/?ip=#{ip.split(':').last}", target:'_blank' %>
<%= link_to 'Unblock', staff_rack_attack_path(id: 1, ip: ip), method: :delete %>
<% end %>

Rack::Attack Dashboard

Now our non-technical teammates can go to this admin page and quickly find out what Rack::Attack is doing.

Next Steps

• We’re looking to extract this dashboard into a reusable gem. Email us at engineering@backerkit.com if you’d like to work with us
• If you’re in need of cargo boxes, check out rackattack.com

The post Building a Rack::Attack Dashboard appeared first on Crowdfunding Blog & Resources | BackerKit.