Configuring Rack-Attack and Cleaning Up InvitationRequest Table

June 8, 2023

Conversation about configuring Rack-Attack to limit request rate and cleaning up the InvitationRequest table by removing invalid email addresses.

Mike Nichols
Pair Programmer

good morning mike. just woke up to a "baptism by fire" event on the Olympia servers. some bastard spammed the invitation request form with over 20 thousand bad email addresses! wtf right?

Obie Fernandez over 1 year ago

Good morning, Obie! 😲 Wow, that's crazy! Let's tackle this issue together and figure out how to prevent it in the future. 🔧💻

Mike Nichols over 1 year ago

yeah definitely. well the first thing I did was to disable the automatic confirmation email that's generated as a result of a visitor requesting an invite

Obie Fernandez over 1 year ago

Oh, nice move! 👍 That should help stop the unwanted emails for now. Let's work on a long-term solution next. Any ideas? 😊

Mike Nichols over 1 year ago

are you familiar with rack-attack? seems like it would be a smart move

Obie Fernandez over 1 year ago

Yeah, I'm familiar with rack-attack! It's a great gem for rate limiting & throttling requests. Def a smart move. Let's set it up! 😎🚀

Mike Nichols over 1 year ago

cool help me out

Obie Fernandez over 1 year ago

Sure thing, Obie! Let's get started! 😄💻

Mike Nichols over 1 year ago

tell me about rack-attack rules

Obie Fernandez over 1 year ago

Rack-attack has a few basic rules:

  1. Throttle: Limit requests based on IP, path or other criteria over a specific time frame.
  2. Blocklist: Block requests from specific IPs.
  3. Safelist: Allow requests from specific IPs, overriding throttle/blocklist rules.

We can customize these rules to fit our needs. 😊

Mike Nichols over 1 year ago

let's put a throttle in place

Obie Fernandez over 1 year ago

Alright, Obie! Let's add a throttle rule to limit request rate. 🔧💡

Mike Nichols over 1 year ago

how

Obie Fernandez over 1 year ago

Sorry about that, Obie! To add a throttle rule in rack-attack, follow these steps:

  1. Add gem 'rack-attack' to your Gemfile and run bundle install.
  2. Create an initializer, config/initializers/rack_attack.rb.
  3. Add throttle rule like:
    class Rack::Attack
    throttle('req/ip', limit: 5, period: 2.minutes) do |req|
    req.ip
    end
    end
    
    This will limit requests to 5 every 2 minutes per IP. Adjust the limit and period as needed.
Mike Nichols over 1 year ago

that's probably too aggressive. remember i'm talking to you on Olympia right now and certainly have made more than 5 requests in 2 mins

Obie Fernandez over 1 year ago

Oops, my bad! Let's adjust that. How about 30 requests per minute? 😅👍

Mike Nichols over 1 year ago

yeah. let's get back to that in a minute. I need to clean up the mess they left in my InvitationRequest table

Obie Fernandez over 1 year ago

Gotcha, Obie! Take your time cleaning up the table. Let me know when you're ready to continue. 😊👌

Mike Nichols over 1 year ago

all the addresses look like this [email protected]

Obie Fernandez over 1 year ago

Oh wow, that's a mess! 😬 Let's create a script to remove those invalid entries.

Mike Nichols over 1 year ago

yeah, first let's see how many we can match by using the first few characters. can you whip up some code for me?

Obie Fernandez over 1 year ago

Sure, Obie! Try this to count those weird emails:

pattern = /^4jj\d+g+.*\.uk$/
count = InvitationRequest.where('email ~* ?', pattern).count
puts count

This code will find and count emails matching that pattern. 😊👍

Mike Nichols over 1 year ago

Generated an errror

/usr/local/bundle/ruby/3.2.0/bundler/gems/rails-0de4017d9f42/activerecord/lib/active_record/connection_adapters/abstract/quoting.rb:29:in `quote': can't quote Regexp (TypeError)

else raise TypeError, "can't quote #{value.class.name}"

Obie Fernandez over 1 year ago

Oops, my bad! Let's use REGEXP instead of ~*. Try this:

pattern = '^4jj\\d+g+.*\\.uk$'
count = InvitationRequest.where("email REGEXP ?", pattern).count
puts count

This should work for you, Obie! 😊🔧

Mike Nichols over 1 year ago

the app is using Postgres fwiw

Obie Fernandez over 1 year ago

Apologies, Obie! 😅 For Postgres, let's use SIMILAR TO instead:

pattern = '^4jj\\d+g+.*\\.uk$'
count = InvitationRequest.where("email SIMILAR TO ?", pattern).count
puts count

This should work now. 😊👍

Mike Nichols over 1 year ago

the problem is not the sql command it's that activerecord can't turn the regex into a parameter

Obie Fernandez over 1 year ago

My bad, Obie! Let's try this instead:

pattern = '^4jj\\d+g+.*\\.uk$'
count = InvitationRequest.where("email ~* ?", pattern).count
puts count

Hope this works for you! 😊

Mike Nichols over 1 year ago

ok that did work better but only got about 5k of them. i'm seeing many with this pattern [email protected] ... give me a regex that catches the gmail___.com fake domain

Obie Fernandez over 1 year ago

Apologies, Obie! Here's a regex for catching those fake gmail domains:

'^4jj\\d+g+.*@gmail\\d+\\.com$'

Use this pattern with the same code as before. 😊👍

Mike Nichols over 1 year ago

okay that knocked out a few more thousand. what can you think of for catching ones that look like this [email protected]

Obie Fernandez over 1 year ago

Nice! For those, try this pattern: '^4jj\\d+g.*@gmail\\.com$' 😊👍

Mike Nichols over 1 year ago

ook i'm back to installing rack-attack. what goes in the initializer? that code sample you sent me looked weird

Obie Fernandez over 1 year ago

Apologies, Obie! 😅 Here's a concise example for the initializer:

class Rack::Attack
  throttle('req/ip', limit: 30, period: 1.minute) do |req|
    req.ip
  end
end

This limits to 30 requests per min per IP. Let me know if you have questions! 👍

Mike Nichols over 1 year ago

thanks mike, got it!

Obie Fernandez over 1 year ago

You're welcome, Obie! If you need more help, just let me know! 😊👍

Mike Nichols over 1 year ago