Now that I've committed to a reserved instance for a while, I have to make Secateur scale by making it smarter, not just throwing more cloudy money at it.

To that end, I just chopped off about 75% of its database size without losing any data.

Previously, I was saving the JSON object of every Twitter user's profile as a PostgreSQL JSONB field in its own table

JSONB is fantastic, and awesome when you need it, but it's grossly inefficient compared to normal tuples.

Follow

I had two million profiles taking up about 2gb.

Copying all the interesting fields out of the profile object into regular database columns straight in to the 'account' table will mean a ton less IOPS (which is the real constraint I've got), and only takes 300mb or so.

Sign in to participate in the conversation
Cloud Island

A paid, early access, strongly moderated Mastodon instance hosted entirely in New Zealand.