Beyond the Single PostgreSQL Node, Part One

When Disco was the hot new music, the people who built database management systems based their praxis around the then-correct assumption that hardware was extremely expensive, and hence that procuring things like more storage, let alone more machines on which they would run, could be counted on to require significant capital resources. This could only happen on the time scale of quarters, even if it was deemed worthwhile to fight the fights needed to procure them.

The above conditions informed the practice of hardening single nodes against failure without adding significant hardware resources in the process. This practice continues, and it's a good thing to have.

However.

Over the past few decades, hardware resources have been dropping precipitously in cost even when acquired via capital expenditure, and tax (read subsidy here) regimes have been altered to put a massive thumb on the scale to favor operational expenditures. This in turn has driven technologies that made it easy to make those operational expenditures, and to make it possible to make such expenditures in small increments.

Now, doubling a node's storage or lighting a powerful new node and having it working in production can and does happen on the scale of minutes, not least because the first chunk of money spent in order to do so is well within the discretion of an individual developer, and a junior one at that.

Topic Transition!

That's a lot of paragraphs on history and political economy, and you may be wondering how they're pertinent to the matter at hand. Wonder no more, as we've gotten there.

Now that it's cheap and easy to light even a database node, generally a pretty expensive type of node to light, we can use techniques to approach problems that would have been untenably profligate in the Disco era. Among these are replication.

It's not all sweetness and light, though. The proliferation of new nodes has created new problems. When you switch the database to that hot new hardware, lots of new systems can need to know about that change, and there can be a large and complex problem of coordination. You might consider DNS as a way to promulgate this news, and now, as jwz once said of regexes, you have two problems.

Fortunately, there's a way to isolate the places where the news needs to go, and a way to do it that normally involves zero downtime deployment: database-aware proxies, specifically pgbouncer. It's pretty safe to put all your eggs in that basket because pgbouncer has been hammered on hard enough that failure of the underlying node is far and away the most common source of failure for it, i.e. you can often get away with having just one node, and you can in dire emergency know that lighting a new node to replace it is a matter of the configuration management you already have in place, and an IP takeover, which can happen on the scale of seconds with practice.

How, you may ask?

That's the cliffhanger for the next installment of Beyond the Single PostgreSQL Node.

Thanks as always to my employer, Aiven, for providing me with the resources to do this work.

Comments

Post a Comment

Popular posts from this blog

Cool Stuff in PostgreSQL 10: Transition Table Triggers

Last time , we created a heirarchy of tables for logging, but we don't have anything they'd log, and we don't have a way to ensure the logging happens automagically.  This time, we'll fix those problems. First, a thing to log: CREATE TABLE foo(     id BIGSERIAL PRIMARY KEY,     t TEXT ); Next, let's make an efficient trigger function.  You may have seen one of these before, but it was probably row-based, so it couldn't be super efficient.  This one is statement-based, and uses the new transition table feature in PostgreSQL 10.  We only need to build one trigger function because our logging infrastructure is schema-agnostic.  We could have separated it into three trigger functions, but that's just more code to maintain.  It starts off as usual: CREATE OR REPLACE FUNCTION log() RETURNS TRIGGER LANGUAGE plpgsql AS $$ BEGIN     IF TG_OP NOT IN ('INSERT', 'UPDATE', 'DELETE') THEN    ...
Read more

Cool Stuff in PostgreSQL 10: Auto-logging

We started off by creating a logging infrastructure, then arranging for a single table to use it. Rather than repeat that work for each table, let's use a relatively although not completely new feature: `EVENT TRIGGER`. The idea here is that we fire a trigger on `CREATE TABLE` and see to it that the table is logged. We'll write the trigger first, even though in reality, we'd need to load the function it calls first. ```sql CREATE EVENT TRIGGER add_logger ON ddl_command_end WHEN tag IN ('create table') EXECUTE PROCEDURE add_logger(); COMMENT ON EVENT TRIGGER add_logger IS 'Ensure that each table which is not a log gets logged'; ``` The magic happens inside `add_logger()`, but it's magic we've already seen. First, we'll get the table's name and schema using `pg_event_trigger_ddl_commands()`, filtering out tables which are already log tables. The test here is crude and string-based, but we could easily go to schema-based...
Read more

Cool Stuff in PostgreSQL 10: Partitioned Audit Table

If you've dealt with PostgreSQL's answer to partitioning in the past, you may have put a lot of work into it.  Happily, you have a lot less to bother with as of PostgreSQL 10, which is not out yet as of this writing. Let's dive right in with a table that's a little like an audit trail. CREATE TABLE IF NOT EXISTS the_log (     "timestamp" timestamp with time zone DEFAULT now() NOT NULL,     "user" text NOT NULL DEFAULT CURRENT_USER,     action text NOT NULL,     table_schema text NOT NULL,     table_name text NOT NULL,     old_row jsonb,     new_row jsonb,     CONSTRAINT the_log_check CHECK (         (old_row IS NOT NULL)::integer +         (new_row IS NOT NULL)::integer > 0     ) ) PARTITION BY LIST(table_schema); COMMENT ON CONSTRAINT the_log_check ON the_log IS 'Make...
Read more