# Scaling Applications

## Change Notifiers

Oban uses PubSub notifications for communication between nodes, like job inserts, pausing queues,
resuming queues, and metrics for Web. The default notifier is `Oban.Notifiers.Postgres`, which
sends all messages through the database. This provides transactional consistency, but Postgres'
notifications adds up at scale because each one requires a separate query.

If you're clustered, consider switching to an alternative notifier like `Oban.Notifiers.PG`. That
keeps notifications out of the db, reduces total queries, and allows larger messages, with the
tradeoff that notifications from within a database transaction may be sent even if the transaction
is rolled back. As long as you have a functional Distributed Erlang cluster, switching is a
single line change to your Oban config.

```diff
 config :my_app, Oban,
+  notifier: Oban.Notifiers.PG,
```

If you're not clustered, consider using [`Oban.Notifiers.Phoenix`][onp] to send notifications
through an alternative service like Redis.

[onp]: https://github.com/sorentwo/oban_notifiers_phoenix

## Disable Triggers

Inserting jobs emits a trigger notification to let queues know there are jobs to process
immediately, without waiting up to 1s for the next polling interval. Triggers may create many
notifications for active queues.

Evaluate if you need sub-second job dispatch. Without it, jobs may wait up to 1s before running,
but that’s not a concern for busy queues since they’re constantly fetching and dispatching.

Disable triggers in your Oban configuration:

```diff
 config :my_app, Oban,
+  insert_trigger: false,
```

## Tuning Autovacuum

Default autovacuum settings are sized for general purpose tables and may fall behind on
`oban_jobs`, where state transitions update a row and pruning deletes in bulk. Dead tuples
accumulate between vacuums and planner statistics go stale, which effects critical operations like
job fetching.

Tune autovacuum for the `oban_jobs` table alone rather than the whole database:

```sql
ALTER TABLE oban_jobs SET (
  -- Vacuum earlier on large tables
  autovacuum_vacuum_scale_factor = 0.02,
  autovacuum_vacuum_threshold = 50,

  -- Keep stats fresh for the planner
  autovacuum_analyze_scale_factor = 0.02,
  autovacuum_analyze_threshold = 100,

  -- Handle insert-heavy spikes
  autovacuum_vacuum_insert_scale_factor = 0.02,
  autovacuum_vacuum_insert_threshold = 1000,

  -- Make autovacuum push harder with little sleeping
  autovacuum_vacuum_cost_limit = 2000,
  autovacuum_vacuum_cost_delay = 1,

  -- Leave headroom on pages for locality and fewer page splits
  fillfactor = 85
);
```

Check `n_dead_tup` and `last_autovacuum` in `pg_stat_user_tables` to confirm vacuum is keeping up,
and adjust the scale factors to match table size and write load.

## Limit Uniqueness

Frequently, people set uniqueness for jobs that don’t really need it. Not you, of course.
Before setting uniqueness, ensure the following, in a very checklist type fashion:

1. Evaluate whether it’s necessary for your workload
2. Always set a `keys` option so that uniqueness isn’t based on the full `args` or `meta`
3. Avoid setting a `period` at all if possible, use `period: :infinity` instead

If you're still committed to setting uniquness for your jobs, consider tweaking your
configuration as follows:

```diff
use Oban.Worker, unique: [
-   period: {1, :hour},
+   period: :infinity,
+   keys: [:some_key]
```

> #### 🌟 Pro Uniqueness {: .tip}
>
> Oban Pro uses an [alternative mechanism for unique jobs][uniq] that works for bulk inserts, and
> is designed for speed, correctness, scalability, and simplicity. Uniqueness is enforced and makes insertion entirely safe between processes and nodes, without the load added
> by multiple queries.

[uniq]: https://oban.pro/docs/pro/1.5.0-rc.4/Oban.Pro.Engines.Smart.html#module-enhanced-unique

## Apply Reindexing

To stop oban_jobs indexes from taking up so much space on disk, use the
`Oban.Reindexer` plugin to rebuild indexes periodically. The Postgres transactional
model applies to indexes as well as tables. That leaves bloat from inserting, updating, and
deleting jobs that auto-vacuuming won’t always fix.

The reindexer rebuilds key indexes on a fixed schedule, concurrently. Concurrent rebuilds are low
impact, they don’t lock the table, and they free up space while optimizing indexes.

The `Oban.Reindexer` plugin is part of OSS Oban. It runs every day at midnight by
default, but it accepts a cron-style schedule and you can tweak it to run less frequently.

```diff
config :my_app, Oban,
+  reindexer: [schedule: "@weekly"],
   …
```

## Optimize Staging

Applications with a large number of scheduled or retryable jobs may benefit from an index tailored
to staging. The index isn't necessary for most applications, and maintaining it adds overhead to
inserts and updates, so verify that staging queries are a bottleneck before adding it.

Create the index concurrently to avoid locking the `oban_jobs` table. Concurrent index operations
must run outside of a transaction and without Ecto's migration lock:

```elixir
defmodule MyApp.Repo.Migrations.AddObanJobsStagingIndex do
  use Ecto.Migration

  @disable_ddl_transaction true
  @disable_migration_lock true

  def change do
    create_if_not_exists index(
      :oban_jobs,
      [:scheduled_at],
      concurrently: true,
      where: "state IN ('scheduled', 'retryable')"
    )
  end
end
```

## Increase Pruning

Ensuring you are using the `Pruner` plugin, and that you prune _aggressively_. Pruning
periodically deletes `completed`, `cancelled`, and `discarded` jobs. Your application
and database will benefit from keeping the jobs table small. Aim to retain as few jobs
as necessary for uniqueness and historic introspection.

For example, to limit historic jobs to 1 day:

```diff
 config :my_app, Oban,
+  pruner: [max_age: {1, :day}],
   …
```

## Consider Pooling

Oban uses connections from your application Repo’s pool to talk to the database. When that pool
is busy, it can starve Oban of connections and you’ll see timeout errors. Likewise, if Oban is
extremely busy (as it should be), it can starve your application of connections. A good solution
for this is to set up another pool that’s exclusively for Oban’s internal use. The dedicated
pool isolates Oban’s queries from the rest of the application.

Start by defining a new `ObanRepo`:

```elixir
defmodule MyApp.ObanRepo do
   use Ecto.Repo,
     adapter: Ecto.Adapters.Postgres,
     otp_app: :my_app
end
```

Then add a function to your main Repo

```diff
defmodule MyApp.Repo do
  use Ecto.Repo,
     adapter: Ecto.Adapters.Postgres,
     otp_app: :my_app
  
+ def oban_repo do
+   if in_transaction?() do
+     MyApp.Repo
+   else
+     MyApp.ObanRepo
+   end
+ end
end
```

Then switch the configured `repo`, and use the repo's `dynamic_repo` option to ensure the same
repo is used within a transaction:

```diff
 config :my_app, Oban,
-  repo: MyApp.Repo,
+  repo: {MyApp.ObanRepo, dynamic_repo: {MyApp.Repo, :oban_repo, []}},
   ...
```

## Scale Concurrency

In a busy system with high concurrency all of the record keeping after jobs run causes pool
contention, despite the individual queries being very quick. Fetching jobs uses a single query
per queue. However, acking when a job finishes takes a single connection for each job.

Improve the ratio between executing jobs and available connections by scaling up your Ecto
`pool_size` and minimizing concurrency between all queues.

```diff
config :my_app, Repo,
-  pool_size: 10,
+  pool_size: 50,
   …

 config :my_app, Oban,
   queues: [
-    events: 200,
+    events: 50,
-    emails: 100,
+    emails: 25,
   …
```

Using a dedicated pool with a known number of constant connections can also help the ratio. It’s
not necessary for most applications, but a dedicated database can help maintain predictable
performance.