> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Squid SDK tips and troubleshooting

> Diagnose slow indexing, batch behavior, reorgs, and Squid SDK resource use.

Use this guide when a squid works but is slow, consumes unpredictable resources,
or behaves differently after a restart or reorg.

| Symptom                                                | First check                                                                                                                    |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| Historical sync is slow                                | Run the same processor configuration with an empty batch handler to separate ingestion time from processing and database time. |
| Memory rises suddenly                                  | Check for mutable global state, unbounded collections, and operations that grow faster than linearly with the batch size.      |
| Results differ between clean syncs                     | Remove any correctness state that survives only in process memory and compare clean runs again.                                |
| `block.transactions` or another collection is empty    | Add a data request or related-data flag. `setFields()` only controls fields on items that were requested.                      |
| An external consumer sees duplicate or replaced blocks | Treat messages from unfinalized blocks as provisional and reconcile them by block height and hash.                             |
| A processor appears stuck                              | Enable processor debug logs for a short reproduction near the chain head.                                                      |
| The squid fails with a specific error message          | Look it up in [Common errors](#common-errors).                                                                                 |

## Separate ingestion from processing

Before changing RPC settings or scaling the database, measure where time is
spent. Temporarily replace the body of the
[batch handler](../../reference/tron-processor/context-interfaces)
with a block counter:

```ts theme={"system"}
processor.run(
  new TypeormDatabase({ stateSchema: 'squid_ingestion_probe' }),
  async (ctx) => {
    ctx.log.info({ blocks: ctx.blocks.length }, 'ingestion probe')
  }
)
```

Run the probe over the same block range and in the same environment as the slow
run:

* If throughput improves substantially, optimize transformations and database
  access. Start with [batch processing](./batch-processing)
  and [query optimization](/en/cloud/resources/query-optimization).
* If throughput stays similar, investigate the data source, network connection,
  RPC rate limits, and RPC provider.

<Warning>
  An empty handler still advances processor status. Use a disposable database or
  a unique `stateSchema`, as shown above. Do not replace a production handler and
  reuse its processor status.
</Warning>

Compare like with like. A historical range served by the SQD Network and a
near-head range served in real time have different performance characteristics.

## Request only data you use

Narrow requests by contract and block range whenever possible. Disable optional
fields that are not used — this reduces data transfer and the amount of data
held in each batch.

[`setFields()`](../../reference/tron-processor/field-selection)
controls the shape of returned items. It selects fields on items that were
already requested elsewhere in the configuration; it does not add new data items
to the batch by itself. Request each data item type and its related data
explicitly, and enable only the fields your handler reads.

## Make batch handlers deterministic

Batch boundaries are an implementation detail and can change between runs. The
same block range can arrive as several different batch groupings.

* Do not let results depend on mutable module-level or global state carried from
  one batch to the next.
* Persist correctness-critical state in the database, or derive it again from
  the current batch and durable data.
* Keep in-memory caches bounded and safe to rebuild after a restart.
* Test important changes with multiple clean syncs and compare the resulting
  rows or a stable export.

Batch handlers should also be safe to retry. `TypeormDatabase` can run a
handler more than once for the same data when a transaction is retried or the
chain reorganizes. See the
[`typeorm-store` transaction notes](../../reference/data-stores/typeorm-store#store-interface).

## Keep batch work bounded

Processors can receive large batches, and the number of blocks and selected
items per batch varies. Keep memory and CPU use close to linear in the number
of selected items:

* collect IDs in a `Set`, then load matching rows with one `findBy()` call;
* update entities in a `Map`, then save them as an array;
* avoid nested scans over the complete batch;
* clear temporary collections at the end of each handler call;
* load test with realistic event density, not only a short quiet range.

Do not replace batching with a database read or write per event. The
[batch-processing patterns](./batch-processing#implementation-patterns)
show how to load and persist entities in groups.

## Run periodic work intentionally

By default, a processor fetches only blocks that contain requested items. Add
`includeAllBlocks()` when logic must run on every block:

```ts theme={"system"}
processor
  .includeAllBlocks()

processor.run(new TypeormDatabase(), async (ctx) => {
  for (const block of ctx.blocks) {
    if (block.header.height % 100 === 0) {
      await updatePeriodicState(ctx, block)
    }
  }
})
```

[`includeAllBlocks()`](../../reference/tron-processor/general#include-all-blocks)
accepts an optional block range, so limit it when the requirement is historical
only. Fetching every block increases work.

## Make external side effects reorg-safe

When a reorg replaces unfinalized blocks, a compatible database rolls back SQL
changes made for orphaned blocks and the processor runs the handler again on
the canonical chain. Messages already sent to Kafka, HTTP requests, and other
external effects are not rolled back.

For effects derived from unfinalized blocks:

* include block height and block hash in the external record;
* make writes idempotent;
* let consumers replace or retract provisional data when the canonical hash at
  a height changes;
* delay irreversible actions until the required confirmation or finality
  threshold has been reached.

The [unfinalized blocks and reorgs guide](./unfinalized-blocks)
explains which blocks are considered hot and how database rollback works.

## Debug a processor near the failure

Use a narrow debug namespace before enabling all logs:

```bash theme={"system"}
SQD_DEBUG=sqd:processor*
```

Debug output can be large. If the issue occurs near the chain head, first sync
close to the head with the normal log level, then restart with debug logging for
a short reproduction. See
[overriding the log level](../../reference/logger#overriding-the-log-level)
and [Cloud logging](/en/cloud/resources/logging).

For slow or oversized GraphQL requests, use the
[query optimization procedure](/en/cloud/resources/query-optimization), add
[schema indexes](../../reference/schema-files/indexes-constraints), paginate
collections, and configure
[OpenReader DoS protection](../../reference/openreader/configuration/dos-protection).

## Common errors

### Processor errors

#### Log decoding errors on Tron

* Verify that the log topics and data match the expected TRC20 or smart contract event format.

* Check that contract addresses are correctly formatted (Tron uses base58 encoding for addresses).

* Ensure that event data is being decoded according to the contract ABI.

### Database errors

#### `QueryFailedError: relation "..." does not exist`

It is likely that the generated migrations in the `db/migrations` folder are outdated and do not match the schema file.
Recreate the migrations from scratch as detailed on [this page](../writing-to-postgres#updating-after-schema-changes). The relevant commands are listed in the [`squid-typeorm-migration` reference](../../reference/typeorm-migration).

#### `Query runner already released. Cannot run queries anymore`, or `too late to perform db updates, make sure you haven't forgot to await on db query`

If your squid [saves its data to a database](../writing-to-postgres), all operations with `ctx.store` are asynchronous. Make sure you `await` on all `store` operations like `upsert`, `update`, `find`, `save` etc.

You may find the [require-await](https://eslint.org/docs/latest/rules/require-await) eslint rule to be helpful.

#### `QueryFailedError: invalid byte sequence for encoding "UTF8": 0x00`

PostgreSQL doesn't support storing `NULL (\0x00)` characters in text fields. Usually the error occurs when a raw bytes string (like `UIntArray` or `Bytes`) is inserted into a `String` field. If this is the case, use hex encoding, e.g. using [`util-internal-hex`](https://github.com/subsquid/squid/tree/master/util/util-internal-hex) library. For addresses, use appropriate encoding for your chain type.

### GraphQL errors

#### `response might exceed the size limit`

Make sure the input query has limits set or the entities are decorated with `@cardinality`. We recommend using `XXXConnection` queries for pagination. For configuring limits and max response sizes, see [DoS protection](../../reference/openreader/configuration/dos-protection).