Chunking Jobs
Sometimes it's useful to split up the processing of something into multiple jobs, like processing a large file or scraping data from a paginated API. Laravel haystack uses the Laravel Chunkable Job package and allows you to split a job up into chunks.
Firstly, install the
laravel-chunkable-jobs
package using Composer with the command below.composer require sammyjo20/laravel-chunkable-jobs
Next, just create a job, remove the
handle
method and extend the ChunkableHaystackJob
class. It's important that you extend this class and not ChunkableJob
as it will use Haystack's methods to keep processing on the same Haystack.You do not need to add the
StackableJob
interface or Stackable
trait since the ChunkableHaystackJob
will already add this for you.<?php
use Sammyjo20\ChunkableJobs\Chunk;
use Sammyjo20\LaravelHaystack\ChunkableHaystackJob;
class GetPageOfPokemon extends ChunkableHaystackJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public function defineChunk(): ?Chunk
{
$response = Http::asJson()->get('https://pokeapi.co/api/v2/pokemon');
$count = $response->json('count'); // 1154
return new Chunk(totalItems: $count, chunkSize: 1, startingPosition: 1);
}
protected function handleChunk(Chunk $chunk): void
{
$response = Http::asJson()->get(sprintf('https://pokeapi.co/api/v2/pokemon?limit=%s&offset=%s', $chunk->limit, $chunk->offset));
$data = $response->json();
// Store data of response
}
}