Resources Cleaner

⚠️⚠️⚠️⚠️ WARNING: very dangerous process ⚠️⚠️⚠️⚠️

This process is capable of mass deleting an arbitrarily large number of news, images, videos, ... from the database and the persistence backend (e.g. AWS S3).

The operation of this process is HIGHLY SENSITIVE to the configuration of the php-resource-cleaner-class setting, so it MUST BE TREATED WITH GREAT CARE in production environments.

Strategy

Checking whether certain resources are in use is expensive and not scalable unless additional structures are added (e.g. a certain index indicating which news items use which images). Therefore, a certain nearby window in the past time is reviewed, for example, all the news created a week ago (that is, between 7 and 6 days ago). As it is only a window, only the resources will be analyzed and deleted in one day, and as it is not very long ago (in the example 7 days ago), its use should only be reviewed for all the content created/updated in these days (since resources that have not been modified cannot use a resource created later).

Configuration

By default the process is conservative, in the sense that it will only delete news that has never been published, with an import code, etc...

Not all behavior can be modified (for security), but certain behaviors can be modified by overloading the App\Bundle\CmsAdminBundle\Services\ResourceCleanerService\CleanerConfigBase class using the php-resource-cleaner-class setting, for example:

return new class() extends return new \App\Bundle\CmsAdminBundle\Services\ResourceCleanerService\CleanerConfigBase {

    public function cleanupUnusedPagesDecorateQuery(\DateTime $createdFrom, \DateTime $createdTo, QueryBuilder $queryBuilder): QueryBuilder
    {
        return parent::cleanupUnusedPagesDecorateQuery($createdFrom, $createdTo, $queryBuilder)
                ->andWhere('p.deleteThis IS TRUE');
    }
    
    public function cleanupUnusedImagesPostFilterMustBeCleaned($image): bool
    {
        return parent::cleanupUnusedImagesPostFilterMustBeCleaned($image)
            && in_array($imagine->getSource(), ['shutterstock', 'getty', '500px'], true);
    }
    
    ...
};

Running

⚠️ MAKE SURE ⚠️ that you have backup copies of the database and that the downtime period is limited (e.g. in the event of a contingency, everything could be recovered except, perhaps, a few hours of work).

Above all initially, it is good to configure it:

1. automatically in pre-production environments.
2. manually and controlled in production environments.

Execution:

$ ./app/admin/console app:resources:clean --help
Description:
  Clean resources (news, images, videos, ...) not used or whatever.

Usage:
  app:resources:clean [options]

Options:
  -a, --dateFrom=DATEFROM  Check all resources and changes from that date expressed using the `strtotime` format.
  -b, --dateTo=DATETO      Check all resources and changes up to that date expressed using the `strtotime` format.
  -f, --force[=FORCE]      By default only one month ago is enabled, to enable longer time intervals, force.
  ...

For example:

$ ./app/admin/console app:resources:clean --dateFrom='3 days ago' --dateTo='0 days ago'
INFO: clean resources updated after 2023-12-24T11:54:11+01:00 (3 days ago)
INFO: === running ThisVeryDangerousProcess from 2023-12-24T11:54:11+01:00 to 2023-12-27T11:54:11+01:00
WARNING: 28 pages will be deleted
INFO: 2 content model types found
INFO: 0 resources of type image in use since 2023-12-24T11:54:11+01:00
INFO: 0 resources of type video in use since 2023-12-24T11:54:11+01:00
WARNING: 99 images will be deleted
INFO: removing "uploads:2023/12/27/658bf46b7a2ab.jpeg" resource
INFO: removing "uploads:2023/12/27/658bf472ef273.jpeg" resource
INFO: removing "uploads:2023/12/27/658bf47422f11.jpeg" resource
...
INFO: removing "uploads:2023/12/27/original_658bf5d813ffe.jpeg" resource
INFO: removing "uploads:2023/12/27/658bf5dc45108.jpeg" resource
INFO: removing "uploads:2023/12/27/original_658bf5dc45108.jpeg" resource
INFO: removing "uploads:2023/12/27/658bf5ec53785.jpeg" resource
WARNING: 0 videos will be deleted
INFO: ThisVeryDangerousProcess end ===
INFO: {"deletedPages":28,"deletedResources":{"image":99,"video":0}}