Commit Graph

149 Commits

Author SHA1 Message Date
Mark Felder 0847d9ebaf Oban queue simplification 2024-05-27 13:48:17 -04:00
Mark Felder ede414094f RichMedia refactor
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-05-07 19:54:56 -04:00
Haelwenn (lanodan) Monnier a299ddb10e
ReceiverWorker: Make sure non-{:ok, _} is returned as {:error, …}
Otherwise an error like `{:signature, {:error, {:error, :not_found}}}` ends up considered a success.
2024-04-17 07:43:47 +02:00
marcin mikołajczak b7c625db0f Merge remote-tracking branch 'origin/develop' into link-verification 2024-03-07 13:13:49 +01:00
Haelwenn (lanodan) Monnier 6af6a9704d
RemoteFetcherWorker: Make sure {:error, _} is returned on failure
Otherwise jobs are considered a success.
2024-02-23 01:59:30 +01:00
Mark Felder 91c83a82a0 Fix atom leak in background worker
The only permitted values are "blocks_import", "follow_import", "mutes_import" of which we already have the equivalent atoms defined.
2024-02-14 18:03:54 -05:00
Haelwenn 3c65a2899d Merge branch 'handle_object_fetch_failures' into 'develop'
Handle object fetch failures gracefully

See merge request pleroma/pleroma!4015
2024-01-19 18:43:00 +00:00
Mark Felder 12c052551b Allow the Remote Fetcher to attempt fetching an unreachable instance 2024-01-14 14:05:43 -05:00
Mark Felder f17f92105b Oban jobs should be discarded on permanent errors 2023-12-28 23:52:59 -05:00
Mark Felder 3acfdb6f8a Retire the Pleroma.Web.Federator.Publisher module 2023-12-28 14:53:09 -05:00
Mark Felder ad0a5deb67 Prevent requeuing Remote Fetcher jobs that exceed thread depth 2023-12-27 22:28:41 -05:00
Mark Felder a2708f7fe3 Leverage existing atoms as return errors for the object fetcher 2023-12-27 22:01:59 -05:00
Mark Felder 9c0040124a Skip remote fetch jobs for unreachable instances 2023-12-26 16:28:05 -05:00
Mark Felder 5f6966cd9f Remove mistaken duplicate fetch 2023-12-26 16:23:41 -05:00
Mark Felder c6b38441f1 Cancel remote fetch jobs for deleted objects 2023-12-26 16:05:44 -05:00
Mark Felder d472bafec1 Mark instances as unreachable when returning a 403 from an object fetch
This is a definite sign the instance is blocked and they are enforcing authorized_fetch
2023-12-26 15:54:21 -05:00
Mark Felder 223c1bac8d Cancel the job if the signature is still invalid after a refetch of the public key 2023-12-10 12:55:41 -05:00
Mark Felder 82724f6664 Do not retry fetching deleted objects 2023-12-09 17:48:33 -05:00
Mark Felder 4039106500 Fix the req_headers formatting 2023-12-08 21:51:36 -05:00
Mark Felder 074b31d9ab Optimistic Inbox
Rework inbound federation to accept requests optimistically. The HTTP Signatures Plug will not attempt to fetch the actor or key and will fail early.

If the signature cannot be validated we pass the required data into the Oban job with a reduced priority and increase the timeout to 20 seconds. The Oban job will handle the actor and key fetching before attempting to validate the activity again. This job will be retried 5 times by default.

Another welcome side effect is that actors who change their keys can federate to Pleroma instances immediately instead of needing to wait the default value of 86400s / 24 hours before the key will be fetched again.
2023-12-08 17:45:20 -05:00
Lain Soykaf d3f8950588 B MeiliSearch, SearchIndexingWorker: Use Config.Getting, make tests async. 2023-11-12 14:49:50 +04:00
Lain Soykaf 0c5cc51983 Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into pleroma-meilisearch 2023-11-12 13:53:18 +04:00
Mark Felder bf426c53b4 Fix digest email processing, consolidate Oban queues
The email related jobs can all share a single Oban queue
2023-11-07 15:14:36 -05:00
marcin mikołajczak 50e7706b26 Verify link ownership with rel="me"
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-11-02 13:30:46 +01:00
Haelwenn 41f2ee69a8 Merge branch 'from/upstream-develop/tusooa/backup-status' into 'develop'
Detail backup states

Closes #3024

See merge request pleroma/pleroma!3809
2023-06-27 12:08:11 +00:00
Haelwenn (lanodan) Monnier e17265a7a2 TransmogrifierWorker: Remove obsolete worker 2023-05-05 11:11:26 +02:00
anemone f463b7570e Set background worker timeout to 15 minutes 2023-03-23 23:14:52 -07:00
tusooa 1babd0798f
Stop oban from retrying if validating errors occur when processing incoming data 2023-03-01 21:03:30 -05:00
lain e853cfe7c3 Revert "Merge branch 'copyright-bump' into 'develop'"
This reverts merge request !3825
2023-01-02 20:38:50 +00:00
marcin mikołajczak 10886eeaa2 Bump copyright year
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-01-01 12:13:06 +01:00
tusooa e4ac2a7cd6
Detail backup states 2022-12-24 00:04:32 -05:00
Ekaterina Vaartis 398141da68 Merge remote-tracking branch 'upstream/develop' into meilisearch 2022-12-20 21:00:07 +03:00
Mark Felder 8be7f87e1f Define sane Oban Worker timeouts 2022-11-11 13:46:27 -05:00
Ekaterina Vaartis 5ac6763238 Make add_to_index and remove_from_index report errors 2022-10-10 20:19:09 +03:00
Ekaterina Vaartis d89dc5518b Fix meilisearch tests and jobs for oban 2022-10-10 20:19:09 +03:00
Ekaterina Vaartis 2bc21c6f18 Use oban for search indexing 2022-10-10 20:19:09 +03:00
Tusooa Zhu d487e0160c
Treat containment failure as cancel in ReceiverWorker 2022-08-08 08:41:33 -04:00
Tusooa Zhu a0166e92fa
Treat MRF rejects as success in Oban worker 2022-08-06 00:33:18 -04:00
Tusooa Zhu 221cb3fb81
Allow users to create backups without providing email address
Ref: backup-without-email
2022-05-07 00:23:55 -04:00
Sean King 17aa3644be
Copyright bump for 2022 2022-02-25 23:11:42 -07:00
lain 6e3df11693 Merge branch 'instance-deletion' into 'develop'
AdminAPI: endpoint to delete all content from a remote instance

See merge request pleroma/pleroma!3483
2021-11-15 14:24:13 +00:00
Alex Gleason 6a6e42c9bf
PollWorker defensive checks 2021-07-18 11:10:22 -05:00
Alex Gleason 0114754db2
MastodonAPI: Support poll notification 2021-07-17 22:19:38 -05:00
Alex Gleason f67d00d12b
Add Instance.delete_users_and_activities/1 to delete all content from a remote instance 2021-07-17 14:06:35 -05:00
feld 008499f65a Merge branch 'develop' into 'fix/2412-filters'
# Conflicts:
#   CHANGELOG.md
2021-02-03 14:23:23 +00:00
feld c3dd860a02 Merge branch 'fix/2449-scheduled-poll-bug' into 'develop'
Fix for scheduled post with poll

Closes #2449

See merge request pleroma/pleroma!3294
2021-02-03 14:22:23 +00:00
feld 6a2d3fb9a3 Merge branch 'fix/2473-purge-expired-activities-duplicates' into 'develop'
fix and delete purge activities duplicates

Closes #2473

See merge request pleroma/pleroma!3285
2021-02-02 17:22:23 +00:00
Alexander Strizhakov 0dc68c157f
fix for scheduled post with poll 2021-02-01 18:22:26 +03:00
Mark Felder 13d79c281f Make attachment cleanup jobs a noop if the setting is disabled. 2021-01-28 14:42:20 -06:00
Alexander Strizhakov 6c987c7670
fix and delete purge activities duplicates 2021-01-28 16:57:41 +03:00