Commit Graph

661 Commits

Author SHA1 Message Date
Mark Felder c20ac6d1ad Add missing foreign key indexes 2024-05-30 10:53:00 -04:00
Mark Felder 0847d9ebaf Oban queue simplification 2024-05-27 13:48:17 -04:00
marcin mikołajczak 2e76ceb5b4 Merge remote-tracking branch 'origin/develop' into status-notification-type
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-05-18 11:30:25 +02:00
feld 53ef576739 Merge branch 'instance_rules' into 'develop'
Instance rules

See merge request pleroma/pleroma!3669
2024-05-15 20:29:04 +00:00
Mark Felder ede414094f RichMedia refactor
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-05-07 19:54:56 -04:00
marcin mikołajczak ccc3ac241f Add hint to rules
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-06 11:45:19 +02:00
marcin mikołajczak 01a5f839c5 Merge remote-tracking branch 'origin/develop' into instance_rules 2024-04-06 10:42:23 +02:00
marcin mikołajczak d415686bb9 Allow to group bookmarks in folders
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-01 11:04:01 +01:00
marcin mikołajczak 226e53fdd7 Merge remote-tracking branch 'origin/develop' into status-notification-type
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-01-31 22:19:33 +01:00
marcin mikołajczak 6051715a99 Merge remote-tracking branch 'origin/develop' into instance_rules
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-12-22 14:34:30 +01:00
Mark Felder fb3eb6e0a4 Fix more Logger warn -> warning 2023-12-20 16:24:27 -05:00
lain ee15939d33 Skip transaction to generate the index concurrently 2023-12-09 05:54:02 +00:00
Mark Felder 003d3312fd Permit the index creation to run concurrently 2023-12-08 15:48:00 -05:00
lain 35774d44b2 Merge branch 'generate-unset-user-keys-migration' into 'develop'
Fix GenerateUnsetUserKeys migration

See merge request pleroma/pleroma!3976
2023-11-28 12:34:32 +00:00
marcin mikołajczak fe776d8b33 Fix GenerateUnsetUserKeys migration
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-11-13 14:18:22 +01:00
lain 752bc168f6 Merge branch 'quotes-count' into 'develop'
Count and display post quotes

See merge request pleroma/pleroma!3956
2023-11-12 13:38:09 +00:00
marcin mikołajczak 9a063deacc Count and display post quotes 2023-11-12 13:38:08 +00:00
Mark Felder bf426c53b4 Fix digest email processing, consolidate Oban queues
The email related jobs can all share a single Oban queue
2023-11-07 15:14:36 -05:00
Haelwenn 043a00991d Merge branch 'instance-nodeinfo-metadata' into 'develop'
instances: Store some metadata based on NodeInfo

See merge request pleroma/pleroma!3853
2023-06-27 18:58:04 +00:00
Haelwenn 41f2ee69a8 Merge branch 'from/upstream-develop/tusooa/backup-status' into 'develop'
Detail backup states

Closes #3024

See merge request pleroma/pleroma!3809
2023-06-27 12:08:11 +00:00
Lain Soykaf 22878cf84a B Migrations: Don't remove activity_visibility_index for now. 2023-06-11 16:22:16 +04:00
Lain Soykaf 175ee9e6fe Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into unused_indexes 2023-06-11 16:20:30 +04:00
Haelwenn (lanodan) Monnier fcd49e3985 User: Remove ap_enabled field 2023-05-05 11:11:26 +02:00
Mark Felder d91a823836 Remove unused indexes
These indexes were always listed as unused and several grow quite large.
The most significant impact is the activities_visibility_index which takes many hours to rebuild when restoring the server from backup even on fast hardware.
2023-04-22 16:43:33 +00:00
Haelwenn (lanodan) Monnier 9a2523a09a instances: Store some metadata based on NodeInfo 2023-03-16 09:02:20 +01:00
marcin mikołajczak 78d1105bff Fix down migration
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-02-19 22:02:38 +01:00
marcin mikołajczak 92592c25c2 Merge remote-tracking branch 'pleroma/develop' into status-notification-type
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-02-19 22:02:03 +01:00
lain e853cfe7c3 Revert "Merge branch 'copyright-bump' into 'develop'"
This reverts merge request !3825
2023-01-02 20:38:50 +00:00
marcin mikołajczak 10886eeaa2 Bump copyright year
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-01-01 12:13:06 +01:00
tusooa e4ac2a7cd6
Detail backup states 2022-12-24 00:04:32 -05:00
Lain Soykaf 63d00f8123 Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into update-deps 2022-12-15 17:19:36 -05:00
Mark Felder 8a3b450397 Add migration to remove Quack from ConfigDB 2022-11-11 12:36:34 -05:00
Mark Felder 7c64f705f6 Update to Phoenix 1.6 and chase dependencies
Also bump minimum Elixir to 1.11
2022-11-03 16:13:07 +00:00
Haelwenn 346c130ddc Merge branch 'fix/user-private-key-generation' into 'develop'
User: generate private keys on user creation

See merge request pleroma/pleroma!3737
2022-09-05 15:38:15 +00:00
Hélène cfb1bc967f
Migrations: generate unset user keys
User keys are now generated on user creation instead of "when needed",
to prevent race conditions in federation and a few other issues. This
migration will generate keys missing for local users.
2022-09-05 03:51:35 +02:00
tusooa 20347898e2 Merge branch 'fix/federation-context-issues' into 'develop'
Fix reply context fixing (Pleroma replies to Misskey threads) and removal of context objects

See merge request pleroma/pleroma!3717
2022-09-04 18:43:36 +00:00
Haelwenn 299255b9bb Merge branch 'from/upstream-develop/tusooa/assoc-object-id' into 'develop'
Add function to calculate associated object id

Closes #2307

See merge request pleroma/pleroma!3692
2022-09-03 02:50:40 +00:00
Haelwenn e40c221c31 Merge branch 'from/upstream-develop/tusooa/edits' into 'develop'
Editing

Closes #1429, #2859, and #2288

See merge request pleroma/pleroma!3678
2022-09-03 02:16:42 +00:00
Tusooa Zhu f047088a93
Update thread visibility function 2022-08-20 21:06:12 -04:00
Tusooa Zhu 4e7ed563c0
Lint 2022-08-20 20:43:47 -04:00
Tusooa Zhu 3885ee182a
Switch to associated_object_id index 2022-08-20 20:43:46 -04:00
Tusooa Zhu 06678fb4ad
Add function to calculate associated object id 2022-08-20 20:43:40 -04:00
Hélène 88c1c76d3e
Migrations: delete contexts with BaseMigrator
Due to the lengthiness of this task, the migration has been adapted into
a BaseMigrator migration, running in the background instead.
2022-08-15 01:47:09 +02:00
Hélène c559c240d1
Migrations: delete context objects
These objects represent from 30 to 70% of the rows on the objects table,
based on numbers from a few live instances (single-user, small, large.)

As those pseudo-objects prevent creating objects with those actual IDs,
deleting them is a better solution. This could have happened if an
object used another object's ID as its context.
2022-08-10 02:29:38 +02:00
Haelwenn b2ba307f4d Merge branch 'from/upstream-develop/tusooa/2871-fix-local-public' into 'develop'
local only fixes

Closes #2871

See merge request pleroma/pleroma!3660
2022-08-02 05:39:50 +00:00
tusooa 301ce5bc62 Merge branch 'mute-expiration' into 'develop'
MastoAPI: Show mutes expiration date

See merge request pleroma/pleroma!3682
2022-07-23 00:34:15 +00:00
marcin mikołajczak 597f56b4c4 Use :utc_datetime
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2022-07-16 16:28:22 +02:00
Haelwenn 29f4ab640b Merge branch 'from/upstream-develop/tusooa/report-too-long' into 'develop'
Fix long report notes giving errors on creation

See merge request pleroma/pleroma!3679
2022-07-04 05:21:20 +00:00
Tusooa Zhu 4edc867b87 Merge branch 'develop' into 'from/upstream-develop/tusooa/edits'
# Conflicts:
#   lib/pleroma/constants.ex
2022-07-03 22:24:57 +00:00
Haelwenn eefa981e0a Merge branch 'indexing-hotspots' into 'develop'
Add index hotspots

See merge request pleroma/pleroma!3664
2022-07-03 22:01:37 +00:00