Commit Graph

6993 Commits

Author SHA1 Message Date
Mark Felder 5a5a193877 Fix broken Rich Media parsing when the image URL is a relative path 2024-05-07 19:54:56 -04:00
Mark Felder d21aa1a77c Respect the TTL returned in OpenGraph tags 2024-05-07 19:54:56 -04:00
Mark Felder df0734fcbf Increase the :max_body for Rich Media to 5MB
Websites are increasingly getting more bloated with tricks like inlining content (e.g., CNN.com) which puts pages at or above 5MB. This value may still be too low.
2024-05-07 19:54:56 -04:00
Mark Felder ede414094f RichMedia refactor
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-05-07 19:54:56 -04:00
feld 750fb25f48 Revert "Merge branch 'pleroma-card-image-description' into 'develop'"
This reverts merge request !4101
2024-05-07 23:20:38 +00:00
Mark Felder 06c26bf9c9 Add the absent max_featured_tags to the api spec for /api/v1/instance 2024-05-07 17:46:05 -04:00
Mark Felder b979389958 Add configuration[accounts][max_pinned_statuses] to /api/v2/instance
Also add the absent max_featured_tags to the api spec for /api/v2/instance
2024-05-07 17:45:02 -04:00
Mark Felder 3cad57bf48 Add configuration[statuses][characters_reserved_per_url] to /api/v2/instance
Fixes #3250
2024-05-07 17:25:30 -04:00
Haelwenn 88412daf11 Apply @lanodan's suggestion
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-25 12:34:12 +02:00
lain 50af909c01 Merge branch 'pleroma-card-image-description' into 'develop'
Include image description in status media cards

See merge request pleroma/pleroma!4101
2024-04-19 07:39:05 +00:00
marcin mikołajczak 6f6bede900 Include image description in status media cards
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-19 10:20:31 +04:00
marcin mikołajczak 4f5c4d79c4 FEP-2c59, add "webfinger" to user actor
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-11 17:50:11 +02:00
marcin mikołajczak ccc3ac241f Add hint to rules
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-06 11:45:19 +02:00
marcin mikołajczak 9e6cf45906 /api/v1/accounts/familiar_followers
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-06 11:43:56 +02:00
marcin mikołajczak 01a5f839c5 Merge remote-tracking branch 'origin/develop' into instance_rules 2024-04-06 10:42:23 +02:00
lain 987f44d811 Merge branch 'bookmark-folders' into 'develop'
Fix BookmarkFolderView, add test

See merge request pleroma/pleroma!4096
2024-03-20 13:26:47 +00:00
marcin mikołajczak 37ec645ff2 Fix BookmarkFolderView, add test
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-20 13:24:43 +01:00
Mark Felder 40823462e7 Logger metadata for request path and authenticated user 2024-03-19 12:15:10 -04:00
Mark Felder 7dfd148ff8 Logger metadata for inbound federation requests 2024-03-19 12:15:10 -04:00
Mark Felder c25fda34e7 Skip generating notifications for internal users 2024-03-19 12:11:30 -04:00
Mark Felder 291d531e4c Unify notification push and streaming events for both local and federated activities
This also removes generation of notifications for blocked/filtered/muted users and threads.
2024-03-19 12:11:30 -04:00
Lain Soykaf 4e8a1b40cb Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into transient-validators-defaults 2024-03-19 16:26:02 +04:00
marcin mikołajczak 60c4cb21ea InstanceView: Update features
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-18 14:02:29 +01:00
marcin mikołajczak 918c406a91 Merge remote-tracking branch 'origin/develop' into instance_rules
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-18 13:50:25 +01:00
lain 2b71f4897f Merge branch 'develop' into 'bookmark-folders'
# Conflicts:
#   docs/development/API/differences_in_mastoapi_responses.md
2024-03-18 10:35:02 +00:00
lain cf0aa1238c Merge branch 'public-polls' into 'develop'
Expose nonAnonymous field from Smithereen polls

See merge request pleroma/pleroma!3962
2024-03-18 06:26:22 +00:00
Lain Soykaf caf855cf9c ActivityPub.Publisher: Don't try federating if a user doesn't have an inbox. 2024-03-17 16:57:45 +04:00
Lain Soykaf 0450da88b6 Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into pleroma-fix-3241 2024-03-17 16:42:17 +04:00
lain 95bcd5d28f Merge branch 'force-mention' into 'develop'
Add ForceMention mrf

See merge request pleroma/pleroma!4082
2024-03-17 12:32:14 +00:00
Haelwenn (lanodan) Monnier 48c22a67de
QuestionOptionsValidator: set default AS types 2024-03-15 16:22:18 +01:00
Haelwenn (lanodan) Monnier 8b651fab1d
AttachmentValidator: Set "Link" as default type 2024-03-15 16:22:18 +01:00
marcin mikołajczak c0c4a9ed0d Merge remote-tracking branch 'origin/develop' into instance-contact-account
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-08 12:12:55 +01:00
tusooa 1422082bf2 Apply ledyba's suggestion(s) to 1 file(s) 2024-03-07 04:43:56 +00:00
tusooa 8298b326a7 Merge branch 'instance-v2' into 'develop'
Add some missing fields to instanceV2

See merge request pleroma/pleroma!4081
2024-03-07 01:31:27 +00:00
Kaede Fujisaki 0242c1f691 fmt 2024-03-02 18:34:12 +09:00
Kaede Fujisaki 6ad4acea50 Consider a case when inbox is nil 2024-03-02 18:09:08 +09:00
marcin mikołajczak 9cfa4e67b1 Add ForceMention mrf
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-01 18:16:09 +01:00
marcin mikołajczak d415686bb9 Allow to group bookmarks in folders
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-01 11:04:01 +01:00
marcin mikołajczak acb9e46074 Add some missing fields to instanceV2
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-02-27 13:25:26 +01:00
marcin mikołajczak c592a0e58d Merge remote-tracking branch 'origin/develop' into HEAD
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-02-22 21:05:20 +01:00
Mark Felder 72fc41d891 Prevent publisher jobs from erroring if the connection pool is full
A full pool is a soft-error. Snooze the job for 30 seconds and try again.
2024-02-22 14:14:40 -05:00
Haelwenn (lanodan) Monnier 7d624c4750
StealEmojiPolicy: Sanitize shortcodes
Closes: https://git.pleroma.social/pleroma/pleroma/-/issues/3245
2024-02-20 09:09:54 +01:00
Mark Felder 7e99d0619d Force more frequent full_sweep GC runs on the Websocket processes
Websocket processes seem to be the primary culprit for Binary memory allocation bloat.
2024-02-17 09:53:52 -05:00
feld 4dd8a1a1ca Merge branch 'websocket-refactor' into 'develop'
Websocket refactor to use Phoenix.Socket.Transport

See merge request pleroma/pleroma!4064
2024-02-15 14:36:54 +00:00
lain d3208d2250 Merge branch 'oauth-nickname' into 'develop'
Use User.full_nickname/1 in oauth html template

See merge request pleroma/pleroma!4054
2024-02-15 09:35:39 +00:00
lain f2f455f283 Merge branch 'frontend-caching' into 'develop'
FrontendStatic should have reasonable caching

See merge request pleroma/pleroma!3685
2024-02-15 09:23:47 +00:00
Haelwenn d19642d7eb Merge branch 'bugfix-ccworks' into 'develop'
Bugfix for ccworks AP bridge

Closes #3234

See merge request pleroma/pleroma!4043
2024-02-15 01:30:22 +00:00
Mark Felder 64ad451a7b Websocket refactor to use Phoenix.Socket.Transport
This will make us compatible with Cowboy and Bandit
2024-02-14 15:27:07 -05:00
marcin mikołajczak 0fcdcc2300 Use User.full_nickname/1 in oauth html template
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-02-09 16:46:45 +01:00
Mark Felder 9f2319e50d RichMedia.Helpers: move the validate_page_url/1 function to the Parser module
This will ensure that the page validation happens in Parser.parse/1 so it can be called from anywhere and still filter invalid URLs.
2024-02-06 18:34:02 -05:00
Mark Felder 6b7b443ff9 Pleroma.Web.RichMedia.Parser: Remove test-specific codepaths
Also consolidate Tesla mocks into the HttpRequestMock module.

Tests were not exercising the real codepaths. The Rich Media Preview only works with https, but most of these tests were only mocking http.
2024-02-06 18:33:54 -05:00
Mark Felder 0cc038b67c Ensure URLs with IP addresses for the host do not generate previews 2024-02-05 00:09:37 -05:00
Mark Felder 579561e97b URI.authority is deprecated 2024-02-04 23:49:07 -05:00
Mark Felder 04fc4eddaa Fix Rich Media Previews for updated activities
The Rich Media Previews were not regenerated when a post was updated due to a cache invalidation issue. They are now cached by the activity id so they can be evicted with the other activity cache objects in the :scrubber_cache.
2024-02-04 23:47:04 -05:00
marcin mikołajczak 226e53fdd7 Merge remote-tracking branch 'origin/develop' into status-notification-type
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-01-31 22:19:33 +01:00
Mark Felder 518a38577b Fix dialyzer errors due to deprecated usage of put_layout/2 2024-01-31 14:55:11 -05:00
Mark Felder 97c4d3bcc9 Pleroma.Web.Plugs.RateLimiter.Supervisor: dialyzer error
lib/pleroma/web/plugs/rate_limiter/supervisor.ex:12:no_return
Function init/1 has no local return.
2024-01-31 13:12:56 -05:00
Mark Felder 92992c022d Pleroma.Web.OAuth.OAuthController: dialyzer error
validate_scopes/2 can never receive a map as it is only called in one place with a guard requiring a list

lib/pleroma/web/o_auth/o_auth_controller.ex:615:guard_fail
The guard test:

is_map(_params :: maybe_improper_list())

can never succeed.
2024-01-31 11:29:06 -05:00
Mark Felder 5e8bedcca0 Pleroma.Web.PleromaAPI.MascotController: fix dialyzer error due to bad error match
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:37:pattern_match
The pattern can never match the type.

Pattern:
{:content_type, _}

Type:
{:error, _}

________________________________________________________________________________
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:40:pattern_match
The pattern can never match the type.

Pattern:
{:upload, {:error, _}}

Type:
{:error, _}
2024-01-31 11:15:48 -05:00
Mark Felder ed2f8e45e9 Pleroma.Web.MastodonAPI.SearchController: fix dialyzer errors
Add a separate Pagination.paginate_list/2 function instead of overloading paginate/4 and complicating its matching and @spec
2024-01-31 11:12:41 -05:00
Mark Felder c6f783c551 Pleroma.Web.ControllerHelper: fix @spec to resolve dialyzer errors
lib/pleroma/web/admin_api/controllers/user_controller.ex:333:no_return
Function index/2 has no local return.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:357:unused_fun
Function maybe_parse_filters/1 will never be called.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:366:no_return
Function page_params/1 has no local return.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:368:call
The function call will not succeed.

Pleroma.Web.ControllerHelper.fetch_integer_param(_params :: any(), :page, 1)

breaks the contract
(map(), String.t(), integer() | nil) :: integer() | nil
2024-01-31 11:01:37 -05:00
Mark Felder 225afe05b3 Pleroma.Web.TwitterAPI.UtilController: fix dialyzer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder b8db67dafa Pleroma.Web.MastodonAPI.StatusController: fix dialzyer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder 9760149886 Pleroma.Web.PleromaAPI.UserImportController: fix dialyzer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder c39e4dd214 Pleroma.Web.PleromaAPI.NotificationController: fix dialyzer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder e157fd60ee Pleroma.Web.PleromaAPI.MascotController: fix dialyzer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder 9d16393d8b Pleroma.Web.PleromaAPI.EmojiPackController: fix dialyzer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder f1aeb80518 Pleroma.Web.PleromaAPI.EmojiFileController: fix dialyzer errors with replace_params: false 2024-01-31 10:13:02 -05:00
Mark Felder 4d20fbc6d9 Pleroma.Web.PleromaAPI.ChatController: fix dialyzer errors with replace_params: false 2024-01-31 10:12:59 -05:00
Mark Felder fdddba100e Pleroma.Web.MastodonAPI.SearchController: fix dialyzer errors with replace_params: false 2024-01-30 18:40:46 -05:00
Mark Felder 324fd08455 Pleroma.Web.MastodonAPI.ScheduledActivityController: fix dialyzer errors with replace_params: false 2024-01-30 18:33:23 -05:00
Mark Felder d3c6acd2f2 Pleroma.Web.MastodonAPI.PollController: fix dialyzer errors with replace_params: false 2024-01-30 18:31:13 -05:00
Mark Felder 0a6b2c9587 Pleroma.Web.MastodonAPI.NotificationController: fix dialyzer errors with replace_params: false 2024-01-30 18:27:31 -05:00
Mark Felder 90c9f38f40 Pleroma.Web.MastodonAPI.MediaController: fix dialyzer errors with replace_params: false 2024-01-30 18:09:11 -05:00
Mark Felder 9741f045e4 Pleroma.Web.MastodonAPI.ListController: fix dialyzer errors with replace_params: false 2024-01-30 18:05:28 -05:00
Mark Felder 14de8376ad Pleroma.Web.MastodonAPI.FollowRequestController: fix dialyzer errors with replace_params: false 2024-01-30 17:56:43 -05:00
Mark Felder f400224a00 Pleroma.Web.AdminAPI.UserController: fix dialyzer errors with replace_params: false 2024-01-30 17:51:40 -05:00
Mark Felder dd916e0b4c Pleroma.Web.AdminAPI.ReportController: fix dialyzer errors with replace_params: false 2024-01-30 17:24:26 -05:00
Mark Felder b84a33a101 Pleroma.Web.AdminAPI.RelayController: fix dialyzer errors with replace_params: false 2024-01-30 17:13:14 -05:00
Mark Felder fb6f53fc12 Pleroma.Web.AdminAPI.MediaProxyCacheController: fix dialyzer errors with replace_params: false 2024-01-30 17:11:43 -05:00
Mark Felder ea26dcd800 Pleroma.Web.AdminAPI.InviteController: fix dialyzer errors with replace_params: false 2024-01-30 17:06:37 -05:00
Mark Felder 85c9397ec7 Pleroma.Web.AdminAPI.InstanceDocumentController: fix dialyzer errors with replace_params: false 2024-01-30 17:04:12 -05:00
Mark Felder ef1f301756 Pleroma.Web.AdminAPI.ConfigController: fix dialyzer errors with replace_params: false 2024-01-30 16:59:28 -05:00
Mark Felder 81c8592d69 Pleroma.Web.MastodonAPI.AccountController: fix dialyzer errors with replace_params: false 2024-01-30 16:50:00 -05:00
Mark Felder cfe7438b2f Pleroma.Web.MastodonAPI.DomainBlockController: dialyzer fixes via :replace_params 2024-01-30 14:52:15 -05:00
Mark Felder 608466d098 Modify our CastAndValidate plug to include the new functionality provided by the :replace_params config option
This allows us to configure Open API Spex to not overwrite the params with the casted versions which violates the Plug.Conn.t() contract

https://github.com/open-api-spex/open_api_spex/issues/92
https://github.com/open-api-spex/open_api_spex/pull/425
2024-01-30 14:49:55 -05:00
Mark Felder 8476eb1842 Revert "Pleroma.Web.AdminAPI.MediaProxyCacheController: dialyzer errors"
This reverts commit 8b02c85810.
2024-01-30 14:20:02 -05:00
Mark Felder 88a35b2865 Revert "Pleroma.Web.TwitterAPI.UtilController: dialyzer fixes"
This reverts commit 17f4251b19.
2024-01-30 14:19:54 -05:00
Mark Felder 674ae51d6a Revert "Pleroma.Web.PleromaAPI.UserImportController: Dialyzer errors"
This reverts commit 52e18a6249.
2024-01-30 14:19:41 -05:00
Mark Felder 4a80a285d1 Revert "Pleroma.Web.PleromaAPI.NotificationController: dialyzer errors"
This reverts commit 26a95e5787.
2024-01-30 14:19:32 -05:00
Mark Felder 1e76ceacd5 Revert "Pleroma.Web.AdminAPI.ConfigController: dialyzer error"
This reverts commit 60d89cb404.
2024-01-30 14:19:13 -05:00
Mark Felder 2c8e4f32c6 Revert "Pleroma.Web.PleromaAPI.MascotController: dialyzer errors"
This reverts commit 9c8055d4b3.
2024-01-30 14:18:36 -05:00
Mark Felder b1a6102a85 Revert "Pleroma.Web.PleromaAPI.EmojiPackController: dialyzer errors"
This reverts commit 77bf617c4b.
2024-01-30 14:15:40 -05:00
Mark Felder 4a9ed4682a Revert "Pleroma.Web.PleromaAPI.EmojiFileController: dialyzer errors"
This reverts commit dc912dc590.
2024-01-30 14:15:17 -05:00
Mark Felder b709fc4dfe Revert "Pleroma.Web.PleromaAPI.ChatController: Dialyzer error"
This reverts commit 8d64eedbec.
2024-01-30 14:15:05 -05:00
Mark Felder 589456f0ba Revert "Pleroma.Web.AdminAPI.InviteController: dialyzer errors"
This reverts commit db87be126e.
2024-01-30 14:14:56 -05:00
Mark Felder 4227db0871 Revert "Pleroma.Web.AdminAPI.UserController: dialyzer errors"
This reverts commit d92c3d927d.
2024-01-30 14:14:39 -05:00
Mark Felder ac06a47689 Revert "Pleroma.Web.AdminAPI.RelayController: dialyzer errors"
This reverts commit 94838ed941.
2024-01-30 14:14:22 -05:00
Mark Felder da5e0fca4f Revert "Pleroma.Web.AdminAPI.ReportController: dialyzer errors"
This reverts commit bfe626d578.
2024-01-30 14:14:07 -05:00
Mark Felder 39241107d9 Revert "Pleroma.Web.AdminAPI.InstanceDocumentController: fix dialyzer error"
This reverts commit a3024dd5ac.
2024-01-30 14:13:56 -05:00