Mark Felder
5a5a193877
Fix broken Rich Media parsing when the image URL is a relative path
2024-05-07 19:54:56 -04:00
Mark Felder
d21aa1a77c
Respect the TTL returned in OpenGraph tags
2024-05-07 19:54:56 -04:00
Mark Felder
df0734fcbf
Increase the :max_body for Rich Media to 5MB
...
Websites are increasingly getting more bloated with tricks like inlining content (e.g., CNN.com) which puts pages at or above 5MB. This value may still be too low.
2024-05-07 19:54:56 -04:00
Mark Felder
ede414094f
RichMedia refactor
...
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.
Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.
Implementation notes:
- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering
Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-05-07 19:54:56 -04:00
feld
750fb25f48
Revert "Merge branch 'pleroma-card-image-description' into 'develop'"
...
This reverts merge request !4101
2024-05-07 23:20:38 +00:00
Mark Felder
06c26bf9c9
Add the absent max_featured_tags to the api spec for /api/v1/instance
2024-05-07 17:46:05 -04:00
Mark Felder
b979389958
Add configuration[accounts][max_pinned_statuses] to /api/v2/instance
...
Also add the absent max_featured_tags to the api spec for /api/v2/instance
2024-05-07 17:45:02 -04:00
Mark Felder
3cad57bf48
Add configuration[statuses][characters_reserved_per_url] to /api/v2/instance
...
Fixes #3250
2024-05-07 17:25:30 -04:00
Haelwenn
88412daf11
Apply @lanodan's suggestion
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-25 12:34:12 +02:00
lain
50af909c01
Merge branch 'pleroma-card-image-description' into 'develop'
...
Include image description in status media cards
See merge request pleroma/pleroma!4101
2024-04-19 07:39:05 +00:00
marcin mikołajczak
6f6bede900
Include image description in status media cards
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-19 10:20:31 +04:00
marcin mikołajczak
4f5c4d79c4
FEP-2c59, add "webfinger" to user actor
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-11 17:50:11 +02:00
marcin mikołajczak
ccc3ac241f
Add hint to rules
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-06 11:45:19 +02:00
marcin mikołajczak
9e6cf45906
/api/v1/accounts/familiar_followers
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-04-06 11:43:56 +02:00
marcin mikołajczak
01a5f839c5
Merge remote-tracking branch 'origin/develop' into instance_rules
2024-04-06 10:42:23 +02:00
lain
987f44d811
Merge branch 'bookmark-folders' into 'develop'
...
Fix BookmarkFolderView, add test
See merge request pleroma/pleroma!4096
2024-03-20 13:26:47 +00:00
marcin mikołajczak
37ec645ff2
Fix BookmarkFolderView, add test
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-20 13:24:43 +01:00
Mark Felder
40823462e7
Logger metadata for request path and authenticated user
2024-03-19 12:15:10 -04:00
Mark Felder
7dfd148ff8
Logger metadata for inbound federation requests
2024-03-19 12:15:10 -04:00
Mark Felder
c25fda34e7
Skip generating notifications for internal users
2024-03-19 12:11:30 -04:00
Mark Felder
291d531e4c
Unify notification push and streaming events for both local and federated activities
...
This also removes generation of notifications for blocked/filtered/muted users and threads.
2024-03-19 12:11:30 -04:00
Lain Soykaf
4e8a1b40cb
Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into transient-validators-defaults
2024-03-19 16:26:02 +04:00
marcin mikołajczak
60c4cb21ea
InstanceView: Update features
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-18 14:02:29 +01:00
marcin mikołajczak
918c406a91
Merge remote-tracking branch 'origin/develop' into instance_rules
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-18 13:50:25 +01:00
lain
2b71f4897f
Merge branch 'develop' into 'bookmark-folders'
...
# Conflicts:
# docs/development/API/differences_in_mastoapi_responses.md
2024-03-18 10:35:02 +00:00
lain
cf0aa1238c
Merge branch 'public-polls' into 'develop'
...
Expose nonAnonymous field from Smithereen polls
See merge request pleroma/pleroma!3962
2024-03-18 06:26:22 +00:00
Lain Soykaf
caf855cf9c
ActivityPub.Publisher: Don't try federating if a user doesn't have an inbox.
2024-03-17 16:57:45 +04:00
Lain Soykaf
0450da88b6
Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into pleroma-fix-3241
2024-03-17 16:42:17 +04:00
lain
95bcd5d28f
Merge branch 'force-mention' into 'develop'
...
Add ForceMention mrf
See merge request pleroma/pleroma!4082
2024-03-17 12:32:14 +00:00
Haelwenn (lanodan) Monnier
48c22a67de
QuestionOptionsValidator: set default AS types
2024-03-15 16:22:18 +01:00
Haelwenn (lanodan) Monnier
8b651fab1d
AttachmentValidator: Set "Link" as default type
2024-03-15 16:22:18 +01:00
marcin mikołajczak
c0c4a9ed0d
Merge remote-tracking branch 'origin/develop' into instance-contact-account
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-08 12:12:55 +01:00
tusooa
1422082bf2
Apply ledyba's suggestion(s) to 1 file(s)
2024-03-07 04:43:56 +00:00
tusooa
8298b326a7
Merge branch 'instance-v2' into 'develop'
...
Add some missing fields to instanceV2
See merge request pleroma/pleroma!4081
2024-03-07 01:31:27 +00:00
Kaede Fujisaki
0242c1f691
fmt
2024-03-02 18:34:12 +09:00
Kaede Fujisaki
6ad4acea50
Consider a case when inbox is nil
2024-03-02 18:09:08 +09:00
marcin mikołajczak
9cfa4e67b1
Add ForceMention mrf
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-01 18:16:09 +01:00
marcin mikołajczak
d415686bb9
Allow to group bookmarks in folders
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-01 11:04:01 +01:00
marcin mikołajczak
acb9e46074
Add some missing fields to instanceV2
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-02-27 13:25:26 +01:00
marcin mikołajczak
c592a0e58d
Merge remote-tracking branch 'origin/develop' into HEAD
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-02-22 21:05:20 +01:00
Mark Felder
72fc41d891
Prevent publisher jobs from erroring if the connection pool is full
...
A full pool is a soft-error. Snooze the job for 30 seconds and try again.
2024-02-22 14:14:40 -05:00
Haelwenn (lanodan) Monnier
7d624c4750
StealEmojiPolicy: Sanitize shortcodes
...
Closes: https://git.pleroma.social/pleroma/pleroma/-/issues/3245
2024-02-20 09:09:54 +01:00
Mark Felder
7e99d0619d
Force more frequent full_sweep GC runs on the Websocket processes
...
Websocket processes seem to be the primary culprit for Binary memory allocation bloat.
2024-02-17 09:53:52 -05:00
feld
4dd8a1a1ca
Merge branch 'websocket-refactor' into 'develop'
...
Websocket refactor to use Phoenix.Socket.Transport
See merge request pleroma/pleroma!4064
2024-02-15 14:36:54 +00:00
lain
d3208d2250
Merge branch 'oauth-nickname' into 'develop'
...
Use User.full_nickname/1 in oauth html template
See merge request pleroma/pleroma!4054
2024-02-15 09:35:39 +00:00
lain
f2f455f283
Merge branch 'frontend-caching' into 'develop'
...
FrontendStatic should have reasonable caching
See merge request pleroma/pleroma!3685
2024-02-15 09:23:47 +00:00
Haelwenn
d19642d7eb
Merge branch 'bugfix-ccworks' into 'develop'
...
Bugfix for ccworks AP bridge
Closes #3234
See merge request pleroma/pleroma!4043
2024-02-15 01:30:22 +00:00
Mark Felder
64ad451a7b
Websocket refactor to use Phoenix.Socket.Transport
...
This will make us compatible with Cowboy and Bandit
2024-02-14 15:27:07 -05:00
marcin mikołajczak
0fcdcc2300
Use User.full_nickname/1 in oauth html template
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-02-09 16:46:45 +01:00
Mark Felder
9f2319e50d
RichMedia.Helpers: move the validate_page_url/1 function to the Parser module
...
This will ensure that the page validation happens in Parser.parse/1 so it can be called from anywhere and still filter invalid URLs.
2024-02-06 18:34:02 -05:00
Mark Felder
6b7b443ff9
Pleroma.Web.RichMedia.Parser: Remove test-specific codepaths
...
Also consolidate Tesla mocks into the HttpRequestMock module.
Tests were not exercising the real codepaths. The Rich Media Preview only works with https, but most of these tests were only mocking http.
2024-02-06 18:33:54 -05:00
Mark Felder
0cc038b67c
Ensure URLs with IP addresses for the host do not generate previews
2024-02-05 00:09:37 -05:00
Mark Felder
579561e97b
URI.authority is deprecated
2024-02-04 23:49:07 -05:00
Mark Felder
04fc4eddaa
Fix Rich Media Previews for updated activities
...
The Rich Media Previews were not regenerated when a post was updated due to a cache invalidation issue. They are now cached by the activity id so they can be evicted with the other activity cache objects in the :scrubber_cache.
2024-02-04 23:47:04 -05:00
marcin mikołajczak
226e53fdd7
Merge remote-tracking branch 'origin/develop' into status-notification-type
...
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-01-31 22:19:33 +01:00
Mark Felder
518a38577b
Fix dialyzer errors due to deprecated usage of put_layout/2
2024-01-31 14:55:11 -05:00
Mark Felder
97c4d3bcc9
Pleroma.Web.Plugs.RateLimiter.Supervisor: dialyzer error
...
lib/pleroma/web/plugs/rate_limiter/supervisor.ex:12:no_return
Function init/1 has no local return.
2024-01-31 13:12:56 -05:00
Mark Felder
92992c022d
Pleroma.Web.OAuth.OAuthController: dialyzer error
...
validate_scopes/2 can never receive a map as it is only called in one place with a guard requiring a list
lib/pleroma/web/o_auth/o_auth_controller.ex:615:guard_fail
The guard test:
is_map(_params :: maybe_improper_list())
can never succeed.
2024-01-31 11:29:06 -05:00
Mark Felder
5e8bedcca0
Pleroma.Web.PleromaAPI.MascotController: fix dialyzer error due to bad error match
...
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:37:pattern_match
The pattern can never match the type.
Pattern:
{:content_type, _}
Type:
{:error, _}
________________________________________________________________________________
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:40:pattern_match
The pattern can never match the type.
Pattern:
{:upload, {:error, _}}
Type:
{:error, _}
2024-01-31 11:15:48 -05:00
Mark Felder
ed2f8e45e9
Pleroma.Web.MastodonAPI.SearchController: fix dialyzer errors
...
Add a separate Pagination.paginate_list/2 function instead of overloading paginate/4 and complicating its matching and @spec
2024-01-31 11:12:41 -05:00
Mark Felder
c6f783c551
Pleroma.Web.ControllerHelper: fix @spec to resolve dialyzer errors
...
lib/pleroma/web/admin_api/controllers/user_controller.ex:333:no_return
Function index/2 has no local return.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:357:unused_fun
Function maybe_parse_filters/1 will never be called.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:366:no_return
Function page_params/1 has no local return.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:368:call
The function call will not succeed.
Pleroma.Web.ControllerHelper.fetch_integer_param(_params :: any(), :page, 1)
breaks the contract
(map(), String.t(), integer() | nil) :: integer() | nil
2024-01-31 11:01:37 -05:00
Mark Felder
225afe05b3
Pleroma.Web.TwitterAPI.UtilController: fix dialyzer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
b8db67dafa
Pleroma.Web.MastodonAPI.StatusController: fix dialzyer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
9760149886
Pleroma.Web.PleromaAPI.UserImportController: fix dialyzer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
c39e4dd214
Pleroma.Web.PleromaAPI.NotificationController: fix dialyzer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
e157fd60ee
Pleroma.Web.PleromaAPI.MascotController: fix dialyzer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
9d16393d8b
Pleroma.Web.PleromaAPI.EmojiPackController: fix dialyzer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
f1aeb80518
Pleroma.Web.PleromaAPI.EmojiFileController: fix dialyzer errors with replace_params: false
2024-01-31 10:13:02 -05:00
Mark Felder
4d20fbc6d9
Pleroma.Web.PleromaAPI.ChatController: fix dialyzer errors with replace_params: false
2024-01-31 10:12:59 -05:00
Mark Felder
fdddba100e
Pleroma.Web.MastodonAPI.SearchController: fix dialyzer errors with replace_params: false
2024-01-30 18:40:46 -05:00
Mark Felder
324fd08455
Pleroma.Web.MastodonAPI.ScheduledActivityController: fix dialyzer errors with replace_params: false
2024-01-30 18:33:23 -05:00
Mark Felder
d3c6acd2f2
Pleroma.Web.MastodonAPI.PollController: fix dialyzer errors with replace_params: false
2024-01-30 18:31:13 -05:00
Mark Felder
0a6b2c9587
Pleroma.Web.MastodonAPI.NotificationController: fix dialyzer errors with replace_params: false
2024-01-30 18:27:31 -05:00
Mark Felder
90c9f38f40
Pleroma.Web.MastodonAPI.MediaController: fix dialyzer errors with replace_params: false
2024-01-30 18:09:11 -05:00
Mark Felder
9741f045e4
Pleroma.Web.MastodonAPI.ListController: fix dialyzer errors with replace_params: false
2024-01-30 18:05:28 -05:00
Mark Felder
14de8376ad
Pleroma.Web.MastodonAPI.FollowRequestController: fix dialyzer errors with replace_params: false
2024-01-30 17:56:43 -05:00
Mark Felder
f400224a00
Pleroma.Web.AdminAPI.UserController: fix dialyzer errors with replace_params: false
2024-01-30 17:51:40 -05:00
Mark Felder
dd916e0b4c
Pleroma.Web.AdminAPI.ReportController: fix dialyzer errors with replace_params: false
2024-01-30 17:24:26 -05:00
Mark Felder
b84a33a101
Pleroma.Web.AdminAPI.RelayController: fix dialyzer errors with replace_params: false
2024-01-30 17:13:14 -05:00
Mark Felder
fb6f53fc12
Pleroma.Web.AdminAPI.MediaProxyCacheController: fix dialyzer errors with replace_params: false
2024-01-30 17:11:43 -05:00
Mark Felder
ea26dcd800
Pleroma.Web.AdminAPI.InviteController: fix dialyzer errors with replace_params: false
2024-01-30 17:06:37 -05:00
Mark Felder
85c9397ec7
Pleroma.Web.AdminAPI.InstanceDocumentController: fix dialyzer errors with replace_params: false
2024-01-30 17:04:12 -05:00
Mark Felder
ef1f301756
Pleroma.Web.AdminAPI.ConfigController: fix dialyzer errors with replace_params: false
2024-01-30 16:59:28 -05:00
Mark Felder
81c8592d69
Pleroma.Web.MastodonAPI.AccountController: fix dialyzer errors with replace_params: false
2024-01-30 16:50:00 -05:00
Mark Felder
cfe7438b2f
Pleroma.Web.MastodonAPI.DomainBlockController: dialyzer fixes via :replace_params
2024-01-30 14:52:15 -05:00
Mark Felder
608466d098
Modify our CastAndValidate plug to include the new functionality provided by the :replace_params config option
...
This allows us to configure Open API Spex to not overwrite the params with the casted versions which violates the Plug.Conn.t() contract
https://github.com/open-api-spex/open_api_spex/issues/92
https://github.com/open-api-spex/open_api_spex/pull/425
2024-01-30 14:49:55 -05:00
Mark Felder
8476eb1842
Revert "Pleroma.Web.AdminAPI.MediaProxyCacheController: dialyzer errors"
...
This reverts commit 8b02c85810
.
2024-01-30 14:20:02 -05:00
Mark Felder
88a35b2865
Revert "Pleroma.Web.TwitterAPI.UtilController: dialyzer fixes"
...
This reverts commit 17f4251b19
.
2024-01-30 14:19:54 -05:00
Mark Felder
674ae51d6a
Revert "Pleroma.Web.PleromaAPI.UserImportController: Dialyzer errors"
...
This reverts commit 52e18a6249
.
2024-01-30 14:19:41 -05:00
Mark Felder
4a80a285d1
Revert "Pleroma.Web.PleromaAPI.NotificationController: dialyzer errors"
...
This reverts commit 26a95e5787
.
2024-01-30 14:19:32 -05:00
Mark Felder
1e76ceacd5
Revert "Pleroma.Web.AdminAPI.ConfigController: dialyzer error"
...
This reverts commit 60d89cb404
.
2024-01-30 14:19:13 -05:00
Mark Felder
2c8e4f32c6
Revert "Pleroma.Web.PleromaAPI.MascotController: dialyzer errors"
...
This reverts commit 9c8055d4b3
.
2024-01-30 14:18:36 -05:00
Mark Felder
b1a6102a85
Revert "Pleroma.Web.PleromaAPI.EmojiPackController: dialyzer errors"
...
This reverts commit 77bf617c4b
.
2024-01-30 14:15:40 -05:00
Mark Felder
4a9ed4682a
Revert "Pleroma.Web.PleromaAPI.EmojiFileController: dialyzer errors"
...
This reverts commit dc912dc590
.
2024-01-30 14:15:17 -05:00
Mark Felder
b709fc4dfe
Revert "Pleroma.Web.PleromaAPI.ChatController: Dialyzer error"
...
This reverts commit 8d64eedbec
.
2024-01-30 14:15:05 -05:00
Mark Felder
589456f0ba
Revert "Pleroma.Web.AdminAPI.InviteController: dialyzer errors"
...
This reverts commit db87be126e
.
2024-01-30 14:14:56 -05:00
Mark Felder
4227db0871
Revert "Pleroma.Web.AdminAPI.UserController: dialyzer errors"
...
This reverts commit d92c3d927d
.
2024-01-30 14:14:39 -05:00
Mark Felder
ac06a47689
Revert "Pleroma.Web.AdminAPI.RelayController: dialyzer errors"
...
This reverts commit 94838ed941
.
2024-01-30 14:14:22 -05:00
Mark Felder
da5e0fca4f
Revert "Pleroma.Web.AdminAPI.ReportController: dialyzer errors"
...
This reverts commit bfe626d578
.
2024-01-30 14:14:07 -05:00
Mark Felder
39241107d9
Revert "Pleroma.Web.AdminAPI.InstanceDocumentController: fix dialyzer error"
...
This reverts commit a3024dd5ac
.
2024-01-30 14:13:56 -05:00