30 to 70% of the objects in the object table are simple JSON objects
containing a single field, 'id', being the context's ID. The reason for
the creation of an object per context seems to be an old relic from the
StatusNet era, and has only been used nowadays as an helper for threads
in Pleroma-FE via the `pleroma.conversation_id` field in status views.
An object per context was created, and its numerical ID (table column)
was used and stored as 'context_id' in the object and activity along
with the full 'context' URI/string.
This commit removes this field and stops creation of objects for each
context, which will also allow incoming activities to use activity IDs
as contexts, something which was not possible before, or would have been
very broken under most circumstances.
The `pleroma.conversation_id` field has been reimplemented in a way to
maintain backwards-compatibility by calculating a CRC32 of the full
context URI/string in the object, instead of relying on the row ID for
the created context object.
This implements fully_qualify_emoji/1, which will return the
fully-qualified version of an emoji if it knows of one, or return the
emoji unmodified if not.
This code generates combinations per emoji: for each FE0F, all possible
combinations of the character being removed or staying will be
generated. This is made as an attempt to find all partially-qualified
and unqualified versions of a fully-qualified emoji.
I have found *no cases* for which this would be a problem, after
browsing the entire emoji list in emoji-test.txt. This is safe, and,
sadly, most likely the sanest too.
Tries fully-qualifying emoji when receiving them, by adding the emoji
variation sequence to the received reaction emoji.
This issue arises when other instance software, such as Misskey, tries
reacting with emoji that have unqualified or minimally qualified
variants, like a red heart. Pleroma only accepts fully qualified emoji
in emoji reactions, and refused those emoji. Now, Pleroma will attempt
to properly qualify them first, and reject them if checks still fail.
This commit contains changes to tests proposed by lanodan.
Co-authored-by: Haelwenn <contact+git.pleroma.social@hacktivis.me>
This was done by floatingghost as part of a bigger commit in Akkoma.
See <37ae047e16/lib/pleroma/application.ex (L83)>.
As explained in <https://ihatebeinga.live/objects/860d23e1-dc64-4b07-8b4d-020b9c56cff6>
> there are so many caches that clearing them all can nuke the supervisor, which by default will become an hero if it gets more than 3 restarts in <5 seconds
And further down the thread
> essentially we've got like 11 caches (37ae047e16/lib/pleroma/application.ex (L165))
> then in test we fetch them all (https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/test/support/data_case.ex#L50) and call clear on them
> so if this clear fails on any 3 of them, the pleroma supervisor itself will die
How it fails?
> idk maybe cachex dies, maybe :ets does a weird thing
> it doesn't log anything, it just consistently dies during cache clearing so i figured it had to be that
> honestly my best bet is locksmith and queuing
> https://github.com/whitfin/cachex/blob/master/lib/cachex/actions/clear.ex#L26
> clear is thrown into a locksmith transaction
> locksmith says
> >If the process is already in a transactional context, the provided function will be executed immediately. Otherwise the required keys will be locked until the provided function has finished executing.
> so if we get 2 clears too close together, maybe it locks, then doesn't like the next clear?
It is possible for an earlier Update to be received by us later.
For this, we now
(1) only allows Updates to poll counts if there is no updated field,
or the updated field is the same as the last updated date or
creation date;
(2) does not allow updating anything if the updated field
is older than the last updated date or creation date;
(3) allows updating updatable fields otherwise (normal updates);
(4) if only the updated field is changed, it does not create
a new history item on its own.
In Create validator we do not validate the object data,
but that is because the object itself will go through the
pipeline again, which is not the case for Update. Thus,
we added validation for objects in Update activities.
I renamed some tags before, but forgot to rename the pipelines
I also had some tags which I forgot to add to the config, description, etc.
These have now been done/added
I used keyword_list[:key], but if the key doesn't exist, it will return nil. I actually expect a list and further down the code I use that list.
I believe the key should always be present, but in case it's not, it's better to return an empty list instead of nil. That way the code wont fail further down the line.
During attachment upload Pleroma returns a "description" field. Pleroma-fe has an MR to use that to pre-fill the image description field, <https://git.pleroma.social/pleroma/pleroma-fe/-/merge_requests/1399>
* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE
* If a description is already present (e.g. because a previous module added it), it will use that
* Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
* If no description is found, it will simply return nil, just like before
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so
This was taken from an MR i did on Pleroma and isn't finished yet.
I first focussed on getting things working
Now that they do and we know what tags there are, I put some thought in providing better names
I use the form <what_it_controls>_<what_it_allows_you_to_do>
:statuses_read => :messages_read
:status_delete => :messages_delete
:user_read => :users_read
:user_deletion => :users_delete
:user_activation => :users_manage_activation_state
:user_invite => :users_manage_invites
:user_tag => :users_manage_tags
:user_credentials => :users_manage_credentials
:report_handle => :reports_manage_reports
:emoji_management => :emoji_manage_emoji
Tries fully-qualifying emoji when receiving them, by adding the emoji
variation sequence to the received reaction emoji.
This issue arises when other instance software, such as Misskey, tries
reacting with emoji that have unqualified or minimally qualified
variants, like a red heart. Pleroma only accepts fully qualified emoji
in emoji reactions, and refused those emoji. Now, Pleroma will attempt
to properly qualify them first, and reject them if checks still fail.
Deactivated users are only visible to users privileged with :user_activation since fc317f3b17
Here we also make sure the users who are deactivated get the status deactivated for users who are allowed to see these users
Instead of `Pleroma.User.all_superusers()` we now use `Pleroma.User.all_superusers(:report_handle)`
I also changed it for sending emails, but there were no tests.
This should eventually replace the Pleroma.User.all_superusers/0 function
* I added a new param `is_privileged` in User.query
* Now we can fetch all users with a specified privilege
Before we deleted the notifications, but that was a side effect and didn't always trigger any more.
Now we just hide them when an unprivileged user asks them.
This reverts commit 89667189b8 and cdc5bbe836.
This is a side effect when changing user role.
The goal was to not have report notifications when someone isn't admin or moderator any more.
But this won't be triggered when we change the privilege tags for a role, so we can't use this sollution any more.
There was another solution to filter out report notifications during fetch.
It wasn't merged because this seemed 'cleaner' at the time, but now it seems the better sollution.
I'll add it in the next commit.
I still had three endpoints I didn't really know what to do with them. I added them under separate tags
* :instance_delete
* :moderation_log_read
* :stats_read
I also checked and these are the last changes done by MR https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3480/diffs this is trying to fix
The list of TLS versions was added by
8bd2b6eb13 when hackney version was
pinned to 1.15.2. Later hackney version was upgraded
(166455c884) but the list of TLS
versions wasn't removed. From the hackney point of view, this list has
been replaced by the OTP defaults since 0.16.0
(734694ea4e24f267864c459a2f050e943adc6694).
It looks like the same issue already occurred before:
0cb7b0ea84.
A way to test this issue (where example.com is an ActivityPub site
which uses TLSv1.3 only):
$ PLEROMA_CONFIG_PATH=/path/to/config.exs pleroma start_iex
Erlang/OTP 22 [erts-10.7.2.16] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]
Erlang/OTP 22 [erts-10.7.2.16] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]
Interactive Elixir (1.10.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(pleroma@127.0.0.1)2> Pleroma.Object.Fetcher.fetch_and_contain_remote_object_from_id("https://example.com/@/Nick/")
{:error,
{:tls_alert,
{:protocol_version,
'TLS client: In state hello received SERVER ALERT: Fatal - Protocol Version\n'}}}
With this patch, the output is the expected one:
iex(pleroma@127.0.0.1)3> Pleroma.Object.Fetcher.fetch_and_contain_remote_object_from_id("https://example.com/@/Nick/")
{:error,
{:ok,
%{
"@context" => [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
%{
"Emoji" => "toot:Emoji",
"Hashtag" => "as:Hashtag",
"atomUri" => "ostatus:atomUri",
"conversation" => "ostatus:conversation",
"featured" => "toot:featured",
"focalPoint" => %{"@container" => "@list", "@id" => "toot:focalPoint"},
"inReplyToAtomUri" => "ostatus:inReplyToAtomUri",
"manuallyApprovesFollowers" => "as:manuallyApprovesFollowers",
"movedTo" => "as:movedTo",
"ostatus" => "http://ostatus.org#",
"sensitive" => "as:sensitive",
"toot" => "http://joinmastodon.org/ns#"
}
],
"endpoints" => %{"sharedInbox" => "https://example.com/inbox"},
"followers" => "https://example.com/@/Nick/followers",
"following" => nil,
"icon" => %{
"type" => "Image",
"url" => "https://example.com/static/media/[...].png"
},
"id" => "https://example.com/@/Nick/",
"inbox" => "https://example.com/@/Nick/inbox",
"liked" => nil,
"name" => "Nick",
"outbox" => "https://example.com/@/Nick/outbox",
"preferredUsername" => "Nick",
"publicKey" => %{
"id" => "https://example.com/@/Nick/#main-key",
"owner" => "https://example.com/@/Nick/",
"publicKeyPem" => "[...]
},
"summary" => "",
"type" => "Person",
"url" => "https://example.com/@/Nick/"
}}
A way to test the reverse proxy bits of this issue (where example.com allows TLSv1.3 only):
iex(pleroma@127.0.0.1)1> Pleroma.ReverseProxy.Client.Hackney.request("GET", "https://example.com", [], [])
{:error,
{:tls_alert,
{:protocol_version,
'TLS client: In state hello received SERVER ALERT: Fatal - Protocol Version\n'}}}
* rejected_shortcodes is defined as a list of strings in the
configuration description. As such, database-based configuration was
led to handle those settings as strings, and not as the actually
expected type, Regex.
* This caused each message passing through this MRF, if a rejected
shortcode was set and the emoji did not exist already on the instance,
to fail federating, as an exception was raised, swiftly caught and
mostly silenced.
* This commit fixes the issue by introducing new behavior: strings are
now handled as perfect matches for an emoji shortcode (meaning that if
the emoji-to-be-pulled's shortcode is in the blacklist, it will be
rejected), while still supporting Regex types as before.
When someone isn't a superuser any more, they shouldn't see the reporsts any more either.
Here we delete the report notifications from a user when that user gets updated from being a superuser to a non-superuser.
elixir gettext current does not fully support fallback to another language [0].
But it might in the future. We adapt it so that all languages in Accept-Language
headers are received by Pleroma.Web.Gettext. User.languages is now a comma-separated
list.
[0]: https://github.com/elixir-gettext/gettext/issues/303
For an example, here, zh is not supported, but zh_Hans and zh_Hant
are. If the user asks for zh, we should choose a variant for them
instead of fallbacking to default.
Some browsers (e.g. Firefox) does not allow users to customize
their language codes. For example, there is no zh-Hans, but only
zh, zh-CN, zh-TW, zh-HK, etc. This provides a workaround for
those users suffering from bad design decisions.
Even though latest PleromaFE supports displaying these properly, mobile
apps still exist, so I think we should offer a workaround to those who
want it.
0: Use the CommonAPI unfollow function to make sure the
unfollow activity is federated.
1: Limit the follow and unfollow to local followers only,
while let the romote servers decide whether to move their followers.
Ref: emit-move
Mastodon uses the Reject activity also for the purpose of removing
a follower, in addition to reject a follow request. We should
also update the original Follow activity in this case.
Notes:
- QuestionValidator didn't have a :name field but that seems like a mistake
- `_fields` functions can't inherit others because of some Validators
- bto/bcc fields were absent in activities, also seems like a mistake
(Well IIRC we don't or barely support bto/bcc anyway)
* Policies were put under a new module (Pleroma.Web.ActivityPub.MRF.Policy instead of Pleroma.Web.ActivityPub.MRF), but this wasn't changed in the Pleroma.Web.ActivityPub.MRF @mrf_config_descriptions
* I don't have a unit test to prevent similar problems in the future because I don't find a proper way to do it
* The descriptions in the unit tests are defined in the unit tests, so if someone changes module names in the code, the tests wont see it
* The list is generated in Pleroma.Docs.Generator.list_behaviour_implementations, but I can't do a check in the when clause of the function to see if the provided module is a behaviour or not.
Backport of: https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3509
The original approach to search in GIN indexes is to use
`to_tsvector(text)` in the WHERE clause of the query. According to
postgres docs [pdoc], this method does not make use of the index,
while `to_tsvector(config, text)` does. This commit changed the
query to use the two-argument `to_tsvector()`.
[pdoc]: https://www.postgresql.org/docs/12/textsearch-tables.html
To obtain the search config in use, we make a query to the db first.
The `::regconfig::oid` hack is needed because Postgrex does not support
regconfig type directly [postgrexbug]. I use the conversion from and to
`oid` instead of `text` because I tested in the actual DB and querying
using the conversion via `text` is slow just as the one-argument
`to_tsvector()` variant.
[postgrexbug]: https://github.com/elixir-ecto/postgrex/issues/502
Backport of: https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3519
Closes: https://git.pleroma.social/pleroma/pleroma/-/issues/2758
* To see what front ends are installed, it ls static/frontends. When this folder doesn't exists yet, it will return an empty array.
* Installing still works since the folder is created during installation already
Backport of: https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3510
The original approach to search in GIN indexes is to use
`to_tsvector(text)` in the WHERE clause of the query. According to
postgres docs [pdoc], this method does not make use of the index,
while `to_tsvector(config, text)` does. This commit changed the
query to use the two-argument `to_tsvector()`.
[pdoc]: https://www.postgresql.org/docs/12/textsearch-tables.html
To obtain the search config in use, we make a query to the db first.
The `::regconfig::oid` hack is needed because Postgrex does not support
regconfig type directly [postgrexbug]. I use the conversion from and to
`oid` instead of `text` because I tested in the actual DB and querying
using the conversion via `text` is slow just as the one-argument
`to_tsvector()` variant.
[postgrexbug]: https://github.com/elixir-ecto/postgrex/issues/502
BUG: https://git.pleroma.social/pleroma/pleroma/-/issues/2758
* To see what front ends are installed, it ls static/frontends. When this folder doesn't exists yet, it will return an empty array.
* Installing still works since the folder is created during installation already
* Policies were put under a new module (Pleroma.Web.ActivityPub.MRF.Policy instead of Pleroma.Web.ActivityPub.MRF), but this wasn't changed in the Pleroma.Web.ActivityPub.MRF @mrf_config_descriptions
* I don't have a unit test to prevent similar problems in the future because I don't find a proper way to do it
* The descriptions in the unit tests are defined in the unit tests, so if someone changes module names in the code, the tests wont see it
* The list is generated in Pleroma.Docs.Generator.list_behaviour_implementations, but I can't do a check in the when clause of the function to see if the provided module is a behaviour or not.
Hélène
Mark Felder
![https://example.com/static/media/[...].png](https://example.com/static/media/[...].png){kind=link}