Merge branch 'develop' into feature/digest-email

CHANGELOG.md

@@ -4,12 +4,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
+### Security
+- OStatus: eliminate the possibility of a protocol downgrade attack.
+- OStatus: prevent following locked accounts, bypassing the approval process.
+
 ### Changed
 - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
+- **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
 - Configuration: OpenGraph and TwitterCard providers enabled by default
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
+- NodeInfo: Return `mailerEnabled` in `metadata`
 - Mastodon API: Unsubscribe followers when they unfollow a user
 - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
 
@@ -26,6 +32,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Rich Media: Parser failing when no TTL can be found by image TTL setters
 - Rich Media: The crawled URL is now spliced into the rich media data.
 - ActivityPub S2S: sharedInbox usage has been mostly aligned with the rules in the AP specification.
+- ActivityPub S2S: remote user deletions now work the same as local user deletions.
+- Not being able to access the Mastodon FE login page on private instances
+- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag
 
 ### Added
 - MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
@@ -41,6 +50,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Add support for categories for custom emojis by reusing the group feature. <https://github.com/tootsuite/mastodon/pull/11196>
 - Mastodon API: Add support for muting/unmuting notifications
 - Mastodon API: Add support for the `blocked_by` attribute in the relationship API (`GET /api/v1/accounts/relationships`). <https://github.com/tootsuite/mastodon/pull/10373>
+- Mastodon API: Add support for the `domain_blocking` attribute in the relationship API (`GET /api/v1/accounts/relationships`).
 - Mastodon API: Add `pleroma.deactivated` to the Account entity
 - Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
 - Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
@@ -48,6 +58,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
 - Admin API: Added support for `tuples`.
+- Admin API: Added endpoints to run mix tasks pleroma.config migrate_to_db & pleroma.config migrate_from_db
 - Added synchronization of following/followers counters for external users
 - Configuration: `enabled` option for `Pleroma.Emails.Mailer`, defaulting to `false`.
 - Configuration: Pleroma.Plugs.RateLimiter `bucket_name`, `params` options.
@@ -56,6 +67,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - ActivityPub: Add an internal service actor for fetching ActivityPub objects.
 - ActivityPub: Optional signing of ActivityPub object fetches.
 - Admin API: Endpoint for fetching latest user's statuses
+- Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation.
 
 ### Changed
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text

config/config.exs

@@ -542,7 +542,8 @@
   relation_id_action: {60_000, 2},
   statuses_actions: {10_000, 15},
   status_id_action: {60_000, 3},
-  password_reset: {1_800_000, 5}
+  password_reset: {1_800_000, 5},
+  account_confirmation_resend: {8_640_000, 5}
 
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.

config/test.exs

@@ -29,7 +29,8 @@
   email: "admin@example.com",
   notify_email: "noreply@example.com",
   skip_thread_containment: false,
-  federating: false
+  federating: false,
+  external_user_synchronization: false
 
 config :pleroma, :activitypub, sign_object_fetches: false
 

docs/api/admin_api.md

@@ -575,6 +575,29 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
     - 404 Not Found `"Not found"`
   - On success: 200 OK `{}`
 
+
+## `/api/pleroma/admin/config/migrate_to_db`
+### Run mix task pleroma.config migrate_to_db
+Copy settings on key `:pleroma` to DB.
+- Method `GET`
+- Params: none
+- Response:
+
+```json
+{}
+```
+
+## `/api/pleroma/admin/config/migrate_from_db`
+### Run mix task pleroma.config migrate_from_db
+Copy all settings from DB to `config/prod.exported_from_db.secret.exs` with deletion from DB.
+- Method `GET`
+- Params: none
+- Response:
+
+```json
+{}
+```
+
 ## `/api/pleroma/admin/config`
 ### List config settings
 List config settings only works with `:pleroma => :instance => :dynamic_configuration` setting to `true`.

docs/api/pleroma_api.md

@@ -245,6 +245,14 @@ See [Admin-API](Admin-API.md)
 - PATCH `/api/v1/pleroma/accounts/update_banner`: Set/clear user banner image
 - PATCH `/api/v1/pleroma/accounts/update_background`: Set/clear user background image
 
+## `/api/v1/pleroma/accounts/confirmation_resend`
+### Resend confirmation email
+* Method `POST`
+* Params:
+    * `email`: email of that needs to be verified
+* Authentication: not required
+* Response: 204 No Content 
+
 ## `/api/v1/pleroma/mascot`
 ### Gets user mascot image
 * Method `GET`

lib/mix/tasks/pleroma/config.ex

@@ -15,7 +15,7 @@ defmodule Mix.Tasks.Pleroma.Config do
 
       mix pleroma.config migrate_to_db
 
-  ## Transfers config from DB to file.
+  ## Transfers config from DB to file `config/env.exported_from_db.secret.exs`
 
       mix pleroma.config migrate_from_db ENV
   """

lib/mix/tasks/pleroma/database.ex

@@ -8,6 +8,7 @@ defmodule Mix.Tasks.Pleroma.Database do
   alias Pleroma.Repo
   alias Pleroma.User
   require Logger
+  require Pleroma.Constants
   import Mix.Pleroma
   use Mix.Task
 
@@ -99,10 +100,15 @@ def run(["prune_objects" | args]) do
       NaiveDateTime.utc_now()
       |> NaiveDateTime.add(-(deadline * 86_400))
 
-    public = "https://www.w3.org/ns/activitystreams#Public"
-
     from(o in Object,
-      where: fragment("?->'to' \\? ? OR ?->'cc' \\? ?", o.data, ^public, o.data, ^public),
+      where:
+        fragment(
+          "?->'to' \\? ? OR ?->'cc' \\? ?",
+          o.data,
+          ^Pleroma.Constants.as_public(),
+          o.data,
+          ^Pleroma.Constants.as_public()
+        ),
       where: o.inserted_at < ^time_deadline,
       where:
         fragment("split_part(?->>'actor', '/', 3) != ?", o.data, ^Pleroma.Web.Endpoint.host())

lib/pleroma/activity/search.ex

@@ -9,6 +9,8 @@ defmodule Pleroma.Activity.Search do
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Visibility
 
+  require Pleroma.Constants
+
   import Ecto.Query
 
   def search(user, search_query, options \\ []) do
@@ -39,7 +41,7 @@ def maybe_restrict_author(query, _), do: query
   defp restrict_public(q) do
     from([a, o] in q,
       where: fragment("?->>'type' = 'Create'", a.data),
-      where: "https://www.w3.org/ns/activitystreams#Public" in a.recipients
+      where: ^Pleroma.Constants.as_public() in a.recipients
     )
   end
 

lib/pleroma/constants.ex

@@ -0,0 +1,9 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Constants do
+  use Const
+
+  const(as_public, do: "https://www.w3.org/ns/activitystreams#Public")
+end

lib/pleroma/flake_id.ex

@@ -66,6 +66,16 @@ def from_integer(integer) do
   @spec get :: binary
   def get, do: to_string(:gen_server.call(:flake, :get))
 
+  # checks that ID is is valid FlakeID
+  #
+  @spec is_flake_id?(String.t()) :: boolean
+  def is_flake_id?(id), do: is_flake_id?(String.to_charlist(id), true)
+  defp is_flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: is_flake_id?(cs, true)
+  defp is_flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: is_flake_id?(cs, true)
+  defp is_flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: is_flake_id?(cs, true)
+  defp is_flake_id?([], true), do: true
+  defp is_flake_id?(_, _), do: false
+
   # -- Ecto.Type API
   @impl Ecto.Type
   def type, do: :uuid

lib/pleroma/http/connection.ex

@@ -11,6 +11,7 @@ defmodule Pleroma.HTTP.Connection do
     connect_timeout: 10_000,
     recv_timeout: 20_000,
     follow_redirect: true,
+    force_redirect: true,
     pool: :federation
   ]
   @adapter Application.get_env(:tesla, :adapter)

lib/pleroma/object/fetcher.ex

@@ -114,7 +114,7 @@ defp maybe_date_fetch(headers, date) do
     end
   end
 
-  def fetch_and_contain_remote_object_from_id(id) do
+  def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
     Logger.info("Fetching object #{id} via AP")
 
     date =
@@ -141,4 +141,9 @@ def fetch_and_contain_remote_object_from_id(id) do
         {:error, e}
     end
   end
+
+  def fetch_and_contain_remote_object_from_id(%{"id" => id}),
+    do: fetch_and_contain_remote_object_from_id(id)
+
+  def fetch_and_contain_remote_object_from_id(_id), do: {:error, "id must be a string"}
 end

lib/pleroma/plugs/set_format_plug.ex

@@ -0,0 +1,24 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Plugs.SetFormatPlug do
+  import Plug.Conn, only: [assign: 3, fetch_query_params: 1]
+
+  def init(_), do: nil
+
+  def call(conn, _) do
+    case get_format(conn) do
+      nil -> conn
+      format -> assign(conn, :format, format)
+    end
+  end
+
+  defp get_format(conn) do
+    conn.private[:phoenix_format] ||
+      case fetch_query_params(conn) do
+        %{query_params: %{"_format" => format}} -> format
+        _ -> nil
+      end
+  end
+end

lib/pleroma/signature.ex

@@ -15,7 +15,7 @@ def key_id_to_actor_id(key_id) do
       |> Map.put(:fragment, nil)
 
     uri =
-      if String.ends_with?(uri.path, "/publickey") do
+      if not is_nil(uri.path) and String.ends_with?(uri.path, "/publickey") do
         Map.put(uri, :path, String.replace(uri.path, "/publickey", ""))
       else
         uri

lib/pleroma/upload.ex

@@ -228,7 +228,14 @@ defp url_from_spec(%__MODULE__{name: name}, base_url, {:file, path}) do
           ""
         end
 
-    [base_url, "media", path]
+    prefix =
+      if is_nil(Pleroma.Config.get([__MODULE__, :base_url])) do
+        "media"
+      else
+        ""
+      end
+
+    [base_url, prefix, path]
     |> Path.join()
   end
 

lib/pleroma/user.ex

@@ -115,7 +115,9 @@ def ap_following(%User{} = user), do: "#{ap_id(user)}/following"
 
   def user_info(%User{} = user, args \\ %{}) do
     following_count =
-      if args[:following_count], do: args[:following_count], else: following_count(user)
+      if args[:following_count],
+        do: args[:following_count],
+        else: user.info.following_count || following_count(user)
 
     follower_count =
       if args[:follower_count], do: args[:follower_count], else: user.info.follower_count
@@ -227,6 +229,7 @@ def password_update_changeset(struct, params) do
     |> put_password_hash
   end
 
+  @spec reset_password(User.t(), map) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
   def reset_password(%User{id: user_id} = user, data) do
     multi =
       Multi.new()
@@ -331,6 +334,7 @@ def needs_update?(%User{local: false} = user) do
 
   def needs_update?(_), do: true
 
+  @spec maybe_direct_follow(User.t(), User.t()) :: {:ok, User.t()} | {:error, String.t()}
   def maybe_direct_follow(%User{} = follower, %User{local: true, info: %{locked: true}}) do
     {:ok, follower}
   end
@@ -405,6 +409,8 @@ def follow(%User{} = follower, %User{info: info} = followed) do
 
         {1, [follower]} = Repo.update_all(q, [])
 
+        follower = maybe_update_following_count(follower)
+
         {:ok, _} = update_follower_count(followed)
 
         set_cache(follower)
@@ -424,6 +430,8 @@ def unfollow(%User{} = follower, %User{} = followed) do
 
       {1, [follower]} = Repo.update_all(q, [])
 
+      follower = maybe_update_following_count(follower)
+
       {:ok, followed} = update_follower_count(followed)
 
       set_cache(follower)
@@ -472,7 +480,7 @@ def set_cache(%User{} = user) do
   end
 
   def update_and_set_cache(changeset) do
-    with {:ok, user} <- Repo.update(changeset) do
+    with {:ok, user} <- Repo.update(changeset, stale_error_field: :id) do
       set_cache(user)
     else
       e -> e
@@ -708,32 +716,73 @@ def update_note_count(%User{} = user) do
     |> update_and_set_cache()
   end
 
-  def update_follower_count(%User{} = user) do
+  def maybe_fetch_follow_information(user) do
-    follower_count_query =
+    with {:ok, user} <- fetch_follow_information(user) do
-      User.Query.build(%{followers: user, deactivated: false})
+      user
-      |> select([u], %{count: count(u.id)})
+    else
+      e ->
+        Logger.error("Follower/Following counter update for #{user.ap_id} failed.\n#{inspect(e)}")
 
-    User
+        user
-    |> where(id: ^user.id)
-    |> join(:inner, [u], s in subquery(follower_count_query))
-    |> update([u, s],
-      set: [
-        info:
-          fragment(
-            "jsonb_set(?, '{follower_count}', ?::varchar::jsonb, true)",
-            u.info,
-            s.count
-          )
-      ]
-    )
-    |> select([u], u)
-    |> Repo.update_all([])
-    |> case do
-      {1, [user]} -> set_cache(user)
-      _ -> {:error, user}
     end
   end
 
+  def fetch_follow_information(user) do
+    with {:ok, info} <- ActivityPub.fetch_follow_information_for_user(user) do
+      info_cng = User.Info.follow_information_update(user.info, info)
+
+      changeset =
+        user
+        |> change()
+        |> put_embed(:info, info_cng)
+
+      update_and_set_cache(changeset)
+    else
+      {:error, _} = e -> e
+      e -> {:error, e}
+    end
+  end
+
+  def update_follower_count(%User{} = user) do
+    if user.local or !Pleroma.Config.get([:instance, :external_user_synchronization]) do
+      follower_count_query =
+        User.Query.build(%{followers: user, deactivated: false})
+        |> select([u], %{count: count(u.id)})
+
+      User
+      |> where(id: ^user.id)
+      |> join(:inner, [u], s in subquery(follower_count_query))
+      |> update([u, s],
+        set: [
+          info:
+            fragment(
+              "jsonb_set(?, '{follower_count}', ?::varchar::jsonb, true)",
+              u.info,
+              s.count
+            )
+        ]
+      )
+      |> select([u], u)
+      |> Repo.update_all([])
+      |> case do
+        {1, [user]} -> set_cache(user)
+        _ -> {:error, user}
+      end
+    else
+      {:ok, maybe_fetch_follow_information(user)}
+    end
+  end
+
+  def maybe_update_following_count(%User{local: false} = user) do
+    if Pleroma.Config.get([:instance, :external_user_synchronization]) do
+      {:ok, maybe_fetch_follow_information(user)}
+    else
+      user
+    end
+  end
+
+  def maybe_update_following_count(user), do: user
+
   def remove_duplicated_following(%User{following: following} = user) do
     uniq_following = Enum.uniq(following)
 
@@ -883,19 +932,26 @@ def muted_notifications?(nil, _), do: false
   def muted_notifications?(user, %{ap_id: ap_id}),
     do: Enum.member?(user.info.muted_notifications, ap_id)
 
-  def blocks?(%User{info: info} = _user, %{ap_id: ap_id}) do
+  def blocks?(%User{} = user, %User{} = target) do
-    blocks = info.blocks
+    blocks_ap_id?(user, target) || blocks_domain?(user, target)
-
-    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(info.domain_blocks)
-
-    %{host: host} = URI.parse(ap_id)
-
-    Enum.member?(blocks, ap_id) ||
-      Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
   end
 
   def blocks?(nil, _), do: false
 
+  def blocks_ap_id?(%User{} = user, %User{} = target) do
+    Enum.member?(user.info.blocks, target.ap_id)
+  end
+
+  def blocks_ap_id?(_, _), do: false
+
+  def blocks_domain?(%User{} = user, %User{} = target) do
+    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(user.info.domain_blocks)
+    %{host: host} = URI.parse(target.ap_id)
+    Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
+  end
+
+  def blocks_domain?(_, _), do: false
+
   def subscribed_to?(user, %{ap_id: ap_id}) do
     with %User{} = target <- get_cached_by_ap_id(ap_id) do
       Enum.member?(target.info.subscribers, user.ap_id)

lib/pleroma/user/info.ex

@@ -16,6 +16,8 @@ defmodule Pleroma.User.Info do
     field(:source_data, :map, default: %{})
     field(:note_count, :integer, default: 0)
     field(:follower_count, :integer, default: 0)
+    # Should be filled in only for remote users
+    field(:following_count, :integer, default: nil)
     field(:locked, :boolean, default: false)
     field(:confirmation_pending, :boolean, default: false)
     field(:confirmation_token, :string, default: nil)
@@ -248,7 +250,11 @@ def remote_user_creation(info, params) do
       :uri,
       :hub,
       :topic,
-      :salmon
+      :salmon,
+      :hide_followers,
+      :hide_follows,
+      :follower_count,
+      :following_count
     ])
   end
 
@@ -259,7 +265,11 @@ def user_upgrade(info, params) do
       :source_data,
       :banner,
       :locked,
-      :magic_key
+      :magic_key,
+      :follower_count,
+      :following_count,
+      :hide_follows,
+      :hide_followers
     ])
   end
 
@@ -373,4 +383,14 @@ def remove_reblog_mute(info, ap_id) do
 
     cast(info, params, [:muted_reblogs])
   end
+
+  def follow_information_update(info, params) do
+    info
+    |> cast(params, [
+      :hide_followers,
+      :hide_follows,
+      :follower_count,
+      :following_count
+    ])
+  end
 end

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -23,6 +23,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   import Pleroma.Web.ActivityPub.Visibility
 
   require Logger
+  require Pleroma.Constants
 
   # For Announce activities, we filter the recipients based on following status for any actors
   # that match actual users.  See issue #164 for more information about why this is necessary.
@@ -207,8 +208,6 @@ def stream_out_participations(%Object{data: %{"context" => context}}, user) do
   def stream_out_participations(_, _), do: :noop
 
   def stream_out(activity) do
-    public = "https://www.w3.org/ns/activitystreams#Public"
-
     if activity.data["type"] in ["Create", "Announce", "Delete"] do
       object = Object.normalize(activity)
       # Do not stream out poll replies
@@ -216,7 +215,7 @@ def stream_out(activity) do
         Pleroma.Web.Streamer.stream("user", activity)
         Pleroma.Web.Streamer.stream("list", activity)
 
-        if Enum.member?(activity.data["to"], public) do
+        if get_visibility(activity) == "public" do
           Pleroma.Web.Streamer.stream("public", activity)
 
           if activity.local do
@@ -238,13 +237,8 @@ def stream_out(activity) do
             end
           end
         else
-          # TODO: Write test, replace with visibility test
+          if get_visibility(activity) == "direct",
-          if !Enum.member?(activity.data["cc"] || [], public) &&
+            do: Pleroma.Web.Streamer.stream("direct", activity)
-               !Enum.member?(
-                 activity.data["to"],
-                 User.get_cached_by_ap_id(activity.data["actor"]).follower_address
-               ),
-             do: Pleroma.Web.Streamer.stream("direct", activity)
         end
       end
     end
@@ -514,7 +508,7 @@ def flag(
   end
 
   defp fetch_activities_for_context_query(context, opts) do
-    public = ["https://www.w3.org/ns/activitystreams#Public"]
+    public = [Pleroma.Constants.as_public()]
 
     recipients =
       if opts["user"], do: [opts["user"].ap_id | opts["user"].following] ++ public, else: public
@@ -555,7 +549,7 @@ def fetch_latest_activity_id_for_context(context, opts \\ %{}) do
   end
 
   def fetch_public_activities(opts \\ %{}) do
-    q = fetch_activities_query(["https://www.w3.org/ns/activitystreams#Public"], opts)
+    q = fetch_activities_query([Pleroma.Constants.as_public()], opts)
 
     q
     |> restrict_unlisted()
@@ -646,10 +640,9 @@ defp user_activities_recipients(%{"godmode" => true}) do
 
   defp user_activities_recipients(%{"reading_user" => reading_user}) do
     if reading_user do
-      ["https://www.w3.org/ns/activitystreams#Public"] ++
+      [Pleroma.Constants.as_public()] ++ [reading_user.ap_id | reading_user.following]
-        [reading_user.ap_id | reading_user.following]
     else
-      ["https://www.w3.org/ns/activitystreams#Public"]
+      [Pleroma.Constants.as_public()]
     end
   end
 
@@ -834,7 +827,7 @@ defp restrict_unlisted(query) do
         fragment(
           "not (coalesce(?->'cc', '{}'::jsonb) \\?| ?)",
           activity.data,
-          ^["https://www.w3.org/ns/activitystreams#Public"]
+          ^[Pleroma.Constants.as_public()]
         )
     )
   end
@@ -971,7 +964,7 @@ def fetch_activities_bounded_query(query, recipients, recipients_with_public) do
       where:
         fragment("? && ?", activity.recipients, ^recipients) or
           (fragment("? && ?", activity.recipients, ^recipients_with_public) and
-             "https://www.w3.org/ns/activitystreams#Public" in activity.recipients)
+             ^Pleroma.Constants.as_public() in activity.recipients)
     )
   end
 
@@ -1016,10 +1009,10 @@ defp object_to_user_data(data) do
     user_data = %{
       ap_id: data["id"],
       info: %{
-        "ap_enabled" => true,
+        ap_enabled: true,
-        "source_data" => data,
+        source_data: data,
-        "banner" => banner,
+        banner: banner,
-        "locked" => locked
+        locked: locked
       },
       avatar: avatar,
       name: data["name"],
@@ -1043,6 +1036,71 @@ defp object_to_user_data(data) do
     {:ok, user_data}
   end
 
+  def fetch_follow_information_for_user(user) do
+    with {:ok, following_data} <-
+           Fetcher.fetch_and_contain_remote_object_from_id(user.following_address),
+         following_count when is_integer(following_count) <- following_data["totalItems"],
+         {:ok, hide_follows} <- collection_private(following_data),
+         {:ok, followers_data} <-
+           Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address),
+         followers_count when is_integer(followers_count) <- followers_data["totalItems"],
+         {:ok, hide_followers} <- collection_private(followers_data) do
+      {:ok,
+       %{
+         hide_follows: hide_follows,
+         follower_count: followers_count,
+         following_count: following_count,
+         hide_followers: hide_followers
+       }}
+    else
+      {:error, _} = e ->
+        e
+
+      e ->
+        {:error, e}
+    end
+  end
+
+  defp maybe_update_follow_information(data) do
+    with {:enabled, true} <-
+           {:enabled, Pleroma.Config.get([:instance, :external_user_synchronization])},
+         {:ok, info} <- fetch_follow_information_for_user(data) do
+      info = Map.merge(data.info, info)
+      Map.put(data, :info, info)
+    else
+      {:enabled, false} ->
+        data
+
+      e ->
+        Logger.error(
+          "Follower/Following counter update for #{data.ap_id} failed.\n" <> inspect(e)
+        )
+
+        data
+    end
+  end
+
+  defp collection_private(data) do
+    if is_map(data["first"]) and
+         data["first"]["type"] in ["CollectionPage", "OrderedCollectionPage"] do
+      {:ok, false}
+    else
+      with {:ok, %{"type" => type}} when type in ["CollectionPage", "OrderedCollectionPage"] <-
+             Fetcher.fetch_and_contain_remote_object_from_id(data["first"]) do
+        {:ok, false}
+      else
+        {:error, {:ok, %{status: code}}} when code in [401, 403] ->
+          {:ok, true}
+
+        {:error, _} = e ->
+          e
+
+        e ->
+          {:error, e}
+      end
+    end
+  end
+
   def user_data_from_user_object(data) do
     with {:ok, data} <- MRF.filter(data),
          {:ok, data} <- object_to_user_data(data) do
@@ -1054,7 +1112,8 @@ def user_data_from_user_object(data) do
 
   def fetch_and_prepare_user_from_ap_id(ap_id) do
     with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id),
-         {:ok, data} <- user_data_from_user_object(data) do
+         {:ok, data} <- user_data_from_user_object(data),
+         data <- maybe_update_follow_information(data) do
       {:ok, data}
     else
       e -> Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")

lib/pleroma/web/activity_pub/mrf/hellthread_policy.ex

@@ -4,6 +4,9 @@
 
 defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
   alias Pleroma.User
+
+  require Pleroma.Constants
+
   @moduledoc "Block messages with too much mentions (configurable)"
 
   @behaviour Pleroma.Web.ActivityPub.MRF
@@ -19,12 +22,12 @@ defp delist_message(message, threshold) when threshold > 0 do
         when follower_collection? and recipients > threshold ->
           message
           |> Map.put("to", [follower_collection])
-          |> Map.put("cc", ["https://www.w3.org/ns/activitystreams#Public"])
+          |> Map.put("cc", [Pleroma.Constants.as_public()])
 
         {:public, recipients} when recipients > threshold ->
           message
           |> Map.put("to", [])
-          |> Map.put("cc", ["https://www.w3.org/ns/activitystreams#Public"])
+          |> Map.put("cc", [Pleroma.Constants.as_public()])
 
         _ ->
           message
@@ -51,10 +54,10 @@ defp get_recipient_count(message) do
     recipients = (message["to"] || []) ++ (message["cc"] || [])
     follower_collection = User.get_cached_by_ap_id(message["actor"]).follower_address
 
-    if Enum.member?(recipients, "https://www.w3.org/ns/activitystreams#Public") do
+    if Enum.member?(recipients, Pleroma.Constants.as_public()) do
       recipients =
         recipients
-        |> List.delete("https://www.w3.org/ns/activitystreams#Public")
+        |> List.delete(Pleroma.Constants.as_public())
         |> List.delete(follower_collection)
 
       {:public, length(recipients)}

lib/pleroma/web/activity_pub/mrf/keyword_policy.ex

@@ -3,6 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
+  require Pleroma.Constants
+
   @moduledoc "Reject or Word-Replace messages with a keyword or regex"
 
   @behaviour Pleroma.Web.ActivityPub.MRF
@@ -31,12 +33,12 @@ defp check_reject(%{"object" => %{"content" => content, "summary" => summary}} =
   defp check_ftl_removal(
          %{"to" => to, "object" => %{"content" => content, "summary" => summary}} = message
        ) do
-    if "https://www.w3.org/ns/activitystreams#Public" in to and
+    if Pleroma.Constants.as_public() in to and
          Enum.any?(Pleroma.Config.get([:mrf_keyword, :federated_timeline_removal]), fn pattern ->
            string_matches?(content, pattern) or string_matches?(summary, pattern)
          end) do
-      to = List.delete(to, "https://www.w3.org/ns/activitystreams#Public")
+      to = List.delete(to, Pleroma.Constants.as_public())
-      cc = ["https://www.w3.org/ns/activitystreams#Public" | message["cc"] || []]
+      cc = [Pleroma.Constants.as_public() | message["cc"] || []]
 
       message =
         message

lib/pleroma/web/activity_pub/mrf/reject_non_public.ex

@@ -10,7 +10,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
 
   @behaviour Pleroma.Web.ActivityPub.MRF
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   @impl true
   def filter(%{"type" => "Create"} = object) do
@@ -19,8 +19,8 @@ def filter(%{"type" => "Create"} = object) do
     # Determine visibility
     visibility =
       cond do
-        @public in object["to"] -> "public"
+        Pleroma.Constants.as_public() in object["to"] -> "public"
-        @public in object["cc"] -> "unlisted"
+        Pleroma.Constants.as_public() in object["cc"] -> "unlisted"
         user.follower_address in object["to"] -> "followers"
         true -> "direct"
       end

lib/pleroma/web/activity_pub/mrf/simple_policy.ex

@@ -8,6 +8,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
   @moduledoc "Filter activities depending on their origin instance"
   @behaviour MRF
 
+  require Pleroma.Constants
+
   defp check_accept(%{host: actor_host} = _actor_info, object) do
     accepts =
       Pleroma.Config.get([:mrf_simple, :accept])
@@ -89,14 +91,10 @@ defp check_ftl_removal(%{host: actor_host} = _actor_info, object) do
     object =
       with true <- MRF.subdomain_match?(timeline_removal, actor_host),
            user <- User.get_cached_by_ap_id(object["actor"]),
-           true <- "https://www.w3.org/ns/activitystreams#Public" in object["to"] do
+           true <- Pleroma.Constants.as_public() in object["to"] do
-        to =
+        to = List.delete(object["to"], Pleroma.Constants.as_public()) ++ [user.follower_address]
-          List.delete(object["to"], "https://www.w3.org/ns/activitystreams#Public") ++
-            [user.follower_address]
 
-        cc =
+        cc = List.delete(object["cc"], user.follower_address) ++ [Pleroma.Constants.as_public()]
-          List.delete(object["cc"], user.follower_address) ++
-            ["https://www.w3.org/ns/activitystreams#Public"]
 
         object
         |> Map.put("to", to)

lib/pleroma/web/activity_pub/mrf/tag_policy.ex

@@ -19,7 +19,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
      - `mrf_tag:disable-any-subscription`: Reject any follow requests
   """
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   defp get_tags(%User{tags: tags}) when is_list(tags), do: tags
   defp get_tags(_), do: []
@@ -70,9 +70,9 @@ defp process_tag(
        ) do
     user = User.get_cached_by_ap_id(actor)
 
-    if Enum.member?(to, @public) do
+    if Enum.member?(to, Pleroma.Constants.as_public()) do
-      to = List.delete(to, @public) ++ [user.follower_address]
+      to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
-      cc = List.delete(cc, user.follower_address) ++ [@public]
+      cc = List.delete(cc, user.follower_address) ++ [Pleroma.Constants.as_public()]
 
       object =
         object
@@ -103,9 +103,10 @@ defp process_tag(
        ) do
     user = User.get_cached_by_ap_id(actor)
 
-    if Enum.member?(to, @public) or Enum.member?(cc, @public) do
+    if Enum.member?(to, Pleroma.Constants.as_public()) or
-      to = List.delete(to, @public) ++ [user.follower_address]
+         Enum.member?(cc, Pleroma.Constants.as_public()) do
-      cc = List.delete(cc, @public)
+      to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
+      cc = List.delete(cc, Pleroma.Constants.as_public())
 
       object =
         object

lib/pleroma/web/activity_pub/publisher.ex

@@ -11,6 +11,8 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
   alias Pleroma.Web.ActivityPub.Relay
   alias Pleroma.Web.ActivityPub.Transmogrifier
 
+  require Pleroma.Constants
+
   import Pleroma.Web.ActivityPub.Visibility
 
   @behaviour Pleroma.Web.Federator.Publisher
@@ -117,8 +119,6 @@ defp get_cc_ap_ids(ap_id, recipients) do
     |> Enum.map(& &1.ap_id)
   end
 
-  @as_public "https://www.w3.org/ns/activitystreams#Public"
-
   defp maybe_use_sharedinbox(%User{info: %{source_data: data}}),
     do: (is_map(data["endpoints"]) && Map.get(data["endpoints"], "sharedInbox")) || data["inbox"]
 
@@ -145,7 +145,7 @@ def determine_inbox(
       type == "Delete" ->
         maybe_use_sharedinbox(user)
 
-      @as_public in to || @as_public in cc ->
+      Pleroma.Constants.as_public() in to || Pleroma.Constants.as_public() in cc ->
         maybe_use_sharedinbox(user)
 
       length(to) + length(cc) > 1 ->

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -19,6 +19,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @doc """
   Modifies an incoming AP object (mastodon format) to our internal format.
@@ -102,8 +103,7 @@ def fix_explicit_addressing(object) do
 
     follower_collection = User.get_cached_by_ap_id(Containment.get_actor(object)).follower_address
 
-    explicit_mentions =
+    explicit_mentions = explicit_mentions ++ [Pleroma.Constants.as_public(), follower_collection]
-      explicit_mentions ++ ["https://www.w3.org/ns/activitystreams#Public", follower_collection]
 
     fix_explicit_addressing(object, explicit_mentions, follower_collection)
   end
@@ -115,11 +115,11 @@ def fix_implicit_addressing(%{"to" => to, "cc" => cc} = object, followers_collec
 
     if followers_collection not in recipients do
       cond do
-        "https://www.w3.org/ns/activitystreams#Public" in cc ->
+        Pleroma.Constants.as_public() in cc ->
           to = to ++ [followers_collection]
           Map.put(object, "to", to)
 
-        "https://www.w3.org/ns/activitystreams#Public" in to ->
+        Pleroma.Constants.as_public() in to ->
           cc = cc ++ [followers_collection]
           Map.put(object, "cc", cc)
 
@@ -480,8 +480,7 @@ def handle_incoming(
          {:ok, %User{} = follower} <- User.get_or_fetch_by_ap_id(follower),
          {:ok, activity} <- ActivityPub.follow(follower, followed, id, false) do
       with deny_follow_blocked <- Pleroma.Config.get([:user, :deny_follow_blocked]),
-           {_, false} <-
+           {_, false} <- {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
-             {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
            {_, false} <- {:user_locked, User.locked?(followed)},
            {_, {:ok, follower}} <- {:follow, User.follow(follower, followed)},
            {_, {:ok, _}} <-
@@ -609,13 +608,13 @@ def handle_incoming(
     with %User{ap_id: ^actor_id} = actor <- User.get_cached_by_ap_id(object["id"]) do
       {:ok, new_user_data} = ActivityPub.user_data_from_user_object(object)
 
-      banner = new_user_data[:info]["banner"]
+      banner = new_user_data[:info][:banner]
-      locked = new_user_data[:info]["locked"] || false
+      locked = new_user_data[:info][:locked] || false
 
       update_data =
         new_user_data
         |> Map.take([:name, :bio, :avatar])
-        |> Map.put(:info, %{"banner" => banner, "locked" => locked})
+        |> Map.put(:info, %{banner: banner, locked: locked})
 
       actor
       |> User.upgrade_changeset(update_data)
@@ -656,20 +655,7 @@ def handle_incoming(
       nil ->
         case User.get_cached_by_ap_id(object_id) do
           %User{ap_id: ^actor} = user ->
-            {:ok, followers} = User.get_followers(user)
+            User.delete(user)
-
-            Enum.each(followers, fn follower ->
-              User.unfollow(follower, user)
-            end)
-
-            {:ok, friends} = User.get_friends(user)
-
-            Enum.each(friends, fn followed ->
-              User.unfollow(user, followed)
-            end)
-
-            User.invalidate_cache(user)
-            Repo.delete(user)
 
           nil ->
             :error
@@ -1090,10 +1076,6 @@ def upgrade_user_from_ap_id(ap_id) do
         PleromaJobQueue.enqueue(:transmogrifier, __MODULE__, [:user_upgrade, user])
       end
 
-      if Pleroma.Config.get([:instance, :external_user_synchronization]) do
-        update_following_followers_counters(user)
-      end
-
       {:ok, user}
     else
       %User{} = user -> {:ok, user}
@@ -1126,27 +1108,4 @@ def maybe_fix_user_object(data) do
     data
     |> maybe_fix_user_url
   end
-
-  def update_following_followers_counters(user) do
-    info = %{}
-
-    following = fetch_counter(user.following_address)
-    info = if following, do: Map.put(info, :following_count, following), else: info
-
-    followers = fetch_counter(user.follower_address)
-    info = if followers, do: Map.put(info, :follower_count, followers), else: info
-
-    User.set_info_cache(user, info)
-  end
-
-  defp fetch_counter(url) do
-    with {:ok, %{body: body, status: code}} when code in 200..299 <-
-           Pleroma.HTTP.get(
-             url,
-             [{:Accept, "application/activity+json"}]
-           ),
-         {:ok, data} <- Jason.decode(body) do
-      data["totalItems"]
-    end
-  end
 end

lib/pleroma/web/activity_pub/utils.ex

@@ -18,6 +18,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @supported_object_types ["Article", "Note", "Video", "Page", "Question", "Answer"]
   @supported_report_states ~w(open closed resolved)
@@ -418,7 +419,7 @@ def make_follow_data(
       "type" => "Follow",
       "actor" => follower_id,
       "to" => [followed_id],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "object" => followed_id,
       "state" => "pending"
     }
@@ -510,7 +511,7 @@ def make_announce_data(
       "actor" => ap_id,
       "object" => id,
       "to" => [user.follower_address, object.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => object.data["context"]
     }
 
@@ -530,7 +531,7 @@ def make_unannounce_data(
       "actor" => ap_id,
       "object" => activity.data,
       "to" => [user.follower_address, activity.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => context
     }
 
@@ -547,7 +548,7 @@ def make_unlike_data(
       "actor" => ap_id,
       "object" => activity.data,
       "to" => [user.follower_address, activity.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => context
     }
 
@@ -556,7 +557,7 @@ def make_unlike_data(
 
   def add_announce_to_object(
         %Activity{
-          data: %{"actor" => actor, "cc" => ["https://www.w3.org/ns/activitystreams#Public"]}
+          data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]}
         },
         object
       ) do
@@ -765,7 +766,7 @@ defp get_updated_targets(
        ) do
     cc = Map.get(data, "cc", [])
     follower_address = User.get_cached_by_ap_id(data["actor"]).follower_address
-    public = "https://www.w3.org/ns/activitystreams#Public"
+    public = Pleroma.Constants.as_public()
 
     case visibility do
       "public" ->

lib/pleroma/web/activity_pub/visibility.ex

@@ -8,14 +8,14 @@ defmodule Pleroma.Web.ActivityPub.Visibility do
   alias Pleroma.Repo
   alias Pleroma.User
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   @spec is_public?(Object.t() | Activity.t() | map()) :: boolean()
   def is_public?(%Object{data: %{"type" => "Tombstone"}}), do: false
   def is_public?(%Object{data: data}), do: is_public?(data)
   def is_public?(%Activity{data: data}), do: is_public?(data)
   def is_public?(%{"directMessage" => true}), do: false
-  def is_public?(data), do: @public in (data["to"] ++ (data["cc"] || []))
+  def is_public?(data), do: Pleroma.Constants.as_public() in (data["to"] ++ (data["cc"] || []))
 
   def is_private?(activity) do
     with false <- is_public?(activity),
@@ -73,10 +73,10 @@ def get_visibility(object) do
     cc = object.data["cc"] || []
 
     cond do
-      @public in to ->
+      Pleroma.Constants.as_public() in to ->
         "public"
 
-      @public in cc ->
+      Pleroma.Constants.as_public() in cc ->
         "unlisted"
 
       # this should use the sql for the object's activity

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -379,6 +379,16 @@ def status_delete(%{assigns: %{user: user}} = conn, %{"id" => id}) do
     end
   end
 
+  def migrate_to_db(conn, _params) do
+    Mix.Tasks.Pleroma.Config.run(["migrate_to_db"])
+    json(conn, %{})
+  end
+
+  def migrate_from_db(conn, _params) do
+    Mix.Tasks.Pleroma.Config.run(["migrate_from_db", Pleroma.Config.get(:env), "true"])
+    json(conn, %{})
+  end
+
   def config_show(conn, _params) do
     configs = Pleroma.Repo.all(Config)
 

lib/pleroma/web/auth/authenticator.ex

@@ -21,8 +21,7 @@ def get_user(plug), do: implementation().get_user(plug)
   def create_from_registration(plug, registration),
     do: implementation().create_from_registration(plug, registration)
 
-  @callback get_registration(Plug.Conn.t()) ::
+  @callback get_registration(Plug.Conn.t()) :: {:ok, Registration.t()} | {:error, any()}
-              {:ok, Registration.t()} | {:error, any()}
   def get_registration(plug), do: implementation().get_registration(plug)
 
   @callback handle_error(Plug.Conn.t(), any()) :: any()

lib/pleroma/web/common_api/common_api.ex

@@ -300,8 +300,7 @@ def pin(id_or_ap_id, %{ap_id: user_ap_id} = user) do
            }
          } = activity <- get_by_id_or_ap_id(id_or_ap_id),
          true <- Visibility.is_public?(activity),
-         %{valid?: true} = info_changeset <-
+         %{valid?: true} = info_changeset <- User.Info.add_pinnned_activity(user.info, activity),
-           User.Info.add_pinnned_activity(user.info, activity),
          changeset <-
            Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset),
          {:ok, _user} <- User.update_and_set_cache(changeset) do

lib/pleroma/web/common_api/utils.ex

@@ -19,11 +19,17 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Pleroma.Web.MediaProxy
 
   require Logger
+  require Pleroma.Constants
 
   # This is a hack for twidere.
   def get_by_id_or_ap_id(id) do
     activity =
-      Activity.get_by_id_with_object(id) || Activity.get_create_by_object_ap_id_with_object(id)
+      with true <- Pleroma.FlakeId.is_flake_id?(id),
+           %Activity{} = activity <- Activity.get_by_id_with_object(id) do
+        activity
+      else
+        _ -> Activity.get_create_by_object_ap_id_with_object(id)
+      end
 
     activity &&
       if activity.data["type"] == "Create" do
@@ -66,7 +72,7 @@ def attachments_from_ids_descs(ids, descs_str) do
   @spec get_to_and_cc(User.t(), list(String.t()), Activity.t() | nil, String.t()) ::
           {list(String.t()), list(String.t())}
   def get_to_and_cc(user, mentioned_users, inReplyTo, "public") do
-    to = ["https://www.w3.org/ns/activitystreams#Public" | mentioned_users]
+    to = [Pleroma.Constants.as_public() | mentioned_users]
     cc = [user.follower_address]
 
     if inReplyTo do
@@ -78,7 +84,7 @@ def get_to_and_cc(user, mentioned_users, inReplyTo, "public") do
 
   def get_to_and_cc(user, mentioned_users, inReplyTo, "unlisted") do
     to = [user.follower_address | mentioned_users]
-    cc = ["https://www.w3.org/ns/activitystreams#Public"]
+    cc = [Pleroma.Constants.as_public()]
 
     if inReplyTo do
       {Enum.uniq([inReplyTo.data["actor"] | to]), cc}

lib/pleroma/web/fallback_redirect_controller.ex

@@ -0,0 +1,77 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Fallback.RedirectController do
+  use Pleroma.Web, :controller
+  require Logger
+  alias Pleroma.User
+  alias Pleroma.Web.Metadata
+
+  def api_not_implemented(conn, _params) do
+    conn
+    |> put_status(404)
+    |> json(%{error: "Not implemented"})
+  end
+
+  def redirector(conn, _params, code \\ 200)
+
+  # redirect to admin section
+  # /pleroma/admin -> /pleroma/admin/
+  #
+  def redirector(conn, %{"path" => ["pleroma", "admin"]} = _, _code) do
+    redirect(conn, to: "/pleroma/admin/")
+  end
+
+  def redirector(conn, _params, code) do
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_file(code, index_file_path())
+  end
+
+  def redirector_with_meta(conn, %{"maybe_nickname_or_id" => maybe_nickname_or_id} = params) do
+    with %User{} = user <- User.get_cached_by_nickname_or_id(maybe_nickname_or_id) do
+      redirector_with_meta(conn, %{user: user})
+    else
+      nil ->
+        redirector(conn, params)
+    end
+  end
+
+  def redirector_with_meta(conn, params) do
+    {:ok, index_content} = File.read(index_file_path())
+
+    tags =
+      try do
+        Metadata.build_tags(params)
+      rescue
+        e ->
+          Logger.error(
+            "Metadata rendering for #{conn.request_path} failed.\n" <>
+              Exception.format(:error, e, __STACKTRACE__)
+          )
+
+          ""
+      end
+
+    response = String.replace(index_content, "<!--server-generated-meta-->", tags)
+
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_resp(200, response)
+  end
+
+  def index_file_path do
+    Pleroma.Plugs.InstanceStatic.file_path("index.html")
+  end
+
+  def registration_page(conn, params) do
+    redirector(conn, params)
+  end
+
+  def empty(conn, _params) do
+    conn
+    |> put_status(204)
+    |> text("")
+  end
+end

lib/pleroma/web/mastodon_api/mastodon_api_controller.ex

@@ -4,6 +4,9 @@
 
 defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper, only: [json_response: 3]
+
   alias Ecto.Changeset
   alias Pleroma.Activity
   alias Pleroma.Bookmark
@@ -46,6 +49,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @rate_limited_relations_actions ~w(follow unfollow)a
 
@@ -74,6 +78,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   plug(RateLimiter, :app_account_creation when action == :account_register)
   plug(RateLimiter, :search when action in [:search, :search2, :account_search])
   plug(RateLimiter, :password_reset when action == :password_reset)
+  plug(RateLimiter, :account_confirmation_resend when action == :account_confirmation_resend)
 
   @local_mastodon_name "Mastodon-Local"
 
@@ -1220,10 +1225,9 @@ def user_favourites(%{assigns: %{user: for_user}} = conn, %{"id" => id} = params
 
       recipients =
         if for_user do
-          ["https://www.w3.org/ns/activitystreams#Public"] ++
+          [Pleroma.Constants.as_public()] ++ [for_user.ap_id | for_user.following]
-            [for_user.ap_id | for_user.following]
         else
-          ["https://www.w3.org/ns/activitystreams#Public"]
+          [Pleroma.Constants.as_public()]
         end
 
       activities =
@@ -1839,6 +1843,16 @@ def password_reset(conn, params) do
     end
   end
 
+  def account_confirmation_resend(conn, params) do
+    nickname_or_email = params["email"] || params["nickname"]
+
+    with %User{} = user <- User.get_by_nickname_or_email(nickname_or_email),
+         {:ok, _} <- User.try_send_confirmation_email(user) do
+      conn
+      |> json_response(:no_content, "")
+    end
+  end
+
   def try_render(conn, target, params)
       when is_binary(target) do
     case render(conn, target, params) do

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -50,13 +50,13 @@ def render("relationship.json", %{user: %User{} = user, target: %User{} = target
       id: to_string(target.id),
       following: User.following?(user, target),
       followed_by: User.following?(target, user),
-      blocking: User.blocks?(user, target),
+      blocking: User.blocks_ap_id?(user, target),
-      blocked_by: User.blocks?(target, user),
+      blocked_by: User.blocks_ap_id?(target, user),
       muting: User.mutes?(user, target),
       muting_notifications: User.muted_notifications?(user, target),
       subscribing: User.subscribed_to?(user, target),
       requested: requested,
-      domain_blocking: false,
+      domain_blocking: User.blocks_domain?(user, target),
       showing_reblogs: User.showing_reblogs?(user, target),
       endorsed: false
     }

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -222,7 +222,7 @@ def render("status.json", %{activity: %{data: %{"object" => _object}} = activity
       if user.local do
         Pleroma.Web.Router.Helpers.o_status_url(Pleroma.Web.Endpoint, :notice, activity)
       else
-        object.data["external_url"] || object.data["id"]
+        object.data["url"] || object.data["external_url"] || object.data["id"]
       end
 
     %{

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -165,6 +165,7 @@ def raw_nodeinfo do
         },
         accountActivationRequired: Config.get([:instance, :account_activation_required], false),
         invitesEnabled: Config.get([:instance, :invites_enabled], false),
+        mailerEnabled: Config.get([Pleroma.Emails.Mailer, :enabled], false),
         features: features,
         restrictedNicknames: Config.get([Pleroma.User, :restricted_nicknames]),
         skipThreadContainment: Config.get([:instance, :skip_thread_containment], false)

lib/pleroma/web/oauth/oauth_controller.ex

@@ -365,8 +365,7 @@ def registration_details(%Plug.Conn{} = conn, %{"authorization" => auth_attrs})
   def register(%Plug.Conn{} = conn, %{"authorization" => _, "op" => "connect"} = params) do
     with registration_id when not is_nil(registration_id) <- get_session_registration_id(conn),
          %Registration{} = registration <- Repo.get(Registration, registration_id),
-         {_, {:ok, auth}} <-
+         {_, {:ok, auth}} <- {:create_authorization, do_create_authorization(conn, params)},
-           {:create_authorization, do_create_authorization(conn, params)},
          %User{} = user <- Repo.preload(auth, :user).user,
          {:ok, _updated_registration} <- Registration.bind_to_user(registration, user) do
       conn

lib/pleroma/web/oauth/token.ex

@@ -44,8 +44,7 @@ def get_by_refresh_token(%App{id: app_id} = _app, token) do
     |> Repo.find_resource()
   end
 
-  @spec exchange_token(App.t(), Authorization.t()) ::
+  @spec exchange_token(App.t(), Authorization.t()) :: {:ok, Token.t()} | {:error, Changeset.t()}
-          {:ok, Token.t()} | {:error, Changeset.t()}
   def exchange_token(app, auth) do
     with {:ok, auth} <- Authorization.use_token(auth),
          true <- auth.app_id == app.id do

lib/pleroma/web/ostatus/activity_representer.ex

@@ -9,6 +9,7 @@ defmodule Pleroma.Web.OStatus.ActivityRepresenter do
   alias Pleroma.Web.OStatus.UserRepresenter
 
   require Logger
+  require Pleroma.Constants
 
   defp get_href(id) do
     with %Object{data: %{"external_url" => external_url}} <- Object.get_cached_by_ap_id(id) do
@@ -34,7 +35,7 @@ defp get_mentions(to) do
     Enum.map(to, fn id ->
       cond do
         # Special handling for the AP/Ostatus public collections
-        "https://www.w3.org/ns/activitystreams#Public" == id ->
+        Pleroma.Constants.as_public() == id ->
           {:link,
            [
              rel: "mentioned",

lib/pleroma/web/ostatus/handlers/follow_handler.ex

@@ -9,14 +9,18 @@ defmodule Pleroma.Web.OStatus.FollowHandler do
   alias Pleroma.Web.XML
 
   def handle(entry, doc) do
-    with {:ok, actor} <- OStatus.find_make_or_update_user(doc),
+    with {:ok, actor} <- OStatus.find_make_or_update_actor(doc),
          id when not is_nil(id) <- XML.string_from_xpath("/entry/id", entry),
          followed_uri when not is_nil(followed_uri) <-
            XML.string_from_xpath("/entry/activity:object/id", entry),
          {:ok, followed} <- OStatus.find_or_make_user(followed_uri),
+         {:locked, false} <- {:locked, followed.info.locked},
          {:ok, activity} <- ActivityPub.follow(actor, followed, id, false) do
       User.follow(actor, followed)
       {:ok, activity}
+    else
+      {:locked, true} ->
+        {:error, "It's not possible to follow locked accounts over OStatus"}
     end
   end
 end

lib/pleroma/web/ostatus/handlers/note_handler.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Web.OStatus.NoteHandler do
   require Logger
+  require Pleroma.Constants
 
   alias Pleroma.Activity
   alias Pleroma.Object
@@ -49,7 +50,7 @@ def get_people_mentions(entry) do
   def get_collection_mentions(entry) do
     transmogrify = fn
       "http://activityschema.org/collection/public" ->
-        "https://www.w3.org/ns/activitystreams#Public"
+        Pleroma.Constants.as_public()
 
       group ->
         group
@@ -110,7 +111,7 @@ def handle_note(entry, doc \\ nil, options \\ []) do
     with id <- XML.string_from_xpath("//id", entry),
          activity when is_nil(activity) <- Activity.get_create_by_object_ap_id_with_object(id),
          [author] <- :xmerl_xpath.string('//author[1]', doc),
-         {:ok, actor} <- OStatus.find_make_or_update_user(author),
+         {:ok, actor} <- OStatus.find_make_or_update_actor(author),
          content_html <- OStatus.get_content(entry),
          cw <- OStatus.get_cw(entry),
          in_reply_to <- XML.string_from_xpath("//thr:in-reply-to[1]/@ref", entry),
@@ -126,7 +127,7 @@ def handle_note(entry, doc \\ nil, options \\ []) do
          to <- make_to_list(actor, mentions),
          date <- XML.string_from_xpath("//published", entry),
          unlisted <- XML.string_from_xpath("//mastodon:scope", entry) == "unlisted",
-         cc <- if(unlisted, do: ["https://www.w3.org/ns/activitystreams#Public"], else: []),
+         cc <- if(unlisted, do: [Pleroma.Constants.as_public()], else: []),
          note <-
            CommonAPI.Utils.make_note_data(
              actor.ap_id,

lib/pleroma/web/ostatus/handlers/unfollow_handler.ex

@@ -9,7 +9,7 @@ defmodule Pleroma.Web.OStatus.UnfollowHandler do
   alias Pleroma.Web.XML
 
   def handle(entry, doc) do
-    with {:ok, actor} <- OStatus.find_make_or_update_user(doc),
+    with {:ok, actor} <- OStatus.find_make_or_update_actor(doc),
          id when not is_nil(id) <- XML.string_from_xpath("/entry/id", entry),
          followed_uri when not is_nil(followed_uri) <-
            XML.string_from_xpath("/entry/activity:object/id", entry),

lib/pleroma/web/ostatus/ostatus.ex

@@ -56,7 +56,7 @@ def remote_follow_path do
 
   def handle_incoming(xml_string, options \\ []) do
     with doc when doc != :error <- parse_document(xml_string) do
-      with {:ok, actor_user} <- find_make_or_update_user(doc),
+      with {:ok, actor_user} <- find_make_or_update_actor(doc),
            do: Pleroma.Instances.set_reachable(actor_user.ap_id)
 
       entries = :xmerl_xpath.string('//entry', doc)
@@ -120,7 +120,7 @@ def handle_incoming(xml_string, options \\ []) do
   end
 
   def make_share(entry, doc, retweeted_activity) do
-    with {:ok, actor} <- find_make_or_update_user(doc),
+    with {:ok, actor} <- find_make_or_update_actor(doc),
          %Object{} = object <- Object.normalize(retweeted_activity),
          id when not is_nil(id) <- string_from_xpath("/entry/id", entry),
          {:ok, activity, _object} = ActivityPub.announce(actor, object, id, false) do
@@ -138,7 +138,7 @@ def handle_share(entry, doc) do
   end
 
   def make_favorite(entry, doc, favorited_activity) do
-    with {:ok, actor} <- find_make_or_update_user(doc),
+    with {:ok, actor} <- find_make_or_update_actor(doc),
          %Object{} = object <- Object.normalize(favorited_activity),
          id when not is_nil(id) <- string_from_xpath("/entry/id", entry),
          {:ok, activity, _object} = ActivityPub.like(actor, object, id, false) do
@@ -264,11 +264,18 @@ def maybe_update_ostatus(doc, user) do
     end
   end
 
-  def find_make_or_update_user(doc) do
+  def find_make_or_update_actor(doc) do
     uri = string_from_xpath("//author/uri[1]", doc)
 
-    with {:ok, user} <- find_or_make_user(uri) do
+    with {:ok, %User{} = user} <- find_or_make_user(uri),
+         {:ap_enabled, false} <- {:ap_enabled, User.ap_enabled?(user)} do
       maybe_update(doc, user)
+    else
+      {:ap_enabled, true} ->
+        {:error, :invalid_protocol}
+
+      _ ->
+        {:error, :unknown_user}
     end
   end
 

lib/pleroma/web/ostatus/ostatus_controller.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Web.OStatus.OStatusController do
   use Pleroma.Web, :controller
 
+  alias Fallback.RedirectController
   alias Pleroma.Activity
   alias Pleroma.Object
   alias Pleroma.User
@@ -12,42 +13,44 @@ defmodule Pleroma.Web.OStatus.OStatusController do
   alias Pleroma.Web.ActivityPub.ActivityPubController
   alias Pleroma.Web.ActivityPub.ObjectView
   alias Pleroma.Web.ActivityPub.Visibility
+  alias Pleroma.Web.Endpoint
   alias Pleroma.Web.Federator
+  alias Pleroma.Web.Metadata.PlayerView
   alias Pleroma.Web.OStatus
   alias Pleroma.Web.OStatus.ActivityRepresenter
   alias Pleroma.Web.OStatus.FeedRepresenter
+  alias Pleroma.Web.Router
   alias Pleroma.Web.XML
 
   plug(Pleroma.Web.FederatingPlug when action in [:salmon_incoming])
 
+  plug(
+    Pleroma.Plugs.SetFormatPlug
+    when action in [:feed_redirect, :object, :activity, :notice]
+  )
+
   action_fallback(:errors)
 
+  def feed_redirect(%{assigns: %{format: "html"}} = conn, %{"nickname" => nickname}) do
+    with {_, %User{} = user} <-
+           {:fetch_user, User.get_cached_by_nickname_or_id(nickname)} do
+      RedirectController.redirector_with_meta(conn, %{user: user})
+    end
+  end
+
+  def feed_redirect(%{assigns: %{format: format}} = conn, _params)
+      when format in ["json", "activity+json"] do
+    ActivityPubController.call(conn, :user)
+  end
+
   def feed_redirect(conn, %{"nickname" => nickname}) do
-    case get_format(conn) do
+    with {_, %User{} = user} <- {:fetch_user, User.get_cached_by_nickname(nickname)} do
-      "html" ->
+      redirect(conn, external: OStatus.feed_path(user))
-        with %User{} = user <- User.get_cached_by_nickname_or_id(nickname) do
-          Fallback.RedirectController.redirector_with_meta(conn, %{user: user})
-        else
-          nil -> {:error, :not_found}
-        end
-
-      "activity+json" ->
-        ActivityPubController.call(conn, :user)
-
-      "json" ->
-        ActivityPubController.call(conn, :user)
-
-      _ ->
-        with %User{} = user <- User.get_cached_by_nickname(nickname) do
-          redirect(conn, external: OStatus.feed_path(user))
-        else
-          nil -> {:error, :not_found}
-        end
     end
   end
 
   def feed(conn, %{"nickname" => nickname} = params) do
-    with %User{} = user <- User.get_cached_by_nickname(nickname) do
+    with {_, %User{} = user} <- {:fetch_user, User.get_cached_by_nickname(nickname)} do
       query_params =
         Map.take(params, ["max_id"])
         |> Map.merge(%{"whole_db" => true, "actor_id" => user.ap_id})
@@ -65,8 +68,6 @@ def feed(conn, %{"nickname" => nickname} = params) do
       conn
       |> put_resp_content_type("application/atom+xml")
       |> send_resp(200, response)
-    else
-      nil -> {:error, :not_found}
     end
   end
 
@@ -97,93 +98,82 @@ def salmon_incoming(conn, _) do
     |> send_resp(200, "")
   end
 
-  def object(conn, %{"uuid" => uuid}) do
+  def object(%{assigns: %{format: format}} = conn, %{"uuid" => _uuid})
-    if get_format(conn) in ["activity+json", "json"] do
+      when format in ["json", "activity+json"] do
-      ActivityPubController.call(conn, :object)
+    ActivityPubController.call(conn, :object)
-    else
+  end
-      with id <- o_status_url(conn, :object, uuid),
-           {_, %Activity{} = activity} <-
-             {:activity, Activity.get_create_by_object_ap_id_with_object(id)},
-           {_, true} <- {:public?, Visibility.is_public?(activity)},
-           %User{} = user <- User.get_cached_by_ap_id(activity.data["actor"]) do
-        case get_format(conn) do
-          "html" -> redirect(conn, to: "/notice/#{activity.id}")
-          _ -> represent_activity(conn, nil, activity, user)
-        end
-      else
-        {:public?, false} ->
-          {:error, :not_found}
 
-        {:activity, nil} ->
+  def object(%{assigns: %{format: format}} = conn, %{"uuid" => uuid}) do
-          {:error, :not_found}
+    with id <- o_status_url(conn, :object, uuid),
-
+         {_, %Activity{} = activity} <-
-        e ->
+           {:activity, Activity.get_create_by_object_ap_id_with_object(id)},
-          e
+         {_, true} <- {:public?, Visibility.is_public?(activity)},
+         %User{} = user <- User.get_cached_by_ap_id(activity.data["actor"]) do
+      case format do
+        "html" -> redirect(conn, to: "/notice/#{activity.id}")
+        _ -> represent_activity(conn, nil, activity, user)
       end
+    else
+      reason when reason in [{:public?, false}, {:activity, nil}] ->
+        {:error, :not_found}
+
+      e ->
+        e
     end
   end
 
-  def activity(conn, %{"uuid" => uuid}) do
+  def activity(%{assigns: %{format: format}} = conn, %{"uuid" => _uuid})
-    if get_format(conn) in ["activity+json", "json"] do
+      when format in ["json", "activity+json"] do
-      ActivityPubController.call(conn, :activity)
+    ActivityPubController.call(conn, :activity)
-    else
+  end
-      with id <- o_status_url(conn, :activity, uuid),
-           {_, %Activity{} = activity} <- {:activity, Activity.normalize(id)},
-           {_, true} <- {:public?, Visibility.is_public?(activity)},
-           %User{} = user <- User.get_cached_by_ap_id(activity.data["actor"]) do
-        case format = get_format(conn) do
-          "html" -> redirect(conn, to: "/notice/#{activity.id}")
-          _ -> represent_activity(conn, format, activity, user)
-        end
-      else
-        {:public?, false} ->
-          {:error, :not_found}
 
-        {:activity, nil} ->
+  def activity(%{assigns: %{format: format}} = conn, %{"uuid" => uuid}) do
-          {:error, :not_found}
+    with id <- o_status_url(conn, :activity, uuid),
-
+         {_, %Activity{} = activity} <- {:activity, Activity.normalize(id)},
-        e ->
+         {_, true} <- {:public?, Visibility.is_public?(activity)},
-          e
+         %User{} = user <- User.get_cached_by_ap_id(activity.data["actor"]) do
+      case format do
+        "html" -> redirect(conn, to: "/notice/#{activity.id}")
+        _ -> represent_activity(conn, format, activity, user)
       end
+    else
+      reason when reason in [{:public?, false}, {:activity, nil}] ->
+        {:error, :not_found}
+
+      e ->
+        e
     end
   end
 
-  def notice(conn, %{"id" => id}) do
+  def notice(%{assigns: %{format: format}} = conn, %{"id" => id}) do
     with {_, %Activity{} = activity} <- {:activity, Activity.get_by_id_with_object(id)},
          {_, true} <- {:public?, Visibility.is_public?(activity)},
          %User{} = user <- User.get_cached_by_ap_id(activity.data["actor"]) do
-      case format = get_format(conn) do
+      cond do
-        "html" ->
+        format == "html" && activity.data["type"] == "Create" ->
-          if activity.data["type"] == "Create" do
+          %Object{} = object = Object.normalize(activity)
-            %Object{} = object = Object.normalize(activity)
 
-            Fallback.RedirectController.redirector_with_meta(conn, %{
+          RedirectController.redirector_with_meta(
+            conn,
+            %{
               activity_id: activity.id,
               object: object,
-              url:
+              url: Router.Helpers.o_status_url(Endpoint, :notice, activity.id),
-                Pleroma.Web.Router.Helpers.o_status_url(
-                  Pleroma.Web.Endpoint,
-                  :notice,
-                  activity.id
-                ),
               user: user
-            })
+            }
-          else
+          )
-            Fallback.RedirectController.redirector(conn, nil)
-          end
 
-        _ ->
+        format == "html" ->
+          RedirectController.redirector(conn, nil)
+
+        true ->
           represent_activity(conn, format, activity, user)
       end
     else
-      {:public?, false} ->
+      reason when reason in [{:public?, false}, {:activity, nil}] ->
         conn
         |> put_status(404)
-        |> Fallback.RedirectController.redirector(nil, 404)
+        |> RedirectController.redirector(nil, 404)
-
-      {:activity, nil} ->
-        conn
-        |> Fallback.RedirectController.redirector(nil, 404)
 
       e ->
         e
@@ -204,13 +194,13 @@ def notice_player(conn, %{"id" => id}) do
         "content-security-policy",
         "default-src 'none';style-src 'self' 'unsafe-inline';img-src 'self' data: https:; media-src 'self' https:;"
       )
-      |> put_view(Pleroma.Web.Metadata.PlayerView)
+      |> put_view(PlayerView)
       |> render("player.html", url)
     else
       _error ->
         conn
         |> put_status(404)
-        |> Fallback.RedirectController.redirector(nil, 404)
+        |> RedirectController.redirector(nil, 404)
     end
   end
 
@@ -248,6 +238,8 @@ def errors(conn, {:error, :not_found}) do
     render_error(conn, :not_found, "Not found")
   end
 
+  def errors(conn, {:fetch_user, nil}), do: errors(conn, {:error, :not_found})
+
   def errors(conn, _) do
     render_error(conn, :internal_server_error, "Something went wrong")
   end

lib/pleroma/web/rich_media/parsers/ttl/aws_signed_url.ex

@@ -19,8 +19,7 @@ defp is_aws_signed_url(nil), do: nil
   defp is_aws_signed_url(image) when is_binary(image) do
     %URI{host: host, query: query} = URI.parse(image)
 
-    if String.contains?(host, "amazonaws.com") and
+    if String.contains?(host, "amazonaws.com") and String.contains?(query, "X-Amz-Expires") do
-         String.contains?(query, "X-Amz-Expires") do
       image
     else
       nil

lib/pleroma/web/router.ex

@@ -196,6 +196,8 @@ defmodule Pleroma.Web.Router do
 
     get("/config", AdminAPIController, :config_show)
     post("/config", AdminAPIController, :config_update)
+    get("/config/migrate_to_db", AdminAPIController, :migrate_to_db)
+    get("/config/migrate_from_db", AdminAPIController, :migrate_from_db)
   end
 
   scope "/", Pleroma.Web.TwitterAPI do
@@ -412,6 +414,12 @@ defmodule Pleroma.Web.Router do
 
     get("/accounts/search", SearchController, :account_search)
 
+    post(
+      "/pleroma/accounts/confirmation_resend",
+      MastodonAPIController,
+      :account_confirmation_resend
+    )
+
     scope [] do
       pipe_through(:oauth_read_or_public)
 
@@ -694,7 +702,7 @@ defmodule Pleroma.Web.Router do
     post("/auth/password", MastodonAPIController, :password_reset)
 
     scope [] do
-      pipe_through(:oauth_read_or_public)
+      pipe_through(:oauth_read)
       get("/web/*path", MastodonAPIController, :index)
     end
   end
@@ -731,68 +739,3 @@ defmodule Pleroma.Web.Router do
     options("/*path", RedirectController, :empty)
   end
 end
-
-defmodule Fallback.RedirectController do
-  use Pleroma.Web, :controller
-  require Logger
-  alias Pleroma.User
-  alias Pleroma.Web.Metadata
-
-  def api_not_implemented(conn, _params) do
-    conn
-    |> put_status(404)
-    |> json(%{error: "Not implemented"})
-  end
-
-  def redirector(conn, _params, code \\ 200) do
-    conn
-    |> put_resp_content_type("text/html")
-    |> send_file(code, index_file_path())
-  end
-
-  def redirector_with_meta(conn, %{"maybe_nickname_or_id" => maybe_nickname_or_id} = params) do
-    with %User{} = user <- User.get_cached_by_nickname_or_id(maybe_nickname_or_id) do
-      redirector_with_meta(conn, %{user: user})
-    else
-      nil ->
-        redirector(conn, params)
-    end
-  end
-
-  def redirector_with_meta(conn, params) do
-    {:ok, index_content} = File.read(index_file_path())
-
-    tags =
-      try do
-        Metadata.build_tags(params)
-      rescue
-        e ->
-          Logger.error(
-            "Metadata rendering for #{conn.request_path} failed.\n" <>
-              Exception.format(:error, e, __STACKTRACE__)
-          )
-
-          ""
-      end
-
-    response = String.replace(index_content, "<!--server-generated-meta-->", tags)
-
-    conn
-    |> put_resp_content_type("text/html")
-    |> send_resp(200, response)
-  end
-
-  def index_file_path do
-    Pleroma.Plugs.InstanceStatic.file_path("index.html")
-  end
-
-  def registration_page(conn, params) do
-    redirector(conn, params)
-  end
-
-  def empty(conn, _params) do
-    conn
-    |> put_status(204)
-    |> text("")
-  end
-end

lib/pleroma/web/twitter_api/controllers/util_controller.ex

@@ -15,11 +15,11 @@ defmodule Pleroma.Web.TwitterAPI.UtilController do
   alias Pleroma.Plugs.AuthenticationPlug
   alias Pleroma.User
   alias Pleroma.Web
-  alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.CommonAPI
-  alias Pleroma.Web.OStatus
   alias Pleroma.Web.WebFinger
 
+  plug(Pleroma.Plugs.SetFormatPlug when action in [:config, :version])
+
   def help_test(conn, _params) do
     json(conn, "ok")
   end
@@ -60,27 +60,25 @@ def remote_follow(%{assigns: %{user: user}} = conn, %{"acct" => acct}) do
       %Activity{id: activity_id} = Activity.get_create_by_object_ap_id(object.data["id"])
       redirect(conn, to: "/notice/#{activity_id}")
     else
-      {err, followee} = OStatus.find_or_make_user(acct)
+      with {:ok, followee} <- User.get_or_fetch(acct) do
-      avatar = User.avatar_url(followee)
-      name = followee.nickname
-      id = followee.id
-
-      if !!user do
         conn
-        |> render("follow.html", %{error: err, acct: acct, avatar: avatar, name: name, id: id})
+        |> render(follow_template(user), %{
-      else
-        conn
-        |> render("follow_login.html", %{
           error: false,
           acct: acct,
-          avatar: avatar,
+          avatar: User.avatar_url(followee),
-          name: name,
+          name: followee.nickname,
-          id: id
+          id: followee.id
         })
+      else
+        {:error, _reason} ->
+          render(conn, follow_template(user), %{error: :error})
       end
     end
   end
 
+  defp follow_template(%User{} = _user), do: "follow.html"
+  defp follow_template(_), do: "follow_login.html"
+
   defp is_status?(acct) do
     case Pleroma.Object.Fetcher.fetch_and_contain_remote_object_from_id(acct) do
       {:ok, %{"type" => type}} when type in ["Article", "Note", "Video", "Page", "Question"] ->
@@ -94,50 +92,53 @@ defp is_status?(acct) do
   def do_remote_follow(conn, %{
         "authorization" => %{"name" => username, "password" => password, "id" => id}
       }) do
-    followee = User.get_cached_by_id(id)
+    with %User{} = followee <- User.get_cached_by_id(id),
-    avatar = User.avatar_url(followee)
+         {_, %User{} = user, _} <- {:auth, User.get_cached_by_nickname(username), followee},
-    name = followee.nickname
+         {_, true, _} <- {
-
+           :auth,
-    with %User{} = user <- User.get_cached_by_nickname(username),
+           AuthenticationPlug.checkpw(password, user.password_hash),
-         true <- AuthenticationPlug.checkpw(password, user.password_hash),
+           followee
-         %User{} = _followed <- User.get_cached_by_id(id),
+         },
-         {:ok, follower} <- User.follow(user, followee),
+         {:ok, _follower, _followee, _activity} <- CommonAPI.follow(user, followee) do
-         {:ok, _activity} <- ActivityPub.follow(follower, followee) do
       conn
       |> render("followed.html", %{error: false})
     else
       # Was already following user
       {:error, "Could not follow user:" <> _rest} ->
-        render(conn, "followed.html", %{error: false})
+        render(conn, "followed.html", %{error: "Error following account"})
 
-      _e ->
+      {:auth, _, followee} ->
         conn
         |> render("follow_login.html", %{
           error: "Wrong username or password",
           id: id,
-          name: name,
+          name: followee.nickname,
-          avatar: avatar
+          avatar: User.avatar_url(followee)
         })
+
+      e ->
+        Logger.debug("Remote follow failed with error #{inspect(e)}")
+        render(conn, "followed.html", %{error: "Something went wrong."})
     end
   end
 
   def do_remote_follow(%{assigns: %{user: user}} = conn, %{"user" => %{"id" => id}}) do
-    with %User{} = followee <- User.get_cached_by_id(id),
+    with {:fetch_user, %User{} = followee} <- {:fetch_user, User.get_cached_by_id(id)},
-         {:ok, follower} <- User.follow(user, followee),
+         {:ok, _follower, _followee, _activity} <- CommonAPI.follow(user, followee) do
-         {:ok, _activity} <- ActivityPub.follow(follower, followee) do
       conn
       |> render("followed.html", %{error: false})
     else
       # Was already following user
       {:error, "Could not follow user:" <> _rest} ->
-        conn
+        render(conn, "followed.html", %{error: "Error following account"})
-        |> render("followed.html", %{error: false})
+
+      {:fetch_user, error} ->
+        Logger.debug("Remote follow failed with error #{inspect(error)}")
+        render(conn, "followed.html", %{error: "Could not find user"})
 
       e ->
         Logger.debug("Remote follow failed with error #{inspect(e)}")
-
+        render(conn, "followed.html", %{error: "Something went wrong."})
-        conn
-        |> render("followed.html", %{error: inspect(e)})
     end
   end
 
@@ -152,67 +153,70 @@ def notifications_read(%{assigns: %{user: user}} = conn, %{"id" => notification_
     end
   end
 
+  def config(%{assigns: %{format: "xml"}} = conn, _params) do
+    instance = Pleroma.Config.get(:instance)
+
+    response = """
+    <config>
+    <site>
+    <name>#{Keyword.get(instance, :name)}</name>
+    <site>#{Web.base_url()}</site>
+    <textlimit>#{Keyword.get(instance, :limit)}</textlimit>
+    <closed>#{!Keyword.get(instance, :registrations_open)}</closed>
+    </site>
+    </config>
+    """
+
+    conn
+    |> put_resp_content_type("application/xml")
+    |> send_resp(200, response)
+  end
+
   def config(conn, _params) do
     instance = Pleroma.Config.get(:instance)
 
-    case get_format(conn) do
+    vapid_public_key = Keyword.get(Pleroma.Web.Push.vapid_config(), :public_key)
-      "xml" ->
-        response = """
-        <config>
-          <site>
-            <name>#{Keyword.get(instance, :name)}</name>
-            <site>#{Web.base_url()}</site>
-            <textlimit>#{Keyword.get(instance, :limit)}</textlimit>
-            <closed>#{!Keyword.get(instance, :registrations_open)}</closed>
-          </site>
-        </config>
-        """
 
-        conn
+    uploadlimit = %{
-        |> put_resp_content_type("application/xml")
+      uploadlimit: to_string(Keyword.get(instance, :upload_limit)),
-        |> send_resp(200, response)
+      avatarlimit: to_string(Keyword.get(instance, :avatar_upload_limit)),
+      backgroundlimit: to_string(Keyword.get(instance, :background_upload_limit)),
+      bannerlimit: to_string(Keyword.get(instance, :banner_upload_limit))
+    }
 
-      _ ->
+    data = %{
-        vapid_public_key = Keyword.get(Pleroma.Web.Push.vapid_config(), :public_key)
+      name: Keyword.get(instance, :name),
+      description: Keyword.get(instance, :description),
+      server: Web.base_url(),
+      textlimit: to_string(Keyword.get(instance, :limit)),
+      uploadlimit: uploadlimit,
+      closed: bool_to_val(Keyword.get(instance, :registrations_open), "0", "1"),
+      private: bool_to_val(Keyword.get(instance, :public, true), "0", "1"),
+      vapidPublicKey: vapid_public_key,
+      accountActivationRequired:
+        bool_to_val(Keyword.get(instance, :account_activation_required, false)),
+      invitesEnabled: bool_to_val(Keyword.get(instance, :invites_enabled, false)),
+      safeDMMentionsEnabled: bool_to_val(Pleroma.Config.get([:instance, :safe_dm_mentions]))
+    }
 
-        uploadlimit = %{
+    managed_config = Keyword.get(instance, :managed_config)
-          uploadlimit: to_string(Keyword.get(instance, :upload_limit)),
-          avatarlimit: to_string(Keyword.get(instance, :avatar_upload_limit)),
-          backgroundlimit: to_string(Keyword.get(instance, :background_upload_limit)),
-          bannerlimit: to_string(Keyword.get(instance, :banner_upload_limit))
-        }
-
-        data = %{
-          name: Keyword.get(instance, :name),
-          description: Keyword.get(instance, :description),
-          server: Web.base_url(),
-          textlimit: to_string(Keyword.get(instance, :limit)),
-          uploadlimit: uploadlimit,
-          closed: if(Keyword.get(instance, :registrations_open), do: "0", else: "1"),
-          private: if(Keyword.get(instance, :public, true), do: "0", else: "1"),
-          vapidPublicKey: vapid_public_key,
-          accountActivationRequired:
-            if(Keyword.get(instance, :account_activation_required, false), do: "1", else: "0"),
-          invitesEnabled: if(Keyword.get(instance, :invites_enabled, false), do: "1", else: "0"),
-          safeDMMentionsEnabled:
-            if(Pleroma.Config.get([:instance, :safe_dm_mentions]), do: "1", else: "0")
-        }
 
+    data =
+      if managed_config do
         pleroma_fe = Pleroma.Config.get([:frontend_configurations, :pleroma_fe])
+        Map.put(data, "pleromafe", pleroma_fe)
+      else
+        data
+      end
 
-        managed_config = Keyword.get(instance, :managed_config)
+    json(conn, %{site: data})
-
-        data =
-          if managed_config do
-            data |> Map.put("pleromafe", pleroma_fe)
-          else
-            data
-          end
-
-        json(conn, %{site: data})
-    end
   end
 
+  defp bool_to_val(true), do: "1"
+  defp bool_to_val(_), do: "0"
+  defp bool_to_val(true, val, _), do: val
+  defp bool_to_val(_, _, val), do: val
+
   def frontend_configurations(conn, _params) do
     config =
       Pleroma.Config.get(:frontend_configurations, %{})
@@ -221,20 +225,16 @@ def frontend_configurations(conn, _params) do
     json(conn, config)
   end
 
-  def version(conn, _params) do
+  def version(%{assigns: %{format: "xml"}} = conn, _params) do
     version = Pleroma.Application.named_version()
 
-    case get_format(conn) do
+    conn
-      "xml" ->
+    |> put_resp_content_type("application/xml")
-        response = "<version>#{version}</version>"
+    |> send_resp(200, "<version>#{version}</version>")
+  end
 
-        conn
+  def version(conn, _params) do
-        |> put_resp_content_type("application/xml")
+    json(conn, Pleroma.Application.named_version())
-        |> send_resp(200, response)
-
-      _ ->
-        json(conn, version)
-    end
   end
 
   def emoji(conn, _params) do

lib/pleroma/web/twitter_api/twitter_api.ex

@@ -15,6 +15,8 @@ defmodule Pleroma.Web.TwitterAPI.TwitterAPI do
 
   import Ecto.Query
 
+  require Pleroma.Constants
+
   def create_status(%User{} = user, %{"status" => _} = data) do
     CommonAPI.post(user, data)
   end
@@ -286,7 +288,7 @@ def search(_user, %{"q" => query} = params) do
       from(
         [a, o] in Activity.with_preloaded_object(Activity),
         where: fragment("?->>'type' = 'Create'", a.data),
-        where: "https://www.w3.org/ns/activitystreams#Public" in a.recipients,
+        where: ^Pleroma.Constants.as_public() in a.recipients,
         where:
           fragment(
             "to_tsvector('english', ?->>'content') @@ plainto_tsquery('english', ?)",

lib/pleroma/web/twitter_api/views/activity_view.ex

@@ -19,6 +19,7 @@ defmodule Pleroma.Web.TwitterAPI.ActivityView do
 
   import Ecto.Query
   require Logger
+  require Pleroma.Constants
 
   defp query_context_ids([]), do: []
 
@@ -91,7 +92,7 @@ defp get_user(ap_id, opts) do
       String.ends_with?(ap_id, "/followers") ->
         nil
 
-      ap_id == "https://www.w3.org/ns/activitystreams#Public" ->
+      ap_id == Pleroma.Constants.as_public() ->
         nil
 
       user = User.get_cached_by_ap_id(ap_id) ->

lib/pleroma/web/twitter_api/views/notification_view.ex

@@ -10,6 +10,8 @@ defmodule Pleroma.Web.TwitterAPI.NotificationView do
   alias Pleroma.Web.TwitterAPI.ActivityView
   alias Pleroma.Web.TwitterAPI.UserView
 
+  require Pleroma.Constants
+
   defp get_user(ap_id, opts) do
     cond do
       user = opts[:users][ap_id] ->
@@ -18,7 +20,7 @@ defp get_user(ap_id, opts) do
       String.ends_with?(ap_id, "/followers") ->
         nil
 
-      ap_id == "https://www.w3.org/ns/activitystreams#Public" ->
+      ap_id == Pleroma.Constants.as_public() ->
         nil
 
       true ->

lib/pleroma/web/web_finger/web_finger.ex

@@ -86,11 +86,17 @@ def represent_user(user, "XML") do
     |> XmlBuilder.to_doc()
   end
 
-  defp get_magic_key(magic_key) do
+  defp get_magic_key("data:application/magic-public-key," <> magic_key) do
-    "data:application/magic-public-key," <> magic_key = magic_key
     {:ok, magic_key}
-  rescue
+  end
-    MatchError -> {:error, "Missing magic key data."}
+
+  defp get_magic_key(nil) do
+    Logger.debug("Undefined magic key.")
+    {:ok, nil}
+  end
+
+  defp get_magic_key(_) do
+    {:error, "Missing magic key data."}
   end
 
   defp webfinger_from_xml(doc) do
@@ -187,6 +193,7 @@ def find_lrdd_template(domain) do
     end
   end
 
+  @spec finger(String.t()) :: {:ok, map()} | {:error, any()}
   def finger(account) do
     account = String.trim_leading(account, "@")
 
@@ -220,8 +227,6 @@ def finger(account) do
       else
         with {:ok, doc} <- Jason.decode(body) do
           webfinger_from_json(doc)
-        else
-          {:error, e} -> e
         end
       end
     else

lib/pleroma/web/web_finger/web_finger_controller.ex

@@ -7,6 +7,7 @@ defmodule Pleroma.Web.WebFinger.WebFingerController do
 
   alias Pleroma.Web.WebFinger
 
+  plug(Pleroma.Plugs.SetFormatPlug)
   plug(Pleroma.Web.FederatingPlug)
 
   def host_meta(conn, _params) do
@@ -17,30 +18,28 @@ def host_meta(conn, _params) do
     |> send_resp(200, xml)
   end
 
-  def webfinger(conn, %{"resource" => resource}) do
+  def webfinger(%{assigns: %{format: format}} = conn, %{"resource" => resource})
-    case get_format(conn) do
+      when format in ["xml", "xrd+xml"] do
-      n when n in ["xml", "xrd+xml"] ->
+    with {:ok, response} <- WebFinger.webfinger(resource, "XML") do
-        with {:ok, response} <- WebFinger.webfinger(resource, "XML") do
+      conn
-          conn
+      |> put_resp_content_type("application/xrd+xml")
-          |> put_resp_content_type("application/xrd+xml")
+      |> send_resp(200, response)
-          |> send_resp(200, response)
+    else
-        else
+      _e -> send_resp(conn, 404, "Couldn't find user")
-          _e -> send_resp(conn, 404, "Couldn't find user")
-        end
-
-      n when n in ["json", "jrd+json"] ->
-        with {:ok, response} <- WebFinger.webfinger(resource, "JSON") do
-          json(conn, response)
-        else
-          _e -> send_resp(conn, 404, "Couldn't find user")
-        end
-
-      _ ->
-        send_resp(conn, 404, "Unsupported format")
     end
   end
 
-  def webfinger(conn, _params) do
+  def webfinger(%{assigns: %{format: format}} = conn, %{"resource" => resource})
-    send_resp(conn, 400, "Bad Request")
+      when format in ["json", "jrd+json"] do
+    with {:ok, response} <- WebFinger.webfinger(resource, "JSON") do
+      json(conn, response)
+    else
+      _e ->
+        conn
+        |> put_status(404)
+        |> json("Couldn't find user")
+    end
   end
+
+  def webfinger(conn, _params), do: send_resp(conn, 400, "Bad Request")
 end

mix.exs

@@ -152,6 +152,7 @@ defp deps do
       {:benchee, "~> 1.0"},
       {:esshd, "~> 0.1.0", runtime: Application.get_env(:esshd, :enabled, false)},
       {:ex_rated, "~> 1.3"},
+      {:ex_const, "~> 0.2"},
       {:plug_static_index_html, "~> 1.0.0"},
       {:excoveralls, "~> 0.11.1", only: :test},
       {:mox, "~> 0.5", only: :test}
@@ -191,12 +192,13 @@ defp version(version) do
            tag = String.trim(tag),
            {describe, 0} <- System.cmd("git", ["describe", "--tags", "--abbrev=8"]),
            describe = String.trim(describe),
-           ahead <- String.replace(describe, tag, "") do
+           ahead <- String.replace(describe, tag, ""),
+           ahead <- String.trim_leading(ahead, "-") do
         {String.replace_prefix(tag, "v", ""), if(ahead != "", do: String.trim(ahead))}
       else
         _ ->
           {commit_hash, 0} = System.cmd("git", ["rev-parse", "--short", "HEAD"])
-          {nil, "-0-g" <> String.trim(commit_hash)}
+          {nil, "0-g" <> String.trim(commit_hash)}
       end
 
     if git_tag && version != git_tag do
@@ -208,14 +210,15 @@ defp version(version) do
     # Branch name as pre-release version component, denoted with a dot
     branch_name =
       with {branch_name, 0} <- System.cmd("git", ["rev-parse", "--abbrev-ref", "HEAD"]),
+           branch_name <- String.trim(branch_name),
            branch_name <- System.get_env("PLEROMA_BUILD_BRANCH") || branch_name,
-           true <- branch_name != "master" do
+           true <- branch_name not in ["master", "HEAD"] do
         branch_name =
           branch_name
           |> String.trim()
           |> String.replace(identifier_filter, "-")
 
-        "." <> branch_name
+        branch_name
       end
 
     build_name =
@@ -235,6 +238,17 @@ defp version(version) do
         env_override -> env_override
       end
 
+    # Pre-release version, denoted by appending a hyphen
+    # and a series of dot separated identifiers
+    pre_release =
+      [git_pre_release, branch_name]
+      |> Enum.filter(fn string -> string && string != "" end)
+      |> Enum.join(".")
+      |> (fn
+            "" -> nil
+            string -> "-" <> String.replace(string, identifier_filter, "-")
+          end).()
+
     # Build metadata, denoted with a plus sign
     build_metadata =
       [build_name, env_name]
@@ -245,7 +259,7 @@ defp version(version) do
             string -> "+" <> String.replace(string, identifier_filter, "-")
           end).()
 
-    [version, git_pre_release, branch_name, build_metadata]
+    [version, pre_release, build_metadata]
     |> Enum.filter(fn string -> string && string != "" end)
     |> Enum.join()
   end

mix.lock

@@ -28,6 +28,7 @@
   "ex2ms": {:hex, :ex2ms, "1.5.0", "19e27f9212be9a96093fed8cdfbef0a2b56c21237196d26760f11dfcfae58e97", [:mix], [], "hexpm"},
   "ex_aws": {:hex, :ex_aws, "2.1.0", "b92651527d6c09c479f9013caa9c7331f19cba38a650590d82ebf2c6c16a1d8a", [:mix], [{:configparser_ex, "~> 2.0", [hex: :configparser_ex, repo: "hexpm", optional: true]}, {:hackney, "1.6.3 or 1.6.5 or 1.7.1 or 1.8.6 or ~> 1.9", [hex: :hackney, repo: "hexpm", optional: true]}, {:jsx, "~> 2.8", [hex: :jsx, repo: "hexpm", optional: true]}, {:poison, ">= 1.2.0", [hex: :poison, repo: "hexpm", optional: true]}, {:sweet_xml, "~> 0.6", [hex: :sweet_xml, repo: "hexpm", optional: true]}, {:xml_builder, "~> 0.1.0", [hex: :xml_builder, repo: "hexpm", optional: true]}], "hexpm"},
   "ex_aws_s3": {:hex, :ex_aws_s3, "2.0.1", "9e09366e77f25d3d88c5393824e613344631be8db0d1839faca49686e99b6704", [:mix], [{:ex_aws, "~> 2.0", [hex: :ex_aws, repo: "hexpm", optional: false]}, {:sweet_xml, ">= 0.0.0", [hex: :sweet_xml, repo: "hexpm", optional: true]}], "hexpm"},
+  "ex_const": {:hex, :ex_const, "0.2.4", "d06e540c9d834865b012a17407761455efa71d0ce91e5831e86881b9c9d82448", [:mix], [], "hexpm"},
   "ex_doc": {:hex, :ex_doc, "0.20.2", "1bd0dfb0304bade58beb77f20f21ee3558cc3c753743ae0ddbb0fd7ba2912331", [:mix], [{:earmark, "~> 1.3", [hex: :earmark, repo: "hexpm", optional: false]}, {:makeup_elixir, "~> 0.10", [hex: :makeup_elixir, repo: "hexpm", optional: false]}], "hexpm"},
   "ex_machina": {:hex, :ex_machina, "2.3.0", "92a5ad0a8b10ea6314b876a99c8c9e3f25f4dde71a2a835845b136b9adaf199a", [:mix], [{:ecto, "~> 2.2 or ~> 3.0", [hex: :ecto, repo: "hexpm", optional: true]}, {:ecto_sql, "~> 3.0", [hex: :ecto_sql, repo: "hexpm", optional: true]}], "hexpm"},
   "ex_rated": {:hex, :ex_rated, "1.3.3", "30ecbdabe91f7eaa9d37fa4e81c85ba420f371babeb9d1910adbcd79ec798d27", [:mix], [{:ex2ms, "~> 1.5", [hex: :ex2ms, repo: "hexpm", optional: false]}], "hexpm"},

test/fixtures/tesla_mock/kpherox@mstdn.jp.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0">
+  <Subject>acct:kPherox@mstdn.jp</Subject>
+  <Alias>https://mstdn.jp/@kPherox</Alias>
+  <Alias>https://mstdn.jp/users/kPherox</Alias>
+  <Link rel="http://webfinger.net/rel/profile-page" type="text/html" href="https://mstdn.jp/@kPherox"/>
+  <Link rel="http://schemas.google.com/g/2010#updates-from" type="application/atom+xml" href="https://mstdn.jp/users/kPherox.atom"/>
+  <Link rel="self" type="application/activity+json" href="https://mstdn.jp/users/kPherox"/>
+  <Link rel="http://ostatus.org/schema/1.0/subscribe" template="https://mstdn.jp/authorize_interaction?acct={uri}"/>
+</XRD>

test/fixtures/tesla_mock/wedistribute-article.json

@@ -0,0 +1,18 @@
+{
+  "@context": [
+    "https://www.w3.org/ns/activitystreams"
+  ],
+  "type": "Article",
+  "name": "The end is near: Mastodon plans to drop OStatus support",
+  "content": "<!-- wp:paragraph {\"dropCap\":true} -->\n<p class=\"has-drop-cap\">The days of OStatus are numbered. The venerable protocol has served as a glue between many different types of servers since the early days of the Fediverse, connecting StatusNet (now GNU Social) to Friendica, Hubzilla, Mastodon, and Pleroma.</p>\n<!-- /wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Now that many fediverse platforms support ActivityPub as a successor protocol, Mastodon appears to be drawing a line in the sand. In <a href=\"https://www.patreon.com/posts/mastodon-2-9-and-28121681\">a Patreon update</a>, Eugen Rochko writes:</p>\n<!-- /wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\"><p>...OStatus...has overstayed its welcome in the code...and now that most of the network uses ActivityPub, it's time for it to go. </p><cite>Eugen Rochko, Mastodon creator</cite></blockquote>\n<!-- /wp:quote -->\n\n<!-- wp:paragraph -->\n<p>The <a href=\"https://github.com/tootsuite/mastodon/pull/11205\">pull request</a> to remove Pubsubhubbub and Salmon, two of the main components of OStatus, has already been merged into Mastodon's master branch.</p>\n<!-- /wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Some projects will be left in the dark as a side effect of this. GNU Social and PostActiv, for example, both only communicate using OStatus. While <a href=\"https://mastodon.social/@dansup/102076573310057902\">some discussion</a> exists regarding adopting ActivityPub for GNU Social, and <a href=\"https://notabug.org/diogo/gnu-social/src/activitypub/plugins/ActivityPub\">a plugin is in development</a>, it hasn't been formally adopted yet. We just hope that the <a href=\"https://status.fsf.org/main/public\">Free Software Foundation's instance</a> gets updated in time!</p>\n<!-- /wp:paragraph -->",
+  "summary": "One of the largest platforms in the federated social web is dropping the protocol that it started with.",
+  "attributedTo": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog",
+  "url": "https://wedistribute.org/2019/07/mastodon-drops-ostatus/",
+  "to": [
+    "https://www.w3.org/ns/activitystreams#Public",
+    "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog/followers"
+  ],
+  "id": "https://wedistribute.org/wp-json/pterotype/v1/object/85810",
+  "likes": "https://wedistribute.org/wp-json/pterotype/v1/object/85810/likes",
+  "shares": "https://wedistribute.org/wp-json/pterotype/v1/object/85810/shares"
+}

test/fixtures/tesla_mock/wedistribute-user.json

@@ -0,0 +1,31 @@
+{
+  "@context": [
+    "https://www.w3.org/ns/activitystreams",
+    "https://w3id.org/security/v1",
+    {
+      "manuallyApprovesFollowers": "as:manuallyApprovesFollowers"
+    }
+  ],
+  "type": "Organization",
+  "id": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog",
+  "following": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog/following",
+  "followers": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog/followers",
+  "liked": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog/liked",
+  "inbox": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog/inbox",
+  "outbox": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog/outbox",
+  "name": "We Distribute",
+  "preferredUsername": "blog",
+  "summary": "<p>Connecting many threads in the federated web. We Distribute is an independent publication dedicated to the fediverse, decentralization, P2P technologies, and Free Software!</p>",
+  "url": "https://wedistribute.org/",
+  "publicKey": {
+    "id": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog#publicKey",
+    "owner": "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog",
+    "publicKeyPem": "-----BEGIN PUBLIC KEY-----\r\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1bmUJ+y8PS8JFVi0KugN\r\nFl4pLvLog3V2lsV9ftmCXpveB/WJx66Tr1fQLsU3qYvQFc8UPGWD52zV4RENR1SN\r\nx0O6T2f97KUbRM+Ckow7Jyjtssgl+Mqq8UBZQ/+H8I/1Vpvt5E5hUykhFgwzx9qg\r\nzoIA3OK7alOpQbSoKXo0QcOh6yTRUnMSRMJAgUoZJzzXI/FmH/DtKr7ziQ1T2KWs\r\nVs8mWnTb/OlCxiheLuMlmJNMF+lPyVthvMIxF6Z5gV9d5QAmASSCI628e6uH2EUF\r\nDEEF5jo+Z5ffeNv28953lrnM+VB/wTjl3tYA+zCQeAmUPksX3E+YkXGxj+4rxBAY\r\n8wIDAQAB\r\n-----END PUBLIC KEY-----"
+  },
+  "manuallyApprovesFollowers": false,
+  "icon": {
+    "url": "https://wedistribute.org/wp-content/uploads/2019/02/b067de423757a538.png",
+    "type": "Image",
+    "mediaType": "image/png"
+  }
+}

test/fixtures/users_mock/masto_closed_followers_page.json

@@ -0,0 +1 @@
+{"@context":"https://www.w3.org/ns/activitystreams","id":"http://localhost:4001/users/masto_closed/followers?page=1","type":"OrderedCollectionPage","totalItems":437,"next":"http://localhost:4001/users/masto_closed/followers?page=2","partOf":"http://localhost:4001/users/masto_closed/followers","orderedItems":["https://testing.uguu.ltd/users/rin","https://patch.cx/users/rin","https://letsalllovela.in/users/xoxo","https://pleroma.site/users/crushv","https://aria.company/users/boris","https://kawen.space/users/crushv","https://freespeech.host/users/cvcvcv","https://pleroma.site/users/picpub","https://pixelfed.social/users/nosleep","https://boopsnoot.gq/users/5c1896d162f7d337f90492a3","https://pikachu.rocks/users/waifu","https://royal.crablettesare.life/users/crablettes"]}

test/fixtures/users_mock/masto_closed_following_page.json

@@ -0,0 +1 @@
+{"@context":"https://www.w3.org/ns/activitystreams","id":"http://localhost:4001/users/masto_closed/following?page=1","type":"OrderedCollectionPage","totalItems":152,"next":"http://localhost:4001/users/masto_closed/following?page=2","partOf":"http://localhost:4001/users/masto_closed/following","orderedItems":["https://testing.uguu.ltd/users/rin","https://patch.cx/users/rin","https://letsalllovela.in/users/xoxo","https://pleroma.site/users/crushv","https://aria.company/users/boris","https://kawen.space/users/crushv","https://freespeech.host/users/cvcvcv","https://pleroma.site/users/picpub","https://pixelfed.social/users/nosleep","https://boopsnoot.gq/users/5c1896d162f7d337f90492a3","https://pikachu.rocks/users/waifu","https://royal.crablettesare.life/users/crablettes"]}

test/flake_id_test.exs

@@ -39,4 +39,9 @@ test "ecto type behaviour" do
     assert dump(flake_s) == {:ok, flake}
     assert dump(flake) == {:ok, flake}
   end
+
+  test "is_flake_id?/1" do
+    assert is_flake_id?("9eoozpwTul5mjSEDRI")
+    refute is_flake_id?("http://example.com/activities/3ebbadd1-eb14-4e20-8118-b6f79c0c7b0b")
+  end
 end

test/notification_test.exs

@@ -611,6 +611,64 @@ test "replying to a deleted post without tagging does not generate a notificatio
 
       assert Enum.empty?(Notification.for_user(user))
     end
+
+    test "notifications are deleted if a local user is deleted" do
+      user = insert(:user)
+      other_user = insert(:user)
+
+      {:ok, _activity} =
+        CommonAPI.post(user, %{"status" => "hi @#{other_user.nickname}", "visibility" => "direct"})
+
+      refute Enum.empty?(Notification.for_user(other_user))
+
+      User.delete(user)
+
+      assert Enum.empty?(Notification.for_user(other_user))
+    end
+
+    test "notifications are deleted if a remote user is deleted" do
+      remote_user = insert(:user)
+      local_user = insert(:user)
+
+      dm_message = %{
+        "@context" => "https://www.w3.org/ns/activitystreams",
+        "type" => "Create",
+        "actor" => remote_user.ap_id,
+        "id" => remote_user.ap_id <> "/activities/test",
+        "to" => [local_user.ap_id],
+        "cc" => [],
+        "object" => %{
+          "type" => "Note",
+          "content" => "Hello!",
+          "tag" => [
+            %{
+              "type" => "Mention",
+              "href" => local_user.ap_id,
+              "name" => "@#{local_user.nickname}"
+            }
+          ],
+          "to" => [local_user.ap_id],
+          "cc" => [],
+          "attributedTo" => remote_user.ap_id
+        }
+      }
+
+      {:ok, _dm_activity} = Transmogrifier.handle_incoming(dm_message)
+
+      refute Enum.empty?(Notification.for_user(local_user))
+
+      delete_user_message = %{
+        "@context" => "https://www.w3.org/ns/activitystreams",
+        "id" => remote_user.ap_id <> "/activities/delete",
+        "actor" => remote_user.ap_id,
+        "type" => "Delete",
+        "object" => remote_user.ap_id
+      }
+
+      {:ok, _delete_activity} = Transmogrifier.handle_incoming(delete_user_message)
+
+      assert Enum.empty?(Notification.for_user(local_user))
+    end
   end
 
   describe "for_user" do

test/object/fetcher_test.exs

@@ -110,6 +110,13 @@ test "it can fetch peertube videos" do
       assert object
     end
 
+    test "it can fetch wedistribute articles" do
+      {:ok, object} =
+        Fetcher.fetch_object_from_id("https://wedistribute.org/wp-json/pterotype/v1/object/85810")
+
+      assert object
+    end
+
     test "all objects with fake directions are rejected by the object fetcher" do
       assert {:error, _} =
                Fetcher.fetch_and_contain_remote_object_from_id(

test/plugs/set_format_plug_test.exs

@@ -0,0 +1,38 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Plugs.SetFormatPlugTest do
+  use ExUnit.Case, async: true
+  use Plug.Test
+
+  alias Pleroma.Plugs.SetFormatPlug
+
+  test "set format from params" do
+    conn =
+      :get
+      |> conn("/cofe?_format=json")
+      |> SetFormatPlug.call([])
+
+    assert %{format: "json"} == conn.assigns
+  end
+
+  test "set format from header" do
+    conn =
+      :get
+      |> conn("/cofe")
+      |> put_private(:phoenix_format, "xml")
+      |> SetFormatPlug.call([])
+
+    assert %{format: "xml"} == conn.assigns
+  end
+
+  test "doesn't set format" do
+    conn =
+      :get
+      |> conn("/cofe")
+      |> SetFormatPlug.call([])
+
+    refute conn.assigns[:format]
+  end
+end

test/support/http_request_mock.ex

@@ -51,6 +51,10 @@ def get("https://mastodon.social/users/emelie", _, _, _) do
      }}
   end
 
+  def get("https://mastodon.social/users/not_found", _, _, _) do
+    {:ok, %Tesla.Env{status: 404}}
+  end
+
   def get("https://mastodon.sdf.org/users/rinpatch", _, _, _) do
     {:ok,
      %Tesla.Env{
@@ -301,6 +305,22 @@ def get("https://baptiste.gelez.xyz/~/PlumeDevelopment/this-month-in-plume-june-
      }}
   end
 
+  def get("https://wedistribute.org/wp-json/pterotype/v1/object/85810", _, _, _) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/wedistribute-article.json")
+     }}
+  end
+
+  def get("https://wedistribute.org/wp-json/pterotype/v1/actor/-blog", _, _, _) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/wedistribute-user.json")
+     }}
+  end
+
   def get("http://mastodon.example.org/users/admin", _, _, Accept: "application/activity+json") do
     {:ok,
      %Tesla.Env{
@@ -614,6 +634,15 @@ def get(
      }}
   end
 
+  def get(
+        "https://social.heldscal.la/.well-known/webfinger?resource=invalid_content@social.heldscal.la",
+        _,
+        _,
+        Accept: "application/xrd+xml,application/jrd+json"
+      ) do
+    {:ok, %Tesla.Env{status: 200, body: ""}}
+  end
+
   def get("http://framatube.org/.well-known/host-meta", _, _, _) do
     {:ok,
      %Tesla.Env{
@@ -767,6 +796,14 @@ def get("http://localhost:4001/users/masto_closed/followers", _, _, _) do
      }}
   end
 
+  def get("http://localhost:4001/users/masto_closed/followers?page=1", _, _, _) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/users_mock/masto_closed_followers_page.json")
+     }}
+  end
+
   def get("http://localhost:4001/users/masto_closed/following", _, _, _) do
     {:ok,
      %Tesla.Env{
@@ -775,6 +812,14 @@ def get("http://localhost:4001/users/masto_closed/following", _, _, _) do
      }}
   end
 
+  def get("http://localhost:4001/users/masto_closed/following?page=1", _, _, _) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/users_mock/masto_closed_following_page.json")
+     }}
+  end
+
   def get("http://localhost:4001/users/fuser2/followers", _, _, _) do
     {:ok,
      %Tesla.Env{
@@ -915,6 +960,14 @@ def get("https://info.pleroma.site/activity3.json", _, _, _) do
     {:ok, %Tesla.Env{status: 404, body: ""}}
   end
 
+  def get("https://mstdn.jp/.well-known/webfinger?resource=acct:kpherox@mstdn.jp", _, _, _) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/kpherox@mstdn.jp.xml")
+     }}
+  end
+
   def get(url, query, body, headers) do
     {:error,
      "Not implemented the mock response for get #{inspect(url)}, #{query}, #{inspect(body)}, #{

test/upload/filter/dedupe_test.exs

@@ -25,7 +25,7 @@ test "adds shasum" do
 
     assert {
              :ok,
-             %Pleroma.Upload{id: @shasum, path: "#{@shasum}.jpg"}
+             %Pleroma.Upload{id: @shasum, path: @shasum <> ".jpg"}
            } = Dedupe.filter(upload)
   end
 end

test/upload_test.exs

@@ -122,24 +122,6 @@ test "returns a media url" do
       assert String.starts_with?(url, Pleroma.Web.base_url() <> "/media/")
     end
 
-    test "returns a media url with configured base_url" do
-      base_url = "https://cache.pleroma.social"
-
-      File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
-
-      file = %Plug.Upload{
-        content_type: "image/jpg",
-        path: Path.absname("test/fixtures/image_tmp.jpg"),
-        filename: "image.jpg"
-      }
-
-      {:ok, data} = Upload.store(file, base_url: base_url)
-
-      assert %{"url" => [%{"href" => url}]} = data
-
-      assert String.starts_with?(url, base_url <> "/media/")
-    end
-
     test "copies the file to the configured folder with deduping" do
       File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
 
@@ -266,4 +248,32 @@ test "escapes reserved uri characters" do
                "%3A%3F%23%5B%5D%40%21%24%26%5C%27%28%29%2A%2B%2C%3B%3D.jpg"
     end
   end
+
+  describe "Setting a custom base_url for uploaded media" do
+    setup do
+      Pleroma.Config.put([Pleroma.Upload, :base_url], "https://cache.pleroma.social")
+
+      on_exit(fn ->
+        Pleroma.Config.put([Pleroma.Upload, :base_url], nil)
+      end)
+    end
+
+    test "returns a media url with configured base_url" do
+      base_url = Pleroma.Config.get([Pleroma.Upload, :base_url])
+
+      File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
+
+      file = %Plug.Upload{
+        content_type: "image/jpg",
+        path: Path.absname("test/fixtures/image_tmp.jpg"),
+        filename: "image.jpg"
+      }
+
+      {:ok, data} = Upload.store(file, base_url: base_url)
+
+      assert %{"url" => [%{"href" => url}]} = data
+
+      refute String.starts_with?(url, base_url <> "/media/")
+    end
+  end
 end

test/user_test.exs

@@ -1472,4 +1472,102 @@ test "user with internal-prefixed nickname returns true" do
       assert User.is_internal_user?(user)
     end
   end
+
+  describe "update_and_set_cache/1" do
+    test "returns error when user is stale instead Ecto.StaleEntryError" do
+      user = insert(:user)
+
+      changeset = Ecto.Changeset.change(user, bio: "test")
+
+      Repo.delete(user)
+
+      assert {:error, %Ecto.Changeset{errors: [id: {"is stale", [stale: true]}], valid?: false}} =
+               User.update_and_set_cache(changeset)
+    end
+
+    test "performs update cache if user updated" do
+      user = insert(:user)
+      assert {:ok, nil} = Cachex.get(:user_cache, "ap_id:#{user.ap_id}")
+
+      changeset = Ecto.Changeset.change(user, bio: "test-bio")
+
+      assert {:ok, %User{bio: "test-bio"} = user} = User.update_and_set_cache(changeset)
+      assert {:ok, user} = Cachex.get(:user_cache, "ap_id:#{user.ap_id}")
+      assert %User{bio: "test-bio"} = User.get_cached_by_ap_id(user.ap_id)
+    end
+  end
+
+  describe "following/followers synchronization" do
+    setup do
+      sync = Pleroma.Config.get([:instance, :external_user_synchronization])
+      on_exit(fn -> Pleroma.Config.put([:instance, :external_user_synchronization], sync) end)
+    end
+
+    test "updates the counters normally on following/getting a follow when disabled" do
+      Pleroma.Config.put([:instance, :external_user_synchronization], false)
+      user = insert(:user)
+
+      other_user =
+        insert(:user,
+          local: false,
+          follower_address: "http://localhost:4001/users/masto_closed/followers",
+          following_address: "http://localhost:4001/users/masto_closed/following",
+          info: %{ap_enabled: true}
+        )
+
+      assert User.user_info(other_user).following_count == 0
+      assert User.user_info(other_user).follower_count == 0
+
+      {:ok, user} = Pleroma.User.follow(user, other_user)
+      other_user = Pleroma.User.get_by_id(other_user.id)
+
+      assert User.user_info(user).following_count == 1
+      assert User.user_info(other_user).follower_count == 1
+    end
+
+    test "syncronizes the counters with the remote instance for the followed when enabled" do
+      Pleroma.Config.put([:instance, :external_user_synchronization], false)
+
+      user = insert(:user)
+
+      other_user =
+        insert(:user,
+          local: false,
+          follower_address: "http://localhost:4001/users/masto_closed/followers",
+          following_address: "http://localhost:4001/users/masto_closed/following",
+          info: %{ap_enabled: true}
+        )
+
+      assert User.user_info(other_user).following_count == 0
+      assert User.user_info(other_user).follower_count == 0
+
+      Pleroma.Config.put([:instance, :external_user_synchronization], true)
+      {:ok, _user} = User.follow(user, other_user)
+      other_user = User.get_by_id(other_user.id)
+
+      assert User.user_info(other_user).follower_count == 437
+    end
+
+    test "syncronizes the counters with the remote instance for the follower when enabled" do
+      Pleroma.Config.put([:instance, :external_user_synchronization], false)
+
+      user = insert(:user)
+
+      other_user =
+        insert(:user,
+          local: false,
+          follower_address: "http://localhost:4001/users/masto_closed/followers",
+          following_address: "http://localhost:4001/users/masto_closed/following",
+          info: %{ap_enabled: true}
+        )
+
+      assert User.user_info(other_user).following_count == 0
+      assert User.user_info(other_user).follower_count == 0
+
+      Pleroma.Config.put([:instance, :external_user_synchronization], true)
+      {:ok, other_user} = User.follow(other_user, user)
+
+      assert User.user_info(other_user).following_count == 152
+    end
+  end
 end

test/web/activity_pub/activity_pub_test.exs

@@ -1128,4 +1128,65 @@ test "fetches only public posts for other users" do
       assert result.id == activity.id
     end
   end
+
+  describe "fetch_follow_information_for_user" do
+    test "syncronizes following/followers counters" do
+      user =
+        insert(:user,
+          local: false,
+          follower_address: "http://localhost:4001/users/fuser2/followers",
+          following_address: "http://localhost:4001/users/fuser2/following"
+        )
+
+      {:ok, info} = ActivityPub.fetch_follow_information_for_user(user)
+      assert info.follower_count == 527
+      assert info.following_count == 267
+    end
+
+    test "detects hidden followers" do
+      mock(fn env ->
+        case env.url do
+          "http://localhost:4001/users/masto_closed/followers?page=1" ->
+            %Tesla.Env{status: 403, body: ""}
+
+          _ ->
+            apply(HttpRequestMock, :request, [env])
+        end
+      end)
+
+      user =
+        insert(:user,
+          local: false,
+          follower_address: "http://localhost:4001/users/masto_closed/followers",
+          following_address: "http://localhost:4001/users/masto_closed/following"
+        )
+
+      {:ok, info} = ActivityPub.fetch_follow_information_for_user(user)
+      assert info.hide_followers == true
+      assert info.hide_follows == false
+    end
+
+    test "detects hidden follows" do
+      mock(fn env ->
+        case env.url do
+          "http://localhost:4001/users/masto_closed/following?page=1" ->
+            %Tesla.Env{status: 403, body: ""}
+
+          _ ->
+            apply(HttpRequestMock, :request, [env])
+        end
+      end)
+
+      user =
+        insert(:user,
+          local: false,
+          follower_address: "http://localhost:4001/users/masto_closed/followers",
+          following_address: "http://localhost:4001/users/masto_closed/following"
+        )
+
+      {:ok, info} = ActivityPub.fetch_follow_information_for_user(user)
+      assert info.hide_followers == false
+      assert info.hide_follows == true
+    end
+  end
 end

test/web/activity_pub/transmogrifier_test.exs

@@ -1373,32 +1373,4 @@ test "removes recipient's follower collection from cc", %{user: user} do
       refute recipient.follower_address in fixed_object["to"]
     end
   end
-
-  test "update_following_followers_counters/1" do
-    user1 =
-      insert(:user,
-        local: false,
-        follower_address: "http://localhost:4001/users/masto_closed/followers",
-        following_address: "http://localhost:4001/users/masto_closed/following"
-      )
-
-    user2 =
-      insert(:user,
-        local: false,
-        follower_address: "http://localhost:4001/users/fuser2/followers",
-        following_address: "http://localhost:4001/users/fuser2/following"
-      )
-
-    Transmogrifier.update_following_followers_counters(user1)
-    Transmogrifier.update_following_followers_counters(user2)
-
-    %{follower_count: followers, following_count: following} = User.get_cached_user_info(user1)
-    assert followers == 437
-    assert following == 152
-
-    %{follower_count: followers, following_count: following} = User.get_cached_user_info(user2)
-
-    assert followers == 527
-    assert following == 267
-  end
 end

test/web/admin_api/admin_api_controller_test.exs

@@ -1916,6 +1916,46 @@ test "queues key as atom", %{conn: conn} do
     end
   end
 
+  describe "config mix tasks run" do
+    setup %{conn: conn} do
+      admin = insert(:user, info: %{is_admin: true})
+
+      temp_file = "config/test.exported_from_db.secret.exs"
+
+      Mix.shell(Mix.Shell.Quiet)
+
+      on_exit(fn ->
+        Mix.shell(Mix.Shell.IO)
+        :ok = File.rm(temp_file)
+      end)
+
+      dynamic = Pleroma.Config.get([:instance, :dynamic_configuration])
+
+      Pleroma.Config.put([:instance, :dynamic_configuration], true)
+
+      on_exit(fn ->
+        Pleroma.Config.put([:instance, :dynamic_configuration], dynamic)
+      end)
+
+      %{conn: assign(conn, :user, admin), admin: admin}
+    end
+
+    test "transfer settings to DB and to file", %{conn: conn, admin: admin} do
+      assert Pleroma.Repo.all(Pleroma.Web.AdminAPI.Config) == []
+      conn = get(conn, "/api/pleroma/admin/config/migrate_to_db")
+      assert json_response(conn, 200) == %{}
+      assert Pleroma.Repo.all(Pleroma.Web.AdminAPI.Config) > 0
+
+      conn =
+        build_conn()
+        |> assign(:user, admin)
+        |> get("/api/pleroma/admin/config/migrate_from_db")
+
+      assert json_response(conn, 200) == %{}
+      assert Pleroma.Repo.all(Pleroma.Web.AdminAPI.Config) == []
+    end
+  end
+
   describe "GET /api/pleroma/admin/users/:nickname/statuses" do
     setup do
       admin = insert(:user, info: %{is_admin: true})

test/web/common_api/common_api_utils_test.exs

@@ -360,4 +360,24 @@ test "for direct posts, a reply" do
       assert third_user.ap_id in to
     end
   end
+
+  describe "get_by_id_or_ap_id/1" do
+    test "get activity by id" do
+      activity = insert(:note_activity)
+      %Pleroma.Activity{} = note = Utils.get_by_id_or_ap_id(activity.id)
+      assert note.id == activity.id
+    end
+
+    test "get activity by ap_id" do
+      activity = insert(:note_activity)
+      %Pleroma.Activity{} = note = Utils.get_by_id_or_ap_id(activity.data["object"])
+      assert note.id == activity.id
+    end
+
+    test "get activity by object when type isn't `Create` " do
+      activity = insert(:like_activity)
+      %Pleroma.Activity{} = like = Utils.get_by_id_or_ap_id(activity.id)
+      assert like.data["object"] == activity.data["object"]
+    end
+  end
 end

test/web/fallback_test.exs

@@ -30,6 +30,10 @@ test "GET /api*path", %{conn: conn} do
            |> json_response(404) == %{"error" => "Not implemented"}
   end
 
+  test "GET /pleroma/admin -> /pleroma/admin/", %{conn: conn} do
+    assert redirected_to(get(conn, "/pleroma/admin")) =~ "/pleroma/admin/"
+  end
+
   test "GET /*path", %{conn: conn} do
     assert conn
            |> get("/foo")

test/web/mastodon_api/account_view_test.exs

@@ -231,6 +231,16 @@ test "represent a relationship for the blocking and blocked user" do
                AccountView.render("relationship.json", %{user: user, target: other_user})
     end
 
+    test "represent a relationship for the user blocking a domain" do
+      user = insert(:user)
+      other_user = insert(:user, ap_id: "https://bad.site/users/other_user")
+
+      {:ok, user} = User.block_domain(user, "bad.site")
+
+      assert %{domain_blocking: true, blocking: false} =
+               AccountView.render("relationship.json", %{user: user, target: other_user})
+    end
+
     test "represent a relationship for the user with a pending follow request" do
       user = insert(:user)
       other_user = insert(:user, %{info: %User.Info{locked: true}})

test/web/mastodon_api/mastodon_api_controller_test.exs

@@ -3154,6 +3154,21 @@ test "redirects not logged-in users to the login page", %{conn: conn, path: path
       assert redirected_to(conn) == "/web/login"
     end
 
+    test "redirects not logged-in users to the login page on private instances", %{
+      conn: conn,
+      path: path
+    } do
+      is_public = Pleroma.Config.get([:instance, :public])
+      Pleroma.Config.put([:instance, :public], false)
+
+      conn = get(conn, path)
+
+      assert conn.status == 302
+      assert redirected_to(conn) == "/web/login"
+
+      Pleroma.Config.put([:instance, :public], is_public)
+    end
+
     test "does not redirect logged in users to the login page", %{conn: conn, path: path} do
       token = insert(:oauth_token)
 
@@ -3923,4 +3938,45 @@ test "it returns 400 when user is not local", %{conn: conn, user: user} do
       assert conn.resp_body == ""
     end
   end
+
+  describe "POST /api/v1/pleroma/accounts/confirmation_resend" do
+    setup do
+      setting = Pleroma.Config.get([:instance, :account_activation_required])
+
+      unless setting do
+        Pleroma.Config.put([:instance, :account_activation_required], true)
+        on_exit(fn -> Pleroma.Config.put([:instance, :account_activation_required], setting) end)
+      end
+
+      user = insert(:user)
+      info_change = User.Info.confirmation_changeset(user.info, need_confirmation: true)
+
+      {:ok, user} =
+        user
+        |> Changeset.change()
+        |> Changeset.put_embed(:info, info_change)
+        |> Repo.update()
+
+      assert user.info.confirmation_pending
+
+      [user: user]
+    end
+
+    test "resend account confirmation email", %{conn: conn, user: user} do
+      conn
+      |> assign(:user, user)
+      |> post("/api/v1/pleroma/accounts/confirmation_resend?email=#{user.email}")
+      |> json_response(:no_content)
+
+      email = Pleroma.Emails.UserEmail.account_confirmation_email(user)
+      notify_email = Pleroma.Config.get([:instance, :notify_email])
+      instance_name = Pleroma.Config.get([:instance, :name])
+
+      assert_email_sent(
+        from: {instance_name, notify_email},
+        to: {user.name, user.email},
+        html_body: email.html_body
+      )
+    end
+  end
 end

test/web/mastodon_api/status_view_test.exs

@@ -300,6 +300,16 @@ test "attachments" do
     assert %{id: "2"} = StatusView.render("attachment.json", %{attachment: object})
   end
 
+  test "put the url advertised in the Activity in to the url attribute" do
+    id = "https://wedistribute.org/wp-json/pterotype/v1/object/85810"
+    [activity] = Activity.search(nil, id)
+
+    status = StatusView.render("status.json", %{activity: activity})
+
+    assert status.uri == id
+    assert status.url == "https://wedistribute.org/2019/07/mastodon-drops-ostatus/"
+  end
+
   test "a reblog" do
     user = insert(:user)
     activity = insert(:note_activity)

test/web/ostatus/ostatus_controller_test.exs

@@ -101,160 +101,538 @@ test "returns 404 for a missing feed", %{conn: conn} do
     assert response(conn, 404)
   end
 
-  test "gets an object", %{conn: conn} do
+  describe "GET object/2" do
-    note_activity = insert(:note_activity)
+    test "gets an object", %{conn: conn} do
-    object = Object.normalize(note_activity)
+      note_activity = insert(:note_activity)
-    user = User.get_cached_by_ap_id(note_activity.data["actor"])
+      object = Object.normalize(note_activity)
-    [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
-    url = "/objects/#{uuid}"
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
+      url = "/objects/#{uuid}"
+
+      conn =
+        conn
+        |> put_req_header("accept", "application/xml")
+        |> get(url)
+
+      expected =
+        ActivityRepresenter.to_simple_form(note_activity, user, true)
+        |> ActivityRepresenter.wrap_with_entry()
+        |> :xmerl.export_simple(:xmerl_xml)
+        |> to_string
+
+      assert response(conn, 200) == expected
+    end
+
+    test "redirects to /notice/id for html format", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      object = Object.normalize(note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
+      url = "/objects/#{uuid}"
+
+      conn =
+        conn
+        |> put_req_header("accept", "text/html")
+        |> get(url)
+
+      assert redirected_to(conn) == "/notice/#{note_activity.id}"
+    end
+
+    test "500s when user not found", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      object = Object.normalize(note_activity)
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
+      User.invalidate_cache(user)
+      Pleroma.Repo.delete(user)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
+      url = "/objects/#{uuid}"
+
+      conn =
+        conn
+        |> put_req_header("accept", "application/xml")
+        |> get(url)
+
+      assert response(conn, 500) == ~S({"error":"Something went wrong"})
+    end
+
+    test "404s on private objects", %{conn: conn} do
+      note_activity = insert(:direct_note_activity)
+      object = Object.normalize(note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
+
+      conn
+      |> get("/objects/#{uuid}")
+      |> response(404)
+    end
+
+    test "404s on nonexisting objects", %{conn: conn} do
+      conn
+      |> get("/objects/123")
+      |> response(404)
+    end
+  end
+
+  describe "GET activity/2" do
+    test "gets an activity in xml format", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
 
-    conn =
       conn
       |> put_req_header("accept", "application/xml")
-      |> get(url)
+      |> get("/activities/#{uuid}")
+      |> response(200)
+    end
 
-    expected =
+    test "redirects to /notice/id for html format", %{conn: conn} do
-      ActivityRepresenter.to_simple_form(note_activity, user, true)
+      note_activity = insert(:note_activity)
-      |> ActivityRepresenter.wrap_with_entry()
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
-      |> :xmerl.export_simple(:xmerl_xml)
-      |> to_string
 
-    assert response(conn, 200) == expected
+      conn =
+        conn
+        |> put_req_header("accept", "text/html")
+        |> get("/activities/#{uuid}")
+
+      assert redirected_to(conn) == "/notice/#{note_activity.id}"
+    end
+
+    test "505s when user not found", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
+      User.invalidate_cache(user)
+      Pleroma.Repo.delete(user)
+
+      conn =
+        conn
+        |> put_req_header("accept", "text/html")
+        |> get("/activities/#{uuid}")
+
+      assert response(conn, 500) == ~S({"error":"Something went wrong"})
+    end
+
+    test "404s on deleted objects", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      object = Object.normalize(note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
+
+      conn
+      |> put_req_header("accept", "application/xml")
+      |> get("/objects/#{uuid}")
+      |> response(200)
+
+      Object.delete(object)
+
+      conn
+      |> put_req_header("accept", "application/xml")
+      |> get("/objects/#{uuid}")
+      |> response(404)
+    end
+
+    test "404s on private activities", %{conn: conn} do
+      note_activity = insert(:direct_note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
+
+      conn
+      |> get("/activities/#{uuid}")
+      |> response(404)
+    end
+
+    test "404s on nonexistent activities", %{conn: conn} do
+      conn
+      |> get("/activities/123")
+      |> response(404)
+    end
+
+    test "gets an activity in AS2 format", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
+      url = "/activities/#{uuid}"
+
+      conn =
+        conn
+        |> put_req_header("accept", "application/activity+json")
+        |> get(url)
+
+      assert json_response(conn, 200)
+    end
   end
 
-  test "404s on private objects", %{conn: conn} do
+  describe "GET notice/2" do
-    note_activity = insert(:direct_note_activity)
+    test "gets a notice in xml format", %{conn: conn} do
-    object = Object.normalize(note_activity)
+      note_activity = insert(:note_activity)
-    [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
 
-    conn
+      conn
-    |> get("/objects/#{uuid}")
+      |> get("/notice/#{note_activity.id}")
-    |> response(404)
+      |> response(200)
-  end
+    end
 
-  test "404s on nonexisting objects", %{conn: conn} do
+    test "gets a notice in AS2 format", %{conn: conn} do
-    conn
+      note_activity = insert(:note_activity)
-    |> get("/objects/123")
-    |> response(404)
-  end
 
-  test "gets an activity in xml format", %{conn: conn} do
-    note_activity = insert(:note_activity)
-    [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
-
-    conn
-    |> put_req_header("accept", "application/xml")
-    |> get("/activities/#{uuid}")
-    |> response(200)
-  end
-
-  test "404s on deleted objects", %{conn: conn} do
-    note_activity = insert(:note_activity)
-    object = Object.normalize(note_activity)
-    [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
-
-    conn
-    |> put_req_header("accept", "application/xml")
-    |> get("/objects/#{uuid}")
-    |> response(200)
-
-    Object.delete(object)
-
-    conn
-    |> put_req_header("accept", "application/xml")
-    |> get("/objects/#{uuid}")
-    |> response(404)
-  end
-
-  test "404s on private activities", %{conn: conn} do
-    note_activity = insert(:direct_note_activity)
-    [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
-
-    conn
-    |> get("/activities/#{uuid}")
-    |> response(404)
-  end
-
-  test "404s on nonexistent activities", %{conn: conn} do
-    conn
-    |> get("/activities/123")
-    |> response(404)
-  end
-
-  test "gets a notice in xml format", %{conn: conn} do
-    note_activity = insert(:note_activity)
-
-    conn
-    |> get("/notice/#{note_activity.id}")
-    |> response(200)
-  end
-
-  test "gets a notice in AS2 format", %{conn: conn} do
-    note_activity = insert(:note_activity)
-
-    conn
-    |> put_req_header("accept", "application/activity+json")
-    |> get("/notice/#{note_activity.id}")
-    |> json_response(200)
-  end
-
-  test "only gets a notice in AS2 format for Create messages", %{conn: conn} do
-    note_activity = insert(:note_activity)
-    url = "/notice/#{note_activity.id}"
-
-    conn =
       conn
       |> put_req_header("accept", "application/activity+json")
-      |> get(url)
+      |> get("/notice/#{note_activity.id}")
+      |> json_response(200)
+    end
 
-    assert json_response(conn, 200)
+    test "500s when actor not found", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
+      User.invalidate_cache(user)
+      Pleroma.Repo.delete(user)
 
-    user = insert(:user)
+      conn =
+        conn
+        |> get("/notice/#{note_activity.id}")
 
-    {:ok, like_activity, _} = CommonAPI.favorite(note_activity.id, user)
+      assert response(conn, 500) == ~S({"error":"Something went wrong"})
-    url = "/notice/#{like_activity.id}"
+    end
 
-    assert like_activity.data["type"] == "Like"
+    test "only gets a notice in AS2 format for Create messages", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      url = "/notice/#{note_activity.id}"
 
-    conn =
+      conn =
-      build_conn()
+        conn
-      |> put_req_header("accept", "application/activity+json")
+        |> put_req_header("accept", "application/activity+json")
-      |> get(url)
+        |> get(url)
 
-    assert response(conn, 404)
+      assert json_response(conn, 200)
+
+      user = insert(:user)
+
+      {:ok, like_activity, _} = CommonAPI.favorite(note_activity.id, user)
+      url = "/notice/#{like_activity.id}"
+
+      assert like_activity.data["type"] == "Like"
+
+      conn =
+        build_conn()
+        |> put_req_header("accept", "application/activity+json")
+        |> get(url)
+
+      assert response(conn, 404)
+    end
+
+    test "render html for redirect for html format", %{conn: conn} do
+      note_activity = insert(:note_activity)
+
+      resp =
+        conn
+        |> put_req_header("accept", "text/html")
+        |> get("/notice/#{note_activity.id}")
+        |> response(200)
+
+      assert resp =~
+               "<meta content=\"#{Pleroma.Web.base_url()}/notice/#{note_activity.id}\" property=\"og:url\">"
+
+      user = insert(:user)
+
+      {:ok, like_activity, _} = CommonAPI.favorite(note_activity.id, user)
+
+      assert like_activity.data["type"] == "Like"
+
+      resp =
+        conn
+        |> put_req_header("accept", "text/html")
+        |> get("/notice/#{like_activity.id}")
+        |> response(200)
+
+      assert resp =~ "<!--server-generated-meta-->"
+    end
+
+    test "404s a private notice", %{conn: conn} do
+      note_activity = insert(:direct_note_activity)
+      url = "/notice/#{note_activity.id}"
+
+      conn =
+        conn
+        |> get(url)
+
+      assert response(conn, 404)
+    end
+
+    test "404s a nonexisting notice", %{conn: conn} do
+      url = "/notice/123"
+
+      conn =
+        conn
+        |> get(url)
+
+      assert response(conn, 404)
+    end
   end
 
-  test "gets an activity in AS2 format", %{conn: conn} do
+  describe "feed_redirect" do
-    note_activity = insert(:note_activity)
+    test "undefined format. it redirects to feed", %{conn: conn} do
-    [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
+      note_activity = insert(:note_activity)
-    url = "/activities/#{uuid}"
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
 
-    conn =
+      response =
-      conn
+        conn
-      |> put_req_header("accept", "application/activity+json")
+        |> put_req_header("accept", "application/xml")
-      |> get(url)
+        |> get("/users/#{user.nickname}")
+        |> response(302)
 
-    assert json_response(conn, 200)
+      assert response ==
+               "<html><body>You are being <a href=\"#{Pleroma.Web.base_url()}/users/#{
+                 user.nickname
+               }/feed.atom\">redirected</a>.</body></html>"
+    end
+
+    test "undefined format. it returns error when user not found", %{conn: conn} do
+      response =
+        conn
+        |> put_req_header("accept", "application/xml")
+        |> get("/users/jimm")
+        |> response(404)
+
+      assert response == ~S({"error":"Not found"})
+    end
+
+    test "activity+json format. it redirects on actual feed of user", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
+
+      response =
+        conn
+        |> put_req_header("accept", "application/activity+json")
+        |> get("/users/#{user.nickname}")
+        |> json_response(200)
+
+      assert response["endpoints"] == %{
+               "oauthAuthorizationEndpoint" => "#{Pleroma.Web.base_url()}/oauth/authorize",
+               "oauthRegistrationEndpoint" => "#{Pleroma.Web.base_url()}/api/v1/apps",
+               "oauthTokenEndpoint" => "#{Pleroma.Web.base_url()}/oauth/token",
+               "sharedInbox" => "#{Pleroma.Web.base_url()}/inbox"
+             }
+
+      assert response["@context"] == [
+               "https://www.w3.org/ns/activitystreams",
+               "http://localhost:4001/schemas/litepub-0.1.jsonld",
+               %{"@language" => "und"}
+             ]
+
+      assert Map.take(response, [
+               "followers",
+               "following",
+               "id",
+               "inbox",
+               "manuallyApprovesFollowers",
+               "name",
+               "outbox",
+               "preferredUsername",
+               "summary",
+               "tag",
+               "type",
+               "url"
+             ]) == %{
+               "followers" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/followers",
+               "following" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/following",
+               "id" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}",
+               "inbox" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/inbox",
+               "manuallyApprovesFollowers" => false,
+               "name" => user.name,
+               "outbox" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/outbox",
+               "preferredUsername" => user.nickname,
+               "summary" => user.bio,
+               "tag" => [],
+               "type" => "Person",
+               "url" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}"
+             }
+    end
+
+    test "activity+json format. it returns error whe use not found", %{conn: conn} do
+      response =
+        conn
+        |> put_req_header("accept", "application/activity+json")
+        |> get("/users/jimm")
+        |> json_response(404)
+
+      assert response == "Not found"
+    end
+
+    test "json format. it redirects on actual feed of user", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
+
+      response =
+        conn
+        |> put_req_header("accept", "application/json")
+        |> get("/users/#{user.nickname}")
+        |> json_response(200)
+
+      assert response["endpoints"] == %{
+               "oauthAuthorizationEndpoint" => "#{Pleroma.Web.base_url()}/oauth/authorize",
+               "oauthRegistrationEndpoint" => "#{Pleroma.Web.base_url()}/api/v1/apps",
+               "oauthTokenEndpoint" => "#{Pleroma.Web.base_url()}/oauth/token",
+               "sharedInbox" => "#{Pleroma.Web.base_url()}/inbox"
+             }
+
+      assert response["@context"] == [
+               "https://www.w3.org/ns/activitystreams",
+               "http://localhost:4001/schemas/litepub-0.1.jsonld",
+               %{"@language" => "und"}
+             ]
+
+      assert Map.take(response, [
+               "followers",
+               "following",
+               "id",
+               "inbox",
+               "manuallyApprovesFollowers",
+               "name",
+               "outbox",
+               "preferredUsername",
+               "summary",
+               "tag",
+               "type",
+               "url"
+             ]) == %{
+               "followers" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/followers",
+               "following" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/following",
+               "id" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}",
+               "inbox" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/inbox",
+               "manuallyApprovesFollowers" => false,
+               "name" => user.name,
+               "outbox" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}/outbox",
+               "preferredUsername" => user.nickname,
+               "summary" => user.bio,
+               "tag" => [],
+               "type" => "Person",
+               "url" => "#{Pleroma.Web.base_url()}/users/#{user.nickname}"
+             }
+    end
+
+    test "json format. it returns error whe use not found", %{conn: conn} do
+      response =
+        conn
+        |> put_req_header("accept", "application/json")
+        |> get("/users/jimm")
+        |> json_response(404)
+
+      assert response == "Not found"
+    end
+
+    test "html format. it redirects on actual feed of user", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      user = User.get_cached_by_ap_id(note_activity.data["actor"])
+
+      response =
+        conn
+        |> get("/users/#{user.nickname}")
+        |> response(200)
+
+      assert response ==
+               Fallback.RedirectController.redirector_with_meta(
+                 conn,
+                 %{user: user}
+               ).resp_body
+    end
+
+    test "html format. it returns error when user not found", %{conn: conn} do
+      response =
+        conn
+        |> get("/users/jimm")
+        |> json_response(404)
+
+      assert response == %{"error" => "Not found"}
+    end
   end
 
-  test "404s a private notice", %{conn: conn} do
+  describe "GET /notice/:id/embed_player" do
-    note_activity = insert(:direct_note_activity)
+    test "render embed player", %{conn: conn} do
-    url = "/notice/#{note_activity.id}"
+      note_activity = insert(:note_activity)
+      object = Pleroma.Object.normalize(note_activity)
 
-    conn =
+      object_data =
-      conn
+        Map.put(object.data, "attachment", [
-      |> get(url)
+          %{
+            "url" => [
+              %{
+                "href" =>
+                  "https://peertube.moe/static/webseed/df5f464b-be8d-46fb-ad81-2d4c2d1630e3-480.mp4",
+                "mediaType" => "video/mp4",
+                "type" => "Link"
+              }
+            ]
+          }
+        ])
 
-    assert response(conn, 404)
+      object
-  end
+      |> Ecto.Changeset.change(data: object_data)
+      |> Pleroma.Repo.update()
 
-  test "404s a nonexisting notice", %{conn: conn} do
+      conn =
-    url = "/notice/123"
+        conn
+        |> get("/notice/#{note_activity.id}/embed_player")
 
-    conn =
+      assert Plug.Conn.get_resp_header(conn, "x-frame-options") == ["ALLOW"]
-      conn
-      |> get(url)
 
-    assert response(conn, 404)
+      assert Plug.Conn.get_resp_header(
+               conn,
+               "content-security-policy"
+             ) == [
+               "default-src 'none';style-src 'self' 'unsafe-inline';img-src 'self' data: https:; media-src 'self' https:;"
+             ]
+
+      assert response(conn, 200) =~
+               "<video controls loop><source src=\"https://peertube.moe/static/webseed/df5f464b-be8d-46fb-ad81-2d4c2d1630e3-480.mp4\" type=\"video/mp4\">Your browser does not support video/mp4 playback.</video>"
+    end
+
+    test "404s when activity isn't create", %{conn: conn} do
+      note_activity = insert(:note_activity, data_attrs: %{"type" => "Like"})
+
+      assert conn
+             |> get("/notice/#{note_activity.id}/embed_player")
+             |> response(404)
+    end
+
+    test "404s when activity is direct message", %{conn: conn} do
+      note_activity = insert(:note_activity, data_attrs: %{"directMessage" => true})
+
+      assert conn
+             |> get("/notice/#{note_activity.id}/embed_player")
+             |> response(404)
+    end
+
+    test "404s when attachment is empty", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      object = Pleroma.Object.normalize(note_activity)
+      object_data = Map.put(object.data, "attachment", [])
+
+      object
+      |> Ecto.Changeset.change(data: object_data)
+      |> Pleroma.Repo.update()
+
+      assert conn
+             |> get("/notice/#{note_activity.id}/embed_player")
+             |> response(404)
+    end
+
+    test "404s when attachment isn't audio or video", %{conn: conn} do
+      note_activity = insert(:note_activity)
+      object = Pleroma.Object.normalize(note_activity)
+
+      object_data =
+        Map.put(object.data, "attachment", [
+          %{
+            "url" => [
+              %{
+                "href" => "https://peertube.moe/static/webseed/480.jpg",
+                "mediaType" => "image/jpg",
+                "type" => "Link"
+              }
+            ]
+          }
+        ])
+
+      object
+      |> Ecto.Changeset.change(data: object_data)
+      |> Pleroma.Repo.update()
+
+      assert conn
+             |> get("/notice/#{note_activity.id}/embed_player")
+             |> response(404)
+    end
   end
 end

test/web/ostatus/ostatus_test.exs

@@ -326,6 +326,14 @@ test "handle incoming follows" do
     assert User.following?(follower, followed)
   end
 
+  test "refuse following over OStatus if the followed's account is locked" do
+    incoming = File.read!("test/fixtures/follow.xml")
+    _user = insert(:user, info: %{locked: true}, ap_id: "https://pawoo.net/users/pekorino")
+
+    {:ok, [{:error, "It's not possible to follow locked accounts over OStatus"}]} =
+      OStatus.handle_incoming(incoming)
+  end
+
   test "handle incoming unfollows with existing follow" do
     incoming_follow = File.read!("test/fixtures/follow.xml")
     {:ok, [_activity]} = OStatus.handle_incoming(incoming_follow)
@@ -426,7 +434,7 @@ test "find_or_make_user sets all the nessary input fields" do
                }
     end
 
-    test "find_make_or_update_user takes an author element and returns an updated user" do
+    test "find_make_or_update_actor takes an author element and returns an updated user" do
       uri = "https://social.heldscal.la/user/23211"
 
       {:ok, user} = OStatus.find_or_make_user(uri)
@@ -439,14 +447,56 @@ test "find_make_or_update_user takes an author element and returns an updated us
 
       doc = XML.parse_document(File.read!("test/fixtures/23211.atom"))
       [author] = :xmerl_xpath.string('//author[1]', doc)
-      {:ok, user} = OStatus.find_make_or_update_user(author)
+      {:ok, user} = OStatus.find_make_or_update_actor(author)
       assert user.avatar["type"] == "Image"
       assert user.name == old_name
       assert user.bio == old_bio
 
-      {:ok, user_again} = OStatus.find_make_or_update_user(author)
+      {:ok, user_again} = OStatus.find_make_or_update_actor(author)
       assert user_again == user
     end
+
+    test "find_or_make_user disallows protocol downgrade" do
+      user = insert(:user, %{local: true})
+      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
+
+      assert User.ap_enabled?(user)
+
+      user =
+        insert(:user, %{
+          ap_id: "https://social.heldscal.la/user/23211",
+          info: %{ap_enabled: true},
+          local: false
+        })
+
+      assert User.ap_enabled?(user)
+
+      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
+      assert User.ap_enabled?(user)
+    end
+
+    test "find_make_or_update_actor disallows protocol downgrade" do
+      user = insert(:user, %{local: true})
+      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
+
+      assert User.ap_enabled?(user)
+
+      user =
+        insert(:user, %{
+          ap_id: "https://social.heldscal.la/user/23211",
+          info: %{ap_enabled: true},
+          local: false
+        })
+
+      assert User.ap_enabled?(user)
+
+      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
+      assert User.ap_enabled?(user)
+
+      doc = XML.parse_document(File.read!("test/fixtures/23211.atom"))
+      [author] = :xmerl_xpath.string('//author[1]', doc)
+      {:error, :invalid_protocol} = OStatus.find_make_or_update_actor(author)
+    end
   end
 
   describe "gathering user info from a user id" do

test/web/push/impl_test.exs

@@ -124,8 +124,7 @@ test "renders body for follow activity" do
     {:ok, _, _, activity} = CommonAPI.follow(user, other_user)
     object = Object.normalize(activity)
 
-    assert Impl.format_body(%{activity: activity}, user, object) ==
+    assert Impl.format_body(%{activity: activity}, user, object) == "@Bob has followed you"
-             "@Bob has followed you"
   end
 
   test "renders body for announce activity" do
@@ -156,7 +155,6 @@ test "renders body for like activity" do
     {:ok, activity, _} = CommonAPI.favorite(activity.id, user)
     object = Object.normalize(activity)
 
-    assert Impl.format_body(%{activity: activity}, user, object) ==
+    assert Impl.format_body(%{activity: activity}, user, object) == "@Bob has favorited your post"
-             "@Bob has favorited your post"
   end
 end

test/web/twitter_api/util_controller_test.exs

@@ -14,6 +14,17 @@ defmodule Pleroma.Web.TwitterAPI.UtilControllerTest do
 
   setup do
     Tesla.Mock.mock(fn env -> apply(HttpRequestMock, :request, [env]) end)
+
+    instance_config = Pleroma.Config.get([:instance])
+    pleroma_fe = Pleroma.Config.get([:frontend_configurations, :pleroma_fe])
+    deny_follow_blocked = Pleroma.Config.get([:user, :deny_follow_blocked])
+
+    on_exit(fn ->
+      Pleroma.Config.put([:instance], instance_config)
+      Pleroma.Config.put([:frontend_configurations, :pleroma_fe], pleroma_fe)
+      Pleroma.Config.put([:user, :deny_follow_blocked], deny_follow_blocked)
+    end)
+
     :ok
   end
 
@@ -31,6 +42,35 @@ test "it returns HTTP 200", %{conn: conn} do
       assert response == "job started"
     end
 
+    test "it imports follow lists from file", %{conn: conn} do
+      user1 = insert(:user)
+      user2 = insert(:user)
+
+      with_mocks([
+        {File, [],
+         read!: fn "follow_list.txt" ->
+           "Account address,Show boosts\n#{user2.ap_id},true"
+         end},
+        {PleromaJobQueue, [:passthrough], []}
+      ]) do
+        response =
+          conn
+          |> assign(:user, user1)
+          |> post("/api/pleroma/follow_import", %{"list" => %Plug.Upload{path: "follow_list.txt"}})
+          |> json_response(:ok)
+
+        assert called(
+                 PleromaJobQueue.enqueue(
+                   :background,
+                   User,
+                   [:follow_import, user1, [user2.ap_id]]
+                 )
+               )
+
+        assert response == "job started"
+      end
+    end
+
     test "it imports new-style mastodon follow lists", %{conn: conn} do
       user1 = insert(:user)
       user2 = insert(:user)
@@ -79,6 +119,33 @@ test "it returns HTTP 200", %{conn: conn} do
 
       assert response == "job started"
     end
+
+    test "it imports blocks users from file", %{conn: conn} do
+      user1 = insert(:user)
+      user2 = insert(:user)
+      user3 = insert(:user)
+
+      with_mocks([
+        {File, [], read!: fn "blocks_list.txt" -> "#{user2.ap_id} #{user3.ap_id}" end},
+        {PleromaJobQueue, [:passthrough], []}
+      ]) do
+        response =
+          conn
+          |> assign(:user, user1)
+          |> post("/api/pleroma/blocks_import", %{"list" => %Plug.Upload{path: "blocks_list.txt"}})
+          |> json_response(:ok)
+
+        assert called(
+                 PleromaJobQueue.enqueue(
+                   :background,
+                   User,
+                   [:blocks_import, user1, [user2.ap_id, user3.ap_id]]
+                 )
+               )
+
+        assert response == "job started"
+      end
+    end
   end
 
   describe "POST /api/pleroma/notifications/read" do
@@ -98,6 +165,18 @@ test "it marks a single notification as read", %{conn: conn} do
       assert Repo.get(Notification, notification1.id).seen
       refute Repo.get(Notification, notification2.id).seen
     end
+
+    test "it returns error when notification not found", %{conn: conn} do
+      user1 = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user1)
+        |> post("/api/pleroma/notifications/read", %{"id" => "22222222222222"})
+        |> json_response(403)
+
+      assert response == %{"error" => "Cannot get notification"}
+    end
   end
 
   describe "PUT /api/pleroma/notification_settings" do
@@ -123,7 +202,63 @@ test "it updates notification settings", %{conn: conn} do
     end
   end
 
-  describe "GET /api/statusnet/config.json" do
+  describe "GET /api/statusnet/config" do
+    test "it returns config in xml format", %{conn: conn} do
+      instance = Pleroma.Config.get(:instance)
+
+      response =
+        conn
+        |> put_req_header("accept", "application/xml")
+        |> get("/api/statusnet/config")
+        |> response(:ok)
+
+      assert response ==
+               "<config>\n<site>\n<name>#{Keyword.get(instance, :name)}</name>\n<site>#{
+                 Pleroma.Web.base_url()
+               }</site>\n<textlimit>#{Keyword.get(instance, :limit)}</textlimit>\n<closed>#{
+                 !Keyword.get(instance, :registrations_open)
+               }</closed>\n</site>\n</config>\n"
+    end
+
+    test "it returns config in json format", %{conn: conn} do
+      instance = Pleroma.Config.get(:instance)
+      Pleroma.Config.put([:instance, :managed_config], true)
+      Pleroma.Config.put([:instance, :registrations_open], false)
+      Pleroma.Config.put([:instance, :invites_enabled], true)
+      Pleroma.Config.put([:instance, :public], false)
+      Pleroma.Config.put([:frontend_configurations, :pleroma_fe], %{theme: "asuka-hospital"})
+
+      response =
+        conn
+        |> put_req_header("accept", "application/json")
+        |> get("/api/statusnet/config")
+        |> json_response(:ok)
+
+      expected_data = %{
+        "site" => %{
+          "accountActivationRequired" => "0",
+          "closed" => "1",
+          "description" => Keyword.get(instance, :description),
+          "invitesEnabled" => "1",
+          "name" => Keyword.get(instance, :name),
+          "pleromafe" => %{"theme" => "asuka-hospital"},
+          "private" => "1",
+          "safeDMMentionsEnabled" => "0",
+          "server" => Pleroma.Web.base_url(),
+          "textlimit" => to_string(Keyword.get(instance, :limit)),
+          "uploadlimit" => %{
+            "avatarlimit" => to_string(Keyword.get(instance, :avatar_upload_limit)),
+            "backgroundlimit" => to_string(Keyword.get(instance, :background_upload_limit)),
+            "bannerlimit" => to_string(Keyword.get(instance, :banner_upload_limit)),
+            "uploadlimit" => to_string(Keyword.get(instance, :upload_limit))
+          },
+          "vapidPublicKey" => Keyword.get(Pleroma.Web.Push.vapid_config(), :public_key)
+        }
+      }
+
+      assert response == expected_data
+    end
+
     test "returns the state of safe_dm_mentions flag", %{conn: conn} do
       option = Pleroma.Config.get([:instance, :safe_dm_mentions])
       Pleroma.Config.put([:instance, :safe_dm_mentions], true)
@@ -210,7 +345,7 @@ test "returns json with custom emoji with tags", %{conn: conn} do
     end
   end
 
-  describe "GET /ostatus_subscribe?acct=...." do
+  describe "GET /ostatus_subscribe - remote_follow/2" do
     test "adds status to pleroma instance if the `acct` is a status", %{conn: conn} do
       conn =
         get(
@@ -230,6 +365,172 @@ test "show follow account page if the `acct` is a account link", %{conn: conn} d
 
       assert html_response(response, 200) =~ "Log in to follow"
     end
+
+    test "show follow page if the `acct` is a account link", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> get("/ostatus_subscribe?acct=https://mastodon.social/users/emelie")
+
+      assert html_response(response, 200) =~ "Remote follow"
+    end
+
+    test "show follow page with error when user cannot fecth by `acct` link", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> get("/ostatus_subscribe?acct=https://mastodon.social/users/not_found")
+
+      assert html_response(response, 200) =~ "Error fetching user"
+    end
+  end
+
+  describe "POST /ostatus_subscribe - do_remote_follow/2 with assigned user " do
+    test "follows user", %{conn: conn} do
+      user = insert(:user)
+      user2 = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/ostatus_subscribe", %{"user" => %{"id" => user2.id}})
+        |> response(200)
+
+      assert response =~ "Account followed!"
+      assert user2.follower_address in refresh_record(user).following
+    end
+
+    test "returns error when user is deactivated", %{conn: conn} do
+      user = insert(:user, info: %{deactivated: true})
+      user2 = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/ostatus_subscribe", %{"user" => %{"id" => user2.id}})
+        |> response(200)
+
+      assert response =~ "Error following account"
+    end
+
+    test "returns error when user is blocked", %{conn: conn} do
+      Pleroma.Config.put([:user, :deny_follow_blocked], true)
+      user = insert(:user)
+      user2 = insert(:user)
+
+      {:ok, _user} = Pleroma.User.block(user2, user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/ostatus_subscribe", %{"user" => %{"id" => user2.id}})
+        |> response(200)
+
+      assert response =~ "Error following account"
+    end
+
+    test "returns error when followee not found", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/ostatus_subscribe", %{"user" => %{"id" => "jimm"}})
+        |> response(200)
+
+      assert response =~ "Error following account"
+    end
+
+    test "returns success result when user already in followers", %{conn: conn} do
+      user = insert(:user)
+      user2 = insert(:user)
+      {:ok, _, _, _} = CommonAPI.follow(user, user2)
+
+      response =
+        conn
+        |> assign(:user, refresh_record(user))
+        |> post("/ostatus_subscribe", %{"user" => %{"id" => user2.id}})
+        |> response(200)
+
+      assert response =~ "Account followed!"
+    end
+  end
+
+  describe "POST /ostatus_subscribe - do_remote_follow/2 without assigned user " do
+    test "follows", %{conn: conn} do
+      user = insert(:user)
+      user2 = insert(:user)
+
+      response =
+        conn
+        |> post("/ostatus_subscribe", %{
+          "authorization" => %{"name" => user.nickname, "password" => "test", "id" => user2.id}
+        })
+        |> response(200)
+
+      assert response =~ "Account followed!"
+      assert user2.follower_address in refresh_record(user).following
+    end
+
+    test "returns error when followee not found", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> post("/ostatus_subscribe", %{
+          "authorization" => %{"name" => user.nickname, "password" => "test", "id" => "jimm"}
+        })
+        |> response(200)
+
+      assert response =~ "Error following account"
+    end
+
+    test "returns error when login invalid", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> post("/ostatus_subscribe", %{
+          "authorization" => %{"name" => "jimm", "password" => "test", "id" => user.id}
+        })
+        |> response(200)
+
+      assert response =~ "Wrong username or password"
+    end
+
+    test "returns error when password invalid", %{conn: conn} do
+      user = insert(:user)
+      user2 = insert(:user)
+
+      response =
+        conn
+        |> post("/ostatus_subscribe", %{
+          "authorization" => %{"name" => user.nickname, "password" => "42", "id" => user2.id}
+        })
+        |> response(200)
+
+      assert response =~ "Wrong username or password"
+    end
+
+    test "returns error when user is blocked", %{conn: conn} do
+      Pleroma.Config.put([:user, :deny_follow_blocked], true)
+      user = insert(:user)
+      user2 = insert(:user)
+      {:ok, _user} = Pleroma.User.block(user2, user)
+
+      response =
+        conn
+        |> post("/ostatus_subscribe", %{
+          "authorization" => %{"name" => user.nickname, "password" => "test", "id" => user2.id}
+        })
+        |> response(200)
+
+      assert response =~ "Error following account"
+    end
   end
 
   describe "GET /api/pleroma/healthcheck" do
@@ -311,5 +612,104 @@ test "it returns HTTP 200", %{conn: conn} do
 
       assert user.info.deactivated == true
     end
+
+    test "it returns returns when password invalid", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/api/pleroma/disable_account", %{"password" => "test1"})
+        |> json_response(:ok)
+
+      assert response == %{"error" => "Invalid password."}
+      user = User.get_cached_by_id(user.id)
+
+      refute user.info.deactivated
+    end
+  end
+
+  describe "GET /api/statusnet/version" do
+    test "it returns version in xml format", %{conn: conn} do
+      response =
+        conn
+        |> put_req_header("accept", "application/xml")
+        |> get("/api/statusnet/version")
+        |> response(:ok)
+
+      assert response == "<version>#{Pleroma.Application.named_version()}</version>"
+    end
+
+    test "it returns version in json format", %{conn: conn} do
+      response =
+        conn
+        |> put_req_header("accept", "application/json")
+        |> get("/api/statusnet/version")
+        |> json_response(:ok)
+
+      assert response == "#{Pleroma.Application.named_version()}"
+    end
+  end
+
+  describe "POST /main/ostatus - remote_subscribe/2" do
+    test "renders subscribe form", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> post("/main/ostatus", %{"nickname" => user.nickname, "profile" => ""})
+        |> response(:ok)
+
+      refute response =~ "Could not find user"
+      assert response =~ "Remotely follow #{user.nickname}"
+    end
+
+    test "renders subscribe form with error when user not found", %{conn: conn} do
+      response =
+        conn
+        |> post("/main/ostatus", %{"nickname" => "nickname", "profile" => ""})
+        |> response(:ok)
+
+      assert response =~ "Could not find user"
+      refute response =~ "Remotely follow"
+    end
+
+    test "it redirect to webfinger url", %{conn: conn} do
+      user = insert(:user)
+      user2 = insert(:user, ap_id: "shp@social.heldscal.la")
+
+      conn =
+        conn
+        |> post("/main/ostatus", %{
+          "user" => %{"nickname" => user.nickname, "profile" => user2.ap_id}
+        })
+
+      assert redirected_to(conn) ==
+               "https://social.heldscal.la/main/ostatussub?profile=#{user.ap_id}"
+    end
+
+    test "it renders form with error when use not found", %{conn: conn} do
+      user2 = insert(:user, ap_id: "shp@social.heldscal.la")
+
+      response =
+        conn
+        |> post("/main/ostatus", %{"user" => %{"nickname" => "jimm", "profile" => user2.ap_id}})
+        |> response(:ok)
+
+      assert response =~ "Something went wrong."
+    end
+  end
+
+  test "it returns new captcha", %{conn: conn} do
+    with_mock Pleroma.Captcha,
+      new: fn -> "test_captcha" end do
+      resp =
+        conn
+        |> get("/api/pleroma/captcha")
+        |> response(200)
+
+      assert resp == "\"test_captcha\""
+      assert called(Pleroma.Captcha.new())
+    end
   end
 end

test/web/web_finger/web_finger_controller_test.exs

@@ -19,6 +19,19 @@ defmodule Pleroma.Web.WebFinger.WebFingerControllerTest do
     :ok
   end
 
+  test "GET host-meta" do
+    response =
+      build_conn()
+      |> get("/.well-known/host-meta")
+
+    assert response.status == 200
+
+    assert response.resp_body ==
+             ~s(<?xml version="1.0" encoding="UTF-8"?><XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0"><Link rel="lrdd" template="#{
+               Pleroma.Web.base_url()
+             }/.well-known/webfinger?resource={uri}" type="application/xrd+xml" /></XRD>)
+  end
+
   test "Webfinger JRD" do
     user = insert(:user)
 
@@ -30,6 +43,16 @@ test "Webfinger JRD" do
     assert json_response(response, 200)["subject"] == "acct:#{user.nickname}@localhost"
   end
 
+  test "it returns 404 when user isn't found (JSON)" do
+    result =
+      build_conn()
+      |> put_req_header("accept", "application/jrd+json")
+      |> get("/.well-known/webfinger?resource=acct:jimm@localhost")
+      |> json_response(404)
+
+    assert result == "Couldn't find user"
+  end
+
   test "Webfinger XML" do
     user = insert(:user)
 
@@ -41,6 +64,26 @@ test "Webfinger XML" do
     assert response(response, 200)
   end
 
+  test "it returns 404 when user isn't found (XML)" do
+    result =
+      build_conn()
+      |> put_req_header("accept", "application/xrd+xml")
+      |> get("/.well-known/webfinger?resource=acct:jimm@localhost")
+      |> response(404)
+
+    assert result == "Couldn't find user"
+  end
+
+  test "Sends a 404 when invalid format" do
+    user = insert(:user)
+
+    assert_raise Phoenix.NotAcceptableError, fn ->
+      build_conn()
+      |> put_req_header("accept", "text/html")
+      |> get("/.well-known/webfinger?resource=acct:#{user.nickname}@localhost")
+    end
+  end
+
   test "Sends a 400 when resource param is missing" do
     response =
       build_conn()

test/web/web_finger/web_finger_test.exs

@@ -40,6 +40,11 @@ test "works for ap_ids" do
   end
 
   describe "fingering" do
+    test "returns error when fails parse xml or json" do
+      user = "invalid_content@social.heldscal.la"
+      assert {:error, %Jason.DecodeError{}} = WebFinger.finger(user)
+    end
+
     test "returns the info for an OStatus user" do
       user = "shp@social.heldscal.la"
 
@@ -81,6 +86,20 @@ test "returns the correctly for json ostatus users" do
       assert data["subscribe_address"] == "https://gnusocial.de/main/ostatussub?profile={uri}"
     end
 
+    test "it work for AP-only user" do
+      user = "kpherox@mstdn.jp"
+
+      {:ok, data} = WebFinger.finger(user)
+
+      assert data["magic_key"] == nil
+      assert data["salmon"] == nil
+
+      assert data["topic"] == "https://mstdn.jp/users/kPherox.atom"
+      assert data["subject"] == "acct:kPherox@mstdn.jp"
+      assert data["ap_id"] == "https://mstdn.jp/users/kPherox"
+      assert data["subscribe_address"] == "https://mstdn.jp/authorize_interaction?acct={uri}"
+    end
+
     test "it works for friendica" do
       user = "lain@squeet.me"