moon
/
honkoma
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'feature/905-dynamic-configuration-in-sql' into 'develop' 

Feature/905 dynamic configuration in sql

Closes #905

See merge request pleroma/pleroma!1195
This commit is contained in:
kaniini 2019-06-14 15:45:05 +00:00
parent b7fc722a2e c2ca1f22a2
commit 270f09dbc2
32 changed files with 940 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -19,6 +19,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Mix Tasks: `mix pleroma.database remove_embedded_objects`
- Mix Tasks: `mix pleroma.database update_users_following_followers_counts`
- Mix Tasks: `mix pleroma.user toggle_confirmed`
- Mix Tasks: `mix pleroma.config migrate_to_db`
- Mix Tasks: `mix pleroma.config migrate_from_db`
- Federation: Support for `Question` and `Answer` objects
- Federation: Support for reports
- Configuration: `poll_limits` option
@ -37,6 +39,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Admin API: added filters (role, tags, email, name) for users endpoint
- Admin API: Endpoints for managing reports
- Admin API: Endpoints for deleting and changing the scope of individual reported statuses
- Admin API: Endpoints to view and change config settings.
- AdminFE: initial release with basic user management accessible at /pleroma/admin/
- Mastodon API: [Scheduled statuses](https://docs.joinmastodon.org/api/rest/scheduled-statuses/)
- Mastodon API: `/api/v1/notifications/destroy_multiple` (glitch-soc extension)

3
config/config.exs
View File

@ -245,7 +245,8 @@
healthcheck: false,
remote_post_retention_days: 90,
skip_thread_containment: true,
limit_to_local_content: :unauthenticated
limit_to_local_content: :unauthenticated,
dynamic_configuration: false
config :pleroma, :markup,
# XXX - unfortunately, inline images must be enabled by default right now, because

3
config/dev.exs
View File

@ -59,3 +59,6 @@
"!!! RUNNING IN LOCALHOST DEV MODE! !!!\nFEDERATION WON'T WORK UNTIL YOU CONFIGURE A dev.secret.exs"
)
end
if File.exists?("./config/dev.migrated.secret.exs"),
do: import_config("./config/dev.migrated.secret.exs")

3
config/prod.exs
View File

@ -63,3 +63,6 @@
# Finally import the config/prod.secret.exs
# which should be versioned separately.
import_config "prod.secret.exs"
if File.exists?("./config/prod.migrated.secret.exs"),
do: import_config("./config/prod.migrated.secret.exs")

108
docs/api/admin_api.md
View File

@ -289,7 +289,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- `limit`: optional, the number of records to retrieve
- `since_id`: optional, returns results that are more recent than the specified id
- `max_id`: optional, returns results that are older than the specified id
- Response:
- Response:
- On failure: 403 Forbidden error `{"error": "error_msg"}` when requested by anonymous or non-admin
- On success: JSON, returns a list of reports, where:
- `account`: the user who has been reported
@ -443,7 +443,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- Params:
- `id`
- Response:
- On failure:
- On failure:
- 403 Forbidden `{"error": "error_msg"}`
- 404 Not Found `"Not found"`
- On success: JSON, Report object (see above)
@ -454,8 +454,8 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- Params:
- `id`
- `state`: required, the new state. Valid values are `open`, `closed` and `resolved`
- Response:
- On failure:
- Response:
- On failure:
- 400 Bad Request `"Unsupported state"`
- 403 Forbidden `{"error": "error_msg"}`
- 404 Not Found `"Not found"`
@ -467,10 +467,10 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- Params:
- `id`
- `status`: required, the message
- Response:
- On failure:
- 400 Bad Request `"Invalid parameters"` when `status` is missing
- 403 Forbidden `{"error": "error_msg"}`
- Response:
- On failure:
- 400 Bad Request `"Invalid parameters"` when `status` is missing
- 403 Forbidden `{"error": "error_msg"}`
- 404 Not Found `"Not found"`
- On success: JSON, created Mastodon Status entity
@ -540,10 +540,10 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- `id`
- `sensitive`: optional, valid values are `true` or `false`
- `visibility`: optional, valid values are `public`, `private` and `unlisted`
- Response:
- On failure:
- Response:
- On failure:
- 400 Bad Request `"Unsupported visibility"`
- 403 Forbidden `{"error": "error_msg"}`
- 403 Forbidden `{"error": "error_msg"}`
- 404 Not Found `"Not found"`
- On success: JSON, Mastodon Status entity
@ -552,8 +552,88 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- Method `DELETE`
- Params:
- `id`
- Response:
- On failure:
- 403 Forbidden `{"error": "error_msg"}`
- Response:
- On failure:
- 403 Forbidden `{"error": "error_msg"}`
- 404 Not Found `"Not found"`
- On success: 200 OK `{}`
## `/api/pleroma/admin/config`
### List config settings
- Method `GET`
- Params: none
- Response:
```json
{
configs: [
{
"key": string,
"value": string or {} or []
}
]
}
```
## `/api/pleroma/admin/config`
### Update config settings
Module name can be passed as string, which starts with `Pleroma`, e.g. `"Pleroma.Upload"`.
Atom or boolean value can be passed with `:` in the beginning, e.g. `":true"`, `":upload"`.
Integer with `i:`, e.g. `"i:150"`.
Compile time settings (need instance reboot):
- all settings by this keys:
- `:hackney_pools`
- `:chat`
- `Pleroma.Web.Endpoint`
- `Pleroma.Repo`
- part settings:
- `Pleroma.Captcha` -> `:seconds_valid`
- `Pleroma.Upload` -> `:proxy_remote`
- `:instance` -> `:upload_limit`
- Method `POST`
- Params:
- `configs` => [
- `key` (string)
- `value` (string, [], {})
- `delete` = true (optional, if parameter must be deleted)
]
- Request (example):
```json
{
configs: [
{
"key": "Pleroma.Upload",
"value": {
"uploader": "Pleroma.Uploaders.Local",
"filters": ["Pleroma.Upload.Filter.Dedupe"],
"link_name": ":true",
"proxy_remote": ":false",
"proxy_opts": {
"redirect_on_failure": ":false",
"max_body_length": "i:1048576",
"http": {
"follow_redirect": ":true",
"pool": ":upload"
}
}
}
}
]
}
- Response:
```json
{
configs: [
{
"key": string,
"value": string or {} or []
}
]
}
```

1
docs/config.md
View File

@ -114,6 +114,7 @@ config :pleroma, Pleroma.Emails.Mailer,
* `remote_post_retention_days`: The default amount of days to retain remote posts when pruning the database.
* `skip_thread_containment`: Skip filter out broken threads. The default is `false`.
* `limit_to_local_content`: Limit unauthenticated users to search for local statutes and users only. Possible values: `:unauthenticated`, `:all` and `false`. The default is `:unauthenticated`.
* `dynamic_configuration`: Allow transferring configuration to DB with the subsequent customization from Admin api.
## :logger

68
lib/mix/tasks/pleroma/config.ex Normal file
View File

@ -0,0 +1,68 @@
defmodule Mix.Tasks.Pleroma.Config do
use Mix.Task
alias Mix.Tasks.Pleroma.Common
alias Pleroma.Repo
alias Pleroma.Web.AdminAPI.Config
@shortdoc "Manages the location of the config"
@moduledoc """
Manages the location of the config.
## Transfers config from file to DB.
mix pleroma.config migrate_to_db
## Transfers config from DB to file.
mix pleroma.config migrate_from_db ENV
"""
def run(["migrate_to_db"]) do
Common.start_pleroma()
if Pleroma.Config.get([:instance, :dynamic_configuration]) do
Application.get_all_env(:pleroma)
|> Enum.reject(fn {k, _v} -> k in [Pleroma.Repo, :env] end)
|> Enum.each(fn {k, v} ->
key = to_string(k) |> String.replace("Elixir.", "")
{:ok, _} = Config.update_or_create(%{key: key, value: v})
Mix.shell().info("#{key} is migrated.")
end)
Mix.shell().info("Settings migrated.")
else
Mix.shell().info(
"Migration is not allowed by config. You can change this behavior in instance settings."
)
end
end
def run(["migrate_from_db", env]) do
Common.start_pleroma()
if Pleroma.Config.get([:instance, :dynamic_configuration]) do
config_path = "config/#{env}.migrated.secret.exs"
{:ok, file} = File.open(config_path, [:write])
Repo.all(Config)
|> Enum.each(fn config ->
mark = if String.starts_with?(config.key, "Pleroma."), do: ",", else: ":"
IO.write(
file,
"config :pleroma, #{config.key}#{mark} #{inspect(Config.from_binary(config.value))}\r\n"
)
{:ok, _} = Repo.delete(config)
Mix.shell().info("#{config.key} deleted from DB.")
end)
File.close(file)
System.cmd("mix", ["format", config_path])
else
Mix.shell().info(
"Migration is not allowed by config. You can change this behavior in instance settings."
)
end
end
end

8
lib/mix/tasks/pleroma/emoji.ex
View File

@ -55,15 +55,13 @@ defmodule Mix.Tasks.Pleroma.Emoji do
are extracted).
"""
@default_manifest Pleroma.Config.get!([:emoji, :default_manifest])
def run(["ls-packs" | args]) do
Application.ensure_all_started(:hackney)
{options, [], []} = parse_global_opts(args)
manifest =
fetch_manifest(if options[:manifest], do: options[:manifest], else: @default_manifest)
fetch_manifest(if options[:manifest], do: options[:manifest], else: default_manifest())
Enum.each(manifest, fn {name, info} ->
to_print = [
@ -88,7 +86,7 @@ def run(["get-packs" | args]) do
{options, pack_names, []} = parse_global_opts(args)
manifest_url = if options[:manifest], do: options[:manifest], else: @default_manifest
manifest_url = if options[:manifest], do: options[:manifest], else: default_manifest()
manifest = fetch_manifest(manifest_url)
@ -298,4 +296,6 @@ defp client do
Tesla.client(middleware)
end
defp default_manifest, do: Pleroma.Config.get!([:emoji, :default_manifest])
end

15
lib/mix/tasks/pleroma/instance.ex
View File

@ -30,6 +30,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
- `--dbuser DBUSER` - the user (aka role) to use for the database connection
- `--dbpass DBPASS` - the password to use for the database connection
- `--indexable Y/N` - Allow/disallow indexing site by search engines
- `--db-configurable Y/N` - Allow/disallow configuring instance from admin part
"""
def run(["gen" | rest]) do
@ -48,7 +49,8 @@ def run(["gen" | rest]) do
dbname: :string,
dbuser: :string,
dbpass: :string,
indexable: :string
indexable: :string,
db_configurable: :string
],
aliases: [
o: :output,
@ -101,6 +103,14 @@ def run(["gen" | rest]) do
"y"
) === "y"
db_configurable? =
Common.get_option(
options,
:db_configurable,
"Do you want to be able to configure instance from admin part? (y/n)",
"y"
) === "y"
dbhost =
Common.get_option(options, :dbhost, "What is the hostname of your database?", "localhost")
@ -144,7 +154,8 @@ def run(["gen" | rest]) do
secret: secret,
signing_salt: signing_salt,
web_push_public_key: Base.url_encode64(web_push_public_key, padding: false),
web_push_private_key: Base.url_encode64(web_push_private_key, padding: false)
web_push_private_key: Base.url_encode64(web_push_private_key, padding: false),
db_configurable?: db_configurable?
)
result_psql =

3
lib/mix/tasks/pleroma/sample_config.eex
View File

@ -16,7 +16,8 @@ config :pleroma, :instance,
notify_email: "<%= notify_email %>",
limit: 5000,
registrations_open: true,
dedupe_media: false
dedupe_media: false,
dynamic_configuration: <%= db_configurable? %>
config :pleroma, :media_proxy,
enabled: false,

3
lib/pleroma/application.ex
View File

@ -31,6 +31,7 @@ def start(_type, _args) do
[
# Start the Ecto repository
%{id: Pleroma.Repo, start: {Pleroma.Repo, :start_link, []}, type: :supervisor},
%{id: Pleroma.Config.TransferTask, start: {Pleroma.Config.TransferTask, :start_link, []}},
%{id: Pleroma.Emoji, start: {Pleroma.Emoji, :start_link, []}},
%{id: Pleroma.Captcha, start: {Pleroma.Captcha, :start_link, []}},
%{
@ -186,7 +187,7 @@ def enabled_hackney_pools do
else
[]
end ++
if Pleroma.Config.get([Pleroma.Uploader, :proxy_remote]) do
if Pleroma.Config.get([Pleroma.Upload, :proxy_remote]) do
[:upload]
else
[]

41
lib/pleroma/config/transfer_task.ex Normal file
View File

@ -0,0 +1,41 @@
defmodule Pleroma.Config.TransferTask do
use Task
alias Pleroma.Web.AdminAPI.Config
def start_link do
load_and_update_env()
if Pleroma.Config.get(:env) == :test, do: Ecto.Adapters.SQL.Sandbox.checkin(Pleroma.Repo)
:ignore
end
def load_and_update_env do
if Pleroma.Config.get([:instance, :dynamic_configuration]) do
Pleroma.Repo.all(Config)
|> Enum.each(&update_env(&1))
end
end
defp update_env(setting) do
try do
key =
if String.starts_with?(setting.key, "Pleroma.") do
"Elixir." <> setting.key
else
setting.key
end
Application.put_env(
:pleroma,
String.to_existing_atom(key),
Config.from_binary(setting.value)
)
rescue
e ->
require Logger
Logger.warn(
"updating env causes error, key: #{inspect(setting.key)}, error: #{inspect(e)}"
)
end
end
end

29
lib/pleroma/emoji.ex
View File

@ -22,7 +22,6 @@ defmodule Pleroma.Emoji do
@ets __MODULE__.Ets
@ets_options [:ordered_set, :protected, :named_table, {:read_concurrency, true}]
@groups Pleroma.Config.get([:emoji, :groups])
@doc false
def start_link do
@ -87,6 +86,8 @@ defp load do
"emoji"
)
emoji_groups = Pleroma.Config.get([:emoji, :groups])
case File.ls(emoji_dir_path) do
{:error, :enoent} ->
# The custom emoji directory doesn't exist,
@ -118,7 +119,7 @@ defp load do
emojis =
Enum.flat_map(
packs,
fn pack -> load_pack(Path.join(emoji_dir_path, pack)) end
fn pack -> load_pack(Path.join(emoji_dir_path, pack), emoji_groups) end
)
true = :ets.insert(@ets, emojis)
@ -129,9 +130,9 @@ defp load do
shortcode_globs = Pleroma.Config.get([:emoji, :shortcode_globs], [])
emojis =
(load_from_file("config/emoji.txt") ++
load_from_file("config/custom_emoji.txt") ++
load_from_globs(shortcode_globs))
(load_from_file("config/emoji.txt", emoji_groups) ++
load_from_file("config/custom_emoji.txt", emoji_groups) ++
load_from_globs(shortcode_globs, emoji_groups))
|> Enum.reject(fn value -> value == nil end)
true = :ets.insert(@ets, emojis)
@ -139,13 +140,13 @@ defp load do
:ok
end
defp load_pack(pack_dir) do
defp load_pack(pack_dir, emoji_groups) do
pack_name = Path.basename(pack_dir)
emoji_txt = Path.join(pack_dir, "emoji.txt")
if File.exists?(emoji_txt) do
load_from_file(emoji_txt)
load_from_file(emoji_txt, emoji_groups)
else
Logger.info(
"No emoji.txt found for pack \"#{pack_name}\", assuming all .png files are emoji"
@ -155,7 +156,7 @@ defp load_pack(pack_dir) do
|> Enum.map(fn {shortcode, rel_file} ->
filename = Path.join("/emoji/#{pack_name}", rel_file)
{shortcode, filename, [to_string(match_extra(@groups, filename))]}
{shortcode, filename, [to_string(match_extra(emoji_groups, filename))]}
end)
end
end
@ -184,21 +185,21 @@ def find_all_emoji(dir, exts) do
|> Enum.filter(fn f -> Path.extname(f) in exts end)
end
defp load_from_file(file) do
defp load_from_file(file, emoji_groups) do
if File.exists?(file) do
load_from_file_stream(File.stream!(file))
load_from_file_stream(File.stream!(file), emoji_groups)
else
[]
end
end
defp load_from_file_stream(stream) do
defp load_from_file_stream(stream, emoji_groups) do
stream
|> Stream.map(&String.trim/1)
|> Stream.map(fn line ->
case String.split(line, ~r/,\s*/) do
[name, file] ->
{name, file, [to_string(match_extra(@groups, file))]}
{name, file, [to_string(match_extra(emoji_groups, file))]}
[name, file | tags] ->
{name, file, tags}
@ -210,7 +211,7 @@ defp load_from_file_stream(stream) do
|> Enum.to_list()
end
defp load_from_globs(globs) do
defp load_from_globs(globs, emoji_groups) do
static_path = Path.join(:code.priv_dir(:pleroma), "static")
paths =
@ -221,7 +222,7 @@ defp load_from_globs(globs) do
|> Enum.concat()
Enum.map(paths, fn path ->
tag = match_extra(@groups, Path.join("/", Path.relative_to(path, static_path)))
tag = match_extra(emoji_groups, Path.join("/", Path.relative_to(path, static_path)))
shortcode = Path.basename(path, Path.extname(path))
external_path = Path.join("/", Path.relative_to(path, static_path))
{shortcode, external_path, [to_string(tag)]}

2
lib/pleroma/instances.ex
View File

@ -13,7 +13,7 @@ def set_consistently_unreachable(url_or_host),
def reachability_datetime_threshold do
federation_reachability_timeout_days =
Pleroma.Config.get(:instance)[:federation_reachability_timeout_days] || 0
Pleroma.Config.get([:instance, :federation_reachability_timeout_days], 0)
if federation_reachability_timeout_days > 0 do
NaiveDateTime.add(

2
lib/pleroma/plugs/uploaded_media.ex
View File

@ -36,7 +36,7 @@ def call(%{request_path: <<"/", @path, "/", file::binary>>} = conn, opts) do
conn
end
config = Pleroma.Config.get([Pleroma.Upload])
config = Pleroma.Config.get(Pleroma.Upload)
with uploader <- Keyword.fetch!(config, :uploader),
proxy_remote = Keyword.get(config, :proxy_remote, false),

6
lib/pleroma/reverse_proxy.ex
View File

@ -146,7 +146,7 @@ defp request(method, url, headers, hackney_opts) do
Logger.debug("#{__MODULE__} #{method} #{url} #{inspect(headers)}")
method = method |> String.downcase() |> String.to_existing_atom()
case :hackney.request(method, url, headers, "", hackney_opts) do
case hackney().request(method, url, headers, "", hackney_opts) do
{:ok, code, headers, client} when code in @valid_resp_codes ->
{:ok, code, downcase_headers(headers), client}
@ -196,7 +196,7 @@ defp chunk_reply(conn, client, opts, sent_so_far, duration) do
duration,
Keyword.get(opts, :max_read_duration, @max_read_duration)
),
{:ok, data} <- :hackney.stream_body(client),
{:ok, data} <- hackney().stream_body(client),
{:ok, duration} <- increase_read_duration(duration),
sent_so_far = sent_so_far + byte_size(data),
:ok <- body_size_constraint(sent_so_far, Keyword.get(opts, :max_body_size)),
@ -377,4 +377,6 @@ defp increase_read_duration({previous_duration, started})
defp increase_read_duration(_) do
{:ok, :no_duration_limit, :no_duration_limit}
end
defp hackney, do: Pleroma.Config.get(:hackney, :hackney)
end

4
lib/pleroma/user.ex
View File

@ -1036,9 +1036,7 @@ def html_filter_policy(%User{info: %{no_rich_text: true}}) do
Pleroma.HTML.Scrubber.TwitterText
end
@default_scrubbers Pleroma.Config.get([:markup, :scrub_policy])
def html_filter_policy(_), do: @default_scrubbers
def html_filter_policy(_), do: Pleroma.Config.get([:markup, :scrub_policy])
def fetch_by_ap_id(ap_id) do
ap_try = ActivityPub.make_user_from_ap_id(ap_id)

2
lib/pleroma/web/activity_pub/publisher.ex
View File

@ -88,7 +88,7 @@ defp should_federate?(inbox, public) do
true
else
inbox_info = URI.parse(inbox)
!Enum.member?(Pleroma.Config.get([:instance, :quarantined_instances], []), inbox_info.host)
!Enum.member?(Config.get([:instance, :quarantined_instances], []), inbox_info.host)
end
end

37
lib/pleroma/web/admin_api/admin_api_controller.ex
View File

@ -10,6 +10,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
alias Pleroma.Web.ActivityPub.ActivityPub
alias Pleroma.Web.ActivityPub.Relay
alias Pleroma.Web.AdminAPI.AccountView
alias Pleroma.Web.AdminAPI.Config
alias Pleroma.Web.AdminAPI.ConfigView
alias Pleroma.Web.AdminAPI.ReportView
alias Pleroma.Web.AdminAPI.Search
alias Pleroma.Web.CommonAPI
@ -362,6 +364,41 @@ def status_delete(%{assigns: %{user: user}} = conn, %{"id" => id}) do
end
end
def config_show(conn, _params) do
configs = Pleroma.Repo.all(Config)
conn
|> put_view(ConfigView)
|> render("index.json", %{configs: configs})
end
def config_update(conn, %{"configs" => configs}) do
updated =
if Pleroma.Config.get([:instance, :dynamic_configuration]) do
updated =
Enum.map(configs, fn
%{"key" => key, "value" => value} ->
{:ok, config} = Config.update_or_create(%{key: key, value: value})
config
%{"key" => key, "delete" => "true"} ->
{:ok, _} = Config.delete(key)
nil
end)
|> Enum.reject(&is_nil(&1))
Pleroma.Config.TransferTask.load_and_update_env()
Mix.Tasks.Pleroma.Config.run(["migrate_from_db", Pleroma.Config.get(:env)])
updated
else
[]
end
conn
|> put_view(ConfigView)
|> render("index.json", %{configs: updated})
end
def errors(conn, {:error, :not_found}) do
conn
|> put_status(404)

144
lib/pleroma/web/admin_api/config.ex Normal file
View File

@ -0,0 +1,144 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.AdminAPI.Config do
use Ecto.Schema
import Ecto.Changeset
alias __MODULE__
alias Pleroma.Repo
@type t :: %__MODULE__{}
schema "config" do
field(:key, :string)
field(:value, :binary)
timestamps()
end
@spec get_by_key(String.t()) :: Config.t() | nil
def get_by_key(key), do: Repo.get_by(Config, key: key)
@spec changeset(Config.t(), map()) :: Changeset.t()
def changeset(config, params \\ %{}) do
config
|> cast(params, [:key, :value])
|> validate_required([:key, :value])
|> unique_constraint(:key)
end
@spec create(map()) :: {:ok, Config.t()} | {:error, Changeset.t()}
def create(%{key: key, value: value}) do
%Config{}
|> changeset(%{key: key, value: transform(value)})
|> Repo.insert()
end
@spec update(Config.t(), map()) :: {:ok, Config} | {:error, Changeset.t()}
def update(%Config{} = config, %{value: value}) do
config
|> change(value: transform(value))
|> Repo.update()
end
@spec update_or_create(map()) :: {:ok, Config.t()} | {:error, Changeset.t()}
def update_or_create(%{key: key} = params) do
with %Config{} = config <- Config.get_by_key(key) do
Config.update(config, params)
else
nil -> Config.create(params)
end
end
@spec delete(String.t()) :: {:ok, Config.t()} | {:error, Changeset.t()}
def delete(key) do
with %Config{} = config <- Config.get_by_key(key) do
Repo.delete(config)
else
nil -> {:error, "Config with key #{key} not found"}
end
end
@spec from_binary(binary()) :: term()
def from_binary(value), do: :erlang.binary_to_term(value)
@spec from_binary_to_map(binary()) :: any()
def from_binary_to_map(binary) do
from_binary(binary)
|> do_convert()
end
defp do_convert([{k, v}] = value) when is_list(value) and length(value) == 1,
do: %{k => do_convert(v)}
defp do_convert(values) when is_list(values), do: for(val <- values, do: do_convert(val))
defp do_convert({k, v} = value) when is_tuple(value),
do: %{k => do_convert(v)}
defp do_convert(value) when is_binary(value) or is_atom(value) or is_map(value),
do: value
@spec transform(any()) :: binary()
def transform(entity) when is_map(entity) do
tuples =
for {k, v} <- entity,
into: [],
do: {if(is_atom(k), do: k, else: String.to_atom(k)), do_transform(v)}
Enum.reject(tuples, fn {_k, v} -> is_nil(v) end)
|> Enum.sort()
|> :erlang.term_to_binary()
end
def transform(entity) when is_list(entity) do
list = Enum.map(entity, &do_transform(&1))
:erlang.term_to_binary(list)
end
def transform(entity), do: :erlang.term_to_binary(entity)
defp do_transform(%Regex{} = value) when is_map(value), do: value
defp do_transform(value) when is_map(value) do
values =
for {key, val} <- value,
into: [],
do: {String.to_atom(key), do_transform(val)}
Enum.sort(values)
end
defp do_transform(value) when is_list(value) do
Enum.map(value, &do_transform(&1))
end
defp do_transform(entity) when is_list(entity) and length(entity) == 1, do: hd(entity)
defp do_transform(value) when is_binary(value) do
value = String.trim(value)
case String.length(value) do
0 ->
nil
_ ->
cond do
String.starts_with?(value, "Pleroma") ->
String.to_existing_atom("Elixir." <> value)
String.starts_with?(value, ":") ->
String.replace(value, ":", "") |> String.to_existing_atom()
String.starts_with?(value, "i:") ->
String.replace(value, "i:", "") |> String.to_integer()
true ->
value
end
end
end
defp do_transform(value), do: value
end

16
lib/pleroma/web/admin_api/views/config_view.ex Normal file
View File

@ -0,0 +1,16 @@
defmodule Pleroma.Web.AdminAPI.ConfigView do
use Pleroma.Web, :view
def render("index.json", %{configs: configs}) do
%{
configs: render_many(configs, __MODULE__, "show.json", as: :config)
}
end
def render("show.json", %{config: config}) do
%{
key: config.key,
value: Pleroma.Web.AdminAPI.Config.from_binary_to_map(config.value)
}
end
end

2
lib/pleroma/web/endpoint.ex
View File

@ -91,7 +91,7 @@ defmodule Pleroma.Web.Endpoint do
Plug.Session,
store: :cookie,
key: cookie_name,
signing_salt: {Pleroma.Config, :get, [[__MODULE__, :signing_salt], "CqaoopA2"]},
signing_salt: Pleroma.Config.get([__MODULE__, :signing_salt], "CqaoopA2"),
http_only: true,
secure: secure_cookies,
extra: extra

5
lib/pleroma/web/oauth/token.ex
View File

@ -14,7 +14,6 @@ defmodule Pleroma.Web.OAuth.Token do
alias Pleroma.Web.OAuth.Token
alias Pleroma.Web.OAuth.Token.Query
@expires_in Pleroma.Config.get([:oauth2, :token_expires_in], 600)
@type t :: %__MODULE__{}
schema "oauth_tokens" do
@ -78,7 +77,7 @@ defp put_refresh_token(changeset, attrs) do
defp put_valid_until(changeset, attrs) do
expires_in =
Map.get(attrs, :valid_until, NaiveDateTime.add(NaiveDateTime.utc_now(), @expires_in))
Map.get(attrs, :valid_until, NaiveDateTime.add(NaiveDateTime.utc_now(), expires_in()))
changeset
|> change(%{valid_until: expires_in})
@ -123,4 +122,6 @@ def is_expired?(%__MODULE__{valid_until: valid_until}) do
end
def is_expired?(_), do: false
defp expires_in, do: Pleroma.Config.get([:oauth2, :token_expires_in], 600)
end

8
lib/pleroma/web/oauth/token/response.ex
View File

@ -4,15 +4,13 @@ defmodule Pleroma.Web.OAuth.Token.Response do
alias Pleroma.User
alias Pleroma.Web.OAuth.Token.Utils
@expires_in Pleroma.Config.get([:oauth2, :token_expires_in], 600)
@doc false
def build(%User{} = user, token, opts \\ %{}) do
%{
token_type: "Bearer",
access_token: token.token,
refresh_token: token.refresh_token,
expires_in: @expires_in,
expires_in: expires_in(),
scope: Enum.join(token.scopes, " "),
me: user.ap_id
}
@ -25,8 +23,10 @@ def build_for_client_credentials(token) do
access_token: token.token,
refresh_token: token.refresh_token,
created_at: Utils.format_created_at(token),
expires_in: @expires_in,
expires_in: expires_in(),
scope: Enum.join(token.scopes, " ")
}
end
defp expires_in, do: Pleroma.Config.get([:oauth2, :token_expires_in], 600)
end

3
lib/pleroma/web/router.ex
View File

@ -202,6 +202,9 @@ defmodule Pleroma.Web.Router do
put("/statuses/:id", AdminAPIController, :status_update)
delete("/statuses/:id", AdminAPIController, :status_delete)
get("/config", AdminAPIController, :config_show)
post("/config", AdminAPIController, :config_update)
end
scope "/", Pleroma.Web.TwitterAPI do

13
priv/repo/migrations/20190518032627_create_config.exs Normal file
View File

@ -0,0 +1,13 @@
defmodule Pleroma.Repo.Migrations.CreateConfig do
use Ecto.Migration
def change do
create table(:config) do
add(:key, :string)
add(:value, :binary)
timestamps()
end
create(unique_index(:config, :key))
end
end

35
test/config/transfer_task_test.exs Normal file
View File

@ -0,0 +1,35 @@
defmodule Pleroma.Config.TransferTaskTest do
use Pleroma.DataCase
setup do
dynamic = Pleroma.Config.get([:instance, :dynamic_configuration])
Pleroma.Config.put([:instance, :dynamic_configuration], true)
on_exit(fn ->
Pleroma.Config.put([:instance, :dynamic_configuration], dynamic)
end)
end
test "transfer config values from db to env" do
refute Application.get_env(:pleroma, :test_key)
Pleroma.Web.AdminAPI.Config.create(%{key: "test_key", value: [live: 2, com: 3]})
Pleroma.Config.TransferTask.start_link()
assert Application.get_env(:pleroma, :test_key) == [live: 2, com: 3]
on_exit(fn ->
Application.delete_env(:pleroma, :test_key)
end)
end
test "non existing atom" do
Pleroma.Web.AdminAPI.Config.create(%{key: "undefined_atom_key", value: [live: 2, com: 3]})
assert ExUnit.CaptureLog.capture_log(fn ->
Pleroma.Config.TransferTask.start_link()
end) =~
"updating env causes error, key: \"undefined_atom_key\", error: %ArgumentError{message: \"argument error\"}"
end
end

13
test/support/factory.ex
View File

@ -310,4 +310,17 @@ def registration_factory do
}
}
end
def config_factory do
%Pleroma.Web.AdminAPI.Config{
key: sequence(:key, &"some_key_#{&1}"),
value:
sequence(
:value,
fn key ->
:erlang.term_to_binary(%{another_key: "#{key}somevalue", another: "#{key}somevalue"})
end
)
}
end
end

54
test/tasks/config_test.exs Normal file
View File

@ -0,0 +1,54 @@
defmodule Mix.Tasks.Pleroma.ConfigTest do
use Pleroma.DataCase
alias Pleroma.Repo
alias Pleroma.Web.AdminAPI.Config
setup_all do
Mix.shell(Mix.Shell.Process)
temp_file = "config/temp.migrated.secret.exs"
dynamic = Pleroma.Config.get([:instance, :dynamic_configuration])
Pleroma.Config.put([:instance, :dynamic_configuration], true)
on_exit(fn ->
Mix.shell(Mix.Shell.IO)
Application.delete_env(:pleroma, :first_setting)
Application.delete_env(:pleroma, :second_setting)
Pleroma.Config.put([:instance, :dynamic_configuration], dynamic)
:ok = File.rm(temp_file)
end)
{:ok, temp_file: temp_file}
end
test "settings are migrated to db" do
assert Repo.all(Config) == []
Application.put_env(:pleroma, :first_setting, key: "value", key2: [Pleroma.Repo])
Application.put_env(:pleroma, :second_setting, key: "value2", key2: [Pleroma.Activity])
Mix.Tasks.Pleroma.Config.run(["migrate_to_db"])
first_db = Config.get_by_key("first_setting")
second_db = Config.get_by_key("second_setting")
refute Config.get_by_key("Pleroma.Repo")
assert Config.from_binary(first_db.value) == [key: "value", key2: [Pleroma.Repo]]
assert Config.from_binary(second_db.value) == [key: "value2", key2: [Pleroma.Activity]]
end
test "settings are migrated to file and deleted from db", %{temp_file: temp_file} do
Config.create(%{key: "setting_first", value: [key: "value", key2: [Pleroma.Activity]]})
Config.create(%{key: "setting_second", value: [key: "valu2", key2: [Pleroma.Repo]]})
Mix.Tasks.Pleroma.Config.run(["migrate_from_db", "temp"])
assert Repo.all(Config) == []
assert File.exists?(temp_file)
{:ok, file} = File.read(temp_file)
assert file =~ "config :pleroma, setting_first:"
assert file =~ "config :pleroma, setting_second:"
end
end

3
test/tasks/instance.exs
View File

@ -36,6 +36,8 @@ test "running gen" do
"--dbpass",
"dbpass",
"--indexable",
"y",
"--db-configurable",
"y"
])
end
@ -53,6 +55,7 @@ test "running gen" do
assert generated_config =~ "database: \"dbname\""
assert generated_config =~ "username: \"dbuser\""
assert generated_config =~ "password: \"dbpass\""
assert generated_config =~ "dynamic_configuration: true"
assert File.read!(tmp_path() <> "setup.psql") == generated_setup_psql()
end

172
test/web/admin_api/admin_api_controller_test.exs
View File

@ -1292,4 +1292,176 @@ test "returns error when status is not exist", %{conn: conn} do
assert json_response(conn, :bad_request) == "Could not delete"
end
end
describe "GET /api/pleroma/admin/config" do
setup %{conn: conn} do
admin = insert(:user, info: %{is_admin: true})
%{conn: assign(conn, :user, admin)}
end
test "without any settings in db", %{conn: conn} do
conn = get(conn, "/api/pleroma/admin/config")
assert json_response(conn, 200) == %{"configs" => []}
end
test "with settings in db", %{conn: conn} do
config1 = insert(:config)
config2 = insert(:config)
conn = get(conn, "/api/pleroma/admin/config")
%{
"configs" => [
%{
"key" => key1,
"value" => _
},
%{
"key" => key2,
"value" => _
}
]
} = json_response(conn, 200)
assert key1 == config1.key
assert key2 == config2.key
end
end
describe "POST /api/pleroma/admin/config" do
setup %{conn: conn} do
admin = insert(:user, info: %{is_admin: true})
temp_file = "config/test.migrated.secret.exs"
on_exit(fn ->
Application.delete_env(:pleroma, :key1)
Application.delete_env(:pleroma, :key2)
Application.delete_env(:pleroma, :key3)
Application.delete_env(:pleroma, :key4)
Application.delete_env(:pleroma, :keyaa1)
Application.delete_env(:pleroma, :keyaa2)
:ok = File.rm(temp_file)
end)
dynamic = Pleroma.Config.get([:instance, :dynamic_configuration])
Pleroma.Config.put([:instance, :dynamic_configuration], true)
on_exit(fn ->
Pleroma.Config.put([:instance, :dynamic_configuration], dynamic)
end)
%{conn: assign(conn, :user, admin)}
end
test "create new config setting in db", %{conn: conn} do
conn =
post(conn, "/api/pleroma/admin/config", %{
configs: [
%{key: "key1", value: "value1"},
%{
key: "key2",
value: %{
"nested_1" => "nested_value1",
"nested_2" => [
%{"nested_22" => "nested_value222"},
%{"nested_33" => %{"nested_44" => "nested_444"}}
]
}
},
%{
key: "key3",
value: [
%{"nested_3" => ":nested_3", "nested_33" => "nested_33"},
%{"nested_4" => ":true"}
]
},
%{
key: "key4",
value: %{"nested_5" => ":upload", "endpoint" => "https://example.com"}
}
]
})
assert json_response(conn, 200) == %{
"configs" => [
%{
"key" => "key1",
"value" => "value1"
},
%{
"key" => "key2",
"value" => [
%{"nested_1" => "nested_value1"},
%{
"nested_2" => [
%{"nested_22" => "nested_value222"},
%{"nested_33" => %{"nested_44" => "nested_444"}}
]
}
]
},
%{
"key" => "key3",
"value" => [
[%{"nested_3" => "nested_3"}, %{"nested_33" => "nested_33"}],
%{"nested_4" => true}
]
},
%{
"key" => "key4",
"value" => [%{"endpoint" => "https://example.com"}, %{"nested_5" => "upload"}]
}
]
}
assert Application.get_env(:pleroma, :key1) == "value1"
assert Application.get_env(:pleroma, :key2) == [
nested_1: "nested_value1",
nested_2: [
[nested_22: "nested_value222"],
[nested_33: [nested_44: "nested_444"]]
]
]
assert Application.get_env(:pleroma, :key3) == [
[nested_3: :nested_3, nested_33: "nested_33"],
[nested_4: true]
]
assert Application.get_env(:pleroma, :key4) == [
endpoint: "https://example.com",
nested_5: :upload
]
end
test "update config setting & delete", %{conn: conn} do
config1 = insert(:config, key: "keyaa1")
config2 = insert(:config, key: "keyaa2")
conn =
post(conn, "/api/pleroma/admin/config", %{
configs: [
%{key: config1.key, value: "another_value"},
%{key: config2.key, delete: "true"}
]
})
assert json_response(conn, 200) == %{
"configs" => [
%{
"key" => config1.key,
"value" => "another_value"
}
]
}
assert Application.get_env(:pleroma, :keyaa1) == "another_value"
refute Application.get_env(:pleroma, :keyaa2)
end
end
end

183
test/web/admin_api/config_test.exs Normal file
View File

@ -0,0 +1,183 @@
defmodule Pleroma.Web.AdminAPI.ConfigTest do
use Pleroma.DataCase, async: true
import Pleroma.Factory
alias Pleroma.Web.AdminAPI.Config
test "get_by_key/1" do
config = insert(:config)
insert(:config)
assert config == Config.get_by_key(config.key)
end
test "create/1" do
{:ok, config} = Config.create(%{key: "some_key", value: "some_value"})
assert config == Config.get_by_key("some_key")
end
test "update/1" do
config = insert(:config)
{:ok, updated} = Config.update(config, %{value: "some_value"})
loaded = Config.get_by_key(config.key)
assert loaded == updated
end
test "update_or_create/1" do
config = insert(:config)
key2 = "another_key"
params = [
%{key: key2, value: "another_value"},
%{key: config.key, value: "new_value"}
]
assert Repo.all(Config) |> length() == 1
Enum.each(params, &Config.update_or_create(&1))
assert Repo.all(Config) |> length() == 2
config1 = Config.get_by_key(config.key)
config2 = Config.get_by_key(key2)
assert config1.value == Config.transform("new_value")
assert config2.value == Config.transform("another_value")
end
test "delete/1" do
config = insert(:config)
{:ok, _} = Config.delete(config.key)
refute Config.get_by_key(config.key)
end
describe "transform/1" do
test "string" do
binary = Config.transform("value as string")
assert binary == :erlang.term_to_binary("value as string")
assert Config.from_binary(binary) == "value as string"
end
test "list of modules" do
binary = Config.transform(["Pleroma.Repo", "Pleroma.Activity"])
assert binary == :erlang.term_to_binary([Pleroma.Repo, Pleroma.Activity])
assert Config.from_binary(binary) == [Pleroma.Repo, Pleroma.Activity]
end
test "list of strings" do
binary = Config.transform(["string1", "string2"])
assert binary == :erlang.term_to_binary(["string1", "string2"])
assert Config.from_binary(binary) == ["string1", "string2"]
end
test "map" do
binary =
Config.transform(%{
"types" => "Pleroma.PostgresTypes",
"telemetry_event" => ["Pleroma.Repo.Instrumenter"],
"migration_lock" => ""
})
assert binary ==
:erlang.term_to_binary(
telemetry_event: [Pleroma.Repo.Instrumenter],
types: Pleroma.PostgresTypes
)
assert Config.from_binary(binary) == [
telemetry_event: [Pleroma.Repo.Instrumenter],
types: Pleroma.PostgresTypes
]
end
test "complex map with nested integers, lists and atoms" do
binary =
Config.transform(%{
"uploader" => "Pleroma.Uploaders.Local",
"filters" => ["Pleroma.Upload.Filter.Dedupe"],
"link_name" => ":true",
"proxy_remote" => ":false",
"proxy_opts" => %{
"redirect_on_failure" => ":false",
"max_body_length" => "i:1048576",
"http" => %{
"follow_redirect" => ":true",
"pool" => ":upload"
}
}
})
assert binary ==
:erlang.term_to_binary(
filters: [Pleroma.Upload.Filter.Dedupe],
link_name: true,
proxy_opts: [
http: [
follow_redirect: true,
pool: :upload
],
max_body_length: 1_048_576,
redirect_on_failure: false
],
proxy_remote: false,
uploader: Pleroma.Uploaders.Local
)
assert Config.from_binary(binary) ==
[
filters: [Pleroma.Upload.Filter.Dedupe],
link_name: true,
proxy_opts: [
http: [
follow_redirect: true,
pool: :upload
],
max_body_length: 1_048_576,
redirect_on_failure: false
],
proxy_remote: false,
uploader: Pleroma.Uploaders.Local
]
end
test "keyword" do
binary =
Config.transform(%{
"level" => ":warn",
"meta" => [":all"],
"webhook_url" => "https://hooks.slack.com/services/YOUR-KEY-HERE"
})
assert binary ==
:erlang.term_to_binary(
level: :warn,
meta: [:all],
webhook_url: "https://hooks.slack.com/services/YOUR-KEY-HERE"
)
assert Config.from_binary(binary) == [
level: :warn,
meta: [:all],
webhook_url: "https://hooks.slack.com/services/YOUR-KEY-HERE"
]
end
test "complex map with sigil" do
binary =
Config.transform(%{
federated_timeline_removal: [],
reject: [~r/comp[lL][aA][iI][nN]er/],
replace: []
})
assert binary ==
:erlang.term_to_binary(
federated_timeline_removal: [],
reject: [~r/comp[lL][aA][iI][nN]er/],
replace: []
)
assert Config.from_binary(binary) ==
[federated_timeline_removal: [], reject: [~r/comp[lL][aA][iI][nN]er/], replace: []]
end
end
end