Commit Graph

530 Commits

Author SHA1 Message Date
Alex Gleason a69b7f54f8
pipeline: broadcast deletions to all known relays 2023-09-05 21:45:30 -05:00
Alex Gleason e2b88d57d9
pipeline: process kind 5 deletion events 2023-09-05 21:24:59 -05:00
Alex Gleason 22ddc7b1a8
pipeline: remove unnecessary async keyword from trackHashtags 2023-09-05 20:35:58 -05:00
Alex Gleason 48195f02b3
pipeline: don't store event if a deletion for it exists 2023-09-05 20:31:34 -05:00
Alex Gleason 97a3478b1a
cron: change cleanupEvents to every 15 minutes 2023-09-05 17:48:50 -05:00
Alex Gleason a46381849f
db/events: delete FTS rows when deleting events 2023-09-05 17:25:54 -05:00
Alex Gleason a25d6c9755
db: pragma, enable fk constraints, enable autovacuum full 2023-09-05 17:25:14 -05:00
Alex Gleason 02049ed9d1
pipeline: fix isFresh comparison 2023-09-05 17:10:20 -05:00
Alex Gleason 5c02fd0773
db/relays: normalize url before inserting 2023-09-05 16:58:34 -05:00
Alex Gleason 2ff40c8fc5
db/events: rework tags, make event_id a FK to events again, drop value_2 and value_3 2023-09-05 16:55:16 -05:00
Alex Gleason 4708839fd6
db: improve output of migrations, exit on failure 2023-09-05 16:05:02 -05:00
Alex Gleason cdffe42cfd
db/events: delete tags during deleteFilters 2023-09-05 13:23:45 -05:00
Alex Gleason df17f62ecb
db/events: try fixing types in CI? 2023-09-05 12:31:59 -05:00
Alex Gleason aea07fd28a
db/events: fix type error 2023-09-04 22:16:43 -05:00
Alex Gleason 5d19c21f51
cron: delete old remote events every hour 2023-09-04 19:46:39 -05:00
Alex Gleason 08dd5fa4eb
db/events: fix `local: false` filter 2023-09-04 17:17:47 -05:00
Alex Gleason 8ec215402f
db/migrations: fix `down` in 003_events_admin to drop the new "admin" column, not the "relays" table (whoops) 2023-09-04 13:20:07 -05:00
Alex Gleason 25e023aaf2
db/users: add indexes on users.pubkey and users.username 2023-09-04 13:18:11 -05:00
Alex Gleason 561ae9532a
db/events: getFiltersQuery, break to multiple lines for readability 2023-09-04 13:04:20 -05:00
Alex Gleason 1f06035bf2
db/events: getFiltersQuery, improve variable names 2023-09-04 13:03:10 -05:00
Alex Gleason e377d7b3ce
events/db: add a deleteFilters function to delete all events from the local db matching the filters 2023-09-04 12:58:35 -05:00
Alex Gleason 5a3d112396
frontendConfigController: add a #d filter 2023-09-03 19:18:05 -05:00
Alex Gleason 7686371183
Implement frontend configurations 2023-09-03 18:49:45 -05:00
Alex Gleason 3c45a4a3aa
sign: simplify awaitSignedEvent (remove unnecessary conditional, rearrange error, fix crash) 2023-09-02 22:39:35 -05:00
Alex Gleason 93f06fd342
config: preserve query params in Conf.local 2023-09-02 22:38:21 -05:00
Alex Gleason b81091f5da
auth98: requireProof --> requireAdmin 2023-09-02 20:52:17 -05:00
Alex Gleason f7d74c97ca
db/users: add `admin` column 2023-09-02 20:52:02 -05:00
Alex Gleason c8a5da086e
sign: ensure the NIP-46 result matches the template 2023-09-02 20:09:28 -05:00
Alex Gleason 007565b513
eventSchema: ensure the event ID matches its hash 2023-09-02 19:40:28 -05:00
Alex Gleason 7595dfa2e5
auth98: make requireProof obtain the event over NIP-46 2023-09-02 19:33:13 -05:00
Alex Gleason c8d6389132
sign: refactor the unnecessarily complex awaitSignedEvent function 2023-09-02 18:56:42 -05:00
Alex Gleason 61f5acc937
nip98: add a dedicated nip98 module, refactor auth98.ts 2023-09-02 18:32:01 -05:00
Alex Gleason 4c41ce8c0a
@/nip05.ts --> @/utils/nip05.ts 2023-09-02 13:43:01 -05:00
Alex Gleason 4ff27ddc04
Add /api/v1/favourites 2023-09-01 13:14:27 -05:00
Alex Gleason 54481ab96b
Fix unfurl timeout 2023-08-30 21:40:52 -05:00
Alex Gleason b9fc663db4
Switch to my fork of deno-sqlite with FTS support 2023-08-30 17:29:45 -05:00
Alex Gleason 3cd6b6f3f3
db/events: index profiles in search 2023-08-30 17:09:44 -05:00
Alex Gleason e976f51d55
schemas/nostr: add `search` key to filters 2023-08-30 16:19:21 -05:00
Alex Gleason af76d8cbae
Bump kysely-deno-sqlite to v1.0.1, remove `as any` from database type 2023-08-30 16:12:39 -05:00
Alex Gleason b7b5e67118
search: refactor/cleanup searchController 2023-08-30 15:40:18 -05:00
Alex Gleason 080cfe817b
search: support account_id query param 2023-08-30 15:31:07 -05:00
Alex Gleason 799a9c1e9a
search: support searching by account_id 2023-08-30 15:22:39 -05:00
Alex Gleason 8079679f18
search: support MastoAPI params, improve performance, improve value lookup 2023-08-30 15:02:28 -05:00
Alex Gleason 675010ddec
search: fix FTS special characters, optimize search 2023-08-30 14:03:16 -05:00
Alex Gleason df14ff66bc
search: search use FTS to search for statuses 2023-08-30 12:16:04 -05:00
Alex Gleason 5e9a3dd8d1
db/events: only index kind 1 events in search 2023-08-30 12:07:49 -05:00
Alex Gleason 02e1a4ce58
db/events: support "search" filter 2023-08-30 12:04:45 -05:00
Alex Gleason 9b67d7a7c0
Add followers/following endpoints
Fixes https://gitlab.com/soapbox-pub/ditto/-/issues/69
2023-08-30 10:27:45 -05:00
Alex Gleason 728f0bd657
accounts: followers/following & status counts
Fixes https://gitlab.com/soapbox-pub/ditto/-/issues/68
2023-08-30 09:50:51 -05:00
Alex Gleason b27b84e3fb
Add favourited_by and reblogged_by controllers 2023-08-29 18:08:00 -05:00
Alex Gleason 9b761ff960
firehose: apparently the `since` filter is essential 2023-08-29 16:40:23 -05:00
Alex Gleason ad823e587a
accounts: remove unecessary sort call 2023-08-29 15:29:12 -05:00
Alex Gleason d01dbcbfee
relay: add support for NIP-45 COUNT 2023-08-29 15:24:07 -05:00
Alex Gleason fc1ed59002
firehose: use `limit: 0` instead of `since` 2023-08-29 14:53:12 -05:00
Alex Gleason 9d714b6173
Pass viewerPubkey to toStatus in more places 2023-08-29 14:48:28 -05:00
Alex Gleason 2ee29bf1e2
Make favourites stick, refactor some async logic in threads 2023-08-29 14:44:59 -05:00
Alex Gleason d4612d5f21
toStatus: make status counters work (local db only) 2023-08-29 14:13:41 -05:00
Alex Gleason 95761e2eef
schema: add comments 2023-08-29 13:28:16 -05:00
Alex Gleason 2841d4f399
queries: remove unused import 2023-08-29 13:25:58 -05:00
Alex Gleason 77b09baa8c
db/events: don't throw on duplicate events 2023-08-29 13:20:21 -05:00
Alex Gleason ebd933126a
webfinger: fix import order 2023-08-29 13:14:03 -05:00
Alex Gleason d21ec6d241
apps: use AppController type 2023-08-29 13:04:38 -05:00
Alex Gleason 4d211d637e
timelines: add DRY renderStatuses function 2023-08-29 13:01:47 -05:00
Alex Gleason 4216a7931a
Add `paginated` helper function, DRY pagination code 2023-08-29 12:55:00 -05:00
Alex Gleason ce4a330812
Rename timeline controllers, homeController --> homeTimelineController, etc 2023-08-29 12:41:14 -05:00
Alex Gleason ccb8c534ca
Add hashtag timeline 2023-08-28 19:51:21 -05:00
Alex Gleason a075c533e6
Make notifications kind of work 2023-08-28 15:56:38 -05:00
Alex Gleason 2029c73eab
requireAuth --> requirePubkey 2023-08-28 15:38:32 -05:00
Alex Gleason dfbad6b2e7
Add boilerplate notifications controller, fix import order in app.ts 2023-08-28 14:23:27 -05:00
Alex Gleason e18aa878bb
streaming: reorder tag params 2023-08-28 13:37:15 -05:00
Alex Gleason e6d1494a10
streaming: make user stream mostly work, in a kind of hacky way 2023-08-28 13:34:15 -05:00
Alex Gleason e8a7dfef2b
Remove uneeded session ID from auth token 2023-08-28 13:00:00 -05:00
Alex Gleason f25284daa9
streaming: support hashtag timeline 2023-08-28 12:42:28 -05:00
Alex Gleason 2e8b26cf4f
Remove legacy stream.ts module 2023-08-27 10:07:13 -05:00
Alex Gleason 320d2f493e
change .positive() to .nonnegative() !!! 2023-08-26 23:13:48 -05:00
Alex Gleason 0a9b53bbc1
sign: fix parsing connect response from string to JSON 2023-08-26 22:49:32 -05:00
Alex Gleason 655004e775
Rework web signer to use NIP-46 events 2023-08-26 18:03:59 -05:00
Alex Gleason 1806cf2286
filter: treat the admin pubkey as local 2023-08-26 17:31:52 -05:00
Alex Gleason 9cd1ca1861
streaming: don't forget to close the subscription when the socket closes 2023-08-26 15:57:51 -05:00
Alex Gleason c13b7f4af7
subs: allow any object in place of the socket 2023-08-26 15:55:16 -05:00
Alex Gleason 67bba508af
utils/web: make `tags` optional 2023-08-26 15:22:49 -05:00
Alex Gleason 1b2a486c65
pipeline: don't store ephemeral events 2023-08-26 13:40:10 -05:00
Alex Gleason 2aefdc4bd1
kinds: improve detection of legacy kinds 2023-08-26 13:25:32 -05:00
Alex Gleason fdc39297fd
Add kinds module to classify events by kind 2023-08-26 13:18:58 -05:00
Alex Gleason 63def1d62c
utils: add return types (to improve readability) 2023-08-26 12:58:17 -05:00
Alex Gleason 7570b0dee4
utils: new Date().getTime() --> Date.now() 2023-08-26 12:56:07 -05:00
Alex Gleason e17111a859
utils/web: PendingEvent --> EventStub 2023-08-26 12:52:24 -05:00
Alex Gleason 60cecafdb5
Remove admin.ts, move to sign.ts, add createAdminEvent function 2023-08-26 12:48:08 -05:00
Alex Gleason 3c279175bc
instance: actually, put all this under a nostr key 2023-08-26 12:28:25 -05:00
Alex Gleason baf4c00fee
instance: add urls.nostr_relay and pubkey properties to instance 2023-08-26 12:01:46 -05:00
Alex Gleason 887c68f052
config: add comments to all config options 2023-08-26 11:55:16 -05:00
Alex Gleason 9da03aa053
streaming: socket.addEventListener('open', ...) --> socket.onopen, etc 2023-08-25 17:17:26 -05:00
Alex Gleason a93d77c851
streaming: add `stream` param to streaming events 2023-08-25 16:17:45 -05:00
Alex Gleason b60e84d29b
relay: only send if socket is open 2023-08-25 15:09:16 -05:00
Alex Gleason f7cd67c572
Make MastoAPI streaming work for public feeds 2023-08-25 14:59:37 -05:00
Alex Gleason d1117f5513
relay: improve variable names in handleReq 2023-08-25 13:42:58 -05:00
Alex Gleason c18d7b952f
Move filter types into filter.ts 2023-08-25 13:38:21 -05:00
Alex Gleason baace5ea2d
Refactor streaming to use async iterators 2023-08-25 13:35:20 -05:00
Alex Gleason 00c531bbff
relay: close sub when socket is closed 2023-08-24 18:43:14 -05:00
Alex Gleason f9de6932ac
utils: remove unused import 2023-08-24 17:42:28 -05:00
Alex Gleason 2f7914f044
pipeline: skip previously encountered events 2023-08-24 17:39:24 -05:00
Alex Gleason a0dff12ca0
pipeline: don't stream events older than 10 seconds 2023-08-24 17:26:46 -05:00
Alex Gleason f1c465beea
pipeline: optimize database calls 2023-08-24 17:16:44 -05:00
Alex Gleason 658dd397f5
relay: respect "local: true" filter 2023-08-24 15:28:13 -05:00
Alex Gleason a676b71d23
relay: make Nostr streaming work 2023-08-23 23:25:38 -05:00
Alex Gleason bcfc3e2414
Use createEvent in more places 2023-08-19 22:25:45 -05:00
Alex Gleason a82ae40c43
db/events: don't add tags to query unless they exist 2023-08-19 15:26:41 -05:00
Alex Gleason 98b6a4b222
Increase NIP-05 lookup timeout to 2s 2023-08-19 14:56:40 -05:00
Alex Gleason f198779ebe Merge branch 'sync' into 'develop'
Sync the user on verify_credentials

Closes #11

See merge request soapbox-pub/ditto!17
2023-08-19 19:09:09 +00:00
Alex Gleason 23928c0024
Move hono-related functions until utils/web 2023-08-19 14:05:08 -05:00
Alex Gleason eb4aebedd7
Sync the user on verify_credentials
Fixes https://gitlab.com/soapbox-pub/ditto/-/issues/11
2023-08-19 13:57:17 -05:00
Alex Gleason c08c801e71
Make following work 2023-08-19 13:37:44 -05:00
Alex Gleason b087d08306
Make the local timeline correctly filter only local users 2023-08-19 13:01:05 -05:00
Alex Gleason c37dd2c2b5
db/events: skip undefined filter values 2023-08-19 12:59:22 -05:00
Alex Gleason fecd69bf8f
mixer: don't use the client pool for local filters 2023-08-19 12:38:20 -05:00
Alex Gleason 05a7a773fe
getFilters: return immediately if filters are empty 2023-08-19 12:37:42 -05:00
Alex Gleason 719b703e7c
client: remove `publish` function, create statuses through pipeline 2023-08-17 20:54:11 -05:00
Alex Gleason 0158a6979e
Publish events through pipeline 2023-08-17 20:45:50 -05:00
Alex Gleason 85345bc157
firehose: catch pipeline crash 2023-08-17 20:28:23 -05:00
Alex Gleason 0f70f5c76f
Add a RelayError class, improve relay error handling 2023-08-17 20:24:16 -05:00
Alex Gleason dc49c305bd
Move isLocallyFollowed to queries.ts 2023-08-17 19:32:05 -05:00
Alex Gleason 2011ca6e1d
pipeline: remove redundant console.log 2023-08-17 18:08:05 -05:00
Alex Gleason 45abaf14a4
pipeline: refactor, use pipeline from relay 2023-08-17 18:07:25 -05:00
Alex Gleason 51881efde0
Push events from the client through the pipeline 2023-08-17 16:48:58 -05:00
Alex Gleason 87c67c0a39
Add pipeline module 2023-08-17 16:47:22 -05:00
Alex Gleason 299a1a0db0
Refactor some code that uses getFilters to import the whole module under a namespace 2023-08-17 15:41:27 -05:00
Alex Gleason 4f9b5c1431
statusController: remove unnecessary "as Event" 2023-08-17 14:57:46 -05:00
Alex Gleason 4f37a1b9f8
queries: don't unnecessarily sort things 2023-08-17 14:55:00 -05:00
Alex Gleason 7330cd10e2
Refactor getEvent to take an opts object 2023-08-17 14:51:49 -05:00
Alex Gleason 054a399f23
queries: reduce getFollows timeout to 1s 2023-08-17 13:19:36 -05:00
Alex Gleason c516f007a5
queries: remove unnecessary "as Event" assertions 2023-08-17 13:18:46 -05:00
Alex Gleason 5a27d791e3
queries: make getFeed accept a pubkey instead of event3 2023-08-17 13:15:47 -05:00
Alex Gleason 870a6f5261
Add queries module for getting events for certain application needs 2023-08-17 13:12:38 -05:00
Alex Gleason a35bc1eef3
db/events: use opts.limit 2023-08-16 21:58:46 -05:00
Alex Gleason c15194020f
Delete event.ts, use nostr-tools types 2023-08-16 21:53:51 -05:00
Alex Gleason 5515c40df3
Remove deprecated getFilter (singular) functions 2023-08-16 21:41:18 -05:00
Alex Gleason 96641a6fa0
Move limit to GetFilterOpts 2023-08-16 21:38:21 -05:00
Alex Gleason bbaf1cb012
mixer: pass opts through to all getFilters functions 2023-08-16 21:21:29 -05:00
Alex Gleason d4721fb82d
Add mixer module to combine results from DB and pool 2023-08-16 16:12:27 -05:00
Alex Gleason 59b7a3eed8
db/events: perform multiple filters with one union query, greatly simplify logic 2023-08-16 15:32:02 -05:00
Alex Gleason 4602b85afe
db/events: remove unnecessary overloads 2023-08-16 12:57:02 -05:00
Alex Gleason 8f45f3a7ad
Move DittoFilter to @/types.ts, refactor db/events query 2023-08-16 12:54:17 -05:00
Alex Gleason 882a3fe203
Use only "active" relays in the pool 2023-08-16 08:28:52 -05:00
Alex Gleason a0769d7c92
db/relays: add `active` column 2023-08-15 19:08:38 -05:00
Alex Gleason 92f5bea891
db/relays: add "domain" column 2023-08-15 18:59:49 -05:00
Alex Gleason 2e983154bb
db/relays: reformat 2023-08-14 20:41:20 -05:00
Alex Gleason 3da30553cb
firehose: accept kind 10002 events 2023-08-14 19:16:49 -05:00
Alex Gleason 875fcfb09d
firehose: nown --> known 2023-08-14 19:01:56 -05:00
Alex Gleason 45a3e2974e
firehose: expand criteria, improve logging 2023-08-14 19:01:28 -05:00
Alex Gleason b11677573b
db/relays: skip if values are empty 2023-08-14 19:00:54 -05:00
Alex Gleason e3ade42f58
Added script to populate relays from external source 2023-08-14 18:46:35 -05:00
Alex Gleason f13616a740
Bump kysely-deno-sqlite to v1.0.0 (no changes) 2023-08-14 18:27:30 -05:00
Alex Gleason 3b4bfb1d90
Merge remote-tracking branch 'origin/develop' into firehose 2023-08-14 18:25:28 -05:00
Alex Gleason 6c944d40f1
Bump deno-safe-fetch to v1.0.0 2023-08-14 18:13:43 -05:00
Alex Gleason 3a2722e190
Merge remote-tracking branch 'origin/develop' into firehose 2023-08-14 18:06:27 -05:00
Alex Gleason 1ed6fa54df
Use safeFetch implementation 2023-08-14 18:03:24 -05:00
Alex Gleason cd5cf96886
firehose: update comment 2023-08-14 15:39:21 -05:00
Alex Gleason 16ba224124
Get rid of DITTO_RELAY, derive from LOCAL_DOMAIN 2023-08-14 14:40:30 -05:00
Alex Gleason 78f638e633
Add relays to database and start tracking them 2023-08-14 14:11:28 -05:00
Alex Gleason 4a47c25f1e
Add NIP-11 schema 2023-08-14 12:29:54 -05:00
Alex Gleason 1b8bb78717
Rename loopback to firehose 2023-08-14 11:02:09 -05:00
Alex Gleason 007f464d31
db/events: fix import order 2023-08-12 20:04:19 -05:00
Alex Gleason 9da4fb2bba
db/events: add comments 2023-08-12 15:45:58 -05:00
Alex Gleason 4c8a685528
relay: allow local users to post to the relay 2023-08-12 15:24:33 -05:00
Alex Gleason a35ea6ab5d
relay: restrict to local events unless the filter is already narrow 2023-08-12 15:14:43 -05:00
Alex Gleason 8e47c9dda2
relay: refactor into smaller functions 2023-08-12 15:07:07 -05:00
Alex Gleason b2f538ed94
Relay: improve types, DRY 2023-08-12 14:41:07 -05:00
Alex Gleason 075da543b0
Make relay only return local events 2023-08-12 14:32:57 -05:00
Alex Gleason 3593d5420d
Relay: limit to 100 events per filter 2023-08-12 14:12:53 -05:00
Alex Gleason b852111ec5
Fix getFilters overload order 2023-08-12 13:57:20 -05:00
Alex Gleason 808e8941b6
Relay: make REQ work (doesn't stream yet) 2023-08-12 13:40:21 -05:00
Alex Gleason e2adc7ad1a
Remove unused MetaContent type 2023-08-12 11:49:33 -05:00
Alex Gleason e999d693d0
Rename hexIdSchema back to nostrIdSchema 2023-08-12 11:48:49 -05:00
Alex Gleason 80775d8bf0
Move more Nostr schema stuff into schemas/nostr.ts 2023-08-12 11:48:11 -05:00
Alex Gleason 893542cf58
Reorganize some nostr schema code 2023-08-12 11:30:57 -05:00
Alex Gleason 1f470ffe2d
Add nostr schema for parsing filters 2023-08-11 19:55:16 -05:00
Alex Gleason 9cca801c80
Order tag conditionals alphabetically, improve logic 2023-08-11 12:14:39 -05:00
Alex Gleason 48343c8035
Add events test 2023-08-10 14:56:27 -05:00
Alex Gleason 63772022cf
deno fmt 2023-08-10 14:56:14 -05:00
Alex Gleason 5f37a68b9f
Bump std to v0.198.0 2023-08-10 14:08:47 -05:00
Alex Gleason 14eb3cb43b
Let DB_PATH be configurable 2023-08-10 13:37:56 -05:00
Alex Gleason 8ebd85b760
Improve tag indexing logic 2023-08-09 20:34:21 -05:00
Alex Gleason f127aa7406
Also track events from local users 2023-08-09 14:24:49 -05:00
Alex Gleason 3b3947ea61
Only track events which are locally followed 2023-08-09 14:06:57 -05:00
Alex Gleason 2d2157293c
Try this? 2023-08-09 02:03:18 -05:00
Alex Gleason a1c8d3352f
Simplify 2023-08-09 02:01:27 -05:00
Alex Gleason ad112ff9a6
Use explicit return type for getFilterQuery 2023-08-09 01:53:50 -05:00
Alex Gleason 55c957d10b
Fix type error 2023-08-09 01:28:38 -05:00
Alex Gleason 9671a77bee
Add isFollowed function 2023-08-09 01:18:43 -05:00
Alex Gleason 0bfd9d83c1
Add getFilters function 2023-08-09 00:22:20 -05:00
Alex Gleason b408838ae0
Support querying by tags 2023-08-08 23:54:17 -05:00
Alex Gleason 6c96240602
Remove foreign key constraint from tags.event_id 2023-08-08 22:35:43 -05:00
Alex Gleason cd68da5b93
Minor refactoring 2023-08-08 19:31:14 -05:00
Alex Gleason 295b16e943
Add a very nice getFilter function 2023-08-08 15:35:37 -05:00
Alex Gleason 97a250c1f7
Add kysely migrations 2023-08-07 01:47:58 -05:00
Alex Gleason eb78a213f6
deno lint 2023-08-07 00:51:11 -05:00
Alex Gleason 4cbdda401a
Remove unused builder.ts 2023-08-07 00:50:31 -05:00
Alex Gleason 3cb5f91d3b
Refactor db.ts to use kysely statements 2023-08-07 00:50:12 -05:00
Alex Gleason ecc9db86dd
Import kysely-deno-sqlite as a dep 2023-08-06 23:30:46 -05:00
Alex Gleason 7c2f290775
Refactor kysely lib into separate files 2023-08-06 21:57:26 -05:00