Merge pull request #458 from NuSkooler/webserver-improvements

Bump version to 0.0.14-beta + major web server updates

UPGRADE.md

@@ -30,7 +30,12 @@ npm install # or simply 'yarn'
 1. Check [TROUBLESHOOTING](TROUBLESHOOTING.md) first.
 2. Report your issue on Xibalba BBS, hop in #`enigma-bbs` on FreeNode and chat, or [file a issue on GitHub](https://github.com/NuSkooler/enigma-bbs/issues) if you believe you've found a bug or missing feature.
 
-# 0.0.12-beta to 0.0.13-beta
+# Version to Version Notes
+> :warning: Be sure to inspect these notes during any upgrades!
+
+## 0.0.13-beta to 0.0.14-beta
+
+## 0.0.12-beta to 0.0.13-beta
 * To enable the new Waiting for Caller (WFC) support, please see [WFC](docs/modding/wfc.md).
 * :exclamation: The SSH server's `ssh2` module has gone through a major upgrade. Existing users will need to comment out two SSH KEX algorithms from their `config.hjson` if present else clients such as NetRunner will not be able to connect over SSH. Comment out `diffie-hellman-group-exchange-sha256` and `diffie-hellman-group-exchange-sha1`
 * Gopher configuration change. See [WHATSNEW](WHATSNEW.md)
@@ -60,7 +65,7 @@ npm install # or simply 'yarn'
 
 In addition to these, there are also new options for `term.cp437TermList` and `term.utf8TermList`. Under most circumstances these should not need to be changed. If you want to customize these lists, more information is available in `config_default.js`
 
-# 0.0.11-beta to 0.0.12-beta
+## 0.0.11-beta to 0.0.12-beta
 * Be aware that `master` is now mainline! This means all `git pull`'s will yield the latest version. See [WHATSNEW](WHATSNEW.md) for more information.
 * **BREAKING CHANGE** There is no longer a `prompt.hjson` file. Prompts are now simply part of the menu set in the `prompts` section. If you have an existing system you will need to add your `prompt.hjson` to your `menu.hjson`'s `includes` section at a minimum. Example:
 ```hjson
@@ -79,14 +84,14 @@ In addition to these, there are also new options for `term.cp437TermList` and `t
 sqlite3 db/file.sqlite3 < ./misc/update/tables_update_2020-11-29.sql
 ```
 
-# 0.0.10-alpha to 0.0.11-beta
+## 0.0.10-alpha to 0.0.11-beta
 * Node.js 12.x LTS is now in use. Follow standard Node.js upgrade procedures (e.g.: `nvm install 12 && nvm use 12`).
 
-# 0.0.9-alpha to 0.0.10-alpha
+## 0.0.9-alpha to 0.0.10-alpha
 * Security related files such as private keys and certs are now looked for in `config/security` by default.
 * Default archive handler for zip files has switched to InfoZip due to a bug in the latest p7Zip packages causing "volume not found" errors. Ensure you have the InfoZip `zip` and `unzip` commands in ENiGMA's path. You can switch back to 7Zip by overriding `archiveHandler` for `application/zip` in your `config.hjson` under `fileTypes` to `7Zip`.
 
-# 0.0.8-alpha to 0.0.9-alpha
+## 0.0.8-alpha to 0.0.9-alpha
 * Development is now against Node.js 10.x LTS. Follow your standard upgrade path to update to Node 10.x before using 0.0.9-alpha!
 * The property `justify` found on various views previously had `left` and `right` values swapped (oops!); you will need to adjust any custom `theme.hjson` that use one or the other and swap them as well.
 * Possible breaking changes in FSE: The MCI code `%TL13` for error indicator is now `%TL4`. This is part of a cleanup and standardization on "custom ranges". You may need to update your `theme.hjson` and related artwork.
@@ -113,7 +118,7 @@ webSocket: {
 * Similar to the last item, `defaults.general.passwordChar` in `theme.hjson` is now just `defaults.passwordChar`.
 
 
-# 0.0.7-alpha to 0.0.8-alpha
+## 0.0.7-alpha to 0.0.8-alpha
 ENiGMA 0.0.8-alpha comes with some structure changes:
 * Configuration files are defaulted to `./config`. Related, the `--config` option now points to a configuration **directory**
 * `./mods/art` has been moved to `./art/general`
@@ -130,17 +135,17 @@ With the above changes, you'll need to to at least:
 * Move any certificates, pub/private keys, etc. from `./misc` to `./config`
 * Specify user modules as `@userModule:my_module_name`
 
-# 0.0.6-alpha to 0.0.7-alpha
+## 0.0.6-alpha to 0.0.7-alpha
 No issues
 
-# 0.0.5-alpha to 0.0.6-alpha
+## 0.0.5-alpha to 0.0.6-alpha
 No issues
 
-# 0.0.4-alpha to 0.0.5-alpha
+## 0.0.4-alpha to 0.0.5-alpha
 No issues
 
-# 0.0.1-alpha to 0.0.4-alpha
+## 0.0.1-alpha to 0.0.4-alpha
-## Node.js 6.x+ LTS is now **required**
+### Node.js 6.x+ LTS is now **required**
 You will need to upgrade Node.js to [6.x+](https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V6.md). If using [nvm](https://github.com/creationix/nvm) (you should be!) the process will go something like this:
 ```bash
 nvm install 6
@@ -150,7 +155,7 @@ nvm alias default 6
 ### ES6
 Newly written code will use ES6 and a lot of code has started the migration process. Of note is the `MenuModule` class. If you have created a mod that inherits from `MenuModule`, you will need to upgrade your class to ES6.
 
-## Manual Database Upgrade
+### Manual Database Upgrade
 A few upgrades need to be made to your SQLite databases:
 
 ```bash
@@ -159,8 +164,8 @@ sqlite3 db/message.sqlite
 sqlite> INSERT INTO message_fts(message_fts) VALUES('rebuild');
 ```
 
-## Archiver Changes
+### Archiver Changes
 If you have overridden or made additions to archivers in your `config.hjson` you will need to update them. See [Archive Configuration](docs/archive.md) and `core/config.js`
 
-## File Base Configuration
+### File Base Configuration
 As 0.0.4-alpha contains file bases, you'll want to create a suitable configuration if you wish to use the feature. See [File Base Configuration](docs/file_base.md).

WHATSNEW.md

@@ -1,6 +1,13 @@
 # Whats New
 This document attempts to track **major** changes and additions in ENiGMA½. For details, see GitHub.
 
+## 0.0.14-beta
+* The [Web Server](/docs/_docs/servers/contentservers/web-server.md) has made some possibly breaking changes:
+    * `/static/` prefixes are no longer required. This was a ugly hack.
+    * Some internal routes such as those used for password resets live within `/_internal/`.
+    * Routes for the file base now default to `/_f/` prefixed instead of just `/f/`. If `/f/` is in your `config.hjson` you are encouraged to update it!
+    * Finally, the system will search for `index.html` and `index.htm` in that order, if another suitable route cannot be established.
+
 ## 0.0.13-beta
 * **Note for contributors**: ENiGMA has switched to [Prettier](https://prettier.io) for formatting/style. Please see [CONTRIBUTING](CONTRIBUTING.md) and the Prettier website for more information.
 * Removed terminal `cursor position reports` from most locations in the code. This should greatly increase the number of terminal programs that work with Enigma 1/2. For more information, see [Issue #222](https://github.com/NuSkooler/enigma-bbs/issues/222). This may also resolve other issues, such as [Issue #365](https://github.com/NuSkooler/enigma-bbs/issues/365), and [Issue #320](https://github.com/NuSkooler/enigma-bbs/issues/320). Anyone that previously had terminal incompatibilities please re-check and let us know!

core/config_default.js

@@ -945,8 +945,10 @@ module.exports = () => {
             ],
 
             web: {
-                path: '/f/',
+                //  if you change the /_f/ prefix here, ensure something
-                routePath: '/f/[a-zA-Z0-9]+$',
+                //  non-colliding with other routes is utilized
+                path: '/_f/',
+                routePath: '^/_f/[a-zA-Z0-9]+$',
                 expireMinutes: 1440, //  1 day
             },
 

core/servers/content/web.js

@@ -15,6 +15,7 @@ const fs = require('graceful-fs');
 const paths = require('path');
 const mimeTypes = require('mime-types');
 const forEachSeries = require('async/forEachSeries');
+const findSeries = require('async/findSeries');
 
 const ModuleInfo = (exports.moduleInfo = {
     name: 'Web',
@@ -74,14 +75,6 @@ exports.getModule = class WebServerModule extends ServerModule {
         this.enableHttps = config.contentServers.web.https.enabled || false;
 
         this.routes = {};
-
-        if (this.isEnabled() && config.contentServers.web.staticRoot) {
-            this.addRoute({
-                method: 'GET',
-                path: '/static/.*$',
-                handler: this.routeStaticFile.bind(this),
-            });
-        }
     }
 
     buildUrl(pathAndQuery) {
@@ -210,13 +203,21 @@ exports.getModule = class WebServerModule extends ServerModule {
     }
 
     routeRequest(req, resp) {
-        const route = _.find(this.routes, r => r.matchesRequest(req));
+        let route = _.find(this.routes, r => r.matchesRequest(req));
 
-        if (!route && '/' === req.url) {
+        if (route) {
-            return this.routeIndex(req, resp);
+            return route.handler(req, resp);
+        } else {
+            this.tryStaticRoute(req, resp, wasHandled => {
+                if (!wasHandled) {
+                    this.tryRouteIndex(req, resp, wasHandled => {
+                        if (!wasHandled) {
+                            return this.fileNotFound(resp);
+                        }
+                    });
+                }
+            });
         }
-
-        return route ? route.handler(req, resp) : this.accessDenied(resp);
     }
 
     respondWithError(resp, code, bodyText, title) {
@@ -256,27 +257,57 @@ exports.getModule = class WebServerModule extends ServerModule {
         return this.respondWithError(resp, 404, 'File not found.', 'File Not Found');
     }
 
-    routeIndex(req, resp) {
+    tryRouteIndex(req, resp, cb) {
-        const filePath = paths.join(Config().contentServers.web.staticRoot, 'index.html');
+        const tryFiles = Config().contentServers.web.tryFiles || [
-        return this.returnStaticPage(filePath, resp);
+            'index.html',
+            'index.htm',
+        ];
+
+        findSeries(
+            tryFiles,
+            (tryFile, nextTryFile) => {
+                const fileName = paths.join(
+                    req.url.substr(req.url.lastIndexOf('/', 1)),
+                    tryFile
+                );
+                const filePath = this.resolveStaticPath(fileName);
+
+                fs.stat(filePath, (err, stats) => {
+                    if (err || !stats.isFile()) {
+                        return nextTryFile(null, false);
+                    }
+
+                    const headers = {
+                        'Content-Type':
+                            mimeTypes.contentType(paths.basename(filePath)) ||
+                            mimeTypes.contentType('.bin'),
+                        'Content-Length': stats.size,
+                    };
+
+                    const readStream = fs.createReadStream(filePath);
+                    resp.writeHead(200, headers);
+                    readStream.pipe(resp);
+
+                    return nextTryFile(null, true);
+                });
+            },
+            (_, wasHandled) => {
+                return cb(wasHandled);
+            }
+        );
     }
 
-    routeStaticFile(req, resp) {
+    tryStaticRoute(req, resp, cb) {
-        const fileName = req.url.substr(req.url.indexOf('/', 1));
+        const fileName = req.url.substr(req.url.lastIndexOf('/', 1));
         const filePath = this.resolveStaticPath(fileName);
-        return this.returnStaticPage(filePath, resp);
-    }
-
-    returnStaticPage(filePath, resp) {
-        const self = this;
 
         if (!filePath) {
-            return this.fileNotFound(resp);
+            return cb(false);
         }
 
         fs.stat(filePath, (err, stats) => {
             if (err || !stats.isFile()) {
-                return self.fileNotFound(resp);
+                return cb(false);
             }
 
             const headers = {
@@ -288,7 +319,9 @@ exports.getModule = class WebServerModule extends ServerModule {
 
             const readStream = fs.createReadStream(filePath);
             resp.writeHead(200, headers);
-            return readStream.pipe(resp);
+            readStream.pipe(resp);
+
+            return cb(true);
         });
     }
 

core/user_2fa_otp_web_register.js

@@ -76,7 +76,7 @@ module.exports = class User2FA_OTPWebRegister {
                 (token, textTemplate, htmlTemplate, callback) => {
                     const webServer = getWebServer();
                     const registerUrl = webServer.instance.buildUrl(
-                        `/enable_2fa_otp?token=${token}&otpType=${otpType}`
+                        `/_internal/enable_2fa_otp?token=${token}&otpType=${otpType}`
                     );
 
                     const replaceTokens = s => {
@@ -170,7 +170,7 @@ module.exports = class User2FA_OTPWebRegister {
                     return User2FA_OTPWebRegister.accessDenied(webServer, resp);
                 }
 
-                const postUrl = webServer.instance.buildUrl('/enable_2fa_otp');
+                const postUrl = webServer.instance.buildUrl('/_internal/enable_2fa_otp');
                 const config = Config();
                 return webServer.instance.routeTemplateFilePage(
                     _.get(config, 'users.twoFactorAuth.otp.registerPageTemplate'),
@@ -296,12 +296,12 @@ ${backupCodes}
         [
             {
                 method: 'GET',
-                path: '^\\/enable_2fa_otp\\?token\\=[a-f0-9]+&otpType\\=[a-zA-Z0-9_]+$',
+                path: /^\/_internal\/enable_2fa_otp\?token=[a-f0-9]+&otpType=[a-zA-Z0-9_]+$/,
                 handler: User2FA_OTPWebRegister.routeRegisterGet,
             },
             {
                 method: 'POST',
-                path: '^\\/enable_2fa_otp$',
+                path: /^\/_internal\/enable_2fa_otp$/,
                 handler: User2FA_OTPWebRegister.routeRegisterPost,
             },
         ].forEach(r => {

core/web_password_reset.js

@@ -121,7 +121,7 @@ class WebPasswordReset {
                     const sendMail = require('./email.js').sendMail;
 
                     const resetUrl = webServer.instance.buildUrl(
-                        `/reset_password?token=${
+                        `/_internal/reset_password?token=${
                             user.properties[UserProps.EmailPwResetToken]
                         }`
                     );
@@ -194,13 +194,13 @@ class WebPasswordReset {
             {
                 //  this is the page displayed to user when they GET it
                 method: 'GET',
-                path: '^\\/reset_password\\?token\\=[a-f0-9]+$', //  Config.contentServers.web.forgotPasswordPageTemplate
+                path: /^\/_internal\/reset_password\?token=[a-f0-9]+$/,
                 handler: WebPasswordReset.routeResetPasswordGet,
             },
             //  POST handler for performing the actual reset
             {
                 method: 'POST',
-                path: '^\\/reset_password$',
+                path: /^\/_internal\/reset_password$/,
                 handler: WebPasswordReset.routeResetPasswordPost,
             },
         ].forEach(r => {
@@ -269,7 +269,7 @@ class WebPasswordReset {
                 );
             }
 
-            const postResetUrl = webServer.instance.buildUrl('/reset_password');
+            const postResetUrl = webServer.instance.buildUrl('/_internal/reset_password');
 
             const config = Config();
             return webServer.instance.routeTemplateFilePage(

docs/_docs/servers/contentservers/web-server.md

@@ -8,7 +8,7 @@ ENiGMA½ comes with a built in *content server* for supporting both HTTP and HTT
 
 By default the web server is not enabled. To enable it, you will need to at a minimum configure two keys in the `contentServers.web` section of `config.hjson`:
 
-```hjson
+```js
 contentServers: {
     web: {
         domain: bbs.yourdomain.com
@@ -58,15 +58,7 @@ If you don't have a TLS certificate for your domain, a good source for a certifi
 > :information_source: Keep in mind that the SSL certificate provided by Let's Encrypt's Certbot is by default stored in a privileged location; if your ENIGMA instance is not running as root (which it should not be!), you'll need to copy the SSL certificate somewhere else in order for ENIGMA to use it.
 
 ## Static Routes
-
+Static files live relative to the `contentServers.web.staticRoot` path which defaults to `enigma-bbs/www`. This is also commonly known as your "public root".
-Static files live relative to the `contentServers.web.staticRoot` path which defaults to `enigma-bbs/www`.
-
-`index.html, favicon.ico`, and any error pages like `404.html` are accessible from the route path. Other static assets hosted by the web server must be referenced from `/static/`, for example:
-
-```html
-<a href="/static/about.html"> Example Link
-```
 
 ## Custom Error Pages
-
 Customized error pages can be created for [HTTP error codes](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_Error) by providing a `<error_code>.html` file in the *static routes* area. For example: `404.html`.

package.json

@@ -1,6 +1,6 @@
 {
     "name": "enigma-bbs",
-    "version": "0.0.13-beta",
+    "version": "0.0.14-beta",
     "description": "ENiGMA½ Bulletin Board System",
     "author": "Bryan Ashby <bryan@l33t.codes>",
     "license": "BSD-2-Clause",