Meine Güte, was für'n Blah!

9 Suchtreffer

2020-06-13

“Master” branch and racism

For some time there are discussions about terms like “master” and “slave” in IT. The terms were used there e.g. to describe the roles of hard-disks quite some time ago and “master” is used by git to name the default branch of a software project. While the former is outdated due to technical progress the latter it still in wide use – including my own projects.

The current racism debate was a good enough reason to change that. While I don’t think that the term master is inherently racist and in fact predates slavery and is used even now in many other contexts – like master/apprentice, or master in the sense of a original (book, movie, audio-track etc.), or master in terms of accomplishments, or master in context of devotion etc. – it’s not worth the time to argue about some word that might make feel some people uncomfortable. So I wrote a small shell script to automate the process as far as possible (with some help):

#!/bin/sh
#
# Change git `master` branch to `main`.
#
# 2020-06-12  Matthias Watermann  <support@mwat.de>
#-------------------------------------------------------

git branch -m master main || exit
git push -u origin main || exit
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main || exit
#
echo "
See the web-page shown above and change the default branch on GitHub to 'main'"
read -p "press <RETURN> when done." REPLY
#
git branch -d master 2>/dev/null
git push origin --delete master
git push -u origin main --tags
echo

# _EoF_

After running that script in all my private and public repositories I was finally un-mastered …

Since I didn’t find a way to programmatically change the default branch on ‘GitHub’ the script above pauses for a while until I manually changed the default branch there from master to main.

After running this script in all my git projects there was a single problem remaining: How to make sure that a new git project would start with main as its default branch?

Well, for the locally created git repositories the easiest way I found is setting a command alias:

git config --global alias.init '!git init && git symbolic-ref HEAD refs/heads/main'

Lamentably that doesn’t work. Apparently one can not ‘rename’ an already existing git command. So try it with a new command:

git config --global alias.new '!git init && git symbolic-ref HEAD refs/heads/main'

Now it works and a new git repository gets created with main as its default branch.

Now, what remains is configuring GitHub to use main as the default branch’s name. As far as I see, however, that’s not possible at the moment. Whenever a new GitHub repository is created the default branch is named master. Which means that after creating the new repository one has to create some file (e.g. .gitignore) to make sure that the branch is actually in use and only then change the default branch’s name to something else, e.g. main. – A few days later Microsoft (the now owner of GitHub) declared that they will change the master name.

As a side note: Much more important than to get rid of master branches in software development I feel it necessary to replace terms like blacklist and whitelist which clearly have a racist connotation: black==bad and white==good. However, since I personally don’t use those terms in my development there’s nothing I can do about (apart from talking to others about it).

#Computer #Git @GitHub #Language #Programming #Racism

2020-03-01

[*]

Whitespace

Purpose

Whitespace (TABulators, NewLines, and SPaces) are characters in e.g. HTML pages which don’t have syntactic significance but are just supposed to ease the reading by humans. Whether you put one space between </p> and the following <p> doesn’t make a difference as far as parsing and rendering the HTML page is concerned – it just increases the filesize and therefore the amount of bandwidth needed, the time used for transfer and interpretation, the amount of memory used in both the sending server and the receiving user/browser. In the end one could say: the more whitespace there is in your HTML pages the more expensive it is for all parties involved.

When writing Nele and Kaliber (both of which are essentially web-servers) I realised that I basically implemented the same code for removing superfluous whitespace before delivering the HTML pages to the remote user. So I extracted that code and refactored it to become the simple middleware plugin you can see here.

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/whitespace

Usage

There are two ways to use this library:

as a middleware plugin which then will be used automatically;
by calling the Remove(…) function from your code whenever it suits you.

In both cases you have to use this library by

import "github.com/mwat56/whitespace"

To use it as middleware for your web-server you call

whitespace.Wrap(aHandler http.Handler) http.Handler

where aHandler is the page-handler you implemented to handle (generate and send) your web-pages. The function’s return value can then be used to set up your http.Server instance.

If instead you want to use the library manually you can call

whitespace.Remove(aPage []byte) []byte

where aPage is the HTML page you prepared and the function’s return value is that very page with all superfluous whitespace removed.

Libraries

No external libraries were used building whitespace.

Licence

    Copyright © 2020 M.Watermann, 10247 Berlin, Germany
                    All rights reserved
                EMail : <support@mwat.de>

2019-05-23

[*]

Nele Blog

Nele Blog

Purpose

The purpose of this package/application was twofold initially. On one hand I needed a project to learn the (then to me new) Go language, and on the other hand I wanted a project, that lead me into different domains, like user authentication, configuration, data formats, error handling, filesystem access, data logging, os, network, regex, templating etc. – And, I wanted no external dependencies (like databases etc.). – And, I didn’t care for Windows™ compatibility since I left the MS-platform about 25 years ago after using it in the 80s and early 90s of the last century. (But who, in his right mind, would want to run a web-service on such a platform anyway?)

That’s how I ended up with this little blog-system (for lack of a better word; or: diary, notes, …). It’s a system that lets you write and add articles from both the command line and a web-interface. It provides options to add, modify and delete entries using a user/password list for authentication when accessing certain URLs in this system. Articles can be added, edited (e.g. for correcting typos etc.), or removed altogether. If you don’t like the styles coming with the package you can, of course, change them according to your preferences in your own installation.

The articles you write are then available on the net as web-pages.

It is not, however, a discussion platform. It’s supposed to be used as a publication platform, not some kind of social media. So I intentionally didn’t bother with comments or discussion threading.

Features

Markdown support
Multiple user accounts supported
No database (like SQLite, MariaDB, etc.) required
No JavaScript dependency
No cookies needed
Privacy aware
Simplicity of use

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/nele

Usage

After downloading this package you go to its directory and compile

go build app/nele.go

which should produce an executable binary. On my system it looked (at a certain point in time) like this:

$ ls -l
total 11420
drwxrwxr-x 12 matthias matthias     4096 Mai 23 18:35 .
drwxrwxr-x 12 matthias matthias     4096 Mai 23 17:58 ..
-rw-rw-r--  1 matthias matthias      474 Apr 27 00:21 addTest.md
drwxrwxr-x  3 matthias matthias     4096 Mai 23 18:14 app
drwxrwxr-x  2 matthias matthias     4096 Mai 23 18:14 certs
-rw-rw-r--  1 matthias matthias     6583 Mai 23 18:14 cmdline.go
-rw-rw-r--  1 matthias matthias    10149 Mai 23 18:20 config.go
-rw-rw-r--  1 matthias matthias     1846 Mai 23 18:14 config_test.go
drwxrwxr-x  2 matthias matthias     4096 Mai 23 18:14 css
-rw-rw-r--  1 matthias matthias      823 Mai 23 18:14 doc.go
drwxrwxr-x  2 matthias matthias     4096 Mai 23 18:14 fonts
drwxrwxr-x  8 matthias matthias     4096 Mai 23 18:10 .git
drwxrwxr-x  3 matthias matthias     4096 Mai 23 17:58 .github
-rw-rw-r--  1 matthias matthias      123 Mai 23 17:58 .gitignore
-rw-------  1 matthias matthias      507 Mai 23 18:11 go.mod
-rw-------  1 matthias matthias     4004 Mai 23 18:11 go.sum
-rw-rw-r--  1 matthias matthias     5010 Mai 23 18:18 hashfile.db
drwxrwxr-x  2 matthias matthias     4096 Mai 23 18:14 img
-rw-rw-r--  1 matthias matthias    32474 Mai 23 17:58 LICENSE
-rwxrwxr-x  1 matthias matthias 11149115 Mai 23 18:19 nele
-rw-rw-r--  1 matthias matthias    21803 Mai 23 18:14 pagehandler.go
-rw-rw-r--  1 matthias matthias      619 Mai 23 18:22 pagehandler_test.go
-rw-rw-r--  1 matthias matthias     9313 Mai 23 18:14 posting.go
drwxrwxr-x  8 matthias matthias     4096 Mai 23 18:01 postings
-rw-rw-r--  1 matthias matthias    15319 Mai 23 18:14 posting_test.go
-rw-rw-r--  1 matthias matthias     8240 Mai 23 18:14 postlist.go
-rw-rw-r--  1 matthias matthias     7279 Mai 23 18:23 postlist_test.go
-rw-rw-r--  1 matthias matthias       70 Mai 23 18:19 pwaccess.db
-rw-rw-r--  1 matthias matthias    22792 Mai 23 18:35 README.md
-rw-rw-r--  1 matthias matthias    10435 Mai 23 18:24 regex.go
-rw-rw-r--  1 matthias matthias     8190 Mai 23 18:14 regex_test.go
drwxrwxr-x  2 matthias matthias     4096 Mai 23 17:58 static
-rw-rw-r--  1 matthias matthias     3656 Mai 23 18:14 tags.go
-rw-rw-r--  1 matthias matthias     3811 Mai 23 17:58 template_vars.md
-rw-rw-r--  2 matthias matthias     3109 Mai 23 18:16 TODO.md
drwxrwxr-x  3 matthias matthias     4096 Mai 23 18:14 views
-rw-rw-r--  1 matthias matthias     6787 Mai 23 18:14 views.go
-rw-rw-r--  1 matthias matthias     6009 Mai 23 18:14 views_test.go
$ _

You can reduce the binary’s size by stripping it:

$ strip nele
$ ls -l nele
-rwxrwxr-x 1 matthias matthias 8146912 Mai 23 18:38 nele
$ _

As you can see the binary lost about 3MB of its weight.

Let’s start with the command line:

$ ./nele -h

Usage: bin/nele-linux-386 [OPTIONS]

-accessLog string
		<filename> Name of the access logfile to write to
		(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/access.log")
-blogName string
		Name of this Blog (shown on every page)
	(default "Meine Güte, was für'n Blah!")
-certKey string
	<fileName> the name of the TLS certificate's private key
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/certs/server.key")
-certPem string
	<fileName> the name of the TLS certificate PEM
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/certs/server.pem")
-dataDir string
	<dirName> the directory with CSS, IMG, JS, POSTINGS, STATIC, VIEWS sub-directories
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele")
-errorLog string
	<filename> Name of the error logfile to write to
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/error.log")
-gzip
	(optional) use gzip compression for server responses (default true)
-hashFile string
	<fileName> (optional) the name of a file storing #hashtags and @mentions
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/hashfile.db")
-ini string
	<fileName> the path/filename of the INI file to use
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/nele.ini")
-lang string
	(optional) the default language to use  (default "de")
-listen string
	the host's IP to listen at  (default "127.0.0.1")
-logStack
	<boolean> Log a stack trace for recovered runtime errors  (default true)
-maxfilesize string
	max. accepted size of uploaded files (default "10MB")
-pa
	(optional) posting add: write a posting from the commandline
-pageView
	(optional) use page preview images for links (default true)
-pf string
	<fileName> (optional) post file: name of a file to add as new posting
-port int
	<portNumber> the IP port to listen to  (default 8181)
-realm string
	(optional) <hostName> name of host/domain to secure by BasicAuth
	(default "This Host")
-theme string
	<name> the display theme to use ('light' or 'dark')
	(default "dark")
-ua string
	<userName> (optional) user add: add a username to the password file
-uc string
	<userName> (optional) user check: check a username in the password file
-ud string
	<userName> (optional) user delete: remove a username from the password file
-uf string
	<fileName> (optional) user passwords file storing user/passwords for BasicAuth
	(default "/home/matthias/devel/Go/src/github.com/mwat56/nele/pwaccess.db")
-ul
	(optional) user list: show all users in the password file
-uu string
	<userName> (optional) user update: update a username in the password file

Most options can be set in an INI file to keep the command-line short ;-)

$ _

However, to just run the program you’ll usually don’t need any of those options to input on the commandline. There is an INI file called nele.ini coming with the package, where you can store the most common settings:

$ cat nele.ini
# Nele's default configuration file

[Default]

# Name of the optional logfile to write to.
# NOTE: A relative path/name will be combined with `datadir` (below).
accessLog = ./access.log

# Name of this Blog (shown on every page).
blogName = "Meine Güte, was für'n Blah!"

# path-/filename of the TLS certificate's private key to enable
# TLS/HTTPS (if empty standard HTTP is used).
# NOTE: A relative path/name will be combined with `datadir` (below).
certKey = ./certs/server.key

# path-/filename of TLS (server) certificate to enable TLS/HTTPS
# (if empty standard HTTP is used).
# NOTE: A relative path/name will be combined with `datadir` (below).
certPem = ./certs/server.pem

# The directory root for the "css", "fonts", "img", "postings",
# "static", and "views" sub-directories.
# NOTE: This should be an _absolute_ path name.
dataDir = ./

# Name of the optional logfile to write to.
# NOTE: A relative path/name will be combined with `datadir` (above).
errorLog =  ./error.log

# Use gzip compression for server responses.
gzip = true

# The file to store #hashtags and @mentions.
# NOTE: A relative path/name will be combined with `datadir` (above).
hashFile = ./hashfile.db

# The default UI language to use ("de" or "en").
lang = de

# The host's IP number to listen at.
listen = 127.0.0.1

# Whether or not log a stack trace for recovered runtime errors.
# NOTE: This is merely a debugging aid and should normally be `false`.
logStack = true

# Use preview images of linked pages.
# NOTE: This feature depends on the external `wkhtmltoimage` binary;
# for more details see: https://godoc.org/github.com/mwat56/pageview
pageView = true

# The IP port to listen to.
port = 8181

# Accepted size of uploaded files.
maxfilesize = 10MB

# Password file for HTTP Basic Authentication.
# NOTE: a relative path/name will be combined with `datadir` (above).
passFile = ./pwaccess.db

# Name of host/domain to secure by BasicAuth.
realm = "This Host"

# Web/display theme ("dark" or "light").
theme = dark

# _EoF_
$ _

The program, when started, will first look for the INI file in five different places:

in your (i.e. the current user’s) directory (./nele.ini),
in the computer’s main config directory (/etc/nele.ini"),
in the current user’s home directory (e.g. $HOME/.nele.ini),
in the current user’s configuration directory (e.g. $HOME/.config/nele.ini),
in the -ini <filename> commandline option (if given).

All these files (if they exist) are read in the given order at startup before finally parsing the commandline options shown earlier. So each step overwrites the previous one, the commandline options having the highest priority. – But let’s look at some of the commandline options more closely.

Commandline postings

You can post an article directly from the commandline.

./nele -pa allows you to write an article/posting directly on the commandline.

$ ./nele -pa
This is
a test
posting directly
from the commandline.
<Ctrl-D>
2019/05/06 14:57:30 ./nele wrote 54 bytes in a new posting
$ _

./nele -pf <fileName> allows you to include an already existing text file (with possibly some Markdown markup) into the system.

$ ./nele -pf addTest.md
2019/05/06 15:09:27 ./nele stored 474 bytes in a new posting
$ _

These two options (-pa and -pf) are only usable from the commandline.

Authentication

Why, you may ask, would you need an username/password file anyway? Well, you remember me mentioning that you can add, edit and delete articles? You wouldn’t want anyone on the net being able to do that, now, would you? For that reason, whenever there’s no password file given (either in the INI file or the command-line) all functionality requiring authentication will be disabled. (Better safe than sorry, right?)

Note that the password file generated and used by this system resembles the htpasswd used by the Apache web-server, but both files are not interchangeable because the actual encryption algorithms used by both are different.

User/password file & handling

Only usable from the commandline are the -uXX options, most of which need a username and the name of the password file to use. – Note that whenever you’re prompted to input a password this will not be echoed to the console.

The -ua option allows you to add an user/password pair:

$ ./nele -ua testuser1 -uf pwaccess.db

 password:
repeat pw:
    added 'testuser1' to list
$ _

Again: The password input is not echoed to the console, therefore you don’t see it.

Since we have the passfile setting already in our INI file (see above) we can forget the -uf option for the next options.

With -uc you can check a user’s password:

$ ./nele -uc testuser1

 password:
    'testuser1' password check successful
$ _

This -uc you’ll probably never actually use, it was just easy to implement.

If you want to remove an user account the -ud will do the trick (i.e. delete a user):

$ ./nele -ud testuser1
    removed 'testuser1' from list
$ _

When you want to know which users are stored in your password file -ul is your friend:

$ ./nele -ul
matthias

$ _

Since we deleted the testuser1 before only one entry remains.

That only leaves -uu to update (change) a user’s password.

$ ./nele -ua testuser2

 password:
repeat pw:
    added 'testuser2' to list

$ ./nele -uu testuser2

 password:
repeat pw:
    updated user 'testuser2' in list

$ ./nele -ul
matthias
testuser2

$ _

First we added (-ua) a new user, then we updated the password (-uu), and finally we asked for the list of users (-ul).

Page/link previews

If you set the pageView INI- or commandline-option to true there will be a preview image generated – by way of calling the external wkhtmltoimage commandline utility. Those image files are stored locally (in the ./img/ directory) and may be used as often as you want.

The commandline utility wkhtmltoimage is required for this pageView option to work. Under Linux this utility is usually part of your distribution. If not, you can download wkhtmltoimage from the web and install it. Sometimes the package from the download page above is more recent than the version in your Linux distribution. If in doubt, I’d suggest to test both versions to determine which one to use and works best for you.

Generating a preview image usually takes between one and five seconds, depending on the actual web-page in question, however, it can take considerably longer. To avoid hanging the program the wkhtmltoimage executable is called with an one minute timeout.

And, finally, not all web-pages can be rendered properly and turned into an image. In such case wkhtmltoimage usually just crashes and the link in your posting just remains as is (i.e. a normal text link w/o preview).

Configuration

The system’s configuration takes two steps:

Prepare the required files and directories.
Customise the INI file and/or prepare a script with all needed commandline arguments.
You most probably want to customise the files ./views/imprint.gohtml, ./views/licence.gohtml, and ./views/privacy.gohtml according to your personal requirements.

URLs

The system uses a number of slightly different URL groups.

Static URLs

First, there are the static files served from the css, img, and static directories. The actual location of which you can configure with the datadir INI entry and/or commandline option.

Common URLs

Second, there are the URLs any normal user might see and use:

/ defines the logical root of the presentation; it’s effectively the same as /n/ (see below).
/faq, /imprint, /licence, and /privacy serve static files which have to be filled with content according to your personal and legal needs.
/hl/tagname allows the users to search for #tagname (but you’ll input it without the number sign # because that has a special meaning in an URL). Provided the given tagname was actually used in one or more of your articles a list of the respective postings will be shown.
/m/ shows the articles of the current month. One can, however, specify the month one is interested in by adding a data part defining the month one wants to see (/m/yyyy-mm), like /m/2019-04 to see the articles from April 2019.
/ml/mentionedname allows the users to search for @mentionedname (but one will input it without the at sign @ because that has a special meaning in an URL). Provided the given mentionedname was actually used in one or more of your articles a list of the respective articles will be shown.
/n/ gives you the newest 30 articles. The number of articles to show can be added to the URL like /n/5 to see only five articles, or /n/100 to see a hundred. If one want to see the articles in slices of, say, 10 per page (instead of the default 30/page) one can use the URL /n/10,10 and to see the second slice use /n/10,20, the third with /n/10,30 and so on. However, as long as there are more articles available, there will be a »» link at the bottom of the page to ease the navigation for the reader.
/p/1234567890abcdef shows a single article/posting (the ID is automatically generated). This kind of URL your users will see when they choose on another page to see the single article per page by selecting the leading [*] link in the overview page(s).
/s/searchterm can be used to search for articles containing a certain word or expression. All existing articles will be searched for the given searchterm.
/w/ shows the articles of the current week. One can, however, specify the week one is interested in by adding a data part defining the week to see (/w/yyyy-mm-dd), like /w/2019-04-13 to see the articles from the week in April 2019 containing the 13th.

Internal URLs

And, third, there’s a group of URLs your users won’t see or use, because by design they are reserved for you, the author of your postings. These URLs are protected by an authentication mechanism called BasicAuth (which is supported by browsers for at least twenty years); this is where the username/password file comes in. Only users whose credentials (i.e. username and password) are stored in the password file will be given access to the following URLs. So don’t forget to set up an appropriate password file. If you forget that (or the file is not accessible for the program) everybody on the net could read, modify, or delete your articles, or add new ones – which you might not like; therefore the system disables all options that might modify your system.

/ap/ add a new posting. A simple Web form will allow you to input whatever is on your mind.
/dp/234567890abcdef1 lets you change an article/posting’s date/time if you feel the need for cosmetic or other reasons. Since you don’t usually know/remember the article ID you’ll first go to show the article/posting on a single page (/p/234567890abcdef1) by selecting the respective [*] link on the index page and then just prepend the p by a d in the URL.
/ep/34567890abcdef12 lets you edit the article/posting’s text identified by 34567890abcdef12, e.g. to fix typos or correct the grammar.
/il/ (init list): Assuming you configured the hashfile INI-/commandline-option this shows you a simple HTML form by which you can start a background process re-initialising the hashlist. It clears the current list and reads all postings to extract the #hashtags and @mentions. Note: You will barely (if ever) need this option; it’s mostly a debugging aid.
/pv/ (pageView): Assuming you set the pageview INI-/commandline-option to true this shows you a simple HTML form by which you can start a background process checking all postings for page preview images. Again, this was implemented as a debugging aid and you won’t usually use this option.
/rp/4567890abcdef123 lets you remove (delete) the article/posting identified by 4567890abcdef123 altogether. Note that there’s no undo feature: Once you’ve deleted an article/posting it’s gone.
/share/https://some.host.domain/somepage lets you share another page URL. Whatever you write after the initial /share/ is considered a remote URL, and a new article will be created and shown for you to edit.
/si/ (store image): This shows you a simple HTML form by which you can upload image files into your /img/ directory. Once the upload is done you (i.e. the user) will be presented an edit page in which the uploaded image is used.
/ss/ (store static): This shows you a simple HTML form by which you can upload static files into your /static/ directory. Once the upload is done you (i.e. the user) will be presented an edit page in which the uploaded file is used.
/xt/ (eXchange tag): This shows you a simple HTML form by which you can exchange a #hashtag/@mention with another one, or correct its writing. Note that the search for the term to replace is done case-insensitive while the replacement string gets inserted as you write it.

Files

Right at the start I mentioned that I wanted to avoid external dependencies – like databases for example. Well, that’s not exactly true (or even possible), because there is one “database” that’s always already there, regardless of the operating system: the filesystem. The trick is to figure out how to best use it for our own purposes. The solution I came up with here is to use sort of a timestamp as ID and filename for the articles, and use part of that very timestamp as ID and name for the directory names as well.

Both directory- and file-names are automatically handled by the system. Each directory can hold up to 52 days worth of articles. After extensive experimentation – with hundreds of thousands of automatically generated (and deleted) test files – that number seemed to be a reasonable compromise between directories not growing too big (search times) and keeping the number of directories used low (about seven per year).

All this data (files and directories) will be created under the directory you configure either in the INI file (entry datadir) or on the commandline (option -datadir). Under that directory the program expects several sub-directories:

css/ for stylesheet files,
fonts/ for font files,
img/ for image files,
postings/ directory root for the articles,
static/ for static files (like e.g. PDF files),
views/ for page templates

Apart from setting that datadir option to your liking you don’t have to worry about it any more.

As mentioned before, it’s always advisable to use absolute pathnames, not relative one. The latter are converted into absolute ones (based on datadir) by the system, but they depend on where you are in the filesystem when you start the program or write the commandline options. You can use ./nele -h to see which directories the program will use (see the example above).

CSS

In the CSS directory (datadir/css) there are currently four files that are used automatically (i.a. hardcoded) by the system: stylesheet.css with some basic styling rules and dark.css and light.css with different settings for mainly colours, thus implementing two different themes for the web-presentation, and there’s the fonts.css file setting up the custom fonts to use. The theme INI setting and the -theme commandline option determine which of the two dark and light styles to actually use.

Fonts

The datadir/fonts/ directory contains some freely available fonts used by the CSS files.

Images

The datadir/img/ directory can be used to store, well, images to which you then can link in your articles. You can put there whatever images you like either from the command-line or by using the system’s /si URL.

Additionally any page preview images are stored here (if use set the pageView INI- or commandline-option to true).

Postings

The datadir/ directory is the base for storing all the articles. The system creates subdirectories as needed to store new articles. This directory structure is not accessed via a direct URL but used internally by the system.

Static

The datadir/static/ directory can be used to store, well, static files to which you then can link in your articles. You can put there whatever file you like either from the command-line or by using the system’s /ss URL.

Views

The datadir/views/ directory holds the templates with which the final HTML pages are generated. Provided that you feel at home working with Go templates you might change them as you see fit. I will, however, not provide any support for you changing the default template structure.

A concise overview of the used templates and which variables they use you’ll find in the file template_vars.md

For all the article you write – either on the commandline or with the web-interface – you can use Markdown to enrich the plain text. In fact, the system expects the postings to be using MarkDown syntax if any markup at all.

Libraries

The following external libraries were used building Nele:

Licence

Copyright © 2019, 2020 M.Watermann, 10247 Berlin, Germany
                All rights reserved
            EMail : <support@mwat.de>

2019-05-17

[*]

UploadHandler

UploadHandler

Purpose

Sometimes a web-server application needs a way to accept file uploads from the remote users. This middleware package does just this: it accepts uploads (up to a certain size) and stores them in a configurable directory.

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/uploadhandler

Usage

The main function to call is

// Wrap returns a handler function that includes upload handling,
// wrapping the given `aHandler` and calling it internally.
//
func Wrap(aHandler http.Handler,
    aDestDir, aFieldName, aUpURL, aNextURL string,
    aMaxSize int64, aPager errorhandler.TErrorPager) http.Handler {…}

While at first glance the number of arguments seems to be overwhelming they allow you to fully configure the package’s behaviour when an uploaded file arrives. Let’s look at the arguments one by one:

aHandler is the handler function which you’re already using for your web-server. It will continue to work as it used to before except that a certain URL (configured by aUpURL, see below) will be intercepted if something gets POSTed to it.
aDestDir is the directory where the incoming file is finally stored after processing it.
aFieldName is the name/ID of the form/field your web-page uses to accept the file-upload.
aUpURL is the URL your web-page’s FORM element POSTs its data to. This URL will be intercepted (if it’s accessed by the POST HTTP method) and its data will be processed.
aNextURL is the URL the user gets forwarded to after the file upload was successfully processed.
aMaxSize defines the max. accepted size of uploaded files; if the given value is smaller/equal to zero then a maximal filesize of 8 MB is used. Files bigger than that value will be rejected. Think carefully about which size will suit your actual needs.
aPager is an optional provider of customised error pages (or nil if not needed). – See github.com/mwat56/errorhandler for details about that package.

Here is a very simple example using this package:

func testHandler(aWriter http.ResponseWriter, aRequest *http.Request) {
    // the upload form to show
    page := `<!DOCTYPE html><html><head><title>Go Upload</title></head><body>
    <form action="/up" method="post" enctype="multipart/form-data">
    <p><label for="uploadFile">Filename:</label>
    <input type="file" name="uploadFile" id="uploadFile"></p>
    <p><input type="submit" name="submit" value="Submit"></p>
    </form></body></html>`

    // send it to the remote user:
    aWriter.WriteHeader(200)
    aWriter.Write([]byte(page))
    // POST is handled by the UploadHandler
} // testHandler()

func main() {
    // let the upload handler wrap our own page handler:
    handler := uploadhandler.Wrap(http.HandlerFunc(testHandler),
        "./static", "uploadFile", "up", "/", 10*1024*1024, nil)

    if err := http.ListenAndServe("127.0.0.1:8080", handler); nil != err {
        log.Fatalf("%s Problem: %v", os.Args[0], err)
    }
} // main()

You’ll probably store the required values for e.g. aDestDir and aMaxSize in some kind of config-file, reading them at start of your web-server, and passing them along to the final Wrap(…) call instead of hard-coding them like in the example above. And the values of aUpURL and aFieldName must, obviously, correspond with those you’re actually using in your own application. If you don’t use customised error pages you can pass nil for the aPager argument as done in the example.

So, to add the file-upload functionality to your web-server application all that’s needed is a single Wrap() function call. That’s it.

However, if for some reason you’d like to be a little more “hands on”, you can use another function to get a TUploadHandler instance:

// NewHandler returns a new `tUploadHandler` instance.
func NewHandler(aDestDir, aFieldName string,
    aMaxSize int64) *TUploadHandler {…}

This function call needs only a subset of the arguments passed to the Wrap(…) function:

aDestDir: the directory where the incoming file is finally stored after processing it.
aFieldName: the name/ID of the form/field your web-page uses to accept the file-upload.
aMaxSize: the max. accepted size of uploaded files; if the given value is smaller/equal to zero then a maximal filesize of 8 MB is used.

The NewHandler() function’s result provides the method

// ServeUpload handles the incoming file upload.
func (uh *TUploadHandler) ServeUpload(aWriter http.ResponseWriter,
    aRequest *http.Request) (string, int) {…}

This method does the actual upload handling. It returns a string (holding a possible error message) and an integer (holding the HTTP status code). If the returned status code is 200 (i.e. everything’s alright) then the string return value will be the name of the processed file. In all other cases (i.e. result status != 200) the calling application can react to the return values as it sees fit.

You can use several TUploadHandler instances to serve different URLs and different destination directories etc. Insofar calling NewHandler() and then ServeUpload(…) gives you more flexibility then simply calling Wrap(…). On the other hand, you could call Wrap(…) several times, wrapping one instance within the other and thus react to different URLs and form/fields etc. …

Libraries

The following external libraries were used building UploadHandler:

ErrorHandler

Licence

    Copyright © 2019, 2020 M.Watermann, 10247 Berlin, Germany
                    All rights reserved
                EMail : <support@mwat.de>

2019-05-12

[*]

HashTags

HashTags

Purpose

Sometimes one might want to search and find socalled #hashtags or @mentions in one’s texts (in a broader sense) and store them for later retrieval. This package offers that facility. It provides the THashList class which can be used to parse texts for the occurrence of both #hashtags and @mentions and store the hits in an internal list for later lookup; that list can be both, stored in a file and later read from a file.

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/hashtags

Usage

In principle for each #hashtag or @mention a list of IDs is maintained. These IDs can be any (string) data that identifies the text in which the #hashtag or @mention was found, e.g. a filename or some database record reference. The only condition is that it is unique as far as the program using this package is concerned.

Note that both #hashtag and @mention are stored lower-cased to allow for case-insensitive searches.

To get a THashList instance there’s a simple way:

fName := "mytags.lst"
htl, err := hashtags.New(fName)
if nil != err {
    log.PrintF("Problem loading file '%s': %v", fName, err)
}
    // …
    // do something with the list
    // …
written, err := htl.Store()
if nil != err {
    log.PrintF("Problem writing file '%s': %v", fName, err)
}

The package provides a boolean variable called UseBinaryStorage which is true by default. It determines whether the data written by Store() and read by Load() use plain text (i.e. UseBinaryStorage = false) or a binary data format. The advantage of the plain text format is that it can be inspected by any text related tool (like e.g. diff). The advantage of the binary format is that it is about three to four times as fast when loading/storing data and it uses a few bytes less than the text format. For this reasons it’s used by default (i.e. hashtags.UseBinaryStorage == true); during development of your own application using this package, however, you might want to change to text format for diagnostic purposes.

For more details please refer to the package documentation.

Libraries

No external libraries were used building HashTags.

Licence

    Copyright © 2019, 2020 M.Watermann, 10247 Berlin, Germany
                    All rights reserved
                EMail : <support@mwat.de>

2019-05-08

[*]

PassList

PassList

Purpose

Sometimes there is a need to password-protect your web-server, either in whole or just some parts of it. That’s were this little package comes in. It offers to simply integrate the popular BasicAuth mechanism into your own web-server.

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/passlist

Usage

PassList provides an easy way to handle HTTP Basic Authentication by simply calling the package’s Wrap() function and implementing the TAuthDecider interface which only requires the single function or method

NeedAuthentication(aRequest *http.Request) bool

That function may decide on whatever means necessary whether to grant access (returning true) or deny it (returning false).

For your ease there are two TAuthDecider implementations provided: TAuthSkipper (which generally returns false) and TAuthSkipper (which generally returns true). Just instantiate one of those – or, of course, your own implementation – and pass it to the Wrap() function.

func Wrap(aHandler http.Handler, aRealm, aPasswdFile string, aAuthDecider TAuthDecider) http.Handler

The arguments mean:

aHandler: the HTTP handler you implemented for your web-server; you will use the return value of Wrap() after you called this function.
aRealm: the name of the host/domain to protect (this can be any string you like); it will be shown by most browsers when the username/password is requested.
aPasswdFile: the name of the password file that holds all the username/password pairs to use when authentication is actually required.
aAuthDecider: the deciding function we talked about above.

So, in short: implement the TAuthDecider interface and call passlist.Wrap(…), and you’re done.

However, the package provides a TPassList class with methods to work with a username/password list. It’s fairly well documented, so it shouldn’t be too hard to use it on your own if you don’t like the automatic handling provided by Wrap(). You can create a new instance by either calling passlist.LoadPasswords(aFilename string) (which, as its name says, tries to load the given password file at once), or you call passlist.NewList(aFilename string) (which leaves it to you when to actually read the password file by calling the TPassList object’s Load() method).

There’s an additional convenience function called passlist.Deny() which sends an “Unauthorised” notice to the remote host in case the remote user couldn’t be authenticated; this function is called internally whenever your TAuthDecider required authentication and wasn’t given valid credentials from the remote user.

To further improve the safety of the passwords they are peppered before hashing and storing them. The default pepper value can be read by calling

pepper := passlist.Pepper()

And the pepper value can be changed by calling

myPepper := "This is my common 'pepper' value for the user passwords"
passlist.SetPepper(myPepper)

Please refer to the source code documentation for further details ot the TPassList class.

In the package’s cmd/ folder you’ll find the pwaccess.go program which implements the maintenance of password files with the following options:

-add string
    <username> name of the user to add to the file (prompting for the password)
-chk string
    <username> name of the user whose pass to check (prompting for the password)
-del string
    <username> name of the user to remove from the file
-file string
    <filename> name of the passwordfile to use (default "pwaccess.db")
-lst
    list all current usernames from the list
-q    whether to be quiet or not (suppress screen output)
-upd string
    <username> name of the user to update in the file (prompting for the password)

Password list

This library provides a couple of functions you can use in your own program to maintain your own password list without having to use the TPassList class directly.

AddUser(aUser, aFilename string) reads a password for aUser from the commandline and adds it to aFilename.
CheckUser(aUser, aFilename string) reads a password for aUser from the commandline and compares it with the one stored in aFilename.
DeleteUser(aUser, aFilename string) removes the entry for aUser from the password list stored in aFilename.
ListUsers(aFilename string) reads aFilename and lists all users stored in that file.
UpdateUser(aUser, aFilename string) reads a password for aUser from the commandline and updates the entry in the password list in aFilename.

Libraries

The following external libraries were used building PassList:

Crypto

Licence

Copyright © 2019, 2020  M.Watermann, 10247 Berlin, Germany
                All rights reserved
            EMail : <support@mwat.de>

[*]

Kaliber

Kaliber

Note

Purpose

I love books. Always have. Since I was a kid. Over the years several thousand books gathered in my flat; so many that I sort of ran out of space. Reluctantly I started to investigate and then use eBooks, first on my main desktop computer, later with a dedicated eBook reader.

Soon – and again – there were so many that I started looking for some convenient way to handle, store, and retrieve them for reading. That’s when I became acquainted with Calibre, a great software for working with eBooks. Of course, there were some problems but since that software is actively maintained and extended all the time those problems either went away on their own with the next update or I found some way around them. And sure, it took some work to get all the eBooks into that library system, and since there are always coming new titles the work kind of never ends.

Another question soon became urgent: How to access my books when I’m not at home? As it turned out, Calibre comes bundled with its own web-server. After figuring out how to start it automatically when the machine gets restarted (which happens once in a while when some kernel software upgrade requires it) the server did its job. Quite another question, however, is how it does its job. And there the problem lies. The web-server coming with Calibre serves pages that are heavily dependent on JavaScript; so much so that the pages simply don’t appear or work at all if you have JavaScript disabled in your browser e.g. for privacy or security reasons. For a while I grudgingly activated JavaScript whenever I wanted to access my books remotely. I asked the author of Calibre whether he’d be willing to provide a barrier-free alternative (i.e. without JavaScript) but unfortunately he declined: “Not going to happen.”

Well, due to other projects I didn’t find the time then to do something about it, and it took some more years before I started seriously to look for alternatives. Therefore now there’s Kaliber, a barrier-free web-server for your Calibre book collection. It doesn’t depend on – or even need – JavaScript, but just requires a web-browser capable to read plain HTML on the remote user’s side. So even privacy or security conscious people or those who depend on assisting technologies (like e.g. screen-readers) can now access their library.

As an aside: > I never really understood the desire to multiply the work needed to be done for a web page. > > A server is a server which means it should serve. > When in a restaurant ordering a meal you surely expect it to be brought to you fully prepared and ready to be consumed. You’d probably were seriously annoyed if the waiter/server just brought you the ingredients and left it to you to prepare the meal for yourself. > > For some strange reason, however, that’s exactly what happens on a growing number of web-presentations: Instead of delivering a ready-to-read web-page they send every user just program code and let the user’s machine do the preparing of the page. > In other words: Instead of one server doing the work and delivering it to thousands of remote users nowadays the server forces thousands (or even millions) of remote users to spend time and electricity to just see a single web-page.

Since I couldn’t find a documentation of the database structure and/or API used by Calibre to store its meta-data I had to reverse engineer ways to access the stored book data. The same is true – to a certain lesser degree – for the web-pages served by Kaliber: it’s kind of a mix of Calibre’s normal (i.e. JavaScript based) and mobile pages. The overall layout of the web-pages served by Kaliber is intentionally kept simple (KISS).

Features

Simplicity of use;
Barrier-free (no JavaScript required);
Books list layout either Cover Grid or Data List;
Easy navigation (First, Prev, Next, Last button/links);
Fulltext search as well as datafield-based searches;
Ordered in either ascending or descending direction;
Selectable number of books per page;
Sortable by author, date, language, publisher, rating, series, size, tags, or title;
Anonymised access logging;
Optional user/password based access control.

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/kaliber

Usage

After downloading this package you go to its directory and compile

cd $GOPATH/src/github.com/mwat56/kaliber
go build app/kaliber.go

which should produce an executable binary.

Commandline options

$ ./kaliber -h

Usage: ./kaliber [OPTIONS]

-accessLog string
	<filename> Name of the access logfile to write to
	(default "/home/matthias/devel/Go/src/github.com/mwat56/kaliber/access.log")
-authAll
	<boolean> whether to require authentication for all pages  (default true)
-booksPerPage int
	<number> the default number of books shown per page  (default 24)
-certKey string
	<fileName> the name of the TLS certificate key
-certPem string
	<fileName> the name of the TLS certificate PEM
-dataDir string
	<dirName> the directory with CSS, FONTS, IMG, SESSIONS, and VIEWS sub-directories
	(default "/home/matthias/devel/Go/src/github.com/mwat56/kaliber")
-errorlog string
	<filename> Name of the error logfile to write to
	(default "/home/matthias/devel/Go/src/github.com/mwat56/kaliber/error.log")
-gzip
	<boolean> use gzip compression for server responses (default true)
-ini string
	<fileName> the path/filename of the INI file to use
	(default "/home/matthias/.kaliber.ini")
-lang string
	the default language to use  (default "de")
-libraryName string
	Name of this Library (shown on every page)
	(default "MeiBucks")
-libraryPath string
	<pathname> Path name of/to the Calibre library
	(default "/var/opt/Calibre")
-listen string
	the host's IP to listen at  (default "0")
-logStack
	<boolean> Log a stack trace for recovered runtime errors  (default true)
-port int
	<portNumber> The IP port to listen to  (default 8383)
-realm string
	<hostName> Name of host/domain to secure by BasicAuth
	(default "eBooks Host")
-sessionTTL int
	<seconds> Number of seconds an unused session keeps valid  (default 1200)
-sidName string
	<name> The name of the session ID to use
	(default "sid")
-sqlTrace string
	<filename> Name of the SQL logfile to write to
	(default "/home/matthias/devel/Go/src/github.com/mwat56/kaliber/sqlTrace.sql")
-theme string
	<name> The display theme to use ('light' or 'dark')
	(default "dark")
-ua string
	<userName> User add: add a username to the password file
-uc string
	<userName> User check: check a username in the password file
-ud string
	<userName> User delete: remove a username from the password file
-uf string
	<fileName> Passwords file storing user/passwords for BasicAuth
	(default "/home/matthias/devel/Go/src/github.com/mwat56/kaliber/pwaccess.db")
-ul
	<boolean> User list: show all users in the password file
-uu string
	<userName> User update: update a username in the password file

Most options can be set in an INI file to keep the command-line short ;-)

$ _

As you can see there are quite a few options available, but almost all of them are optional since they come with reasonable default values most of which can be set in the accompanying INI-file (in fact, the “default” values shown above are coming from the INI-file used).

INI file

You don’t have to give all those commandline options listed above every time you want to start Kaliber. There’s an INI file which can take all the options (apart from the user handling options) all in one place:

$ cat kaliber.ini
# Default configuration file for the Kaliber server

[Default]

# Name of the optional access logfile to write to.
# NOTE: a relative path/name will be combined with `dataDir` (below).
accessLog = ./access.log

# Authenticate user for all pages and documents.
# If `false` only the download links need user authentication
# (see `passFile` below).
authAll = false

# Number of documents to show per page.
booksPerPage = 24

# Path-/filename of the TLS certificate's private key to enable
# TLS/HTTPS (if empty standard HTTP is used).
# NOTE: a relative path/name will be combined with `dataDir` (below).
certKey = ./certs/server.key

# Path-/filename of the TLS (server) certificate to enable TLS/HTTPS
# (if empty standard HTTP is used).
# NOTE: A relative path/name will be combined with `dataDir` (below).
certPem = ./certs/server.pem

# The directory root for the "css", "fonts", "img", "sessions",
# and "views" sub-directories.
# NOTE: This should be an _absolute_ path name!
dataDir = ./

# Name of the optional error logfile to write to.
# NOTE: a relative path/name will be combined with `dataDir` (above).
errorLog = ./error.log

# Use gzip compression for server responses.
gzip = true

# The default UI language to use ("de" or "en").
lang = de

# Name of this library (shown on every page).
libraryName = "MeiBucks"

# Path of Calibre library.
# NOTE: this must be the absolute pathname io the Calibre library.
libraryPath = "/var/opt/Calibre"

# The host's IP number to listen at.
listen = 127.0.0.1

# Whether or not log a stack trace for recovered runtime errors.
# NOTE: This is merely a debugging aid and should normally be `false`.
logStack = true

# The host's IP port to listen to.
port = 8383

# Password file for HTTP Basic Authentication.
# NOTE: a relative path/name will be combined with `dataDir` (above).
passFile = ./pwaccess.db

# Name of host/domain to secure by BasicAuth.
realm = "eBooks Host"

# Number of seconds an unused session stays valid.
sessionTTL = 1200

# Name of the session ID field.
sidName = sid

# Optional (debugging) SQL trace file.
# NOTE: a relative path/name will be combined with `dataDir` (above).
sqlTrace = ./sqlTrace.sql

# Default web/display theme to use ("dark" or "light").
theme = dark

# _EoF_
$ _

An INI-file as shown above is looked for at five different places:

in your (i.e. the current user’s) current directory (./kaliber.ini),
in the computer’s main config directory (/etc/kaliber.ini"),
in the current user’s home directory (e.g. $HOME/.kaliber.ini),
in the current user’s configuration directory (e.g. $HOME/.config/kaliber.ini),
in the -ini <filename> commandline option (if given).

Authentication

Why, you may ask, would you need an username/password file anyway? Well, there may be several reasons one of which could be Copyright problems.

If not all your books are in the public domain and Copyright-free in most countries you may not make them publicly available. In that case you’re most likely the only actual remote user allowed to access the books in your library. Depending on your country’s legislation you may or may not include your family members. If in doubt please consult a Copyright expert.

The authAll commandline option (and INI setting) allows you to specify whether access to all pages require user authentication; if that flag is false then only the download links require authentication, if true any access requires a given username/password pair.

Whenever there’s no password file given (either in the INI file passfile or the command-line -uf) all functionality requiring authentication will be disabled which in turn means that everybody can access your library.

Note that the password file generated and used by this system resembles the htpasswd used by the Apache web-server, but both files are not interchangeable because the actual encryption algorithms used by both are different.

User/password file & handling

Only usable from the commandline are the -uXX options, most of which need an username and the name of the password file to use.

$ ./kaliber -ua testuser1 -uf pwaccess.db

 password:
repeat pw:
    added 'testuser1' to list
$ _

Since we have the passfile setting already in our INI file (see above) we can forget the -uf option for the next options.

With -uc you can check a user’s password:

$ ./kaliber -uc testuser1

 password:
    'testuser1' password check successful
$ _

This -uc you’ll probably never actually use – it was just easy to implement.

If you want to remove a user the -ud will do the trick:

$ ./kaliber -ud testuser1
    removed 'testuser1' from list
$ _

When you want to know which users are stored in your password file -ul is your friend:

$ ./kaliber -ul
matthias

$ _

Since we deleted the testuser1 before only one entry remains.

That only leaves -uu to update (change) a user’s password.

$ ./kaliber -ua testuser2

 password:
repeat pw:
    added 'testuser2' to list

$ ./kaliber -uu testuser2

 password:
repeat pw:
    updated user 'testuser2' in list

$ ./kaliber -ul
matthias
testuser2

$ _

First we added (-ua) a new user, then we updated the password (-uu), and finally we asked for the list of users (-ul).

Directory structure

Under the directory given with the datadir = entry in the INI file (or the -datadir commandline option) there are several sub-directories expected:

css: containing the CSS files used,
fonts: containing the fonts used,
img: containing the images used,
sessions: containing the remote users’ session data,
views: the Go templates used to generate the pages.

All of this directories and files are part of the Kaliber package. You can use them as is or customise them as you see fit to suit your needs. However, please note: I will not support any customisations, you’re on your own with that – and you should know what you’re doing.

Caveats

There are some Calibre features which are not available (yet) with Kaliber and not currently supported:

custom columns defined by the respective Calibre user;
different/multiple libraries for the user to switch between;
OPDS formatted access;
book uploads are not planned to be included.

Once I figure out how they are realised by Calibre I expect they find their way into Kaliber as well (provided I find actually time to do it).

Logging

Like almost every other web-server Kaliber writes all access data to a logfile (logfile = in the INI file and -log at the commandline).

As privacy becomes a serious concern for a growing number of people (including law makers) – the IP address is definitely to be considered as personal data – the logging facility anonymises the requesting users by setting the host-part of the respective remote address to zero (0). This option takes care of e.g. European servers who may not without explicit consent of the users store personal data; this includes IP addresses in logfiles and elsewhere (eg. statistical data gathered from logfiles).

Since the generated logfile resembles that of the popular Apache server you can use all tools written for Apache logfiles to analyse the access data.

Libraries

The following external libraries were used building Kaliber:

Licence

    Copyright © 2019, 2020 M.Watermann, 10247 Berlin, Germany
                    All rights reserved
                EMail : <support@mwat.de>

2019-05-02

[*]

ErrorHandler

ErrorHandler

Purpose

The out-of-the-box Go web-server send plain text error messages whenever an HTTP error occurs. This middleware package provides a simple facility to send whatever HTML page you like for error-pages.

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/errorhandler

Usage

This package defines the TErrorPager interface which requires just one method:

TErrorPager interface {
    // GetErrorPage returns an error page for `aStatus`.
    //
    // `aData` is the original error text.
    //
    // `aStatus` is the error number of the actual HTTP error.
    GetErrorPage(aData []byte, aStatus int) []byte
}

If the method’s return value is empty then aData is sent as is to the remote user, otherwise the method’s result is sent thus delighting your users with your customised page.

Once you’ve implemented such a method you call the package’s Wrap() function:

Wrap(aHandler http.Handler, aPager TErrorPager) http.Handler

The arguments are:

aHandler is your original HTTP handler which will be wrapped by this package (and used internally).
aPager is the provider of error message pages as discussed above.

In the sub-directory ./cmd there is the file demo.go which shows the bare minimum of how to integrate this package with your web-server.

Libraries

No external libraries were used building ErrorHandler.

Licence

Copyright © 2019, 2020 M.Watermann, 10247 Berlin, Germany
                All rights reserved
            EMail : <support@mwat.de>

[*]

ApacheLogger

ApacheLogger

Purpose

This package can be used to add a logfile facility to your Go web-server. The format of the generated logfile entries resemble those of the popular Apache web-server (see below).

Installation

You can use Go to install this package for you:

go get -u github.com/mwat56/apachelogger

Usage

To include the automatic logging facility you just call the Wrap() function as shown here:

func main() {
	// the filenames should be taken from the commandline
	// or a config file:
	accessLog := "/dev/stdout"
	errorLog := "/dev/stderr"

	pageHandler := http.NewServeMux()
	pageHandler.HandleFunc("/", myHandler)

	server := http.Server{
		Addr:    "127.0.0.1:8080",
		Handler: apachelogger.Wrap(pageHandler, accessLog, errorLog),
		//       ^^^^^^^^^^^^^^^^^^
	}
	apachelogger.SetErrLog(&server)

	if err := server.ListenAndServe(); nil != err {
		apachelogger.Close()
		log.Fatalf("%s: %v", os.Args[0], err)
	}
} // main()

So you just have to find a way the get/set the name of the desired logfiles – e.g. via a commandline option, or an environment variable, or a config file, whatever suits you best. Then you set up your server like shown above using the call to apachelogger.Wrap() to wrap your original pagehandler with the logging facility.

The creation pattern for a logfile entry is this:

apacheFormatPattern = `%s - %s [%s] "%s %s %s" %d %d "%s" "%s"`

All the placeholders to be seen in the pattern will be filled in with the appropriate values at runtime which are (in order of appearance):

remote IP,
remote user,
date/time of request,
request method,
requested URL,
request protocol,
server status,
served size,
remote referrer,
remote user agent.

It means you can now use all the logfile analysers etc. for Apache logs for your own logfiles as well.

Special Features

As privacy becomes a serious concern for a growing number of people (including law makers) – the IP address is definitely to be considered as personal data – this logging facility anonymises the requesting users by setting the host-part of the respective remote address to zero (0). This option takes care of e.g. European servers who may not without explicit consent of the users store personal data; this includes IP addresses in logfiles and elsewhere (eg. statistical data gathered from logfiles).

While the logging of web-requests is done automatically you can manually add entries to the logfile by calling

apachelogger.Log(aSender, aMessage string)

The aSender argument should give some indication of from where in your program you’re calling the function, and aMessage is the text you want to write to the logfile. To preserve the format of the log-entry neither aSender nor aMessage should contain double-quotes ("). The messages are logged as coming from 127.0.0.1 with an user-agent of mwat56/apachelogger; this should make it easy to find these messages amongst all the ‘normal’ ones.

If you want to automatically log your server’s errors as well you’d call

apachelogger.SetErrLog(aServer *http.Server)

during initialisation of your program. This will write the errors thrown by the server to the errorlog passed to the Wrap() function. Additionally you can call

apachelogger.Err(aSender, aMessage string)

from your own code to write a message to the error log.

If you want to finish the logging altogether you’d call

apachelogger.Close()

Usually you’d only call this function after your server terminated; it’s not possible to restart the logging after calling Close().

To avoid that a panic crashes your program this module catches and recovers such situations. The error/cause of the panic is written to the error logfile for later inspection.

Libraries

No external libraries were used building ApacheLogger.

Licence

    Copyright © 2019, 2020 M.Watermann, 10247 Berlin, Germany
                    All rights reserved
                EMail : <support@mwat.de>

1834 Artikel
504 Hashtags
714 Erwähnungen

#5g #abmahnung #abortion #abuse #advertising #agriculture #ai #air #alcohol #alexander_fleming #algorithm #ambroise_paré #anatomy #andreas_vesalius #anesthesia #animal_husbandry #anonymity #anopensecret #antisemitism #antizionism #apartheid #archive #artikel13 #assault #astronomy #asylum #austerity #authoritarians #autogynephilia #aviation #backdoor #banks #becourageus #bidenerasedwomen #biology #biometrics #bitcoin #blacklivesmatter #blackpantherisaltright #blockchain #border #boristhebutcher #bosnia #boycott #brain #brexit #buddhism #bundesdatenschutzgesetz #burstiness #bürgerversicherung #caa #canada #cancel_culture #cancer #cannabis #capitalism #carbon #cat #causality #causalitydilemma #censorship #census #ceta #charger #chatgpt #children #christianity #church #civil_disobedience #civil_rights #climate_change #cloud #co2 #coal #commerce #communication #communism #computer #conspiracy #copyright #corona #corruption #crime #cross #cryptoleaks #culture #dalit #data_protection #death #deepfake #defamation #democracy #detransition #devops #diasporastudies #dictionary #digitisation #dioscorides #discrimination #diseases #diversity #diy #dmca #docker #drm #drones #drugs #dsgvo #döner #ebooks #eccentricity #echo_chamber #ecology #ecommerce #economy #education #edward_jenner #egypt #eikonal #election #email #eme #emotions #encryption #energy #equality #erasistratus #espionage #ethics #evolution #experimentation #extremism #facial_recognition #fake_news #fascism #fbgc #fbpe #femicide #feminism #file_sharing #filibuster #filterbubble #fingerprints #fire #fisa #fish #food #football #footnotes #forest #fracking #fraud #free_speech #free_tibet #freedom #freedomofexpression #fridays-for-future #future #galen #galileo_galilei #gambling #gay #gdpr #gender #genderwoowoo #genetics #genitalia #genocide #germany #getstonewalloutofourinstitutions #git #github #glyphosat #gnu #go #golang #gratitude #greece #greed #grundeinkommen #grundgesetz #gutachten #hacker #hacking #harassment #hashtag #hashtags #hate #headphone #health #healthcare #herophilus #hindi #hindu #hippocrates #history #holocaust #homophobia #homosexuality #honour #human_genome_project #human_rights #hygiene #ice #identity #ideology #illness #imhotep #immunity #impeachment #imperialism #imsi-catcher #inclusiveness #independence #india #indien #industrialisation #industry #infection #influenza #infowar #infrastructure #intelligence #interdependence #internet #intersex #iot #iran #isaac_newton #islamophobia #isp #israelipalestinianconflict #istandwithjkrowling #istandwithkeirabell #istandwithmaya #italia #italy #james_young_simpson #javascript #jealousy #jewishethics #jewishhistory #jewishidentity #jewishscholars #jews #johannes_kepler #jornalism #joseph_lister #journalism #justice #keepiton #knowledge #kohleausstieg #labour #language #lgb #lgballiance #liability #liberty #library #licence #lifegetsbetter #linguistics #literature #lobbyism #love #maaslos #marijuana #markdown #marketplaces #marriage #masturbation #meat #media #medicaid #medicine #men #mental-health #metadata #metaphysics #metoo #mietpreisbremse #migration #military #minderheitsregierung #mindset #mining #misogyny #missiles #mobbing #mobile_phone #money #motivation #movies #murder #music #muslim #mussels #mutilation #nacktimnetz #narendra_modi #nationalism #nazis #net_neutrality #netherlands #netzdg #neurodiversity #new_zealand #newspapers #niewiedercdu #nlp #nobel_prize #nuclear_weapons #nürnberg #obamacare #observation #obsession #occupation #oil #open_source #openai #oppression #organspende #paedophiles #pandemic #paris_agreement #parliament #parteispenden #password #patent #paternalism #pc #peace #pension #perception #perplexity #personaldevelopment #pharma #philosophy #phishing #phone #physics #physiology #pigs #piracy #poem #police #politicalphilosophy #pollution #populism #pornography #poverty #prison #privacy #privacy_shield #privatisation #programmers #programming #prompt #prompts #propaganda #prostitution #protest #psychiatry #psychology #puberty_blocker #putsch #quicksilver #racism #rain #rain-forest #rape #readme #reality #recycling #refugees #regime_change #relationships #religion #religiousdebate #renaissance #rent #reportit #repression #research #rigaer94 #righttoknow #riots #robot #roma #rome #router #safety #sanctions #sand #satire #saveourinternet #scandals #schufa #science #scifi #sea_weed #secrecy #secularism #security #selbstbestimmungsgesetz #selfid #selfimprovement #sex #sexism #sexuality #sjw #smartphone #smoke #snowflakes #social-media #social_media #software #solar_panel #solidarity #soy #space #spam #sports #staatstrojaner #sterilisation #stgb #stophateforprofit #streisand-effekt #strike #subsidies #sugar #suicide #supremacy #surgery #surveiilance #surveillance #tagname #tanks #tax #technology #telegram #tents #terror #terrorism #thisisnotadrill #tobacco #tolerance #tomato #torture #tq #tracing #tracking #trade #traffic #transgender #transparency #transsexual #trees #trojan_horse #truth #ttip #tuberculosis #tv #twitter #uncertainty #uploadfilter #vaccination #vaccines #victim #video #violence #volkstrojaner #vorratsdatenspeicherung #voting #wall #war #waronwomen #waste #water #watergate #wealth #weapons #weather #whistleblower #white_house #who #wifi #william_harvey #william_thomas_green_morton #wind #wokeness #women #wordstar #writing #www #zensur #zensur-behörden #zensur-maßnahmen #zensurheberrecht #zeppelin #zionism #§218 #§219 @aadhaar @aakashhassan @abigail_shrier @aclu @adl @adolf_eichmann @adolf_hitler @afd @afghanistan @africa @alain_de_botton @albania @alessandro_strumia @alex_jones @alexa @alexandria_ocasio-cortez @alice-salomon-hochschule @alice_schwarzer @alice_weidel @alliancelgb @allie_funk @allison_bailey @alphago @amanda_tapping @amazon @amnesty_international @andrea_voßhoff @andreas_scheuer @android @angela_davis @angela_merkel @anis_amri @anke_ehrhardt @antifa @antoni_comín @anyabike @aol @apple @ard @area @aserbaidschan @asia @assam @att @attac @attila_hildmann @augusto_pinochet @australia @austria @axel_voss @ayodhya @bahrain @balochistan @bangladesh @barack_obama @barcelona @barereality @barry_reay @basel @bavaria @bayer @bbc @beatles @beijing @belarus @berlin @bernie_sanders @betsy_reed @bettina_gaber @betty_steiner @bgh @bill_clinton @bill_gates @bill_oreilly @bitcoin @bjp @bjportraits @bka @blokada @bmw @bnd @bob_murray @bodo_ramelow @bolivia @bollywood @boris_johnson @boris_yeltsin @botswana @brasilia @brazil @brazil's @bsi @buchbinder @bulgaria @bundesfinanzhof @bundestag @bverfg @caityjohnstone @california @cambridge @cambridge_analytica @cameroon @campact @canada @canada's @cargill @carles_puigdemont @carola_rackete @catalonia @cathy_newman @ccc @cdu @charles_ihlenfeld @charles_stross @charlize_theron @chelsea_manning @chemnitz @chennai @christa_peterson @christian_lindner @christine_lambrecht @christoph_heusgen @cia @cia’s @cisco @claus_schenk_graf_von_stauffenberg @clearview @clinton_foundation @coca-cola @columbia @cornelsen @cory_doctorow @costa_rica @croatia @crypto_ag @csu @cuba @dachau @daimler @dalai_lama @dale_o'leary @dan_gillmor @daniel_suarez @darknet @darwin @david_bell @david_cameron @dbd @debra_soh @defectivebydesign @delhi @deniz_yücel @denmark @dentons @deutsche_bahn @deutsche_bank @deutsche_wohnen @dexter @dfb @dgb @dhs @die_linke @die_partei @digitalcourage @dilay_banu_büyükavci @dirgegirl @disney @docstockk @dominic_cummings @donald_trump @donna_hughes @douma @dr_em @drbiden @dreamhost @dresden @drummond_pike @duckduckgo @düsseldorf @ecosia @ecuador @eddie_izzard @edinburghuni @edward_snowden @edward_w_felten @eff @egypt @einar_wegener @ellen_page @elmar_brok @elon_musk’s @elsevier @emmanuel_macron @eritrea @erwin_schrödinger @ethiopia @eu @eugen_gomringer @europe @europeancommission @evo_morales @ezb @facebook @fairplaywomen @fbi @fcc @fdp @fefe @fff @finfisher @finland @firefox @florida @focus @fondofbeetles @forwomenscot @fox_news @frag_den_staat @france @frank-walter_steinmeier @frank_rieger @franz_josef_strauß @françois_hollande @friedrich_merz @frontex @fsf @ftp @g20 @gaby_weber @gchq @gender @george_floyd @george_orwell @george_osborn @george_soros @george_takei @george_w_bush @germaine_greer @germany @gff @ggreenwald @ghana @gids @gimp @gina_rippon @github @glastonbury @glenn_greenwald @god @google @google’s @great_britain @greece @greenpeace @greta_thunberg @groko @grüne @guardian @guinness @gurbanguly_berdimuhamedov @göttingen @günther_oettinger @haiti @halle @hamburg @hannah_arendt @hans-christian_ströbele @hans-georg_maaßen @hans-peter_uhl @harry_benjamin @harvard's @heiko_maas @helensteel12 @helsinki @her @hermann_otto_solms @hillary_clinton @hjoycegender @hohenzollern @honduras @hong_kong @hongkong @horst_seehofer @huawei @hubertus_heil @hungary @ican @iceland @idad @ietf @iglyo @ikea @immunity @imran_khan @india @intercept @internet_archive @ipcc @iran @iraq @ireland @isaac_asimov @israel @istandwithher1 @italy @iwf @jacob_appelbaum @jacob_rees-mogg @jair_bolsonaro @james_comey @james_damore @james_flynn @jan_kalbitzer @jane_philpott @janice_raymond @japan @jared_kushner @jawaharlal_nehru @jean_luc_melenchon @jeff_bezos @jeff_sessions @jennifer_pritzker @jens_spahn @jeremy_corbyn @jeremy_hunt @jesse_williams @jessica12uk @jimmy_kimmel @jitsi @jk_rowling @jo_bartosch @jody_wilson-raybould @joe_biden @joe_rogan @john_le_carré @john_money @john_oliver @john_oliver's @jon_stryker @jonathan_pie @jonathan_pie's @jordan_peterson @jordanbpeterson @joseph_fischer @juan_guaido @judith_butler @julian_assange @julian_assange’s @justin_trudeau @jutta_ditfurth @kali_linux @kamala_harris @karin_strenz @karl_marx @kaschmir @kashmir @kaspersky @katarina_barley @katharina_nocun @kathleen_stock @kathmurray1 @kazakhstan @ke_jie @keir_starmer @keira_bell @kenneth_zucker @kevin_kühnert @kickstarter @kim_jong-un @kkk @klbfax @kolkata @kristina_hänel @kurds @laura_poitras @lawrence_kubie @lawrence_lessig @lebanon @leipzig @lesbianlabour @lesleysemmens @lili_elbe @lilylilymaynard @linkedin @linus_torvalds @linux @lisa_eckhart @lithium @liverpool @london @lufthansa @lybia @mad @magnus_hirschfeld @mahatma_gandhi @malaysia @malta @margaret_hamilton @margaret_thatcher @margarete_stokowski @marine_le_pen @mark_bonham @mark_hamill @mark_weinstein @mark_zuckerberg @markmair @mars @martin_schulz @martin_sonneborn @martina_navratilova @martine_rothblatt @mathias_döpfner @matteo_salvini @mauramaxwell @max_schrems @maya_forstater @mcleaver @medium @mellibeinhorn @mention @mentionedname @mentions @mermaids @mesut_özil @mewe @mexico @michael_flynn @michel_temer @michigan @microsoft @mike_pence @mike_pompeo @minneapolis @minnesota @mirandanewsom @miroandrej @monsanto @mordor @mosaic @mozilla @muhammadali_jinnah @mumbai @munich @mutilation @mwat @myanmar @narendra_modi @nawaz_sharif @nayantara_sehgal @nestle @netflix @new-delhi @new_york @new_york_times @new_zealand @nextcloud @nga @nhs @nicaragua @nicolas_maduro @nigel_farage @nigeria @nils_melzer @noam_chomsky @norbert_röttgen @north_korea @northern_ireland @norway @nsa @nytimes @ofcom @okbiology @okuna @olaf_scholz @oliver_bierhoff @onelook @opcw @openai @openbook @oriol_junqueras @osaka @oskar_lafontaine @oslo @oxfam @oxford @pakistan @palestine @panama @pankhurstem @paris @parler @passie_kracht @patricia_trimble @patrick_breyer @paul_mchugh @paul_ziemiak @pen @penny_pritzker @pentagon @pepsi @peter_altmaier @peter_handke @peter_ramsauer @peter_schaar @philip_sutton @philipp_amthor @philippines @pimeyes @piraten @pokémon @poland @pope_francis @portland @portugal @posteo @postillon @pr_china @privacyint @profalices @public_domain @puerto_rico @qanon @rachel_levine @raf @rainer_meyer @rainer_wendt @rana_ayyub @ranaayyub @rashmee_kumar @raspberry_pi @ray_blanchard @reality_winner @recep_tayyip_erdoğan @reconquista_internet @reinhard_grindel @renate_künast @rex_tillerson @rezo @rheinmetall @ric_weiland @richard_fitzgibbons @richard_grenell @richard_nixon @ricky_gervais @riot @rob_hoogland @robert-koch-institut @robert_muller @robert_stoller @rodrigo_duterte @romania’s @rose_of_dawn @rote_hilfe @rss @rudolf_henke @rupert_murdoch @russia @ruth_hunt @rüdiger_kruse @saddam_hussein @safoora_zargar @samsung @san_francisco @sappfo @saskia_esken @saudi_arabia @scandinavia @sci-hub @scotland @sean_spicer @sebastian_brehm @sebastian_kurz @seerutkchawla @segmtweets @serbia @shelia_jeffreys @sibiria @sibylle_berg @siemens @sigmar_gabriel @signal @silvio_berlusconi @sina @singapore @siri @sjferguson3 @smh @somalia @sonia_kennebeck @south_africa @south_korea @south_sudan @spain @spd @spezialdemokratie @spiegel @srf @startpage @stellaomalley3 @stern @steve_bannon @steven_pinker @stonewall @strato @sun @sundar_pichai @susan_stryker @suzanne_moore @sweden @swipewright @switzerland @symantec @syria @taiga @taiwan @tajikistan @taliban @tamil_nadu @tampax @tavistock @taz @teamviewer @teatea1337 @telegram @terry_pratchett @thailand @the_economist @the_guardian’s @the_intercept @theresa_may @thierry_breton @thomas_de_maizière @thomas_de_maiziére @thomas_szasz @thomson_reuters_foundation @threema @tiananmen @tibby17 @tibet @tiktok @tim_berners-lee @tim_cook @tim_gill @tinder @tonto_1964 @tony_blair @tor @toronto @transgendertrd @turkey @turkmenistan @twisterfilm @twitter @tönnies @uber @ulla_schmidt @uluru @un @united_internet @uno @ursula_k_le_guin @ursula_von_der_leyen @usa @valentinamedici @vanessa_rodel @varavara_rao @venezuela @venezuela’s @verfassungsschutz @verizon @vietnam @viktor_orban @vladimir_putin @volker_kauder @volkswagen @w3c @wallonia @washington_post @washingtonpost @wau_holland @wdr @welt @whatsapp @who @wikileaks @wikileaks' @wikipedia @william_golding @windows @wire @wirecard @wolf @wolfgang_kubicki @wolfgang_schulz @wolfgang_schäuble @womans_place_uk @womenreadwomen @wpath @wwf @xi_jin_ping @xiaomi @xr @xychelsea @xychelsea's @yanis_varoufakis @yaniv @yemen @yonatanzunger @youtube @yugoslavia @zdf @zoom @zürich @évian

“Master” branch and racism

Whitespace

Purpose

Installation

Usage

Libraries

Licence

Nele Blog

Purpose

Features

Installation

Usage

Commandline postings

Authentication

User/password file & handling

Page/link previews

Configuration

URLs

Static URLs

Common URLs

Internal URLs

Files

CSS

Fonts

Images

Postings

Static

Views

Contents

Libraries

Licence

UploadHandler

Purpose

Installation

Usage

Libraries

Licence

HashTags

Purpose

Installation

Usage

Libraries

Licence

PassList

Purpose

Installation

Usage

Password list

Libraries

Licence

Kaliber

Note

Purpose

Features

Installation

Usage

Commandline options

INI file

Authentication

User/password file & handling

Directory structure

Caveats

Logging

Libraries

Licence

ErrorHandler

Purpose

Installation

Usage

Libraries

Licence

ApacheLogger

Purpose

Installation

Usage

Special Features

Libraries

Licence