Posts categorized “Linux”


February 18, 2017
  Generating a Jekyll Site on Mercurial (Hg) Push

As we mentioned in our last post, we plan to share aspects of how we moved to Jekyll. This is the first of these posts.

Background

With a database-based solution, updating is easy; when the user updates content through the user interface, the new content is served when the page or post is requested. However, with a static site, an “update” is technically any change to the underlying files from which the site is generated. Typically, though, this is marked by source control commit and push to a master repository. GitHub Pages, the product for which Jekyll was developed, uses this as a flag to regenerate the site. We weren’t using GitHub*, though - we were using Mercurial (Hg) for our source code control, with the master repository on a different server than the one from which the site is served.

* There were a few reasons we did not wish to host our sites using GitHub, none of which are pertinent to how this works.

Options

With the need to regenerate the site after each site’s master repository receives a push, there were a few different options we considered.

  1. When a push occurs, regenerate the site on the Hg server, then use scp to delete the old files from and copy the new files to the web server.
  2. Set up a sandbox on the Hg server that updates and regnerates each time a push occurs, and run rsync on the web server to check for updates every so often.
  3. When a push occurs, notify the web server, and have it regenerate the site.

The first option has the potential to run afoul of SSH rate limits, plus has the potential to require much more data transfer than option 3. The second option had the advantage of running a process local to the Hg server, but would have required disk space utilization that we didn’t really need; and, as Jekyll regenerates all the pages in a site, rsync would have likely ended up transferring all the data for every update anyway, losing one of its benefits. The third option required Jekyll to be installed on the web server, and uses it for processing, potentially taking cycles that could be used to serve web pages.

Eventually, we decided to go with option 3.

Script All the Things

On the Hg server, in the master repository for each site, we put the following in .hg/hgrc (the following examples are for this site):

hgrc
1
2
[hooks]
incoming = /opt/jobs/notify.tech-blog.sh

…and then, notify.tech-blog.sh

notify.tech-blog.sh
1
2
#!/bin/bash
ssh user@web.server touch /opt/jobs/jekyll/.tech-blog

That is the only logic required on the Hg server. Now, over on the web server, we need logic to regenerate the site and make it live. Since we have multiple sites, we wrote a script that has a few variables, so it could be duplicated for other sites. The following is tech-blog.sh:

tech-blog.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
##
## DJS Consulting Tech Blog Jekyll Build Script
##
## This will check out, build, and replace a Jekyll-generated site. Just update
## the parts under the "Env" heading for the specific site.
##
## Env
REPO=jekyll.tech-blog
DEST=/path/to/public/techblog
TRIGGER=.tech-blog
## /Env
cd /opt/jobs/jekyll
if [ -e $TRIGGER ]
then
rm $TRIGGER
## Check out the site and build it
hg clone ssh://user@hg.server/HgPath/$REPO $REPO
cd $REPO
jekyll build
## Copy it to the proper directory
cd _site
rm -r $DEST/*
cp -r * $DEST
## Clean up
cd ../..
rm -r $REPO
fi

This script isn’t perfect; it needs to check the exit code from the Jekyll build process before whacking the current site (and notifying for a failed build would be a nice addition). However, with Jekyll being the same on both development and production, and a single committer, this is fine for our purposes.

Finally, each script needs to be run to check for the presence of the semaphore (or TRIGGER, as the script calls it). The following cron definition will check every 4 minutes for a change.

crontab
1
*/4 * * * * /opt/jobs/jekyll/tech-blog.sh > /dev/null

Conclusion

Overall, we’re pleased with the results. The inter-server communication is light, only requiring one initiated ssh connection from each server, so we won’t run afoul of rate limits. With the work being done on the destination server, the amount of time where there are no files in the directory (between the rm -r $DEST/* and the time the cp -r * $DEST finishes) is very short; it would have been much longer if the directory were being repopulated across the network, or more complex if we added a staging area on the web server. Each piece can be run separately, and if we’ve committed a post with a future date, we can run the same touch command to make that post appear.

Next time, we’ll discuss our experiences converting a non-WordPress site.

Categorized under , ,
Tagged , , , , , , , , ,


September 3, 2010
  Mono / FastCGI Startup Script

We’ve begun running Mono on some Bit Badger Solutions servers to enable us to support the .NET environment, in addition to the PHP environment most of our other applications use. While Ubuntu has nice packages (and Badgerports even brings brought them up to the latest release), one thing that we were missing was a “conf.d”-type of configuration; my “/applications=” clause of the command was getting really, really long. We decided to see if we could create something similar to Apache / Nginx’s sites-available/sites-enabled paradigm, and we have succeeded!

To begin, you’ll need to create the directories /etc/mono/fcgi/apps-available and /etc/mono/fcgi/apps-enabled. These directories will hold files that will be used define applications. The intent of these directories is to put the actual files in apps-available, then symlink the ones that are enabled from apps-enabled. These files have no name restrictions, but do not put an extra newline character in them. The script will concatenate the contents of that file to create the MONO_FCGI_APPLICATIONS environment variable, which tells the server what applications exist. (The syntax is the same as that for the “/applications=” clause - [domain]:[URL path]:[filesystem path].) Here’s how the site you’re reading now is configured (from the file djs-consulting.com.techblog.conf)…

djs-consulting.com.techblog.conf
1
techblog.djs-consulting.com:/:/path/to/install/base/for/this/site

Finally, what brings it all together is a shell script. This should be named “monoserve” and placed in /etc/init.d. (This borrows heavily from this script a script we found online, which we used until we wrote this one.) Note the group of variables surrounded by the “make changes here” notes - these are the values that are used in starting the server. They are at the top so that you can easily modify this for your own needs.

monoserve
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
#/bin/bash
### BEGIN INIT INFO
# Provides: monoserve.sh
# Required-Start: $local_fs $syslog $remote_fs
# Required-Stop: $local_fs $syslog $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start FastCGI Mono server with hosts
### END INIT INFO
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/mono
NAME=monoserver
DESC=monoserver
## Begin -- MAKE CHANGES HERE --
PROGRAM=fastcgi-mono-server2 # The program which will be started
ADDRESS=127.0.0.1 # The address on which the server will listen
PORT=9001 # The port on which the server will listen
USER=www-data # The user under which the process will run
GROUP=$USER # The group under which the process will run
## End -- MAKE CHANGES HERE --
# Determine the environment
MONOSERVER=$(which $PROGRAM)
MONOSERVER_PID=""
FCGI_CONFIG_DIR=/etc/mono/fcgi/apps-enabled
# Start up the Mono server
start_up() {
get_pid
if [ -z "$MONOSERVER_PID" ]; then
echo "Configured Applications"
echo "-----------------------"
# Construct the application list if the configuration directory exists
if [ -d $FCGI_CONFIG_DIR ]; then
MONO_FCGI_APPLICATIONS=""
for file in $( ls $FCGI_CONFIG_DIR ); do
if [ "$MONO_FCGI_APPLICATIONS" != "" ]; then
MONO_FCGI_APPLICATIONS=$MONO_FCGI_APPLICATIONS,
fi
MONO_FCGI_APPLICATIONS=$MONO_FCGI_APPLICATIONS`cat $FCGI_CONFIG_DIR/$file`
done
export MONO_FCGI_APPLICATIONS
echo -e ${MONO_FCGI_APPLICATIONS//,/"\n"}
else
echo "None (config directory $FCGI_CONFIG_DIR not found)"
fi
echo
# Start the server
start-stop-daemon -S -c $USER:$GROUP -x $MONOSERVER -- /socket=tcp:$ADDRESS:$PORT &
echo "Mono FastCGI Server $PROGRAM started as $USER on $ADDRESS:$PORT"
else
echo "Mono FastCGI Server is already running - PID $MONOSERVER_PID"
fi
}
# Shut down the Mono server
shut_down() {
get_pid
if [ -n "$MONOSERVER_PID" ]; then
kill $MONOSERVER_PID
echo "Mono FastCGI Server stopped"
else
echo "Mono FastCGI Server is not running"
fi
}
# Refresh the PID
get_pid() {
MONOSERVER_PID=$(ps auxf | grep $PROGRAM.exe | grep -v grep | awk '{print $2}')
}
case "$1" in
start)
start_up
;;
stop)
shut_down
;;
restart|force-reload)
shut_down
start_up
;;
status)
get_pid
if [ -z "$MONOSERVER_PID" ]; then
echo "Mono FastCGI Server is not running"
else
echo "Mono FastCGI Server is running - PID $MONOSERVER_PID"
fi
;;
*)
echo "Usage: monoserve (start|stop|restart|force-reload|status)"
;;
esac
exit 0

This needs to be owned by root and be executable (chmod +x monoserve). You can use update-rc.d monoserve defaults to set this to start at boot.

Categorized under , , , ,
Tagged , , , ,


September 10, 2004
  Webshots - Wine Strikes Again!

When I ran Windows as my desktop, I had a program called WebShots that I used to set my desktop wallpaper, and cycle it. They have Windows and Mac versions, but no Linux version yet. They still send me e-mails each week, showing the daily picture selections for each day in the past week. I decided to download the Windows version, and install in under wine to see if it would work. I moved websamp.exe to /home/summersd/.wine/fake_windows, then ran wine C:\\websamp.exe to install the program. wine "C:\\Program Files\\Webshots\\Launcher.exe" then started the desktop control. I used that to disable the tray icon (wine has one, but you can’t see it), and I disabled almost every other “auto update” feature.

I had downloaded a “.wbz” file (which is what is imported into WebShots), and I finally figured out how to import it. Running the launcher program, and following it with the name of the .wbz file, imports it. I may figure out a way to automate that, but for now, I know how to do it.

(Note: This is the end of the “My Linux Adventure” series of posts. After this, I ended up going back to Windows XP, just because it worked and I didn’t have hobbyist time. As of May 2007, I’m running Ubuntu 7.04 on one computer, and Windows Vista on my laptop, which is currently out of commission.)

Categorized under ,


September 8, 2004
  Apache and MySQL Are Back

I was finally able to resolve my problems with Apache and MySQL. When I decided to mount my FAT32 drive under /home/summersd, I inadvertently caused myself some problems. From talking to a Linux guy at work, I found that no processes that weren’t running under my user ID could access those files. The reason is that Linux looks up the entire diretory tree, back to /, to determine if you can access the file. So, although I had -rwxrwxrwx summersd summersd on every file, /home/summersd was drwx—— summersd summersd, and /home was drwxr-xr-x. The permissions on /home/summersd was keeping Apache from seeing /home/summersd/drive_d/wwwroot, and MySQL from seeing or writing to /home/summersd/drive_d/mysql/data. I moved the drive to /mnt/drive_d, with the mount point being owned by “root”, still mounting the drive with my user name, and everything worked.

In the process of reconfiguring Thunderbird, I believe I may have found out how to share the address book across operating systems. The file ~/.thunderbird/default.[something]/prefs.js has a listing of all the preferences and settings. I modified this file to change the location of my mail files, and there is a setting there for an address book (which isn’t shown in the configuration dialog - after all, it is 0.7.3…) I’ll play with that later - right now I’m just elated to have Apache and MySQL working again.

Categorized under , , , , ,


September 8, 2004
  foobar2000 with Wine

foobar2000 is about the best, most organized audio player I’ve found. However, it is a Windows application, and according to what I’ve read, very reliant on Microsoft C++ extensions. I decided to give it a shot under wine, and it works great! There is a repaint problem - sometimes the playlist doesn’t refresh as it should. But, it’s pretty much a start-and-minimize sort of application, so that’s acceptable.

Categorized under ,


September 7, 2004
  ndiswrapper May Have Issues

We’re speedily working towards a Friday deadline at work, so tonight I had some analysis work to do on some COBOL code. Great, I thought, I can use my VSlick setup under wine. I moved my computer from the living room back to our now-empty bedroom (soon to be nursery), and booted it up. Kernel panics galore - never got past the network stuff. When I booted to Windows, I found that the wireless network didn’t reach that far, and I’m guessing that the ndiswrapper folks haven’t tried their driver a lot with a wireless card, but no wireless network. Once I get past that, I may grab the dumps from these kernel panics and see if the developers need them to see what went wrong. So, for tonight, I had to use WXP (in which I actually had to disable the wireless connection - seems Windows doesn’t handle a barely-there wireless connection much better than Linux).

The diagnostics I ran last night never found anything - they ran for about 10 hours. I suppose I’ll just have to wait until I have problems again, then run it right then. Another person from the WBEL users list suggested I check the way I have my hard drives set up; he thinks that a 2GB drive slaved to a 20GB drive may be causing conflicts, which would cause freezes or panics.

Categorized under , ,