/ freebsd

Configuring FreeBSD for Infrastructure

FreeBSD is a robust operating system that has its roots in the Bell Laboratories Unix Operating System. Technically, FreeBSD is derived from the BSD, Berkeley Software Distribution Unix. Now that you're all caught up, why FreeBSD?

I've been obsessing lately with how things work. Well, specifically, I've been obsessing with how computers work (try as I might, I'll never understand my car's cooling system). The CPU clocks, counters, registers, networks, I/O, all of it. I've been getting lost in books such as The Elements of Computing Systems, Structure and Interpretation of Computer Programs, The Unix and Linux System Administration Handbook, and a ton of others (I will include links below). Needless to say, there is a lot to it. But, there was one thing in common with all of the material I was reading and referencing: "Use Unix". You can infer that to mean "Use Linux and/or Unix".

The question of moving from a Linux distribution (CentOS in my case) to FreeBSD for server space was two fold. The PERC H700 card I was using for RAID in my server was acting funny. I had replaced the battery for the RAID card twice, and still problems with performance persisted. I deleted the old array and rebuilt a new RAID 5 system, did fresh install on the host and VMs. Same thing, randomly getting errors about SAS cable connections and really slow performance with disk read and writes. That said, I had a few options, narrowing them down to buy a new RAID card, use individual disks, or use a filesystem that takes advantage of multiple disks of the same performance. Option 2 was out, for now, as the drives are SAS 10k drives. Decent, but not SSD. Option 1 is expensive. So, that left me with the filesystem option.

The literal tl;dr of using a filesystem option is FreeBSD has native ZFS options. Awesome blossom.

The process was so easy, I'm not going to go into too much detail, but here are the main bullets!

  • Download memstick.img
  • Burn to USB (I used Etcher)
  • Boot USB
  • Press ENTER
  • When disk option came up, I picked Auto ZFS
  • Select your RAID => Select your disks => Select INSTALL
  • Proceed through installation, set up users, network, and security measures

Reboot the system and you're ready to login or ssh in. By default you cannot ssh as root, so I recommend making a user as part of the wheel group so you can su to root.

Post Installation

The steps I took post installation were pretty standard of what you'd expect. I installed updates and some standard packages that I wanted to use.

# pkg update -- This will ask you to install the package manager
# pkg install -y sudo vim git

I added my user to sudo with visudo.

I enabled ntpd in /etc/rc.conf

ntpd_enable="YES"
ntpd_sync_on_start="YES"

Then I ran service ntpd start.

Took about two seconds before date synced with NTP.

Next, I wanted to configure a firewall for managing my services that I planned on using.

That was a bit involved, with some reference to the documentation. However, it was well worth it, because I learned a bit of networking via FreeBSD!

firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

I appended those lines to the end of /etc/rc.conf. Next, I created the file that firewall_script is pointing to.

# vim /usr/local/etc/ipfw.rules

Below are the rules I added to the file:

IPF="ipfw -q add"
ipfw -q -f flush

#loopback 
$IPF 10 allow all from any to any via lo0
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# statefull
$IPF 50 check-state
$IPF 60 allow tcp from any to any established
$IPF 70 allow all from any to any out keep-state
$IPF 80 allow icmp from any to any

# open port ftp (20,21), ssh (22), mail (25)
# http (80), dns (53) etc
$IPF 110 allow tcp from any to any 21 in
$IPF 120 allow tcp from any to any 21 out
$IPF 130 allow tcp from any to any 22 in
$IPF 140 allow tcp from any to any 22 out
$IPF 150 allow tcp from any to any 25 in
$IPF 160 allow tcp from any to any 25 out
$IPF 170 allow udp from any to any 53 in
$IPF 175 allow tcp from any to any 53 in
$IPF 180 allow udp from any to any 53 out
$IPF 185 allow tcp from any to any 53 out
$IPF 200 allow tcp from any to any 80 in
$IPF 210 allow tcp from any to any 80 out

# deny and log everything 
$IPF 500 deny log all from any to any

I ransh on the file to enable the new rules. My ssh connection didn't disconnect, which was awesome.

I also created a bridged network, because I had initially planned on firing up a Hypervisor and spinning up VMs. I later decided to go to jails. The bridge still works with jails, though:

# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
# ifconfig bridge0 create
# ifconfig bridge0 addm bce0 addm tap0
# ifconfig bridge0 up

My network adapter is bce0, you can find this with ifconfig if you're following along.

Setting Up Jenkins

I wanted to try getting Jenkins running through jails. The man page isn't too descriptive on what jail is:

The jail utility creates new jails, or modifies or removes existing
jails.  A jail (or "prison") is specified via parameters on the command
line, or in the jail.conf(5) file.

However, the FreeBSD documentation mentions that jail is more of a container service:

Some administrators divide jails into the following two types: “complete” jails, which resemble a real FreeBSD system, and “service” jails, dedicated to one application or service, possibly running with privileges. This is only a conceptual division and the process of building a jail is not affected by it. When creating a “complete” jail there are two options for the source of the userland: use prebuilt binaries (such as those supplied on an install media) or build from source.

Sounds like something that's right up my ally! Yay.

ONWARD TO THE CONFIG!

I ran jail to see if anything needed to be installed. It did not. I read a lot about using ezjail, which makes jail... easy... But, I enjoy doing things the hard way. I was also comforted that doing jails the jail way isn't that much more involved than doing jails the ezjail way. So, onto jail.

The first step recommending the jails process is setting up a directory. This was how I managed my VMs in KVM as well, each VM had their own directory. Anyway, there are some best practices which I decided to stick to:

# mkdir -p /usr/local/jails/jenkins

I pulled the FreeBSD system, the 32bit libraries, and the ports (software packages).

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.2-RELEASE/base.txz -o /tmp/system.txz
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.2-RELEASE/lib32.txz -o /tmp/lib32.txz
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.2-RELEASE/ports.txz -o /tmp/ports.txz

Next, I extracted the system and the files to the jails directory:

# tar -xvf /tmp/system.txz -C /usr/local/jails/jenkins
# tar -xvf /tmp/lib32.txz -C /usr/local/jails/jenkins
# tar -xvf /tmp/ports.txz -C /usr/local/jails/jenkins

After the system was set, I copied my /etc/resolv.conf and /etc/localtime to the jenkins systems.

# cp /etc/resolv.conf /usr/local/jails/jenkins/etc/resolv.conf
# cp /etc/localtime /usr/local/jails/jenkins/etc/localtime

Last, I'm going to edit the hostname of the jails box:

# vim /usr/local/jails/jenkins/etc/rc.conf

hostname="jenkins"

I setup the global jails settings before launching:

# vim /etc/jail.conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

# The jail definition for jenkins
jenkins {
    host.hostname = "jenkins.domain.local";
    path = "/usr/local/jails/jenkins";
    interface = "bridge0";
    ip4.addr = 10.0.0.115;
}

Time to fire it up!

# jail -c jenkins

The -c flag is for create.

After that, I jumped on via jexec!

# jexec jenkins /bin/sh

All good! ... Well, not quite. I couldn't ping out to google.com.

I jumped back out to my host and, after a bit of research, ran the following command:

# jail -m jid=1 allow.raw_sockets=1

I restarted the jail service and jumped back in via jexec. I was able to successfully ping out and pull software down.

# pkg install jenkins
...
# service jenkins start

Alright, time to rock!

I opened my browser and hit up 10.0.0.115:8080/jenkins

...

Page not found :(

Hm...

More research.

Ah... Weird, the Jenkins instance started on 8180 (found this out socketstat -4 -l). I tried the address above with changing the port. Still nothing. Hm... Very puzzl– Oh, DAMN, the firewall. I forgot!

Added:

$IPF 220 allow tcp from any to any 8180 in
$IPF 230 allow tcp from any to any 8180 out

to /usr/local/etc/ipfw.rules and restarted the firewall service.

Refreshed the page and success! Jenkins was asking for the secret in /usr/local/jenkins/secrets/initialAdminPassword. I went through the setup and was ready to rock and roll:

This was such an awesome experience, I had to write down in this post. I felt reinvigorated with my passion for technology. I knew almost nothing and had to pour into documentation, blog posts, forum posts, it was learning all over again. I was able to do this in a couple of hours versus weeks in my first try with Linux. Maybe that speaks to my evolution as a systems engineer or I.T. professional, or maybe FreeBSD is just much more accessible.

Either way, I'm going to stick with it and evolve my jails.conf to have every service I need for my tools. I plan on going deeper into the networking stack and the configuration management stack as well.

Ping us on Twitter or leave some comments below with any feedback, criticism, or questions!

Thanks for reading.

Links to fun books:

Structure and Interpretation won't link properly, sorry :(

The Elements of Computing Systems

Advanced Programming in the Unix Environment

Modern Operating Systems

Computer Science Distilled

Configuring FreeBSD for Infrastructure
Share this