Archive | Ubuntu RSS feed for this section

FlexGet installation : Ubuntu 12.04 How To

15 Jan

When I read posts on LifeHacker and Gizmodo sometimes, I figure that everybody else in the world is downloading free TV episodes via BitTorrent, and doing it using RSS feeds and an automated process. That can’t possibly be true, but I figured that if other people do this, I should be able to figure out how, too. It is pretty easy to find RSS feeds for various TV shows. Therfore, I thought it would be relatively simple to write a script to download episodes from each one. I figured that I would have to figure out how to sort out duplicate episodes and different quality types (720p, 1080p, etc.) somehow, too. It didn’t take me too long to find out that an excellent Python program called FlexGet already does all this. Plus, it runs from the command line, which is perfect for a headless server, like my Ubuntu Lucid Lynx box.

I don’t actually endorse doing this, by the way, but I was really curious to learn how it was done. I wanted to share what I learned, in case anyone was struggling with the configuration. Also, I am in the process of playing with ffmpeg to verify downloaded content. You will find sometimes, depending on how reliable your feed is, that FlexGet has the ability to add “fake” torrents to your path/queue.

Installation

FlexGet’s web site has an excellent walkthrough that describes how to install it on various operating systems. My instructions in this section are not meant to improve on that guide, but merely to summarize it for an Ubuntu Server install.

FlexGet requires Python 2.6. If you don’t already have that installed on your server, try:

$ sudo apt-get install python2.6 python-setuptools

You have to download FlexGet from their web site. Choose the version for Python 2.6. I prefer to copy the URL and download the file via wget.

$ wget http://download.flexget.com/unstable/FlexGet-1.0r1377-py2.6.egg

After you download FlexGet, install it via Python’s easy_install script. You’ll have to specify the name of the FlexGet file you downloaded.

$ sudo easy_install <downloaded egg>

That’s it! Flexget is now installed in the /usr/local/bin directory.

If you run Transmission like I do, you should also install the the TransmisionRPC plugin.

$ sudo easy_install transmissionrpc

Configuration

Configuring FlexGet is deceptively simple. The configuration file is written in a format I had heard of, but never encountered before: YAML. YAML is simple and powerful, which means that it can be deceptively complex. You can write a config file that does the same thing in a million different ways. FlexGet’s Cookbook shows you a few examples to get you started. It took me a lot of experimentation to find a config file format that I liked. Here is what I ended up with. It’s probably more complex than it needs to be, but I like the idea of using presets to customize different types of downloads. Note that, in YAML, each level of indentation is two spaces (no tabs are allowed).

$ nano ~/.flexget/config.yml

presets:
  tv:
    series:
      settings:
        720p:
          timeframe: 8 hours
        720p:
      - series title 1
      - series title 2
      - series title 3

transmissionrpc:
  host: localhost
  port: 9091
  netrc: /home/mjdescy/.netrc
  removewhendone: true
  addpaused: no
  ratio: 1.00

feeds:
  TvTorrent1:
    rss: http://tv.rss.url/
    preset: tv

  TvTorrent2:
    rss: http://tv.rss.url/
    preset: tv

This configuration file sets up a preset called “tv” which tells FlexGet to wait 8 hours before selecting episodes of any of the series listed under the 720p heading. 720p is the default quality setting for FlexGet, and is perfectly adequate for my needs. If FlexGet finds an episode of that quality, it will download it; if it finds episodes above that quality level, at 1080p for example, it will ignore them. If no 720p version exists after the “timeframe” of 8 hours is over, FlexGet will select the entry with the highest quality available. FlexGet won’t download the same episode more than once, unless there is a “proper” or “repack” version—a corrected version of a previously released episode—is available.

At the end of the “tv” preset, the “tranmissionrpc” heading instructs FlexGet to use the transmissionrpc plugin to directly send torrent URLs to transmission-daemon for download. This obviates the need to set a watch directory in transmission-daemon. Note that the “netrc” setting refers to a special username/password file that contains the user name and password required to log into transmission daemon. That file should be in the following format, hidden, in your home directory. Make sure you grant only your account access to read that file (i.e., chmod 600 ~/.netrc).

machine localhost
login username
password password123456789

If you do not run transmission-daemon, you could replace the “transmissionrpc” section with a “download” section, or use another plugin. Check out the flexget Cookbook for more information.

The “feeds” section lists all the TV torrent feeds to look in. I have each feed set to use the “tv” preset that I defined above.

Each RSS feed you list can contain a number of TV series (which is the easiest way to set it up), or just one series (you’ll have to find or build custom feeds for this). Flexget will search all the feeds you list for the series titles listed in the “tv” preset.

This configuration file works very well for me. I hope that it helps you get up and running.

Scheduling

FlexGet can be run manually for debugging purposes (flexget -v), but it should be scheduled to run periodically via cron. Note that there is no need to run FlexGet as root. I run it under my own user account. If you want to operate under the principle of least privilege, you could create a “flexget” account, edit its crontab, and grant that account the permissions it needs to get the job done. For my needs, that isn’t necessary. Therefore, to schedule FlexGet to run every hour, I simply edit my own crontab.

$ crontab -e

Add the following line to the crontab file, and save it.

0,30 * * * * /usr/local/bin/flexget –cron -c /home/mjdescy/.flexget/config.yml > /dev/null 2>&1

This cron job will run FlexGet at the top of every hour and half past the hour. FlexGet will read its options from the configuration file I specified (/home/mjdescy/.flexget/config.yml). I redirect output to /dev/null to prevent cron from sending me emails regarding the command’s output.

Advertisements

nmap vulnerability scanning | HowTo

17 Dec

Port and vulnerability scanners are common tools used by good as bad guys. Performing a port scanning is one of the first operations required to find potential vulnerabilities on a target system. That’s why vulnerability scanners have built-in port scanners. Writing a port scanner is really easy with a few lines of Perl:

#!/usr/bin/perl
use IO::Socket;
while ($ARGV[1] < 65536) {
  print STDOUT "$ARGV[0]:".($ARGV[1] - 1) . " open\n" if \
(IO::Socket::INET->new(PeerAddr=>"$ARGV[0]:" .$ARGV[1]++, Proto=>'tcp', Timeout=>1));
}

(Source: okc2600.com)

However, “real” port scanners offer much more options like evading techniques to work “below the radar” or fingerprinting. Nmap is the best tools for this purpose.

Synergies already exist between different scanning products. A good example is the integration of Nessus with Nmap. Nmap can save the scan results in XML format. The produced XML content can be re-used by Nessus to scan for vulnerabilities. By using this method, the power of Nessus is combined with the one of Nmap. For more information, read this article.

Performing a vulnerability scan  is highly resources consuming. Why not add a simple vulnerability scan feature to Nmap? This primary goal is to save time and be less intrusive. Nmap has a built-in script interpreter called NSE (“Nmap Scripting Engine“) which allows developers to write extensions for Nmap. It comes by default with a lot of scripts. If you’re interested, I posted an introduction article on NSE a few months ago.

Marc Ruef developed a NSE script which adds a basic vulnerability scanner feature to your Nmap. Technically, the script does NOT perform a vulnerability scan by itself. With the powerful fingerprinting feature of Nmap (using the “-sV” flag), the running applications and versions can be detected. Those information are used as lookup keys in a DB export of OSVDB, the Open Source Vulnerability Data Base. The matching entries are displayed in the script output. The script installation is extremely simple, just copy the files in your existing scripts repository (something like “$NMAP_INSTALL_PATH/share/nmap/scripts/“). Invoke it like any standard script:

# nmap -PN -sS -sV --script=vulscan -p80 www.company.tld

Starting Nmap 5.30BETA1 ( http://nmap.org ) at 2010-06-03 11:11 CEST
Nmap scan report for www.company.tld (10.0.0.1)
Host is up (0.00074s latency).
rDNS record for 10.0.0.1: www.company.tld
PORT   STATE SERVICE VERSION
80/tcp open  http    Apache httpd 2.2.11 ((Ubuntu) PHP/5.2.6-3ubuntu4.5 with Suhosin-Patch)
| vulscan: [48] Apache HTTP Server on Debian /usr/doc Directory Information Disclosure
| [143] Apache HTTP Server printenv.pl Multiple Method CGI XSS
| [222] Apache HTTP Server test-cgi Arbitrary File Access

[Stuff Deleted]

| [63895] Apache HTTP Server mod_headers Unspecified Security Issue
| [64023] Apache Tomcat WWW-Authenticate Header Local Host Information Disclosure
| [64020] Apache ActiveMQ Jetty ResourceHandler Crafted Request JSP File Source Disclosure
| [64307] Apache Tomcat Web Application Manager/Host Manager CSRF
| [64517] Apache Open For Business Project (OFBiz) View Profile Section partyId Parameter XSS
| [64518] Apache Open For Business Project (OFBiz) Show Portal Page Section start Parameter XSS
| [64519] Apache Open For Business Project (OFBiz) Control Servlet URI XSS
| [64520] Apache Open For Business Project (OFBiz) ecommerce/control/ViewBlogArticle contentId Parameter XSS
| [64521] Apache Open For Business Project (OFBiz) Web Tools Section entityName Parameter XSS
|_[64522] Apache Open For Business Project (OFBiz) ecommerce/control/contactus Multiple Parameter XSS

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 10.66 seconds

My first impression was disappointing: The scan reported too much vulnerabilities (>500 hits!). Unusable in a real environment. But, after reading the script (remember: RTFM!), Marc was aware of this problem (caused by a naming convention issue between Nmap & OSVDB). He added a correlation feature to reduce those false positives. To activate this option, just pass the following parameter:

# nmap -PN -sS -sV --script=vulscan --script-args vulscancorrelation=1 -p80 www.company.tld

Hopefully, this second test generated much less hits (26) but, side effect, required more time to complete.

This is a very nice feature for Nmap. By using this script, you can quickly have an overview of the potential vulnerabilities on a target host. And, if necessary, use a more classic tool to focus on specific cases. Don’t forget that false positives or false negatives and results must always be analyzed by a competent person.

To keep the vulnerability scanner accurate, the vulnerability DB must be kept up to date. To achieve this, you can automate the update using the CSV export available on osvdb.org (updated daily). First you have to register. Once done, you will be able to download the CSV updates via a permalink generated with your API key.  The upgrade can be fully automated via a simple daily cron and a script:

NMAPHOME=/usr/local/nmap
FILES="object_correlations.txt object_links.txt object_products.txt vulnerabilities.txt"
cd /tmp
wget -o /dev/null http://osvdb.org/file/get_latest_csv/xxxxx/osvdb-csv.latest.tar.gz
for FILE in $FILES
do
	tar xzf osvdb-csv.latest.tar.gz ./osvdb/$FILE
	mv osvdb/$FILE $NMAPHOME/share/nmap/scripts/vulscan
done
rm -rf osvdb
rm osvdb-csv.latest.tar.gz
exit 0

Marc released the version 0.6 is his script and has already a nice todolist (integration with other vulnerability databases). Great job!

fail2ban | How To | Ubuntu

19 Sep

fail2Ban is a very handy tool to prevent alot of unwanted traffic from consuming bandwidth on your servers. It’s a very small and relatively simple IDS Type Tool that comes with some predefined Filters to automatically lock out potentially dangerous or bandwidth consuming type attacks.

This tutorial covers basic installation and setup along with giving you an example of a simple custom filter to help you on the way to writing your own custom Event Based Blocking rules (filters) for use across the applications you run.

The installation is described at http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Debian  I used apt-get and it went painlessly

$ sudo apt-get install fail2ban
Configuration

I’m on Ubuntu 12.04 and the install location is /etc/fail2ban. Log location for jail’s is located /var/log/fail2ban.log.

filters, actions and jails

fail2ban uses the concept of filters, actions and jails.

    • Filters are the regular expressions you want to look for
    • Actions are the steps you want to take when you find something.
    • Jails are what you create to tie together a log file, a filter and an action
.local files

Don’t edit the .conf files you find. Instead, create a .local file of the same name for your settings. The settings that you specify in the .local will override the .conf, and you will not be troubled by upgrades.

Getting Started

fail2ban should have installed itself in /etc/fail2ban. Take a look at the jail.conf  file and you will see some of the jails are already enabled by default, such as the ssh. The first thing you may want to do is enable some of the others.

Enabling a Predefined Jail

Don’t edit the jail.conf. Rather create a jail.local in the same directory to override the .conf settings. You can copy the whole file, but I recommend just copying the relevant section setting the enabled to true. (You can also turn jails off in this fashion for ones that are enabled by default).

#
# HTTP servers
#[apache]

enabled = true
port    = http,https
filter  = apache-auth
logpath = /var/log/apache*/*error.log
maxretry = 6

Since no action is explicitly specified, the default action is taken. That action is defined at the top of the jail.conf and is currently to use ban the hacker’s IP address for a short while using iptables. If you are not using iptables, you may wish to do so or to take a more advanced action.

Advanced Uses

The real power of fail2ban comes when you create custom filters, actions and jails. I use as an example how to look at pure-FTPd logs and issue a block to the router, effectively blocking the offending IP Address from your entire network.

Creating a Custom Filter

The basic goal is to find a specific error message and an IP address associated with it. Examine the log file you are concerned about.

pure-FTPd example

In my case pure-FTPd logs to /var/log/messages and it had entries such as:

# Aug  8 17:43:10 NAS pure-ftpd: (?@91.121.174.74) [WARNING] Sorry, cleartext sessions are not accepted on this server. Please reconnect using SSL/TLS security mechanisms.
# Aug  8 15:05:45 NAS pure-ftpd: (?@192.168.1.148) [WARNING] Authentication failed for user [schmoe]

Then go to the filter.d directory and create a new file named pure-ftpd-TLS.local. You’re looking put in the error message and replace the IP address with the fail2ban reserved word <HOST>. Here’s an example:

[Definition]
failregex = pure-ftpd: \(.+?@<HOST>\) \[WARNING\] Sorry, cleartext sessions are not accepted on this server
            pure-ftpd: \(.+?@<HOST>\) \[WARNING\] Authentication failed for userignoreregex =

The interesting part is the \(.+?@<HOST>\). This is an escaped “(”  a “.+” the characters “?@” and then the keyword <HOST>. The “.+” is important for the regex because there can be a username in front of the “@”. The <HOST> is specific to fail2ban and is required for it to know what IP to take action against.

There are other features you can take advantage of (though not well documented) but this example represents the minimum approach.

Creating a Custom Action

You can think of an action file as a collection of commands. Some you want to issue when fail2ban starts, others when it stops so as to clean up after itself. They are contained in the action.d directory. The predefined iptables-multiport.conf is great one to start with. The iptables-allports.conf takes it a step further and you may want to do that instead.

Banning at the Boarder example

To modify it to work with a tomato based router, simply paste a ssh root@tomato '....'  command in front of the existing iptable commands. And make sure to enable ssh logins on the router and place the hosts public key in the authorized_keys file

change

actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP

to

actionban = ssh root@tomato 'iptables -t nat -I fail2ban-<name> 1 -s <ip> -j DROP'

Note the addtion of -t nat after the iptable command. This is needed because this is being used on a router, as opposed to something that wasn’t forwarding traffic.

Since iptable’s order of inspection is 1) mangle table’s PREROUTING chain, 2) nat table’s PREROUTING, 3) filter table’s FORWARD or back to mangle table’s INPUT chain, it made sense to cut it off at step 2 were I could use one rule for both the firewall itself and the protected network.

See the attached file at the bottom for all the rule changes needed.

Creating a New Jail

To turn on a jail you will create a jail.local file. In mine I changed the default ban action to the new action I just created and created a jail for Pure-FTPd. In both cases you simply use the file name without the extension to identify the filter and action file you want to use,

#
# ACTIONS
#

# (Default banning action)[DEFAULT]

banaction =  iptables-allports-router

#
# JAILS
#

[pure-FTPd]

enabled  = true
port     = all
filter   = pure-ftpd-TLS
logpath  = /var/log/messages
maxretry = 5
Checking the status of a jail
$ sudo fail2ban-client status
Status
|- Number of jail:      2
`- Jail list:           pure-ftpd-TLS, ssh
 
$ sudo fail2ban-client status pure-ftpd-TLS
Status for the jail: pure-ftpd-TLS
|- filter
|  |- File list:        /var/log/messages
|  |- Currently failed: 1
|  `- Total failed:     76
`- action
   |- Currently banned: 0
   |  `- IP list:
   `- Total banned:     8
Troubleshooting
Filters

It’s a good idea to test your filter. For instance, the default filter for pure-ftpd is set up for an older release and does not work. Do this by creating a couple intentionally failed log on attempts, then use the utility fail2ban-regex like so

$ fail2ban-regex /var/log/messages /etc/fail2ban/filter.d/pure-ftpd-TLS.local
Starting fail2ban

If you try and start fail2ban manually or check it’s status, you’ll see a variation of this error message

  $ /etc/init.d/fail2ban start
   * Socket file /var/run/fail2ban/fail2ban.sock is present

This is because you must sudo commands to fail2ban in some distros (ubuntu at least)

Crashing fail2ban

Sometimes fail2ban starts, but aborts as soon as I ask it what it’s status is. The way to see the error is to start the server in console mode, and ask it via another shell what its status is. Do that with

# /usr/bin/fail2ban-server -x -f

In one shell and in another

 $ fail2ban-client status pure-ftpd-TLS

You may find that fail2ban must be run with python2.4 when on an arm architecture, for things such as the DNS-323

$ apt-get install python2.4
$ python2.4 /usr/bin/fail2ban-server -x -f

 rm /usr/bin/python
 ln -s /usr/bin/python2.4  /usr/bin/python
Blocking all ports V/S specific ones

I choose to block all ports. This means if you fail to login to FTP you are totally banned, rather than just be blocked from FTP. This is somewhat heavy handed as I deny all services based on one service’s attempted exploit. However the exterme is to block an entire subnet. The suspicion being that where there is one ‘bot there is probably more.

ssh-keygen | Passwordless SSH | Ubuntu

28 Jul

You can login to a remote Linux server without entering password in 3 simple steps using ssky-keygen and ssh-copy-id as explained in this article.

ssh-keygen creates the public and private keys. ssh-copy-id copies the local-host’s public key to the remote-host’s authorized_keys file. ssh-copy-id also assigns proper permission to the remote-host’s home, ~/.ssh, and ~/.ssh/authorized_keys.

This article also explains 3 minor annoyances of using ssh-copy-id and how to use ssh-copy-id along with ssh-agent.

Step 1: Create public and private keys using ssh-key-gen on local-host

jsmith@local-host$ [Note: You are on local-host here]

jsmith@local-host$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]
Enter passphrase (empty for no passphrase): [Press enter key]
Enter same passphrase again: [Pess enter key]
Your identification has been saved in /home/jsmith/.ssh/id_rsa.
Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.
The key fingerprint is:
33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9 jsmith@local-host

Step 2: Copy the public key to remote-host using ssh-copy-id

jsmith@local-host$ ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host
jsmith@remote-host's password:
Now try logging into the machine, with "ssh 'remote-host'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

Note: ssh-copy-id appends the keys to the remote-host’s .ssh/authorized_key. If you are unable to utilize ssh-copy-id, as some distro’s do not have this shell program, you can manually append the contents of the local .ssh/id_rsa.pub file to the remote server’s .ssh/authorized_keys file. E.G.

jsmith@local-host$ scp /home/jsmith/.ssh/id_rsa.pub user@remote-host:

Connect to the remote-host via ssh and copy the contents of id_rsa.pub to authorized_keys.

user@remote-host$ cat /home/user/id_rsa.pub >> /home/user/.ssh/authorized_keys

Step 3: Login to remote-host without entering the password

jsmith@local-host$ ssh remote-host
Last login: Sun Nov 16 17:22:33 2008 from 192.168.1.2
[Note: SSH did not ask for password.]

jsmith@remote-host$ [Note: You are on remote-host here]


The above 3 simple steps should get the job done in most cases.

We also discussed earlier in detail about performing SSH and SCP from openSSH to openSSHwithout entering password.

If you are using SSH2, we discussed earlier about performing SSH and SCP without password from SSH2 to SSH2 , from OpenSSH to SSH2 and from SSH2 to OpenSSH.

Using ssh-copy-id along with the ssh-add/ssh-agent

When no value is passed for the option -i and If ~/.ssh/identity.pub is not available, ssh-copy-idwill display the following error message.

jsmith@local-host$ ssh-copy-id -i remote-host
/usr/bin/ssh-copy-id: ERROR: No identities found


If you have loaded keys to the ssh-agent using the ssh-add, then ssh-copy-id will get the keys from the ssh-agent to copy to the remote-host. i.e, it copies the keys provided by ssh-add -Lcommand to the remote-host, when you don’t pass option -i to the ssh-copy-id.

jsmith@local-host$ ssh-agent $SHELL

jsmith@local-host$ ssh-add -L
The agent has no identities.

jsmith@local-host$ ssh-add
Identity added: /home/jsmith/.ssh/id_rsa (/home/jsmith/.ssh/id_rsa)

jsmith@local-host$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsJIEILxftj8aSxMa3d8t6JvM79DyBV
aHrtPhTYpq7kIEMUNzApnyxsHpH1tQ/Ow== /home/jsmith/.ssh/id_rsa

jsmith@local-host$ ssh-copy-id -i remote-host
jsmith@remote-host's password:
Now try logging into the machine, with "ssh 'remote-host'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.
[Note: This has added the key displayed by ssh-add -L]

Three Minor Annoyances of ssh-copy-id

Following are few minor annoyances of the ssh-copy-id.

  1. Default public key: ssh-copy-id uses ~/.ssh/identity.pub as the default public key file (i.e when no value is passed to option -i). Instead, I wish it uses id_dsa.pub, or id_rsa.pub, or identity.pub as default keys. i.e If any one of them exist, it should copy that to the remote-host. If two or three of them exist, it should copy identity.pub as default.
  2. The agent has no identities: When the ssh-agent is running and the ssh-add -L returns “The agent has no identities” (i.e no keys are added to the ssh-agent), the ssh-copy-id will still copy the message “The agent has no identities” to the remote-host’s authorized_keys entry.
  3. Duplicate entry in authorized_keys: I wish ssh-copy-id validates duplicate entry on the remote-host’s authorized_keys. If you execute ssh-copy-id multiple times on the local-host, it will keep appending the same key on the remote-host’s authorized_keys file without checking for duplicates. Even with duplicate entries everything works as expected. But, I would like to have my authorized_keys file clutter free.

How to install mod_security | Apache2 | Ubuntu 11.10

3 Apr
What is mod_security?
Mod_security is a filter for requests and responses sent to and from an Apache web server. It is the “snort” of web applications. Check our their official website for more details modsecurity.org.

As an example, lets say “super haxor,” starts up their kiddie “Auto Haxs 4000” script and begins to pummel your web server with every known vulnerability for every known web application – perhaps even vulnerabilities that are not known the public. As mod_security parses each request to your web server, it matches super haxor’s requests to patterns that indicate attempts to exploit SQL injections, command injections, XSS attacks, etc. and it displays a generic error message. The attack attempts from super haxor never touch your web application.

In another scenario, Paul and Larry are doing a penetration test on your web server. They find a page that produces a 404 error, hoping to get the details of the operating system and web server in order to gather information about the box. As the response is returned to Paul and Larry, mod_security matches the server information in the response and changes it to a generic, administrator defined message.

mod_security adds another layer of protection to your web server and frees up your time usually spent surfing apache logs.

How do I install mod_security?

This guide covers installing mod_security on Ubuntu 11.10 for Apache 2. In this example, we are going to install from source.

Use the Source

Download the latest mod security tars from the mod_security site. modsecurity downloads. You will only need the current modsecurity-apache archive.

Now get the necessary packages for compiling mod_security on Ubuntu with this command:

$ sudo apt-get install automake g++ apache2-threaded-dev dpkg-dev libxml2 libxml2-dev

Now compile and install mod_security with the following commands:

$ cd <modsecurity download directory>/apache2
$ ./configure
$ make
$ sudo make install
Apache Conf Files

Now that the mod_security binary is installed in your Apache 2 modules folder, we have to make a few configuration files so that Apache knows to use the module.

Create a file called /etc/apache2/mods-available/security2.load with the following contents:

LoadFile /usr/lib/libxml2.so 
LoadModule security2_module /usr/lib/apache2/modules/mod_security2.so 
<IfModule !mod_security2.c> 
error_mod_security2_is_not_loaded 
</IfModule>
<IfModule mod_security2.c>
Include /etc/apache2/modsecurity_crs/*.conf 
</IfModule> 

Next create a /etc/apache2/modsecurity_crs directory and move all of the Core Rules into it. Be aware that some of the optional rules explained later on may require some of the .data files in addition to the .conf files.

$ sudo mkdir /etc/apache2/modsecurity_crs
$ sudo cp -R <mod_security download directory> /rules/*.conf /etc/apache2/modsecurity_crs/

You should now take a look at the rule files to make sure the settings are as you like them. For the most part I only modified lines in the modsecurity_crs_10_config.conf file. This file will allow you to enable different portions of the engine. I enabled the directives to scan all XML content. In particular you will want to look at the paths that mod_security stores its log files. I changed all of the log directories to the following:

SecUploadDir /var/log/modsecurity/SecUploadDir 
SecAuditLog /var/log/modsecurity/SecAuditLog/modsec_audit.log 
SecAuditLogStorageDir /var/log/modsecurity/SecAuditLogStorageDir 
SecDebugLog /var/log/modsecurity/SecDebugLog/modsec_debug.log 
SecDataDir /var/log/modsecurity/SecDataDir 
SecTmpDir /var/log/modsecurity/SecTmpDir 

After the settings were made I created the directories and set proper permissions with the following commands:

$ sudo mkdir /var/log/modsecurity
$ sudo mkdir /var/log/modsecurity/SecDataDir
$ sudo mkdir /var/log/modsecurity/SecTmpDir
$ sudo mkdir /var/log/modsecurity/SecUploadDir
$ sudo mkdir /var/log/modsecurity/SecAuditLog
$ sudo mkdir /var/log/modsecurity/SecAuditLogStorageDir
$ sudo mkdir /var/log/modsecurity/SecDebugLog
$ sudo chown -R www-data:www-data /var/log/modsecurity
$ sudo chmod -R a-rwx /var/log/modsecurity/
$ sudo chmod -R u+rwx /var/log/modsecurity/

There are also some other rule sets in the modsecurity-apache_2.1.5/rules/optional_rules/ directory. You may want to take a look at them and place them into your /etc/apache2/modsecurity_crs/ if desired.

Enable and Test

You should now have everything in place to run Apache 2 with mod_security. It is time to enable the module and restart apache.

$ sudo a2enmod security2
$ sudo /etc/init.d/apache2 reload

Hopefully Apache 2 restarts fine with no errors. Once Apache 2 has restarted, go ahead and test your web application with mod_security enabled. If you find that your web application is now working improperly, you can debug the mod_security rule that is blocking it by taking a look at the audit log and using the fabulous web application debugging tool Firebug.

Lastly
You may also need to enable mod_unique_id as my error.log for apache was throwing out this requirement. This can be done by
$ sudo a2enmod unique_id
$ sudo /etc/init.d/apache2 reload
Conclusion

If you are deploying an Apache 2 server that is going to be running any sort of web application, I highly recommend that you take a look at mod_security. The hour you spend installing it could save you from a lawsuit or embarrassing explanations to your customers.

Setup L2TP/IPSec VPN | Ubuntu

23 Mar
Configure L2TP/IPSec VPN on Ubuntu

I need a working L2TP/IPSec VPN for my MacBook and iPhone. I used to have PPTP since it is easy to configure. However, by nature, VPN’s without security are prone to “sniffing”. The extra security of IPSec is a nice to have. You need several components in order to run L2TP/IPSec.

IPSec

IPSec encrypts your IP packets to provide encryption and authentication, so no one can decrypt or forge data between your Mac/iPhone and your server. openswan is the preferred daemon to run IPSec. Install it on your Ubuntu server:

$ sudo aptitude install openswan 

There are several ways to handle encryption for IPSec. I use Pre-Shared Key since it is easy to tweak. Change /etc/ipsec.conf to this:

version 2.0
config setup
    nat_traversal=yes
    virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12
    oe=off
    protostack=netkey

conn L2TP-PSK-NAT
    rightsubnet=vhost:%priv
    also=L2TP-PSK-noNAT

conn L2TP-PSK-noNAT
    authby=secret
    pfs=no
    auto=add
    keyingtries=3
    rekey=no
    ikelifetime=8h
    keylife=1h
    type=transport
    left=YOUR.SERVER.IP.ADDRESS
    leftnexthop=YOUR.SERVER.EXTERNAL.IP.ADDRESS
    leftprotoport=17/1701
    right=%any
    rightprotoport=17/%any


and change /etc/ipsec.secrets to

YOUR.SERVER.IP.ADDRESS %any: PSK "YourSharedSecret" 

Remember to change YOUR.SERVER.IP.ADDRESS and YourSharedSecret accordingly. Run the following command for openswan to stop complaining

for each in /proc/sys/net/ipv4/conf/* 
do 
 echo 0 > $each/accept_redirects 
 echo 0 > $each/send_redirects 
done 

Check if IPSec is correctly setup

$ sudo ipsec verify 

Don’t worry about the disabled Opportunistic Encryption Support. Just make sure other checks are passed OK. Then restart openswan by running

$ sudo /etc/init.d/ipsec restart 

Now you can add a L2TP/IPSec connection on your OS X and see if IPSec is working. Use whatever account and password. We are not there yet. The only thing you need to make sure is that you connect to the right server with the right shared secret as specified in /etc/ipsec.secrets on your server. Monitor /var/log/system.log on your OS X by running

$ tail -f /var/log/system.log 

while OS X is trying to connect to your server via L2TP/IPSec. It will fail eventually because we haven’t configured L2TP yet, but if you see a line in the system log saying something like

Apr 30 18:12:48 bender pppd[1507]: IPSec connection established 

IPSec is good to go.

L2TP

L2TP provides a tunnel to send data. It does not provide encryption and authentication though, that is why we need to use it together with IPSec. Interestingly, both Apple and Microsoft tend to refer L2TP as the secure VPN technology but totally ignore the fact that security is provided by IPSec. The commonly used L2TP daemon is xl2tpd from the same buys behindopenswan. Install it by running

sudo aptitude install xl2tpd 

Change /etc/xl2tpd/xl2tpd.conf to

[global]
ipsec saref = yes

[lns default]
ip range = 10.1.2.2-10.1.2.255
local ip = 10.1.2.1
refuse chap = yes
refuse pap = yes
require authentication = yes
ppp debug = yes
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes

ip range is the set of internal IP addresses that will allocate to clients connected. Make sure it does not overlap with your exisiting IP addresses being used, and not in conflict with the ones on the client’s network. Since most home routers use 172.16.X.X and 192.168.X.X range, you might want to avoid that. local ip is the internal IP for the L2TP server. Make sure it is NOT in the ip range allocated to clients.

PPP

I also run PPTP service using PPP, so I would like to use the same daemon to handle user managenet. Install ppp by running

$ sudo aptitude install ppp 

if you do not have it. Create this file /etc/ppp/options.xl2tpd with the following content

require-mschap-v2
ms-dns 8.8.8.8
ms-dns 8.8.4.4
asyncmap 0
auth
crtscts
lock
hide-password
modem
debug
name l2tpd
proxyarp
lcp-echo-interval 30
lcp-echo-failure 4

Note I am using Google Public DNS in the ms-dns field. If you want to use other DNS servers, change the IP addresses accordingly. Add a test user in /etc/ppp/chap-secrets to try out if L2TP works.

# user server password ip test l2tpd testpassword * 

Now restart xl2tpd by running

$ sudo /etc/init.d/xl2tpd restart 

In addition, if you use iptables for firewalling, make sure it forwards packets so you can browse the Interent after connecting to VPN. Run the following command

$ iptables --table nat --append POSTROUTING --jump MASQUERADE 
$ echo 1 > /proc/sys/net/ipv4/ip_forward 
Almost Done

Update the L2TP/IPSec VPN connection on your OS X with the test user account and try connect. If it can connect and authenticate successfully, congrats! You are done. Now go enjoy the better security. However if you are running Ubuntu 9.10 you will probably have to work a little bit more. openswan 2.6.22 in Ubuntu 9.10 does not play well withxl2tpd (though older openswan 2.4.x in Debian 5 and Ubuntu 8.04 should be fine): you can connect via IPSec, but it never talks L2TP. You need to upgrade to openswan 2.6.24. As of now there is no ready-made .deb package for you to upgrade. Time to get your hands dirty compiling from source code!

There are know issues in setting up L2TP/IPSec VPN on a Debian Lenny server. It has openswan 2.4.12. Turns out that version has a  bugtoo, which prevents clients with changing IP address to connect with a shared secret. So the best bet right now is to compile openswan 2.6.24 from source.

Compiling Openswan from Source

SSH into your server and choose a temporary directory to do the following

$ sudo aptitude install libgmp3-dev gawk flex bison 
$ wget http://www.openswan.org/download/openswan-2.6.24.tar.gz 
$ tar xf openswan-2.6.24.tar.gz 
$ cd openswan-2.6.24 
$ make programs 
$ sudo make install 

The process might take a while so please be patient. You need a decent Linux kernel (2.6.6+) for this to work. Read openswan-2.6.24/README if you are using Linux kernel 2.4.x or do not want to use Netkey. You do not need the packaged openswan installed by aptitude anymore. Remove it (but keep all config files) by running

$ sudo aptitude remove openswan 

Then restart the openswan installed from source

$ sudo /etc/init.d/ipsec restart 

Try connect from OS X. It should work now.

One More Thing

For some reason openswan does not start correctly after reboot, so I put the following lines in my /etc/rc.local

$ iptables --table nat --append POSTROUTING --jump MASQUERADE 
$ echo 1 > /proc/sys/net/ipv4/ip_forward 
for each in /proc/sys/net/ipv4/conf/* 
do 
 echo 0 > $each/accept_redirects 
 echo 0 > $each/send_redirects 
done 
$ /etc/init.d/ipsec restart 
Troubleshooting

On the server side you can monitor /var/log/auth.log and see what is going on with the connection. On OS X you can monitor/var/log/system.log. These two should give you enough information to determine which part is malfunctioning in case of failure. Openswan’s mailing list is a good place to go if you cannot figure out what is wrong.

-Do not forget to port forward Port 500,4500 TCP/UDP to your L2TP VPN Server.
-Do not forget to edit your IPTABLES if you have a firewall enabled :
e.g.
> -A INPUT -p 50 -j ACCEPT
> -A INPUT -p udp -d 12.34.56.78 –dport 500 -j ACCEPT
> -A INPUT -p udp -d 12.34.56.78 –dport 4500 -j ACCEPT

The Command Line Challenge

7 Mar

When I started using Linux I avoided the command line as much as possible. Then I started realizing that the command line is in fact very useful. Then I started digging in what you can actually do on the command line and I never stopped learning ever since.But I had a problem. I found it difficult to learn commands when you actually have GUI applications that replace them. It’s hard to get into the environment and become proficient if you only do some tasks on the command line. Back then you had to use it for some things, but distros like Ubuntu have the unofficial goal of preventing the user to go to the command line. I knew that If I really wanted to master the art of the command line I would have to make it my only environment. So I created the Command Line Challenge.

The idea is simple: Use only the command line for a period of time. If you think of this like a game, the levels would be:

  • Easy: 1 day.
  • Medium: 1 week.
  • Hard: 1 month.
  • Ultimate Geek: 6 months.

I started with the easy level just to realize it’s possible to do it at least one week. In order to have a working command line environment for an every day use, you may have to install the following software.

Browsers

I used both lynx and elinks. lynx has more options and is more powerful in general, but elinks has a better rendering and looks. elinks is not able to log into Facebook (a feature rather than a bug maybe?)

Text Editing.

Vim. That’s pretty much everything you need. Actually if you use emacs with lots of plugins to do a lot of stuff you’re probably ready to take the challenge. I recommend Vim because I use it every day. If by using only a  text editor you’re able to learn to develop without an IDE you get bonus points.

Email.

If you’re not using mutt right now then you’re missing a lot. mutt if fast, highly configurable and runs on our command line. There’s a mutt challenge too, that challenge is about that if mutt is able to do everything that you can do in Gmail, but that’s a topic for another post. I find mutt even more powerful. Here’s a guide to keep mutt synchronized with your gmail account.

Music

Frankly, I’m surprised that there are plenty of options to listen music on the console. I guess sysadmins love music too. My favorite choice is cmus. It has vim-like key bindings so it just feels natural if you’re used to mutt or vim. There are plenty of other options likemoc or mp3blaster but if you live the vimian way of life like me stick with cmus. You can also use mpd a nice daemon that plays music, specially useful if you want a music streaming solution. You can control it via vimpc

Chat

Laughing in front of a black screen because somebody told you a joke make the people around you think you’re some kind of a psycho but chatting is well supported in our powerful consoles. If you can use irssi to chat in irc channels, but that’s not all, you can download bitlbee to tunnel different IM protocols to irc. So you can have all your conversations centralized in an irc way. If you don’t like that approach you can use finchan ncurses version of the popular pidign.

Pictures.

Yes. You can see pictures on the command line without a graphical interface. How? Directly from caca labs, comes libcaca! A graphics library that outputs text instead of pixels, so that it can work on older video cards or text terminals. Be sure to check in your distribution because in Arch the package is called libcaca but all the binaries you need to see pictures (cacaview) come in that package too.

Videos.

Videos are just pictures passing by really fast, so videos are also possible. For that you’ll need the fantastic mplayer or vlc. You need to specify to use the caca driver with mplayer like this:

mplayer -vo caca video

With vlc you can use the nvlc to use vlc in a nice ncurses interface. What’s the quality of these videos? Well you can’t ask much, but for anime or cartoons the videos are actually fairly good.

File manager.

Just because you’re on the command line that doesn’t have to stop you from using a file manager. Lots of people use midnight commander even in a graphical environment.  I prefer to use a more vim-friendly approach. I like ranger because I already know all the key bindings that I need. I like it so much it’s my default file manager(shame on you nautilus)

Tmux

Tmux is terminal multiplexer. What does that mean? In simple terms is like a window manager for your terminal. You can have tabs, split windows and a nice status bar among other things. My life is not the same after I met tmux. There’s an excellent tutorial and a book about how can you improve your productivity with tmux.

These are just options so you can dive into the command line directly without being a terrible painful experience. The truly art of the command line is to learn the bash, how to write scripts to avoid repetition and more important, to understand that in UNIX a word is worth more than one hundred clicks. Have a cheatsheet with you with all the basic commands and remember that man is your friend.

I recommend you should go and have a look at Matt Might’s blog. He posted some really interesting articles about what you can do with a UNIX command line. I speciallyrecommend this one.

So, challenge accepted?

 Update: Also I you use twitter I recommend you bti and tyrs. If you think you need to learn the basics before diving in I recommend linuxcommandline.org I believe there’s a new book about it.