FlexGet installation : Ubuntu 12.04 How To

15 Jan

When I read posts on LifeHacker and Gizmodo sometimes, I figure that everybody else in the world is downloading free TV episodes via BitTorrent, and doing it using RSS feeds and an automated process. That can’t possibly be true, but I figured that if other people do this, I should be able to figure out how, too. It is pretty easy to find RSS feeds for various TV shows. Therfore, I thought it would be relatively simple to write a script to download episodes from each one. I figured that I would have to figure out how to sort out duplicate episodes and different quality types (720p, 1080p, etc.) somehow, too. It didn’t take me too long to find out that an excellent Python program called FlexGet already does all this. Plus, it runs from the command line, which is perfect for a headless server, like my Ubuntu Lucid Lynx box.

I don’t actually endorse doing this, by the way, but I was really curious to learn how it was done. I wanted to share what I learned, in case anyone was struggling with the configuration. Also, I am in the process of playing with ffmpeg to verify downloaded content. You will find sometimes, depending on how reliable your feed is, that FlexGet has the ability to add “fake” torrents to your path/queue.


FlexGet’s web site has an excellent walkthrough that describes how to install it on various operating systems. My instructions in this section are not meant to improve on that guide, but merely to summarize it for an Ubuntu Server install.

FlexGet requires Python 2.6. If you don’t already have that installed on your server, try:

$ sudo apt-get install python2.6 python-setuptools

You have to download FlexGet from their web site. Choose the version for Python 2.6. I prefer to copy the URL and download the file via wget.

$ wget http://download.flexget.com/unstable/FlexGet-1.0r1377-py2.6.egg

After you download FlexGet, install it via Python’s easy_install script. You’ll have to specify the name of the FlexGet file you downloaded.

$ sudo easy_install <downloaded egg>

That’s it! Flexget is now installed in the /usr/local/bin directory.

If you run Transmission like I do, you should also install the the TransmisionRPC plugin.

$ sudo easy_install transmissionrpc


Configuring FlexGet is deceptively simple. The configuration file is written in a format I had heard of, but never encountered before: YAML. YAML is simple and powerful, which means that it can be deceptively complex. You can write a config file that does the same thing in a million different ways. FlexGet’s Cookbook shows you a few examples to get you started. It took me a lot of experimentation to find a config file format that I liked. Here is what I ended up with. It’s probably more complex than it needs to be, but I like the idea of using presets to customize different types of downloads. Note that, in YAML, each level of indentation is two spaces (no tabs are allowed).

$ nano ~/.flexget/config.yml

          timeframe: 8 hours
      - series title 1
      - series title 2
      - series title 3

  host: localhost
  port: 9091
  netrc: /home/mjdescy/.netrc
  removewhendone: true
  addpaused: no
  ratio: 1.00

    rss: http://tv.rss.url/
    preset: tv

    rss: http://tv.rss.url/
    preset: tv

This configuration file sets up a preset called “tv” which tells FlexGet to wait 8 hours before selecting episodes of any of the series listed under the 720p heading. 720p is the default quality setting for FlexGet, and is perfectly adequate for my needs. If FlexGet finds an episode of that quality, it will download it; if it finds episodes above that quality level, at 1080p for example, it will ignore them. If no 720p version exists after the “timeframe” of 8 hours is over, FlexGet will select the entry with the highest quality available. FlexGet won’t download the same episode more than once, unless there is a “proper” or “repack” version—a corrected version of a previously released episode—is available.

At the end of the “tv” preset, the “tranmissionrpc” heading instructs FlexGet to use the transmissionrpc plugin to directly send torrent URLs to transmission-daemon for download. This obviates the need to set a watch directory in transmission-daemon. Note that the “netrc” setting refers to a special username/password file that contains the user name and password required to log into transmission daemon. That file should be in the following format, hidden, in your home directory. Make sure you grant only your account access to read that file (i.e., chmod 600 ~/.netrc).

machine localhost
login username
password password123456789

If you do not run transmission-daemon, you could replace the “transmissionrpc” section with a “download” section, or use another plugin. Check out the flexget Cookbook for more information.

The “feeds” section lists all the TV torrent feeds to look in. I have each feed set to use the “tv” preset that I defined above.

Each RSS feed you list can contain a number of TV series (which is the easiest way to set it up), or just one series (you’ll have to find or build custom feeds for this). Flexget will search all the feeds you list for the series titles listed in the “tv” preset.

This configuration file works very well for me. I hope that it helps you get up and running.


FlexGet can be run manually for debugging purposes (flexget -v), but it should be scheduled to run periodically via cron. Note that there is no need to run FlexGet as root. I run it under my own user account. If you want to operate under the principle of least privilege, you could create a “flexget” account, edit its crontab, and grant that account the permissions it needs to get the job done. For my needs, that isn’t necessary. Therefore, to schedule FlexGet to run every hour, I simply edit my own crontab.

$ crontab -e

Add the following line to the crontab file, and save it.

0,30 * * * * /usr/local/bin/flexget –cron -c /home/mjdescy/.flexget/config.yml > /dev/null 2>&1

This cron job will run FlexGet at the top of every hour and half past the hour. FlexGet will read its options from the configuration file I specified (/home/mjdescy/.flexget/config.yml). I redirect output to /dev/null to prevent cron from sending me emails regarding the command’s output.


Linux File/Folder Permissions: Basics 101

12 Jan


In this article, learn to control file access through correct use of file and directory permissions and ownerships. Learn to:

  • Manage access permissions on both regular and special files as well as directories
  • Maintain security using access modes such as suid, sgid, and the sticky bit
  • Change the file creation mask
  • Grant file access to group members

Unless otherwise noted, the examples in this article use Fedora 13 with a 2.6.34 kernel. Your results on other systems may differ.

This article helps you prepare for Objective 104.5 in Topic 104 of the Linux Professional Institute’s Junior Level Administration (LPIC-1) exam 101. The objective has a weight of 3.

We introduced some of the file and group ownership concepts of this article in our previous article “Learn Linux 101: Manage disk quotas.” This article will help you understand those concepts more completely.


To get the most from the articles in this series, you should have a basic knowledge of Linux and a working Linux system on which you can practice the commands covered in this article. Sometimes different versions of a program will format output differently, so your results may not always look exactly like the listings and figures shown here.

Back to top

User and groups

Connect with Ian

Ian is one of our most popular and prolific authors. Browse all of Ian’s articles on developerWorks. Check out Ian’s profile and connect with him, other authors, and fellow readers in My developerWorks.

By now, you know that Linux is a multiuser system and that each user belongs to one primary group and possibly additional groups. It is also possible to log in as one user and become another user using the su or sudo -s commands. Ownership of files in Linux and access authority are closely related to user ids and groups, so let’s review some basic user and group information.

Who am I?

If you have not become another user, your id is still the one you used to log in. If you have become another user, your prompt may include your user id, as most of the examples in this article do. If your prompt does not include your user id, then you can use thewhoami command to check your current effective id. Listing 1 shows some examples where the prompt strings (from the PS1 environment variable) are different from the other examples in this article. Having your id in the prompt string can be a useful feature.
Listing 1. Determining effective user id

/home/ian$ whoami
/home/ian$ exit
$ whoami

What groups am I in?

Similarly, you can find out what groups you are in by using the groups command. You can find out both user and group information using the id command. Add a user id parameter to either groups or id to see information for that user id instead of the current user id. See Listing 2 for some examples. Note that without a userid, the id command will also display SELinux context as well as basic id information.
Listing 2. Determining group membership

[ian@echidna ~]$ id
uid=1000(ian) gid=1000(ian) groups=1000(ian),505(development),8093(editor)
[ian@echidna ~]$ id ian
uid=1000(ian) gid=1000(ian) groups=1000(ian),8093(editor),505(development)
[ian@echidna ~]$ groups
ian development editor
[ian@echidna ~]$ id tom
uid=1012(tom) gid=1012(tom) groups=1012(tom),505(development)
[ian@echidna ~]$ groups tom
tom : tom development
[ian@echidna ~]$ su tom
[tom@echidna ian]$ groups
tom development
[tom@echidna ian]$ groups ian
ian : ian editor development

Back to top

File ownership and permissions

Just as every user has an id and is a member of one primary group, so every file on a Linux system has one owner and one group associated with it.

Ordinary files

Use the ls -l command to display the owner and group.
Listing 3. Determining file ownership

[ian@echidna ~]$ ls -l /bin/bash .bashrc helloworld.C 
-rw-r--r--. 1 ian  ian            124 Mar 31  2010 .bashrc
-rwxr-xr-x. 1 root root        943360 May 21  2010 /bin/bash
-rw-rw-r--. 1 ian  development    116 Nov 30 10:21 helloworld.C

In this particular example, user ian’s .bashrc file is owned by him and is in the ian group, which is his primary group. Similarly, /bin/bash is owned by user root and is in the group root. However, helloworld.C is owned by user ian, but its group is development. User names and groups names come from separate namespaces, so a given name may be both a user name and a group name. In fact, many distributions default to creating a matching group for each new user.

The Linux permission model has three types of permission for each filesystem object. The permissions are read (r), write (w), and execute (x). Write permission includes the ability to alter or delete an object. In addition, these permissions are specified separately for the file’s owner, members of the file’s group, and everyone else.

Referring back to the first column of Listing 3, notice that it contains an eleven-character string. The eleventh character is a recent addition. We’ll discuss it in a moment. The first character describes the type of object (- for an ordinary file in this example) and the remaining nine characters represent three groups of three characters. The first group indicates the read, write, and execute permissions for the file’s owner. A - indicates that the corresponding permission is not granted. So user ian can read and write the .bashrc file, but not execute it; while root can read, write, and execute the /bin/bash file. The second group indicates the read, write, and execute permissions for the file’s group. Members of the development group can read or write ian’s helloworld.C file, while everyone else can only read it. Similarly, members of the root group and everyone else can read or execute the /bin/bash file.


Directories use the same permissions flags as regular files, but they are interpreted differently. Read permission for a directory allows a user with that permission to list the contents of the directory. Write permission means a user with that permission can create or delete files in the directory. Execute permission allows the user to enter the directory and access any subdirectories. Without execute permission, the filesystem objects inside a directory are not accessible. Without read permission, the filesystem objects inside a directory are not viewable in a directory listing, but these objects can still be accessed as long as you know the full path to the object on disk. Listing 4 is a very artificial example to illustrate these points.
Listing 4. Permissions and directories

[ian@echidna ~]$ ls -l /home
total 32
drwxr-x---. 38 editor   editor      12288 Nov 30 10:49 editor
drwxr-x---.  4 greg     development  4096 Nov 30 12:44 greg
drwx------. 21 gretchen gretchen     4096 Nov 30 11:26 gretchen
drwxr-xr-x. 41 ian      ian          4096 Nov 30 10:51 ian
drwx------. 21 ianadmin ianadmin     4096 May 28  2010 ianadmin
d-wx--x--x. 21 tom      tom          4096 Nov 30 11:30 tom
[ian@echidna ~]$ ls -a ~greg/.ba*
/home/greg/.bash_history  /home/greg/.bash_profile
/home/greg/.bash_logout   /home/greg/.bashrc
[ian@echidna ~]$ ls -a ~gretchen
ls: cannot open directory /home/gretchen: Permission denied
[ian@echidna ~]$ ls -a ~tom
ls: cannot open directory /home/tom: Permission denied
[ian@echidna ~]$ head -n 3 ~tom/.bashrc
# .bashrc

# Source global definitions

The first character of a long listing describes the type of object (d for a directory). User greg’s home directory has read and execute permission for members of the development group, so users tom and ian can list the directory. User gretchen’s home directory has neither read nor execute permission for the gretchen group or other users, so user ian cannot access it. User tom’s home has execute but not read permission, so user ian cannot list the contents, but can access objects within the directory if he knows they exist.

Other filesystem objects

The output from ls -l may contain filesystem objects other than files and directories as shown by the first character in the listing. We will see more of these later in this article, but for now, note the possible types of objects.
Table 1. Filesystem object types

Code Object type
Regular file
d Directory
l Symbolic link
c Character special device
b Block special device
s Socket

The eleventh character

The eleventh character in a long listing from the ls command is a recent enhancement, so some distributions may still show only the first ten characters. In other cases, the eleventh character is a space, so you may not notice it. This character specifies whether an alternate access method applies to the file. When the character following the file mode bits is a space, there is no alternate access method. When it is a printing character, then there is such a method. The method might be an access control list for example. GNU ls uses a ‘.’ (dot) character to signify a file with only an SELinux security context. Files with any other combination of alternate access methods are marked with the ‘+’ (plus) character.

Back to top

Changing permissions

Adding permissions

Suppose you create a “Hello world” shell script. When you first create the script, it will usually not be executable. Use the chmodcommand with the +x option to add the execute permissions as shown in Listing 5.
Listing 5. Creating an executable shell script

[ian@echidna ~]$ echo 'echo "Hello world!"'>hello.sh
[ian@echidna ~]$ ls -l hello.sh
-rw-rw-r--. 1 ian ian 20 Nov 30 13:05 hello.sh
[ian@echidna ~]$ ./hello.sh
bash: ./hello.sh: Permission denied
[ian@echidna ~]$ chmod +x hello.sh
[ian@echidna ~]$ ./hello.sh
Hello world!
[ian@echidna ~]$ ls -l hello.sh
-rwxrwxr-x. 1 ian ian 20 Nov 30 13:05 hello.sh

You can use +r to set the read permissions, and +w to set the write permissions in a similar manner. In fact, you can use any combination of rw, and x together. For example, using chmod +rwx would set all the read, write, and execute permissions for a file. This form of chmod adds permissions that are not already set.

Being selective

You may have noticed in the above example that execute permission was set for the owner, group, and others. To be more selective, you may prefix the mode expression with u to set the permission for users, g to set it for groups, and o to set it for others. Specifying a sets the permission for all users, which is equivalent to omitting it. Listing 6 shows how to add user and group write and execute permissions to another copy of the shell script.
Listing 6. Selectively adding permissions

[ian@echidna ~]$ echo 'echo "Hello world!"'>hello2.sh
[ian@echidna ~]$ chmod ug+xw hello2.sh
[ian@echidna ~]$ ls -l hello2.sh
-rwxrwxr--. 1 ian ian 20 Nov 30 13:08 hello2.sh

Removing permissions

Sometimes you need to remove permissions rather than add them. Simply change the + to a -, and you remove any of the specified permissions that are set. Listing 7 shows how to remove all permissions for other users on the two shell scripts.
Listing 7. Removing permissions

[ian@echidna ~]$ ls -l hello*.sh
-rwxrwxr--. 1 ian ian 20 Nov 30 13:08 hello2.sh
-rwxrwxr-x. 1 ian ian 20 Nov 30 13:05 hello.sh
[ian@echidna ~]$ chmod o-xrw hello*.sh
[ian@echidna ~]$ ls -l hello*.sh
-rwxrwx---. 1 ian ian 20 Nov 30 13:08 hello2.sh
-rwxrwx---. 1 ian ian 20 Nov 30 13:05 hello.sh

Note that you can change permissions on more than one file at a time. As with some other commands you met in the articles for topic 103, you can even use the -R (or --recursive) option to operate recursively on directories and files.

Setting permissions

Now that you can add or remove permissions, you may wonder how to set just a specific set of permissions. Do this using =instead of + or -. To set the permissions on the above scripts so that other users have no access rights, you could use chmod o= hello*, instead of the command we used to remove permissions.

If you want to set different permissions for user, group, or other, you can separate different expressions by commas; for example,ug=rwx,o=rx, or you can use numeric permissions, which are described next.

Octal permissions

So far you have used symbols (ugoa and rxw) to specify permissions. There are three possible permissions in each group. You can also set permissions using octal numbers instead of symbols. Permissions set in this way use up to four octal digits. We will look at the first digit when we discuss attributes. The second digit defines user permissions, the third group permissions and the fourth other permissions. Each of these three digits is constructed by adding the desired permissions settings: read (4), write (2), and execute (1). In the example for hello.sh in Listing 5, the script was created with permissions -rw-r–r–, corresponding to octal 644. Setting execute permission for everyone changed the mode to 755.

Using numeric permissions is very handy when you want to set all the permissions at once without giving the same permissions to each of the groups. Use Table 2 as a handy reference for octal permissions.
Table 2. Numeric permissions

Symbolic Octal
rwx 7
rw- 6
r-x 5
r-- 4
-wx 3
-w- 2
--x 1
--- 0

Back to top

Access modes

When you log in, the new shell process runs with your user and group ids. These are the permissions that govern your access to any files on the system. This usually means that you cannot access files belonging to others and cannot access system files. In fact, we as users are totally dependent on other programs to perform operations on our behalf. Because the programs you start inherit your user id, they cannot access any filesystem objects for which you haven’t been granted access.

An important example is the /etc/passwd file, which cannot be changed by normal users directly, because write permission is enabled only for root: However, normal users need to be able to modify /etc/passwd somehow, whenever they need to change their password. So, if the user is unable to modify this file, how can this be done?

suid and sgid

The Linux permissions model has two special access modes called suid (set user id) and sgid (set group id). When an executable program has the suid access modes set, it will run as if it had been started by the file’s owner, rather than by the user who really started it. Similarly, with the sgid access modes set, the program will run as if the initiating user belonged to the file’s group rather than to his own group. Either or both access modes may be set.

Listing 8 shows that the passwd executable is owned by root:
Listing 8. suid access mode on /usr/bin/passwd

[ian@echidna ~]$ ls -l /usr/bin/passwd
-rwsr-xr-x. 1 root root 34368 Apr  6  2010 /usr/bin/passwd

Note that in place of an x in the user’s permission triplet, there’s an s. This indicates that, for this particular program, the suid and executable bits are set. So when passwd runs, it will execute as if the root user had launched it with full superuser access, rather than that of the user who ran it. Because passwd runs with root access, it can modify /etc/passwd.

The suid and sgid bits occupy the same space as the x bits for user and group in a long directory listing. If the file is executable, the suid or sgid bits, if set, will be displayed as lowercase s, otherwise they are displayed as uppercase S.

While suid and sgid are handy, and even necessary in many circumstances, improper use of these access mode can allow breaches of a system’s security. You should have as few suid programs as possible. The passwd command is one of the few thatmust be suid.

Setting suid and sgid

The suid and sgid bits are set and reset symbolically using the letter s; for example, u+s sets the suid access mode, and g-sremoves the sgid mode. In the octal format, suid has the value 4 in the first (high order) digit, while sgid has the value 2.

Directories and sgid

When a directory has the sgid mode enabled, any files or directories created in it will inherit the group id of the directory. This is particularly useful for directory trees that are used by a group of people working on the same project. Listing 9 shows how user greg could set up a directory that all users of the development group could use, along with an example of how user gretchen could create a file in the directory. As created, the file gretchen.txt allows groups members to write to the file, so gretchen useschmod g-w to disable group write capability.
Listing 9. sgid access mode and directories

[greg@echidna ~]$ mkdir lpi101
[greg@echidna ~]$ chmod g+ws lpi101
[greg@echidna ~]$ ls -ld lpi101
drwxrwsr-x. 2 greg development 4096 Nov 30 13:30 lpi101/
[greg@echidna ~]$ su - gretchen
[gretchen@echidna ~]$ touch ~greg/lpi101/gretchen.txt
[gretchen@echidna ~]$ ls -l ~greg/lpi101/gretchen.txt
-rw-rw-r--. 1 gretchen development 0 Nov 30 14:12 /home/greg/lpi101/gretchen.txt
[gretchen@echidna ~]$ chmod g-w ~greg/lpi101/gretchen.txt
[gretchen@echidna ~]$ ls -l ~greg/lpi101/gretchen.txt
-rw-r--r--. 1 gretchen development 0 Nov 30 14:12 /home/greg/lpi101/gretchen.txt

Any member of the development group can now create files in user greg’s lpi101 directory. As Listing 10 shows, other members of the group cannot update the file gretchen.txt. However, they do have write permission to the directory and can therefore delete the file.
Listing 10. sgid access mode and file ownership

[gretchen@echidna ~]$ su - tom
[tom@echidna ~]$ echo "something" >> ~greg/lpi101/gretchen.txt
-bash: /home/greg/lpi101/gretchen.txt: Permission denied
[tom@echidna ~]$ rm ~greg/lpi101/gretchen.txt
rm: remove write-protected regular empty file `/home/greg/lpi101/gretchen.txt'? y
[tom@echidna ~]$ ls -l ~greg/lpi101/
total 0

The sticky bit

You have just seen how anyone with write permission to a directory can delete files in it. This might be acceptable for a workgroup project, but is not desirable for globally shared file space such as the /tmp directory. Fortunately, there is a solution.

The remaining access mode bit is called the sticky bit. It is represented symbolically by t and numerically as a 1 in the high-order octal digit. It is displayed in a long directory listing in the place of the executable flag for other users (the last character), with the same meaning for upper and lower case as for suid and sgid. If set for a directory, it permits only the owning user or the superuser (root) to delete or unlink a file. Listing 11 shows how user greg could set the sticky bit on his lpi101 directory and also shows that this bit is set for /tmp.
Listing 11. Sticky directories

[greg@echidna ~]$ chmod +t lpi101
[greg@echidna ~]$ ls -ld lpi101 /tmp
drwxrwsr-t.  2 greg development  4096 Nov 30 14:16 lpi101
drwxrwxrwt. 24 root root        12288 Nov 30 14:22 /tmp

On a historical note, UNIX® systems used to use the sticky bit on files to hoard executable files in swap space and avoid reloading. Modern Linux kernels ignore the sticky bit if it is set for files.

Access mode summary

Table 3 summarizes the symbolic and octal representation for the three access modes discussed here.
Table 3. Access modes

Access mode Symbolic Octal
suid s with u 4000
sgid s with g 2000
sticky t 1000

Combining this with the earlier permission information, you can see that the full octal representation corresponding to greg’s lpi101 permissions and access modes of drwxrwsr-t is 3775. While the ls command does not display the octal permissions, you can display them using the find command as shown in Listing 12
Listing 12. Printing symbolic and octal permissions

[greg@echidna ~]$ find . -name lpi101 -printf "%M %m %f\n"
drwxrwsr-t 3775 lpi101

Immutable files

The access modes and permissions provide extensive control over who can do what with files and directories. However, they do not prevent things such as inadvertent deletion of files by the root user. Although beyond the scope of LPI Topic 104.5, there are some additional attributes available on various filesystems that provide additional capabilities. One of these is the immutableattribute. If this is set, even root cannot delete the file until the attribute is unset.

Use the lsattr command to see whether the immutable flag (or any other attribute) is set for a file or directory. To make a file immutable, use the chattr command with the -i flag.

Listing 13 shows that user root can create an immutable file but cannot delete it until the immutable flag is removed.
Listing 13. Immutable files

[root@echidna ~]# touch keep.me
[root@echidna ~]# chattr +i keep.me
[root@echidna ~]# lsattr keep.me
----i--------e- keep.me
[root@echidna ~]# rm -f keep.me
rm: cannot remove `keep.me': Operation not permitted
[root@echidna ~]# chattr -i keep.me
[root@echidna ~]# rm -f keep.me

Changing the immutable flag requires root authority, or at least the CAP_LINUX_IMMUTABLE capability. Making files immutable is often done as part of a security or intrusion detection effort. See the capabilities man page (man capabilities) for more information.

Back to top

The file creation mask

When a new file is created, the creation process specifies the permissions that the new file should have. Often, the mode requested is 0666, which makes the file readable and writable by anyone. Directories usually default to 0777. However, this permissive creation is affected by a umask value, which specifies what permissions a user does not want to grant automatically to newly created files or directories. The system uses the umask value to reduce the originally requested permissions. You can view your umask setting with the umask command, as shown in Listing 14.
Listing 14. Displaying octal umask

[ian@echidna ~]$ umask

Remember that the umask specifies which permissions should not be granted. On Linux systems, where users do not have private groups the umask normally defaults to 0022, which removes group and other write permission from new files. Where users have a private group (as on the Fedora system used in these examples) the umask normally defaults to 0002 which removes the write permission for other users. Use the -S option to display the umask symbolically, in a form that shows which are the permissions that are allowed.

Use the umask command to set a umask as well as display one. So, if you would like to keep your files more private and disallow all group or other access to newly created files, you would use a umask value of 0077. Or set it symbolically using umask u=rwx,g=,o=, as illustrated in Listing 15.
Listing 15. Setting the umask

[ian@echidna ~]$ umask -S
[ian@echidna ~]$ umask u=rwx,g=,o=
[ian@echidna ~]$ umask
[ian@echidna ~]$ touch newfile
[ian@echidna ~]$ ls -l newfile
-rw-------. 1 ian ian 0 Nov 30 15:40 newfile

Back to top

Setting file owner and group

File group

To change the group of a file, use the chgrp command with a group name and one or more filenames. You may also use the group number if you prefer. An ordinary user must own the file and also be a member of the group to which the file’s group is being changed. The root user may change files to any group. Listing 16 shows an example.
Listing 16. Changing group ownership

[ian@echidna ~]$ touch file{1,2}
[ian@echidna ~]$ ls -l file*
-rw-rw-r--. 1 ian ian 0 Nov 30 15:54 file1
-rw-rw-r--. 1 ian ian 0 Nov 30 15:54 file2
[ian@echidna ~]$ chgrp development file1
[ian@echidna ~]$ chgrp 505 file2
[ian@echidna ~]$ ls -l file*
-rw-rw-r--. 1 ian development 0 Nov 30 15:54 file1
-rw-rw-r--. 1 ian development 0 Nov 30 15:54 file2

As with many of the commands covered in this tutorial, chgrp has a -R option to allow changes to be applied recursively to all selected files and subdirectories.

Default group

When you studied Access modes above, you learned how setting the sgid mode on a directory causes new files created in that directory to belong to the group of the directory rather than to the group of the user creating the file.

You may also use the newgrp command to temporarily change your primary group to another group of which you are a member. A new shell will be created, and when you exit the shell, your previous group will be reinstated, as shown in Listing 17.
Listing 17. Using newgrp to temporarily change default group

[ian@echidna ~]$ groups
ian development editor
[ian@echidna ~]$ newgrp development
[ian@echidna ~]$ groups
development ian editor
[ian@echidna ~]$ touch file3
[ian@echidna ~]$ ls -l file3
-rw-r--r--. 1 ian development 0 Nov 30 16:00 file3
[ian@echidna ~]$ exit
[ian@echidna ~]$ groups
ian development editor

File owner

The root user can change the ownership of a file using the chown command. In its simplest form, the syntax is like the chgrpcommand, except that a user name or numeric id is used instead of a group name or id. The file’s group may be changed at the same time by adding a colon and a group name or id right after the user name or id. If only a colon is given, then the user’s default group is used. Naturally, the -R option will apply the change recursively. Listing 18 shows an example.
Listing 18. Using chown to changing file ownership

[ian@echidna ~]$ touch file4
[ian@echidna ~]$ su -
[root@echidna ~]# ls -l ~ian/file4
-rw-rw-r--. 1 ian ian 0 Nov 30 16:04 /home/ian/file4
[root@echidna ~]# chown greg ~ian/file4
[root@echidna ~]# ls -l ~ian/file4
-rw-rw-r--. 1 greg ian 0 Nov 30 16:04 /home/ian/file4
[root@echidna ~]# chown tom:gretchen ~ian/file4
[root@echidna ~]# ls -l ~ian/file4
-rw-rw-r--. 1 tom gretchen 0 Nov 30 16:04 /home/ian/file4
[root@echidna ~]# chown :tom ~ian/file4
[root@echidna ~]# ls -l ~ian/file4
-rw-rw-r--. 1 tom tom 0 Nov 30 16:04 /home/ian/file4

An older form of specifying both user and group used a dot instead of a colon. This is no longer recommended as it may cause confusion when names include a dot.

This completes your introduction to file and directory permissions on Linux.

nmap vulnerability scanning | HowTo

17 Dec

Port and vulnerability scanners are common tools used by good as bad guys. Performing a port scanning is one of the first operations required to find potential vulnerabilities on a target system. That’s why vulnerability scanners have built-in port scanners. Writing a port scanner is really easy with a few lines of Perl:

use IO::Socket;
while ($ARGV[1] < 65536) {
  print STDOUT "$ARGV[0]:".($ARGV[1] - 1) . " open\n" if \
(IO::Socket::INET->new(PeerAddr=>"$ARGV[0]:" .$ARGV[1]++, Proto=>'tcp', Timeout=>1));

(Source: okc2600.com)

However, “real” port scanners offer much more options like evading techniques to work “below the radar” or fingerprinting. Nmap is the best tools for this purpose.

Synergies already exist between different scanning products. A good example is the integration of Nessus with Nmap. Nmap can save the scan results in XML format. The produced XML content can be re-used by Nessus to scan for vulnerabilities. By using this method, the power of Nessus is combined with the one of Nmap. For more information, read this article.

Performing a vulnerability scan  is highly resources consuming. Why not add a simple vulnerability scan feature to Nmap? This primary goal is to save time and be less intrusive. Nmap has a built-in script interpreter called NSE (“Nmap Scripting Engine“) which allows developers to write extensions for Nmap. It comes by default with a lot of scripts. If you’re interested, I posted an introduction article on NSE a few months ago.

Marc Ruef developed a NSE script which adds a basic vulnerability scanner feature to your Nmap. Technically, the script does NOT perform a vulnerability scan by itself. With the powerful fingerprinting feature of Nmap (using the “-sV” flag), the running applications and versions can be detected. Those information are used as lookup keys in a DB export of OSVDB, the Open Source Vulnerability Data Base. The matching entries are displayed in the script output. The script installation is extremely simple, just copy the files in your existing scripts repository (something like “$NMAP_INSTALL_PATH/share/nmap/scripts/“). Invoke it like any standard script:

# nmap -PN -sS -sV --script=vulscan -p80 www.company.tld

Starting Nmap 5.30BETA1 ( http://nmap.org ) at 2010-06-03 11:11 CEST
Nmap scan report for www.company.tld (
Host is up (0.00074s latency).
rDNS record for www.company.tld
80/tcp open  http    Apache httpd 2.2.11 ((Ubuntu) PHP/5.2.6-3ubuntu4.5 with Suhosin-Patch)
| vulscan: [48] Apache HTTP Server on Debian /usr/doc Directory Information Disclosure
| [143] Apache HTTP Server printenv.pl Multiple Method CGI XSS
| [222] Apache HTTP Server test-cgi Arbitrary File Access

[Stuff Deleted]

| [63895] Apache HTTP Server mod_headers Unspecified Security Issue
| [64023] Apache Tomcat WWW-Authenticate Header Local Host Information Disclosure
| [64020] Apache ActiveMQ Jetty ResourceHandler Crafted Request JSP File Source Disclosure
| [64307] Apache Tomcat Web Application Manager/Host Manager CSRF
| [64517] Apache Open For Business Project (OFBiz) View Profile Section partyId Parameter XSS
| [64518] Apache Open For Business Project (OFBiz) Show Portal Page Section start Parameter XSS
| [64519] Apache Open For Business Project (OFBiz) Control Servlet URI XSS
| [64520] Apache Open For Business Project (OFBiz) ecommerce/control/ViewBlogArticle contentId Parameter XSS
| [64521] Apache Open For Business Project (OFBiz) Web Tools Section entityName Parameter XSS
|_[64522] Apache Open For Business Project (OFBiz) ecommerce/control/contactus Multiple Parameter XSS

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 10.66 seconds

My first impression was disappointing: The scan reported too much vulnerabilities (>500 hits!). Unusable in a real environment. But, after reading the script (remember: RTFM!), Marc was aware of this problem (caused by a naming convention issue between Nmap & OSVDB). He added a correlation feature to reduce those false positives. To activate this option, just pass the following parameter:

# nmap -PN -sS -sV --script=vulscan --script-args vulscancorrelation=1 -p80 www.company.tld

Hopefully, this second test generated much less hits (26) but, side effect, required more time to complete.

This is a very nice feature for Nmap. By using this script, you can quickly have an overview of the potential vulnerabilities on a target host. And, if necessary, use a more classic tool to focus on specific cases. Don’t forget that false positives or false negatives and results must always be analyzed by a competent person.

To keep the vulnerability scanner accurate, the vulnerability DB must be kept up to date. To achieve this, you can automate the update using the CSV export available on osvdb.org (updated daily). First you have to register. Once done, you will be able to download the CSV updates via a permalink generated with your API key.  The upgrade can be fully automated via a simple daily cron and a script:

FILES="object_correlations.txt object_links.txt object_products.txt vulnerabilities.txt"
cd /tmp
wget -o /dev/null http://osvdb.org/file/get_latest_csv/xxxxx/osvdb-csv.latest.tar.gz
for FILE in $FILES
	tar xzf osvdb-csv.latest.tar.gz ./osvdb/$FILE
	mv osvdb/$FILE $NMAPHOME/share/nmap/scripts/vulscan
rm -rf osvdb
rm osvdb-csv.latest.tar.gz
exit 0

Marc released the version 0.6 is his script and has already a nice todolist (integration with other vulnerability databases). Great job!

fail2ban | How To | Ubuntu

19 Sep

fail2Ban is a very handy tool to prevent alot of unwanted traffic from consuming bandwidth on your servers. It’s a very small and relatively simple IDS Type Tool that comes with some predefined Filters to automatically lock out potentially dangerous or bandwidth consuming type attacks.

This tutorial covers basic installation and setup along with giving you an example of a simple custom filter to help you on the way to writing your own custom Event Based Blocking rules (filters) for use across the applications you run.

The installation is described at http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Debian  I used apt-get and it went painlessly

$ sudo apt-get install fail2ban

I’m on Ubuntu 12.04 and the install location is /etc/fail2ban. Log location for jail’s is located /var/log/fail2ban.log.

filters, actions and jails

fail2ban uses the concept of filters, actions and jails.

    • Filters are the regular expressions you want to look for
    • Actions are the steps you want to take when you find something.
    • Jails are what you create to tie together a log file, a filter and an action
.local files

Don’t edit the .conf files you find. Instead, create a .local file of the same name for your settings. The settings that you specify in the .local will override the .conf, and you will not be troubled by upgrades.

Getting Started

fail2ban should have installed itself in /etc/fail2ban. Take a look at the jail.conf  file and you will see some of the jails are already enabled by default, such as the ssh. The first thing you may want to do is enable some of the others.

Enabling a Predefined Jail

Don’t edit the jail.conf. Rather create a jail.local in the same directory to override the .conf settings. You can copy the whole file, but I recommend just copying the relevant section setting the enabled to true. (You can also turn jails off in this fashion for ones that are enabled by default).

# HTTP servers

enabled = true
port    = http,https
filter  = apache-auth
logpath = /var/log/apache*/*error.log
maxretry = 6

Since no action is explicitly specified, the default action is taken. That action is defined at the top of the jail.conf and is currently to use ban the hacker’s IP address for a short while using iptables. If you are not using iptables, you may wish to do so or to take a more advanced action.

Advanced Uses

The real power of fail2ban comes when you create custom filters, actions and jails. I use as an example how to look at pure-FTPd logs and issue a block to the router, effectively blocking the offending IP Address from your entire network.

Creating a Custom Filter

The basic goal is to find a specific error message and an IP address associated with it. Examine the log file you are concerned about.

pure-FTPd example

In my case pure-FTPd logs to /var/log/messages and it had entries such as:

# Aug  8 17:43:10 NAS pure-ftpd: (?@ [WARNING] Sorry, cleartext sessions are not accepted on this server. Please reconnect using SSL/TLS security mechanisms.
# Aug  8 15:05:45 NAS pure-ftpd: (?@ [WARNING] Authentication failed for user [schmoe]

Then go to the filter.d directory and create a new file named pure-ftpd-TLS.local. You’re looking put in the error message and replace the IP address with the fail2ban reserved word <HOST>. Here’s an example:

failregex = pure-ftpd: \(.+?@<HOST>\) \[WARNING\] Sorry, cleartext sessions are not accepted on this server
            pure-ftpd: \(.+?@<HOST>\) \[WARNING\] Authentication failed for userignoreregex =

The interesting part is the \(.+?@<HOST>\). This is an escaped “(”  a “.+” the characters “?@” and then the keyword <HOST>. The “.+” is important for the regex because there can be a username in front of the “@”. The <HOST> is specific to fail2ban and is required for it to know what IP to take action against.

There are other features you can take advantage of (though not well documented) but this example represents the minimum approach.

Creating a Custom Action

You can think of an action file as a collection of commands. Some you want to issue when fail2ban starts, others when it stops so as to clean up after itself. They are contained in the action.d directory. The predefined iptables-multiport.conf is great one to start with. The iptables-allports.conf takes it a step further and you may want to do that instead.

Banning at the Boarder example

To modify it to work with a tomato based router, simply paste a ssh root@tomato '....'  command in front of the existing iptable commands. And make sure to enable ssh logins on the router and place the hosts public key in the authorized_keys file


actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP


actionban = ssh root@tomato 'iptables -t nat -I fail2ban-<name> 1 -s <ip> -j DROP'

Note the addtion of -t nat after the iptable command. This is needed because this is being used on a router, as opposed to something that wasn’t forwarding traffic.

Since iptable’s order of inspection is 1) mangle table’s PREROUTING chain, 2) nat table’s PREROUTING, 3) filter table’s FORWARD or back to mangle table’s INPUT chain, it made sense to cut it off at step 2 were I could use one rule for both the firewall itself and the protected network.

See the attached file at the bottom for all the rule changes needed.

Creating a New Jail

To turn on a jail you will create a jail.local file. In mine I changed the default ban action to the new action I just created and created a jail for Pure-FTPd. In both cases you simply use the file name without the extension to identify the filter and action file you want to use,


# (Default banning action)[DEFAULT]

banaction =  iptables-allports-router



enabled  = true
port     = all
filter   = pure-ftpd-TLS
logpath  = /var/log/messages
maxretry = 5
Checking the status of a jail
$ sudo fail2ban-client status
|- Number of jail:      2
`- Jail list:           pure-ftpd-TLS, ssh
$ sudo fail2ban-client status pure-ftpd-TLS
Status for the jail: pure-ftpd-TLS
|- filter
|  |- File list:        /var/log/messages
|  |- Currently failed: 1
|  `- Total failed:     76
`- action
   |- Currently banned: 0
   |  `- IP list:
   `- Total banned:     8

It’s a good idea to test your filter. For instance, the default filter for pure-ftpd is set up for an older release and does not work. Do this by creating a couple intentionally failed log on attempts, then use the utility fail2ban-regex like so

$ fail2ban-regex /var/log/messages /etc/fail2ban/filter.d/pure-ftpd-TLS.local
Starting fail2ban

If you try and start fail2ban manually or check it’s status, you’ll see a variation of this error message

  $ /etc/init.d/fail2ban start
   * Socket file /var/run/fail2ban/fail2ban.sock is present

This is because you must sudo commands to fail2ban in some distros (ubuntu at least)

Crashing fail2ban

Sometimes fail2ban starts, but aborts as soon as I ask it what it’s status is. The way to see the error is to start the server in console mode, and ask it via another shell what its status is. Do that with

# /usr/bin/fail2ban-server -x -f

In one shell and in another

 $ fail2ban-client status pure-ftpd-TLS

You may find that fail2ban must be run with python2.4 when on an arm architecture, for things such as the DNS-323

$ apt-get install python2.4
$ python2.4 /usr/bin/fail2ban-server -x -f

 rm /usr/bin/python
 ln -s /usr/bin/python2.4  /usr/bin/python
Blocking all ports V/S specific ones

I choose to block all ports. This means if you fail to login to FTP you are totally banned, rather than just be blocked from FTP. This is somewhat heavy handed as I deny all services based on one service’s attempted exploit. However the exterme is to block an entire subnet. The suspicion being that where there is one ‘bot there is probably more.

Installing paramiko for SSH/SFTP in Python on OS X

20 Aug

This installation was performed on a 2.6GHz Intel Core i7 iMac with 16GB RAM running OSX Lion 10.7.7. I used XCode 4.2 with iOS SDK 5 beta 3 and MacPorts 1.9.2. This should also work on earlier XCode versions and/or on OSX 10.5+.


XCode – http://developer.apple.com/xcode/ (or from your OSX install disk)
MacPorts – http://www.macports.org
Python setuptools – http://pypi.python.org/pypi/setuptools


  1. Install XCode, MacPorts and Python setuptools if you haven’t already
  2. Update MacPorts to the latest release (sudo port -v selfupdate) and update any outdated software (sudo port -v upgrade outdated)
  3. Install GMP, MPFR and libmpc – sudo port -v install gmp mpfr libmpc
  4. Create a symlink to libgmp.dylib – sudo ln -s /opt/local/lib/libgmp.dylib /usr/lib/libgmp.dylib
  5. Download pycrypto source – http://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/pycrypto-2.3.tar.gz
  6. tar – zxvf pycrypto-2.3.tar.gz
  7. cd pycrypto-2.3
  8. cd src
  9. Edit _fastmath.c and change the line #include <gmp.h> to be #include “/opt/local/include/gmp.h” instead
  10. cd ../ (go back to main pycrypto-2.3 directory)
  11. Install using sudo python setup.py install
    1. I have had some issues with the following error message – lipo: can’t figure out the architecture type of: /var/tmp//ccybzR1W.out for example
    2. If you have this issue then install using the following – sudo ARCHFLAGS=’-arch i386 -arch x86_64′ python setup.py install
  12. sudo easy_install paramiko
    1. Again if you get a lipo error, use the following – sudo ARCHFLAGS=’-arch i386 -arch x86_64′ easy_install paramiko

Updating CLOB Columns | Oracle Database

9 Aug

During a recent migration project for one of our customers, I needed to export certain XML rows from a Novell Sentinel 6.1 Oracle 10G R2 Hotfix 4 Database. In particular, I found that it was not possible to export Filter Configuration in Novell Sentinel 6.1 and I needed the 250+ Filters in the new Database and new Sentinel Server. The restrictions in place made it impossible to edit the fields using utilities such as DBVIS or Toad for Oracle. I wanted to be able to write SQL Statements that allowed files to be read into the stream and update the backend database column (CLOB).

With some MSSQL and MySQL skills, I did some reading up on how Oracle can manipulate CLOB/LOB Columns using DBMS_LOB. The result is a simple SQL Statement that results in an SQL UPDATE statement, which updates a column, with XML located in /directory/to/filter.xml.

**** WARNING ****

This is currently unsupported by Novell, but in both DEV, TEST and PROD environments, this is working as expected and hasn’t broken any function within Sentinel 6.1 itself.

**** ****

To perform the below you will need to :

  • Shutdown the Sentinel Service (/opt/novell/sentinel6/bin/sentinel.sh stop)
  • Backup the ESEC Oracle Database

1. Create Directory Object

To start with, you’ll need to place the XML file in a directory and create the File Handler in Oracle for use later. In my case, I copied the text (XML) from the source column in the source database and pasted it to a file. The file handler is “EXAMPLE_LOB_DIR” and the directory is “/directory/to/filter.xml”.


2. Update Column In Database

Now that we have the file handler, we’ll use the chunk of SQL Syntax below, to read the XML File into a SQL UPDATE statement, which updates DATA(column) in the CONFIG(table) with data in the dest_clob(xml file).

dest_clob CLOB;
src_clob BFILE := BFILENAME(‘EXAMPLE_LOB_DIR’, ‘filter.xml’);
dst_offset number := 1 ;
src_offset number := 1 ;
lang_ctx number := DBMS_LOB.DEFAULT_LANG_CTX;
warning number;
— ———————————————————————–

— DBMS_LOB.LoadFromFile(
— DEST_LOB => dest_clob
— , SRC_LOB => src_clob
— );

— ———————————————————————–
   INTO dest_clob
   WHERE UNIT = ‘filter’
— ————————————-
— ————————————-
DEST_LOB => dest_clob
, SRC_BFILE => src_clob
, DEST_OFFSET => dst_offset
, SRC_OFFSET => src_offset
, LANG_CONTEXT => lang_ctx
, WARNING => warning
DBMS_OUTPUT.PUT_LINE(‘Loaded XML File using DBMS_LOB.LoadCLOBFromFile: (ID=1001).’);

ssh-keygen | Passwordless SSH | Ubuntu

28 Jul

You can login to a remote Linux server without entering password in 3 simple steps using ssky-keygen and ssh-copy-id as explained in this article.

ssh-keygen creates the public and private keys. ssh-copy-id copies the local-host’s public key to the remote-host’s authorized_keys file. ssh-copy-id also assigns proper permission to the remote-host’s home, ~/.ssh, and ~/.ssh/authorized_keys.

This article also explains 3 minor annoyances of using ssh-copy-id and how to use ssh-copy-id along with ssh-agent.

Step 1: Create public and private keys using ssh-key-gen on local-host

jsmith@local-host$ [Note: You are on local-host here]

jsmith@local-host$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]
Enter passphrase (empty for no passphrase): [Press enter key]
Enter same passphrase again: [Pess enter key]
Your identification has been saved in /home/jsmith/.ssh/id_rsa.
Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.
The key fingerprint is:
33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9 jsmith@local-host

Step 2: Copy the public key to remote-host using ssh-copy-id

jsmith@local-host$ ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host
jsmith@remote-host's password:
Now try logging into the machine, with "ssh 'remote-host'", and check in:


to make sure we haven't added extra keys that you weren't expecting.

Note: ssh-copy-id appends the keys to the remote-host’s .ssh/authorized_key. If you are unable to utilize ssh-copy-id, as some distro’s do not have this shell program, you can manually append the contents of the local .ssh/id_rsa.pub file to the remote server’s .ssh/authorized_keys file. E.G.

jsmith@local-host$ scp /home/jsmith/.ssh/id_rsa.pub user@remote-host:

Connect to the remote-host via ssh and copy the contents of id_rsa.pub to authorized_keys.

user@remote-host$ cat /home/user/id_rsa.pub >> /home/user/.ssh/authorized_keys

Step 3: Login to remote-host without entering the password

jsmith@local-host$ ssh remote-host
Last login: Sun Nov 16 17:22:33 2008 from
[Note: SSH did not ask for password.]

jsmith@remote-host$ [Note: You are on remote-host here]

The above 3 simple steps should get the job done in most cases.

We also discussed earlier in detail about performing SSH and SCP from openSSH to openSSHwithout entering password.

If you are using SSH2, we discussed earlier about performing SSH and SCP without password from SSH2 to SSH2 , from OpenSSH to SSH2 and from SSH2 to OpenSSH.

Using ssh-copy-id along with the ssh-add/ssh-agent

When no value is passed for the option -i and If ~/.ssh/identity.pub is not available, ssh-copy-idwill display the following error message.

jsmith@local-host$ ssh-copy-id -i remote-host
/usr/bin/ssh-copy-id: ERROR: No identities found

If you have loaded keys to the ssh-agent using the ssh-add, then ssh-copy-id will get the keys from the ssh-agent to copy to the remote-host. i.e, it copies the keys provided by ssh-add -Lcommand to the remote-host, when you don’t pass option -i to the ssh-copy-id.

jsmith@local-host$ ssh-agent $SHELL

jsmith@local-host$ ssh-add -L
The agent has no identities.

jsmith@local-host$ ssh-add
Identity added: /home/jsmith/.ssh/id_rsa (/home/jsmith/.ssh/id_rsa)

jsmith@local-host$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsJIEILxftj8aSxMa3d8t6JvM79DyBV
aHrtPhTYpq7kIEMUNzApnyxsHpH1tQ/Ow== /home/jsmith/.ssh/id_rsa

jsmith@local-host$ ssh-copy-id -i remote-host
jsmith@remote-host's password:
Now try logging into the machine, with "ssh 'remote-host'", and check in:


to make sure we haven't added extra keys that you weren't expecting.
[Note: This has added the key displayed by ssh-add -L]

Three Minor Annoyances of ssh-copy-id

Following are few minor annoyances of the ssh-copy-id.

  1. Default public key: ssh-copy-id uses ~/.ssh/identity.pub as the default public key file (i.e when no value is passed to option -i). Instead, I wish it uses id_dsa.pub, or id_rsa.pub, or identity.pub as default keys. i.e If any one of them exist, it should copy that to the remote-host. If two or three of them exist, it should copy identity.pub as default.
  2. The agent has no identities: When the ssh-agent is running and the ssh-add -L returns “The agent has no identities” (i.e no keys are added to the ssh-agent), the ssh-copy-id will still copy the message “The agent has no identities” to the remote-host’s authorized_keys entry.
  3. Duplicate entry in authorized_keys: I wish ssh-copy-id validates duplicate entry on the remote-host’s authorized_keys. If you execute ssh-copy-id multiple times on the local-host, it will keep appending the same key on the remote-host’s authorized_keys file without checking for duplicates. Even with duplicate entries everything works as expected. But, I would like to have my authorized_keys file clutter free.