TOMUSS install (updated 2021-06-23)

If you want to install quickly without reading this page:
wget https://perso.univ-lyon1.fr/thierry.excoffier/TOMUSS/run
chmod 755 run
mkdir TOMUSS
cd TOMUSS
../run # To configure and launch
# ../run regtests # To run regression tests
# ../run try # To run without authentication and animals (k01...)

This page provide step to step explanations to install and customize TOMUSS version 7.0.0. It does not replace the reference documentation.

It is a bit complicated because TOMUSS is not an integrated application. You must integrate it in your own information system.

Beware: TOMUSS does not support copy past of multiple cells because it is a source of errors. Read the documentation to see how to import data.

Requirements

To run, you needs:

To load the system docker image (without TOMUSS) :
wget https://perso.univ-lyon1.fr/thierry.excoffier/TOMUSS/$IMAGE.tar.gz -O - |
     zcat | docker load
On Debian and derivatives, running these commands under 'root' will install the prerequises. This procedure has been tested with docker run -it ubuntu:20.04 /bin/bash on commit 9348dd1d5946a1f03c9652c1828e9a5139039c05
apt update
apt install catdoc espeak gettext git gnuplot graphviz inkscape iotop \
    ipython3 libaio1 locales make mypy netpbm nginx nodejs poppler-utils \
    pylint3 pylint python-setuptools python3-cffi python3-coverage \
    python3-dev python3-ldap python3-matplotlib python3-networkx \
    python3-pil python3-pip python3-pyclamd python3-pyinotify \
    python3-pypdf2 python3-qrcode python3-reportlab python3-setuptools \
    python3-xlsxwriter rsync strace unzip wget xlsx2csv \
    python3-selenium firefox-geckodriver chromium-chromedriver \
    xnest xvfb i3 # For testing

apt upgrade
ln -s /usr/share/zoneinfo/Europe/Paris /etc/localtime # For regression tests
apt install locales
echo 'export LC_ALL=C.UTF-8' >>/etc/profile
echo 'export TZ=Europe/Paris' >>/etc/profile
echo 'en_US.UTF-8 UTF-8' >/etc/locale.gen
echo 'fr_FR.UTF-8 UTF-8' >>/etc/locale.gen
locale-gen
. /etc/profile # The server must run with UTF-8 enabled

A web server is not required for testing. In production, one is needed to add HTTPS and to offload static files serving.

Getting the sources

To get the last version, run these shell commands:

git clone https://git.code.sf.net/p/tomuss/code tomuss-code
cd tomuss-code
if [ ! -d LOCAL ]
then
	# Only to run regression tests. Create your own LOCAL directory
	# with all your customizations and new functionnalities.
	ln -s LOCAL.template LOCAL
fi

TOMUSS is now installed!

All the following commands and file names assumes your current directory is TOMUSS one.

Verify installation

Run the regression test once to verify if TOMUSS is working.

make regtest1

It should display after 5 minutes: Test fine

If the test fails, run locale and verify if the character encoding is UTF-8.

Never run regression tests on production server, they are destructive.

Testing installation

You needs to initialize localization:

make translations

And launch TOMUSS server:

./tomuss.py regtest

Many messages are displayed because TOMUSS is in debug mode. The prompt does not come back until the server is killed. To verify if it is working correctly, follow the following URL:

On the web interface, enter the letter 'a' in UE search box, and click on a UE title. The table are all empty, because TOMUSS is not yet customized. You can play with it, but it is preferable to customize basic things.

BEWARE: with 'regtest' parameter, there is no authentication. So everybody may see and modify TOMUSS content.

TOMUSS customization

Configurations explanations are from the most critical to the less important.

User information

User information is critical because it is needed to sort teachers and students.

Getting user information from LDAP

The information are retrieved via LDAP protocol, but it is easily modifiable.

Edit the config table, the link «Modify TOMUSS configuration» is in the «Administration» block on column on the right.

Modify the most important lines:

To avoid any problem, restart TOMUSS and reload web pages. To test if it is working, you can:

The information you entered in tables is stored in DBregtest

Getting user information from elsewhere

Assuming you use TOMUSS version 7.0.2 or more recent, you define or redefine an user attribute.

Vectorized database access:

def birth(items):
    students = {item.ID_: item for item in items}
    request = 'select ID, BIRTH from INFORMATION where ID in {}'.format(students)
    for student, birth in database.execute(request):
        students[student]['birth'] = birth
configuration.Users.add_attribute(birth)

Sequencial database access:

def birth(items):
    request = 'select BIRTH from INFORMATION where ID = %s'
    for item in items:
    	item.birth = next(iter(database.execute(request % item.ID_)))
configuration.Users.add_attribute(birth)

It is possible to define multiple attributes with one request. Look at 'firstname', 'surname', 'mail' attributes in entity.py. It is also possible do define a TTL.

Sorting users roles

Edit the ACLS table the link «Modify ACLS» is in the «Administration» block on column on the right.

This table define a tree of user groups. You should add the new lines to define more leaves:
MemberGroup
your-own-loginroots
ldap:CN=all-the-teachers...DC=...teachers
ldap:CN=all-the-students...DC=...students
python:lambda x: x[-1].isdigit()students
python:lambda x: not x[-1].isdigit()teachers

The 2 last lines sort the teachers and students from the last character of the login. It is not a good idea to use this trick

The staff groups contains all the users allowed to see the home page and the unprotected table. Never put students in this group.

If needed, users may be removed from an existing group. For example if the LDAP group of teachers contains people that are also student.

Authentication

TOMUSS currently supports CAS, Unix Passwords, OpenID, FaceBook authentications. The best is to use a CAS (Centralized Authentication Service)

CAS

Edit the file LOCAL/config.py.
You will find: configuration.cas = 'https://cas.domain.org/cas'

Set configuration.cas to the URL of your CAS server.

Unix passwords

In this case, a web server is required as a frontend to do the authentication.

Edit the file LOCAL/config.py.
You will find: configuration.cas = 'https://cas.domain.org/cas'

Replace it with configuration.cas = ""

You must use https because passwords are not crypted by the basic authentication method.

Here an example of .htaccess you must use to force users to authenticate with the web server. The web server redirect request on port 80 on the TOMUSS server port 8888 on the local host.

AuthBasicProvider file
AuthType Basic
AuthName "Your Identifiants"
AuthUserFile /etc/apache/passwd
 
# Set the allow/deny order
Order Deny,Allow

# Indicate that any of the following will satisfy the Deny/Allow
Satisfy any

# First off, deny from all
Deny from all

# or require a valid user
Require valid-user
 
RewriteEngine on
RewriteRule ^(.*) http://127.0.0.1:8888/$1 [P]

Switch to authenticated

Once an authentication method is defined, you kill the TOMUSS server and relaunch it without the 'regtest' parameter. to prevent any problems between server running without or with authentication the data is not stored at the same place. So a copy must be done.

killall tomuss.py
mv DB DB.to_delete
mv DBregtest DB
./tomuss.py

Mails

TOMUSS send mails to contacts when there is a problem. It also manages students mass mailing. Edit the SMTP configuration in the configuration table. The link «Modify TOMUSS configuration» is in the «Administration» block on column on the right.

Sending mails

The default smtpserver host is 127.0.0.1 If there is no SMTP server on your computer, replace this value by the name of an SMTP server. To not lost any mail you may indicate 2 SMTP servers as smtp1.foo.org smtp2.foo.org

Contacts mail

The maintainer define both the contact mail displayed to the users and the address receiving the alerts when the server has a problem.

Courses

It is possible to let teachers define the courses list and the student list. But these information are yet defined in your information system, so let TOMUSS take it.

Students list

If the students list are in LDAP. You indicate in the configuration table the values for ou_ue_starts. The students found in the LDAP group will be added in the good table.

You can also redefine your own access method for student lists. In LOCAL/config.py you can add this:

from .. import inscrits

class MyLDAP(inscrits.LDAP):
    def students(self, ue, year=None, semester=None):
        # The following code must use 'ue' to find students
	# Year and Semester are the ones of the table to fill or update
        for i in range(5):
            yield ('login%d' % i,     # The login (as in LDAP)
                   'firstname%d' % i,
                   'surname%d' % i,
                   'mail%d' % i,
                   'g%d' % i,         # A group information
                   's%d' % i)         # A sequence information
	# For this student, search names and mail in LDAP:
	yield ("k10", "", "", "", "gX", "SY")

inscrits.LDAP = MyLDAP

After restarting TOMUSS server, you can visit any table an you will find 6 students: http://127.0.0.1:8888/2008/Test/UE-foobar

The firstname, surname, mail are indicated in this function in case they are not found in the LDAP.

Courses list

In LOCAL/config.py you can add this to create your own list or courses with their title and managers:

from .. import teacher

def get_ue_dict():
  ues = {}
  for i in range(1,10):
    ue_code = "UE-%d" % i
    ues[ue_code] = teacher.UE(
        ue_code,                   # UE Name
        ['MFirstname MSurname',],  # The names of the UE managers
        u'UE%d Title' % i,         # UE title
        [],                        # DEPRECATED: Departments of UE
        1000+i,                    # A key for an external Database
        ['ue%d.master' % i,],      # Login of teachers
        1,                         # DEPRECATED: #students registered
        0,                         # DEPRECATED: #students EC registered
        ['ue%d_test@test.org' % i],# Teachers mails
        2000+i,                    # Another key for an external Database
        )
  return ues

teacher.get_ue_dict = get_ue_dict

Once it is done, you must run the following command to create the TOMUSS database of UE.

(cd LOCAL ; make)

After TOMUSS server restart, you will find your UE on the home page.

The names and mails of manager are not required if they can be found into LDAP using their logins.

Student pictures

Two possibles way to display pictures.

By an external service

If you have a service sending the picture of a student from its login, then make TOMUSS use this service by adding to LOCAL/config.py

def picture(student_id, ticket):
    return 'http://webservice.my.org/picture?id=' + student_id + '.png'
configuration.picture = picture

By TOMUSS itself

Just copy all the pictures in the PICTURES directory with the names: login.JPG

If it is not possible to change picture encoding or name, make your own version of PLUGINS/picture.py

Year and semester

Most of the TOMUSS URL start by Year/Semester and by default, only one semester allows modifications. This semester is defined by the variable year_semester of the configuration table.

Beware: the year is the real calendar year, it is not the university year.

The default TOMUSS configuration assume that each university year is splitted into 2 semesters, the first one starts in august (Automn), the second one starts in February (Spring). In LOCAL/config.py it is defined as:

set_semesters(
    ('Printemps', -1, [2, 7], '#EEFFEE' ),  # Spring (Semester 2)
    ('Automne'  ,  0, [8,13], '#FFE8D0' ),  # Autumn (Semester 1)
    )

So the university semesters for the year 2012-2013 are:

If you want to create your own semesters:

for SEMESTER_NAME in Automn Spring Grades
do
  echo 'from ...TEMPLATES.Printemps import *' \
       >LOCAL/LOCAL_TEMPLATES/$SEMESTER_NAME.py
done

Beware: indicate a new set_semesters in LOCAL/config.py and modify year_semester in the configuration table.

Il you have no semesters in your university, you can indicate (if Grades.py was created):

configuration.set_semesters(
    ('Grades', 0, [8, 20], '#EEEEEE' ),
    )

Semesters history

The content of semesters in history is displayed to students using 'suivi' servers.

A list of manager year/semester to display must be defined, by default, the semester of the calendar year are defined in LOCAL/config.py:

# Add a server for each semester of the current year.
# You must redefine this to enumerate the semester in use.
for i, semester in enumerate(configuration.semesters):
  configuration.suivi.add(
    time.localtime()[0],      # The current Year
    semester,                 # A semester
    socket.getfqdn() + ':%d', # The user visible URL for the 'suivi' server
    8889 + i)                 # The socket port number of the server

It is easier to manage a simple enumeration:

configuration.suivi.add(2013,"Autumn","https:tomusss.my.org",8889)
configuration.suivi.add(2013,"Spring","https:tomusss.my.org",8890)

If CPU power is not a problem, it is possible to indicate the same socket port number for all the 'suivi' servers. It will use less memory.

The 'suivi' servers do modify any data, so it is possible to replicate them on multiple hosts.

Using 'suivi' servers

Enter some grades into a couple of TOMUSS table.

Start all the 'suivi' servers (and TOMUSS server) in background:

make

See the 'suivi' of a student by one of the following methods:

Other makefile goal are useful : stoptomuss, stopsafe, stopsuivi, restartsuivi,

TOMUSS administration

Once installed, it runs alone. Some administrative actions are displayed to the roots users on the right column on the home page.

Recurrent job

Use crontab to schedule some tasks. It is also possible to use TOMUSS planified tasks, they are configurable via a link on the home page. The can launch shell script or any plugin call (as table cell change).

5 minutes job

It is a good idea to run SCRIPTS/crontab_run.py crontab every five minutes. This script relaunch stalled servers, or launch TOMUSS servers if the computer has rebooted.

Hourly backup

Just to be on the paranoiac side, an hourly backup of the data is a good idea:

rsync --archive --no-g --delete DB/. Distant_Location

SCRIPTS/hour_job can be used to do a backup in a GIT repository.

Night job

It is a good idea to run SCRIPTS/night_job every night. The log of the work is stored in LOGS directory. The script does the following things:

Upgrade

for a simple and secure upgrade:

If there is a problem found after that, you can stop the new servers and restart the old ones. They are working on the same files.

You can create your own installation procedure using SCRIPTS/install as a starting point.

Lost data

It is really hard to lose data with TOMUSS:

These 2 restorations are only manual, not automatic.