Page History

...

• 80 (HTTP)
• 443 (SSL virtual host for HTTPS)
• 446 (SSL virtual host for authenticating via user certificates)
• 447 (SSL virtual host for certificate authentication using "SSLVerifyClient require")
• 8443 (SSL virtual host for SP's back-channel SOAP calls for attributesfrom MIT SPs)
• 8444 (SSL virtual host for back-channel SOAP calls from InCommon SPs)

The terracotta server will listen on the following TCP ports:

• 9510 (client-to-server)
• 9520 (JMX)
• 9530 (server-to-server)
  Note these listeners only need to accept connections from peer servers in the cluster, so these ports should be configured accordingly in the firewall.

...

• Use stock httpd RPM install (standard NIST install)

• Install mod_ssl and mod_auth_kerb RPMs:

  No Format
  # yum install mod_ssl # yum install mod_auth_kerb

...

• Install the server certificate, key, and CA files in /etc/pki/tls/certs/ and /etc/pki/tls/private/, as appropriate, and make sure the paths are correct in ssl.conf, cert-authn.conf, and idp-attr-query.conf (see below). The key file should be readable by only the tomcat user, as the idp webapp also uses it.

• In /etc/httpd/conf/httpd.conf, set ServerName:

  No Format
  ServerName idp.mit.edu:80

  and set the UseCanonicalName option to On:

  No Format
  UseCanonicalName On

• Disable the stock "Welcome" page, by commenting out the lines in /etc/httpd/conf.d/welcome.conf

• In /etc/httpd/conf.d/ssl.conf, set the SSLRandomSeedoptions:

  No Format
  SSLRandomSeed startup file:/dev/urandom 1024 SSLRandomSeed connect file:/dev/urandom 1024

  within the VirtualHostblock, set the ServerName:

  No Format
  ServerName idp.mit.edu:443

  set the SSL cipher suite:

  No Format
  SSLCipherSuite HIGH:MEDIUM:EXP:!aNULL:!SSLv2:+SHA1:+MD5:+HIGH:+MEDIUM:+EXP

  Install the server certificate, key, and CA files in /etc/pki/tls/certs/ and /etc/pki/tls/private/, as appropriate, and set the paths in ssl.conf:

  No Format
  SSLCertificateFile /etc/pki/tls/certs/idp.mit.edu-cert.pem SSLCertificateKeyFile /etc/pki/tls/private/idp.mit.edu-key.pem SSLCertificateChainFile /etc/pki/tls/certs/EquifaxCA.pem SSLCACertificateFile /etc/pki/tls/certs/mitCAclient.pem

  set the SSL options:

  No Format
  SSLOptions +StrictRequire

  configure custom logging:

  No Format
  CustomLog logs/ssl_request_log \ "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

  ensure that all access is via SSL:

  No Format
  <Directory /> SSLRequireSSL </Directory>

  ensure that all rewrite rules are inherited:

  No Format
  RewriteEngine On RewriteOptions inherit

• Install an HTTP keytab in /etc/httpd/conf/keytab; it must be readable by (only) the apache user. This is used for authentication by HTTP/SPNEGO, as configured in /etc/httpd/conf.d/auth_kerb.conf.
• Install the server's InCommon certificate and key files in /usr/local/shibboleth-idp/credentials/ – these are used for back-channel attribute queries by InCommon SPs, and are referenced in /etc/httpd/conf.d/idp-attr-query.conf. Make sure that they are readable by tomcat, as the idp webapp also accesses them. Also install the InCommon CA file in /etc/pki/tls/certs.
• Install these additional conf files from the touchstone locker (/mit/touchstone/config/idp2-core/httpd) in /etc/httpd/conf.d:
  • auth_kerb.conf
    This configures authentication by HTTP/SPNEGO (replaces version installed by mod_auth_kerb).
  • cert-authn.conf
    This sets up the separate virtual hosts for certificate authentication on ports 446 and 447 (for certificate optional and required, respectively).
  • idp-attr-query.conf
    This sets up the vhosts for back-channel attribute queries on ports 8443 and 8444 (from MIT and InCommon SPs, respectively).
  • idp-rewrite.conf
    This adds various rewrite rules for compatibility, etc.
  • proxy_ajp.conf
    Configures the AJP proxy module for the idp webapp (replaces version installed by httpd).
  • ssl.conf (see above)
  • welcome.conf (see above)

• SELinux context mappings must be added for the non-standard ports configured for certificate authentication and attribute query, so that httpd can use them:

  No Format
  # semanage port -a -t http_port_t -p tcp 446 # semanage port -a -t http_port_t -p tcp 447 # semanage port -a -t http_port_t -p tcp 8444

• Install /var/www/cgi-bin/negotiate-error-handler, from /mit/touchstone/cgi/. This script is used as the ErrorDocument 401 handler for HTTP/SPNEGO authentication, and is needed because handling the error behind the AJP proxy would break the Negotiate option.

• Install our standard robots.txt and favicon.ico files in /var/www/html. The robots.txt should disallow all access:

  No Format
  User-agent: * Disallow: /

  Current versions of these files may be found in the touchstone locker, in /mit/touchstone/config/htdocs/.

• Make sure httpd is started at boot time:

  No Format
  # chkconfig httpd on

• Add the following settings to the stock /etc/logrotate.d/httpd configuration file:

  No Format
  daily rotate 100 compress delaycompress

  This will cause the httpd log files in /var/log/httpd/to be rotated daily and compressed, saving 100 days of old logs (in case we need them for quarterly metrics).

  Install JDK and enhanced JCE

• The IdP uses JDK 1.6; download and install the RPM from Sun, or use the version in the downloads directory in the touchstone locker (jdk-6uNN-linux-amd64.rpm, where NN is the update number):

  No Format
  # rpm -Uvh jdk-6uNN-linux-amd64.rpm

• We installed the Bouncy Castle JCE in the 2.1.5 IdPs, but it should no longer be needed. To support additional cryptographic algorithms used by the IdP, download and install the Bouncy Castle JCE jar file (http://polydistortion.net/bc/index.html) in the lib/ext directory of the JRE (/usr/java/latest/jre/lib/ext/). For example:

  No Format
  # cd /usr/java/latest/jre/lib/ext # cp /path/to/bcprov-jdk16-145.jar .

  (Replace the file version number as needed).

Add it as a provider in in the JRE's lib/security/java.security, e.g.:

security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider

(Replace 9 with the next sequential provider number as needed).

• To support use of crypto key sizes larger than 2048 bits, we also add the Unlimited Strength Security Policy to the JVM. Download jce_policy-6.zip from the locker downloads directory, or from Sun (http://java.sun.com/javase/downloads/index.jsp, Other Downloadssection at the bottom). Unzip the policy zip file and copy local_policy.jar and US_export_policy.jar into the JRE's lib/security directory (replacing the versions installed from the JDK RPM).

  No Format
  # cd /tmp # unzip /path/to/jce_policy-6.zip # cd jce # cp *.jar /usr/java/latest/jre/lib/security/

• The MIT CA certificates must be added to the certificate trust store. This can be done by downloading and adding them explicitly to the JRE cacerts trust store, as follows:

  No Format

  # wget 'http://mv.ezproxy.com.ezproxyberklee.flo.org/mitca.crt' # wget 'http://mv.ezproxy.com.ezproxyberklee.flo.org/mitClient.crt' # setenv JAVA_HOME /usr/java/latest # cd $JAVA_HOME/jre/lib/security # cp -p cacerts cacerts.orig # $JAVA_HOME/bin/keytool -import -keystore cacerts -alias mitca -file /path/to/mitca.crt # $JAVA_HOME/bin/keytool -import -keystore cacerts -alias mitclientca -file /path/to/mitClient.crt

  The password for the trust store is "changeit". Answer "yes" to the "Trust this certificate?" prompt

An already-updated cacerts store is available in the touchstone locker, in /mit/touchstone/config/java.

• For convenience, install shell profile scripts in /etc/profile.d that define JAVA_HOME, e.g. java.csh:

  No Format
  setenv JAVA_HOME /usr/java/default if ( "${path}" !~ *${JAVA_HOME}/bin* ) then set path = ( ${JAVA_HOME}/bin $path ) endif

  java.sh:

  No Format
  export JAVA_HOME=/usr/java/default if ! echo $PATH | grep -q ${JAVA_HOME}/bin ; then export PATH=${JAVA_HOME}/bin:$PATH fi

Install Tomcat

• Download current Tomcat 6.0 binary distribution (tested with 6.0.20, available in /mit/touchstone/downloads/apache-tomcat-6.0.20.tar.gz, and install under /usr/local:

  No Format
  # cd /usr/local # tar xzf /path/to/apache-tomcat-6.0.20.tar.gz # rm -f tomcat # ln -s apache-tomcat-6.0.20.tar.gz tomcat

• Create the tomcat user, and change the ownership of the tomcat tree:

  No Format
  # groupadd -g 52 tomcat # useradd -u 52 -g tomcat -c "Tomcat User" -d /usr/local/tomcat -M -s /sbin/nologin tomcat # chown -R tomcat:tomcat /usr/local/apache-tomcat-6.0.20

• Install our version of conf/server.xml (from /mit/touchstone/config/idp2-core/tomcat), which properly configures the AJP connector on port 8009, and disables the HTTP connector on port 8080.

• Install the tomcat init script in /etc/init.d/, and make sure tomcat is started at boot time:

  No Format
  # chkconfig --add tomcat

Install Shibboleth IdP

• Run the idp application installer from our customized binary distribution, available in /mit/touchstone/builds/NIST/idp2-mit/mit-shibboleth-identityprovider-2.x.y-bin.tgz, and the install script contained therein. For example:

  No Format
  # cd /tmp # rm -rf shibboleth-identityprovider-2.* # tar xzf /path/to/mit-shibboleth-identityprovider-2.1.5-bin.tgz # cd shibboleth-identityprovider-2.1.5 # ./install.sh [There should be no need to override the default responses to the installer's questions.]

  By default (because of one of our customizations to the stock Internet2 distribution) this will install under /usr/local/shibboleth-idp/. The installer will not overwrite the configuration files of an existing installation. For a new installation, the installer will generate a keystore, and prompt for its password; currently we do not use this keystore, so the password does not matter. This distribution contains the standard shibboleth-identityprovider binary distribution

  , from the Internet2

  zip file

  (http://shibboleth.internet2.edu/downloads/shibboleth/idp/latest/^{Image Removed})

  , plus the following customizations:

  • MITLoginHandlerMultiAuthnOptionLoginHandler
    This our custom login handler extension, which supports multiple authentication options (certificate, username/password, etc.). The , Duo 2FA, cookie-based custom SSO, etc. The current jar file for the extension is available in the locker, in /mit/touchstone/builds/NIST/shibboleth-idp-ext-MultiAuthnOptionLoginHandlerMITLoginHandler-<version>.jar, where <version> is 1.0 for idp version 2.1.x, and 1.1 for idp version 2.3.x; it is copied into the lib/ subdirectory of the binary distribution.
  • mitlogin
    This provides the custom login pages for MIT users, the web deployment descriptor, etc. It is available as a tarball (/mit/touchstone/builds/NIST/idp2-mit/mitlogin.tgz) which is unpacked into the top-level directory of the binary distribution.
  • MySQL connector
    This is needed for accessing our targeted ID MySQL database. The connector is available in /mit/touchstone/downloads/mysql-connector-java-5.1.NN-bin.jar (current version is 5.1.14).27); it is copied into the lib/ subdirectory of the binary distribution.
  • Delegation extension
  • IDWSF consumer package
  • duo.jar
    This is used for Duo 2FA; it is copied into the lib/ subdirectory of the binary distributionvt-ldap-2.8.5.jar
    This is an updated version of the IdP's LDAP package. We currently use this in version 2.1.5 (instead of version 2.8.4 of the jar file) because the older version does not properly handle certain error conditions with the connection to the LDAP server. This should not be needed in future releases.
• The installer will create and populate /usr/local/shibboleth-idp; the web application (war) file will be in /usr/local/shibboleth-idp/war/idp.war, but the current version of the idp.war will be available in the locker (/mit/touchstone/builds/NIST/idp2-mit/idp.war).

• The idp application, running under Tomcat, needs full access to the install directory, so make sure it is owned by the tomcat user, e.g.:

  No Format
  # chown -R tomcat:tomcat /usr/local/shibboleth-idp

  To ensure that we run the current version of the web application, download the latest idp.war file from the touchstone locker (/mit/touchstone/builds/NIST/idp2-mit/idp.war) and copy it into /usr/local/tomcat/webapps/:

  No Format
  # cp /path/to/idp.war /usr/local/tomcat/webapps/ # chown tomcat:tomcat /usr/local/tomcat/webapps/idp.war

• Copy the idp's endorsed jar files to tomcat's endorsed dir:

  No Format
  # mkdir -p /usr/local/tomcat/endorsed # cp -p /usr/local/shibboleth-idp/lib/endorsed/*.jar /usr/local/tomcat/endorsed/ # chown -R tomcat:tomcat /usr/local/tomcat/endorsed

• Copy in the idp config files for the server, to the conf subdirectory (they are available in /mit/touchstone/config/idp2-core/shibboleth-idp/); these include:
  • attribute-filter.xml
  • attribute-resolver.xml.in
  • authn-config.properties
  • handler.xml
  • internal.xml
  • logging.xml
  • login.config
  • relying-party.xml
  • service.xml
  • tc-config.xml (for terracotta clustering)
    You must replace %%LDAPUID%% and %%LDAPPASSWORD%% in attribute-resolver.xml.in with the principal uid (e.g. touchstone-core-service) and passord (principalCredential attribute) for accessing our LDAP server, and save the resulting file as attribute-resolver.xml. Make sure the file is not world-readable.

Terracotta

(See https://wiki.shibboleth.net/confluence/display/SHIB2/IdPCluster^{Image Removed})

The terracotta software is used to cluster the IdP nodes. Each node must run the terracotta server, as well as the instrumented client (tomcat, in our case). The terracotta server operates in either the active or passive role; only one server should be in the "active/coordinator" state at a time.

Download the terracotta tarball; our current version is in the touchstone locker, in /mit/touchstone/downloads/terracotta-x.y.z.tar.gz. Extract it under /usr/local, create a logs directory for it, make it owned by the tomcat user, and symlink /usr/local/terracotta to it. For example (replace 3.1.1 with the appropriate terracotta version number):

# cd /usr/local
# tar xzf /path/to/terracotta-3.1.1.tar.gz
# mkdir -p terracotta-3.1.1/logs
# chown -R tomcat:tomcat terracotta-3.1.1
# rm -f terracotta
# ln -s terracotta-3.1.1 terracotta

The IdP requires the installation of a couple of Terracotta Integration Modules, and the generation of a boot jar file for Tomcat, which is specific to the Java version:

# setenv TC_HOME /usr/local/terracotta-3.1.1
# setenv TC_INSTALL_DIR $TC_HOME
# setenv JAVA_HOME /usr/java/default
# $TC_HOME/bin/tim-get.sh install tim-vector 2.5.1 org.terracotta.modules
# $TC_HOME/bin/tim-get.sh install tim-tomcat-6.0 2.0.1
# $TC_HOME/bin/make-boot-jar.sh -f /usr/local/shibboleth-idp/conf/tc-config.xml
# chown -R tomcat:tomcat /usr/local/terracotta-3.1.1

...

Install the init script from /mit/touchstone/maint/shibboleth-idp/terracotta/terracotta.init in /etc/init.d, and make sure it is configured to start at boot time. Note that terracotta must be started before tomcat.

# cp /path/to/terracotta.init /etc/init.d/terracotta
# chmod 755 /etc/init.d/terracotta
# chkconfig --add terracotta

To avoid performance impact during business hours, we disable automatic garbage collection of terracotta objects. Instead, we run a nightly cron job to do the garbage collection manually. Since this should only be done on the active/coordinator node, the script, run-dgc-if-active.sh, checks the server mode, then runs the garbage collector if and only if the server is the active node. Both the script and cron file can be obtained in /mit/touchstone/maint/shibboleth-idp/terracotta/; install as follows:

# cp /path/to/run-dgc-if-active.sh /usr/local/shibboleth-idp/bin/
# cp /path/to/run-dgc.cron /etc/cron.d/run-dgc

For more information on maintaining the IdP cluster, see https://wikis-mit-edu.ezproxyberklee.flo.org/confluence/display/TOUCHSTONE/Maintaining+the+terracotta+cluster+on+the+IdPs^{Image Removed}

Targeted ID MySQL database

...

Start up the daemon, and secure the installation:

# /etc/init.d/mysqld start
# mysql_secure_installation

...

Make sure the daemon starts at boot time:

# chkconfig mysqld on

Make sure that you set a firewall rule which allows the peer IdP node to connect to the daemon (on TCP port 3306).

...

Create the shib and (optionally) shibadmin database users, e.g.:

# mysql -u root -p
Enter password: [Supply the root password created above]

mysql> CREATE USER 'shib'@'localhost' IDENTIFIED BY 'PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shib'@'idp-1.mit.edu' IDENTIFIED BY 'PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shib'@'idp-2.mit.edu' IDENTIFIED BY 'PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shibadmin'@'localhost' IDENTIFIED BY 'ADMINPASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shibadmin'@'idp-1.mit.edu' IDENTIFIED BY 'ADMINPASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shibadmin'@'idp-2.mit.edu' IDENTIFIED BY 'ADMINPASSWORD';
Query OK, 0 rows affected (0.00 sec)

...

Install the scripts used to maintain the targeted ID database. This includes the tid-syncd daemon, which is used to synchronize the database between the machines in the cluster, an init script for the daemon, and a database backup script (run out of cron). The software is installed from the source tarball in /mit/touchstone/src/targeted-id-source.tgz.

# mkdir /tmp/targeted-id
# cd /tmp/targeted-id
# tar xzf /path/to/targeted-id-source.tgz
# make install

...

To create a new (empty) database, process the schema file, e.g.:

# mysql -u root -p < /usr/local/targeted-id/etc/tid-init.sql

...

Perform this step to set up the database when adding a new machine to an existing cluster. You should obtain the backup from an existing machine in the cluster.

# mysql -u root -p < /path/to/most-recent-backup.sql

...

The IdP resolver will call a stored function tid to generate and retrieve the targeted ID as the attribute source. The source for this function is in /usr/local/targeted-id/etc/tid.sql. Note that the function uses 2 hard-coded "secret" strings to randomize the generated IDs. It is imperative that you set these 2 strings in the function source before loading it, and that all nodes in a cluster always use the same 2 secret strings. To set the secret strings, make a copy of tid.sql (to, say, tid.sql.private), edit the copy, locate the declarations of mySecret1 and mySecret2, and replace the secret_1 and secret_2 string literals accordingly.

    DECLARE mySecret1 VARCHAR(255) DEFAULT 'secret_1';
    DECLARE mySecret2 VARCHAR(255) DEFAULT 'secret_2';

Make sure the resulting file is only readable by root. Once you have correctly set these strings, you can load the function as follows:

# mysql -u root -p targetedID < /usr/local/targeted-id/etc/tid.sql.private

...

Once you have loaded the database tables and stored function, you must set up the grant tables for the shib and shibadmin database users. The file /usr/local/targeted-id/etc/tid-grants.sql contains the necessary grants for users on the local machine; you should modify this to add the same grants for the users on the peer machine.

# mysql -u root -p < /usr/local/targeted-id/etc/tid-grants.sql

...

In all but exceptional cases, a generated ID will be the consistent result of hashing the username, provider ID, and secret strings, so the ID for a (user, SP) pair will be the same, no matter which node in a cluster it is generated on. This may not be true, though, in the (highly unlikely) case of a hash collision, or if an ID needs to be revoked. To make sure that the databases in a cluster remain in synch, therefore, we employ a tid-syncd daemon which propagates new IDs to the peer(s). This daemon should run on all nodes, and be started at boot time. The daemon logs at the LOCAL5 syslog facility, so /etc/syslog.conf should be adjusted accordingly:

# targeted ID sync daemon
local5.*                                       /var/log/tid-syncd

Start the daemon, and make sure it is started at boot time:

# /etc/init.d/tid-sync start
# chkconfig --add tid-sync start
# chkconfig --add tid-sync

Entitlements MySQL database

Beginning in the fall of 2012, the IdP attribute resolver can generate an eduPersonEntitlement attribute dynamically for a provider, based on parameters retrieved from a local MySQL database. The following steps for initializing this database assumes that MySQL has already been initialized, and the local targeted ID database has already been set up (see above).

Initialize the entitlements database

# cd /usr/local/shibboleth-idp/conf/entitlements
# mysql --defaults-extra-file=/usr/local/targeted-id/etc/root.cnf < entitlements.sql

Load the stored procedure into the database

# cd /usr/local/shibboleth-idp/conf/entitlements
# mysql --defaults-extra-file=/usr/local/targeted-id/etc/root.cnf < select_SP_params.sql

Set up the grant tables

# cd /usr/local/shibboleth-idp/conf/entitlements
# mysql --defaults-extra-file=/usr/local/targeted-id/etc/root.cnf < grants.sql

Firewall

Make sure that the additional port used by the IdP are enabled in the firewall. Use the command "iptables --list -n --line-numbers" to determine the proper rule number; the following example assumes we are inserting rules beginning at number 36. Also replace 18.x.y.z with the appropriate IP address of the peer node in the cluster, not the local host.

# iptables --list -n --line-numbers
# iptables -I RH-Firewall-1-INPUT 36 -m state --state NEW -m tcp -p tcp --dport 8443 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 37 -m state --state NEW -m tcp -p tcp --dport 8444 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 38 -m state --state NEW -m tcp -p tcp --dport 446 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 39 -m state --state NEW -m tcp -p tcp --dport 447 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 40 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 9510 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 41 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 9520 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 42 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 9530 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 4243 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 3306 -j ACCEPT
# /etc/init.d/iptables save