Simplicité instances manager

This document applies to the Simplicité Instances Manager (SIM) software available on instances servers.

Introduction

Architecture

The SIM offers a command line interface (CLI), a webservices interface (API) - built on top of the CLI - and a web user interface (UI) - built on top of the API - to manage several Simplicité® instances on a server:

Instances manager

Using the CLI offers the full features of the SIM, the API and the UI offers only a subset of the CLI features.

Features

The SIM offers the following features:

All above actions are triggered manually using the CLI, the API or thee UI.

The following actions are triggered by a system-level task scheduler (cron table);

Note that the automatic backup can be configured to export the backup files to a remote storage. Otherwise the backup files are only stored locally on the SIM server.

Implementing pre/post action hooks (see bellow) allow you to customize the standard features of the SIM.

UI usage

The instances manager UI is available on <base URL>/ui.

The UI offers uses the API documented bellow and thus offers equivalent - but somehow limited by a simplified user experience - services.

To create a new instance, enter an instance name (or leave empty for an automatic name) then select the version from the Create button:

To manage an existing instance, clic on one instance in the list, then select the action from the Action button:

The Monitor action opens a new tab and allow to monitor a given instance by tracking record of recurring health checks:

API usage

The instances manager API is available on <base URL>/api.

The API services are using HTTP GET methods only and URL-encoded parameters only. All API responses are JSON-encoded.

For all API services, you can get a pretty printed JSON response by appending &prettyprint=true.

By default the authentication method is HTTP basic auth (username + password), but client certficate authentication can also be used. The examples bellow are using the curl command HTTP client, <credentials> is thus either:

Manager API

Get API configuration

curl -k -s <credentials> <base URL>/api[?action=config]

The action parameter can be ommitted in this case as config is the default action.

Get available versions

curl -k -s <credentials> <base URL>/api?action=versions

Instances API

List instances

curl -k -s <credentials> <base URL>/api?action=list[&param=<version>]

The parameters are:

Create instance

curl -k -s <credentials> <base URL>/api?action=add[&name=<instance name>][&param=<version (or instance name to clone)>][&options[<option name>]=<option value>]

The parameters are:

Stop instance

curl -k -s <credentials> <base URL>/api?action=stop&name=<instance name>[&force=<true|false>]

The parameters are:

Start instance

curl -k -s <credentials> <base URL>/api?action=start&name=<instance name>

The parameters are:

Instance health check

curl -k -s <credentials> <base URL>/api?action=health&name=<instance name>

The parameters are:

Instance I/O credentials

curl -k -s <credentials> <base URL>/api?action=iocredentials&name=<instance name>

The parameters are:

Instance last logs

curl -k -s <credentials> <base URL>/api?action=logs&name=<instance name>

The parameters are:

Download instance logs

curl -k -s <credentials> <base URL>/api?action=download&param=logs&name=<instance name>

The parameters are:

Get access logs report

curl -k -s <credentials> <base URL>/api?action=accesslogs&name=<instance name>

The parameters are:

Upgrade instance

curl -k -s <credentials> <base URL>/api?action=upgrade&name=<instance name>

The parameters are:

Upgrade all automatic upgrade instances

curl -k -s <credentials> <base URL>/api?action=upgradeall

Reset instance

curl -k -s <credentials> <base URL>/api?action=reset&name=<instance name>

The parameters are:

Warning: a reset will erase all existing configuration and data.

Delete instance

curl -k -s <credentials> <base URL>/api?action=del&name=<instance name>[&force=<true|false>]

The parameters are:

Clone instance

curl -k -s <credentials> <base URL>/api?action=clone&name=<instance name>&param=<cloned instance name>

The parameters are:

Rename instance

curl -k -s <credentials> <base URL>/api?action=rename&name=<instance name>&param=<new instance name>

The parameters are:

Save instance on server

curl -k -s <credentials> <base URL>/api?action=save&name=<instance name>

The parameters are:

Save all automatic save instances on server

curl -k -s <credentials> <base URL>/api?action=saveall

Download instance data

curl -k -s <credentials> <base URL>/api?action=download&param=data&name=<instance name>

The parameters are:

Download instance webapp

curl -k -s <credentials> <base URL>/api?action=download&param=webapp&name=<instance name>

The parameters are:

Download full instance

curl -k -s <credentials> <base URL>/api?action=download&name=<instance name>

The parameters are:

Once downloaded you can simply unzip the instance and launch it locally:

Windows:

@echo off

set JAVA_HOME=<path to your JDK home, e.g. %CD%\jdk-1.8.0>
set PATH=%JAVA_HOME%\bin;%PATH%

set TOMCAT_ROOT=<path to your instance, e.g. %CD%\tomcat>
set TOMCAT_ADMIN_PORT=8005
set TOMCAT_HTTP_PORT=8080
set TOMCAT_HTTPS_PORT=8443

set JAVA_OPTS=%JAVA_OPTS% -server -Dserver.vendor=tomcat -Dserver.version=9 -Dfile.encoding=UTF-8 -Dgit.basedir=%TOMCAT_ROOT%\webapps\ROOT\WEB-INF\git -Dtomcat.adminport=%TOMCAT_ADMIN_PORT% -Dtomcat.httpport=%TOMCAT_HTTP_PORT% -Dtomcat.httpsport=%TOMCAT_HTTPS_PORT%
if not exist %TOMCAT_ROOT%\work mkdir %TOMCAT_ROOT%\work
if not exist %TOMCAT_ROOT%\temp mkdir %TOMCAT_ROOT%\temp
if not exist %TOMCAT_ROOT%\logs mkdir %TOMCAT_ROOT%\logs
cd %TOMCAT_ROOT%\bin
.\startup.bat
exit

Linux:

#!/bin/bash

export JAVA_HOME=<path to your JDK home, e.g. `pwd`/jdk-1.8.0>
export PATH=$JAVA_HOME/bin:$PATH

export TOMCAT_ROOT=<path to your instance, e.g. `pwd`/tomcat>
export TOMCAT_ADMIN_PORT=8005
export TOMCAT_HTTP_PORT=8080
export TOMCAT_HTTPS_PORT=8443

export JAVA_OPTS="$JAVA_OPTS -server -Dserver.vendor=tomcat -Dserver.version=9 -Dfile.encoding=UTF-8 -Dgit.basedir=$TOMCAT_ROOT/webapps/<instance|'ROOT'>/WEB-INF/git -Dtomcat.adminport=$TOMCAT_ADMIN_PORT -Dtomcat.httpport=$TOMCAT_HTTP_PORT -Dtomcat.httpsport=$TOMCAT_HTTPS_PORT"
[ ! -d $TOMCAT_ROOT/work ] && mkdir $TOMCAT_ROOT/work
[ ! -d $TOMCAT_ROOT/temp ] && mkdir $TOMCAT_ROOT/temp
[ ! -d $TOMCAT_ROOT/logs ] && mkdir $TOMCAT_ROOT/logs
cd $TOMCAT_ROOT/bin
./startup.sh

Instance actions history

curl -k -s <credentials> <base URL>/api?action=history&name=<instance name>

The parameters are:

Requests API

This applies only if the instances manager is configured to handle instance requests.

List requests

curl -k -s <credentials> <base URL>/api?action=requests

CLI usage

The instances manager CLI is available on <base URL>/term if the web shell service is enabled on the server, otherwise a SSH client is required. The available actions are the same as the ones offered from the API services.

CLI actions

Usage:

sim

Get configuration

sim config

Get available versions

sim versions

Upgrade SIM and retreive latest templates

sim refresh

On a standard SIM deployment, this is done automatically on a regular basis using a dedicated cron task (by default every day during the night).

List all instances

sim list [<version|single keyword> or SQL:<sql where clause>]

Example of SQL where clause (i is the table alias for instances, v the table alias for versions and r the table alias for requests):

sim list "SQL:i.version in ('4.0', '5') and i.status = 'started' and i.version_date < v.date"

List one single instance

sim get <name>

Add an instance

sim add <name> <version (or instance name to clone)>

Save an instance

sim save <name>

Note: during the save action the Tomcat process, if running, is suspended to ensure saved data consistency.

Reset an instance

sim reset <name> <version>

Warning: a reset will erase all existing configuration and data.

Delete an instance

sim delete|rm <name>

Clone an instance

sim clone|cp <name> <cloned name>

Rename an instance

sim rename|mv <name> <new name>

Stop/start an instance

sim stop <name>
sim start <name>

Upgrade an instance

sim upgrade|up <name>

View last logs of an instance

sim logs <name>

Download an instance

sim download <name> <type: all|webapp|data|logs>

Get access logs report for an instance

sim accesslogs <name>

Control the Tomcat server of an instance (only applicable on a started instance)

sim tomcat-<status|stop|start|debug> <name>

Using debug is similar to start except that the Tomcat server is then started in JPDA debug mode. The JPDA port is available in the JPDA_ADDRESS environment variable. This port is temporary allowed thru the firewal.

Warning: starting Tomcat in debug mode must only be done when required, it slows down your instance and makes it vulnerable. It is only suitable for ponctual debugging.

Run ant task for an instance

sim ant-<task: purgelogs|purgejobs|purgesupervisions|purgerecyclebin|purgeexports|purgetempfiles|purgeall|buildindex|...> <name> [<param>]

Protect/unprotect an instance

sim protect <name>
sim unprotect <name>

Activate/deactivate nightly upgrade on an instance

sim autoupdate <name>
sim noautoupdate <name>

Activate/deactivate auto-save on an instance

sim autosave <name>
sim noautosave <name>

Set/reset custom URL of an instance

sim seturl <name> <custom URL>
sim reseturl <name>

Set version of an instance

sim setversion <name> <version>

Warning: Only a higher version than current instance version is possible. If set to lower version, the instance will not start anymore.

Set status of an instance

sim setstatus <name> <started|stopped>

Warning: the status is set by start/stop actions, use this only if there was an unexpected issue and the status is incorrect vs actual instance's status.

Show process of an instance

sim ps <name>

Start shell for instance

sim shell|sh <name>

Connect to instance's database

sim database|db <name> [<input file>]

The client tool used to connect to database depends on the database vendor (e.g. mysql client is used for MySQL/MariaDB, psql for PostgreSQL, ...)

Note: with embedded HSQLDB the instance must be stopped before connecting to database

Instance monitoring

sim monitor <name> [<interval in seconds, defaults to 60>]

Monitoring is done by calling intance's /health endpoint. Each time a monitoring error/warning is detected a call to the optional hook monitoring.sh is done with a reason argument.

Run a unit tests shared code on an instance

sim unitttests <name> <unit test shared code name>

SonarQube® analysis for instance's module(s)

sim sonar <name> [<module 1> [<modules 2> ...]]

If no module is specified all modules are analysed.

Note: SonarQube® must be installed and configured

Advanced CLI usage examples

This example gets status of all instances of a given version:

for NAME in `sim ls 5 | awk '{print $1}'`; do sim health $NAME | grep status; done

This example stops all instances with matching names:

sim ls myinstances\* | awk '{print $1}' | xargs -L 1 sim stop

This more advanced example upgrades all started instances of version 4.0or 5 that needs upgrade:

sim list "SQL:i.version in ('4.0', '5') and i.status = 'started' and i.version_date < v.date" | awk '{print $1}' | xargs -L 1 sim upgrade

Etc.

Hooks

It is possible to extend the manager behavior using the following script hooks (located in <manager home>/hooks):

pre-add.sh
post-add.sh
pre-reset.sh
post-reset.sh
pre-upgrade.sh
post-upgrade.sh
pre-save.sh
post-save.sh
pre-start.sh
post-start.sh
pre-stop.sh
post-stop.sh
pre-del.sh
post-del.sh
monitoring.sh

All hooks receive as argument their caller script's arguments (e.g. instance name, version and options for pre/post-add.sh).

Note: if your usage of the hooks is to add/alter elements to instances' Tomcat (configuration files, additional Java JARs or static web components, ...) you need to do it in the pre/post-add.sh but also in the pre/post-upgrade.sh (and maybe also in the pre/post-reset.sh) as upgrade (and reset) actions reinstall Tomcat.

Examples:

This post-start.sh hook looks for a license XML key named license-<instance-name>.xml in the licensekeys folder located in the base $APPS_HOME folder, if present this file is loaded on the instance's I/O endpoint using the appropriate credentials (which are available as environment variables):

#!/bin/bash

XML=$APPS_HOME/licensekeys/license-$USER.xml
if [ -f $XML ]
then
    echo "Importing $XML file..."
    RES=`curl -s -u $SERVICE_USER:$SERVICE_PASSWORD --form service=xmlimport --form file=@$XML $SERVICE_URL`
    if [ "$RES" = "OK" ]
    then
        mv -f $XML $XML.`date +%Y%m%d%H%M%S`
        echo "Done"
    else
        echo "Failed to import $XML: $RES"
    fi
else
    echo "No $XML file"
fi

Note: this license key example can easily be transposed to any data import on the I/O endpoint

Docker CLI actions

As of version 5.12, basic actions are also available to manage instances as Docker containers:

sim docker config
sim docker pull
sim docker list [--json]
sim docker add [--json] [<name|->] [<mysql|postgresql>]
sim docker del [--json] <name>
sim docker start [--json] <name>
sim docker stop [--json] <name>
sim docker save [--json] <name>
sim docker restart [--json] <name>
sim docker inspect <name>
sim docker logs <name> [<nb lines>]
sim docker shell <name>
sim docker database <name>

An optional API and UI gateway is also available to wrap the above CLI.

Note: at that stage the Docker instances relies on the various Simplicité Docker base server images and only supports Traefik as reverse proxy. The above hooks are not applicable.

Configuration

Global settings

The global settings are set as environment variables in the <manager home>/config.sh file.

Core settings

Manager settings
System settings

Warning: these system settings SHOULD NOT be changed unless you know precisely what you are doing.

Reverse proxy settings
SSL settings

Backup settings

SCP destination
FTP destination
OpenStack object storage container destination
Google Cloud object storage bucket destination
AWS S3 object storage bucket destination

E-mail settings

SonarQube settings

Databases settings

MySQL settings
PostgreSQL settings
Oracle settings

Note that other Oracle-related environment variable can be set here if not set otherwise, e.g:

ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${LD_LIBRARY_PATH}
SQLPATH=${ORACLE_HOME}/lib:${SQLPATH}
PATH=${ORACLE_HOME}/bin:${PATH}
NLS_LANG=AMERICAN_AMERICA.UTF8
ORACLE_SID=XE
SQLServer settings

Note that other SQLServer-related environment variable can be set here if not set otherwise:

MSSQL_HOME=/opt/mssql-tools
PATH=${MSSQL_HOME}/bin:${PATH}

Version-level settings

The version-level settings are set as environment variables in the <manager home>/version-<version>.sh file (without the export keyword)

JVM settings

Auto-patching at instance startup

Note: this feature is different from the fact that the instance is marked as nightly updated.

This APPS_AUTOUPGRADE setting just refers to the way an instance is being updated by the SIM:

Instance-level settings

The instance-level settings are set as environment variables in the <instance home>/.simplicite file.

Warning: these variables are set upon creation of the instance based on the global configuration. In general they should not be changed manually unless you know what you are doing.

Example:

SERVICE_CONTEXT=
SERVICE_TOMCATMINMEM=256
SERVICE_TOMCATMAXMEM=1024
SERVICE_URL=http://localhost:10078/io
SERVICE_USER=designer
SERVICE_PASSWORD=d98990faa918c84584ef
SERVICE_DATABASE=oracle
SERVICE_DATABASE_HOST=localhost
SERVICE_DATABASE_PORT=1521
SERVICE_ORACLE_SID=XE
SERVICE_AUTOUPGRADE=true
SERVICE_DEBUG=false
SERVICE_DEBUG_ORIGIN=192.168.0.0

SonarQube

You can add the following variables for instance-specific SonarQube analysis settings:

Git credentials

If you are not using SSH keys and SSH URIs for your remote Git repositories, you can set the username and a password of your remote using:

Note that you can also define a username and a password in each configuration of your modules, these global variables are only used as fallback if no explicit username and password are defined.