Thermostat/UserGuideV1.0

From IcedTea

Jump to: navigation, search

Thermostat Home

Contents

1 Thermostat User Guide (Version 1.0.x)

1.1 Conventions

This guide uses several conventions.

Text that looks like this indicates something code-related

Indented blocks that begin with a $ as in: 

 $ some command with arguments

Indicate a terminal. The $ indicates a user prompt. A # indicates a root prompt. Only text that appears after the prompt should be entered in a terminal. In the example above only some command with arguments should be entered. The $ is the terminal prompt itself. Lines that do not begin with a prompt (either $ or # or something thermostat specific) indicate output.

1.2 Thermostat Basics

1.2.1 Overview of Components

It is important to know that thermostat is a database-backed application consisting of three (3) main components: agent, storage and client (in whatever form). Each component has a distinct role:

agent 
Collects data and sends it to the storage. The data is is collected from both the host machine and the JVM. This service has to run on the hosts where JVMs should get monitored.
storage 
The thermostat data store. In storage all data is recorded so it can be retrieved later for processing or display. At this point thermostat storage clients - be it thermostat agents or thermostat clients - decide of the storage implementation via the connection URL. URLs starting with mongodb:// connect directly to a mongodb NoSQL database. Similarly, http:// URLs require a storage endpoint which is capable of talking HTTP. For production setups this layer will in fact consist of an backing storage component (e.g. mongodb) and a web storage end point in front of it.
client 
Queries storage for data and displays in a sensible form.

When using thermostat, it is important to note that agents and clients will require storage to be up and running. Both components will try to connect to storage via the configured URLs. If the URL endpoint is not reachable, clients and agents will refuse to start. At this point it is worth noting that a stock development build of thermostat uses the empty string as authentication credentials (username and password) with the mongodb URL end point. This works, because thermostat's mongodb storage backend makes use of the localhost exception[1].

Often, thermostat need to identify a specific machine and/or a JVM. The host id (or agent id) uninquely identifies an agent. With a one-agent for each host policy, the host id (or agent id) uniquely identifies a host. The vm id uniquely identifies a VM.

1.2.2 Supported Use Cases

Thermostat's two main use cases are:

  • Local Thermostat Setup. This is useful if you are doing development or testing and want to run everything on your local machine. This mode is not suitable for running in an untrusted environment, but it is easier to set up.
  • Remote Thermostat Setup Across a Few Machines. This is useful if you want to run thermostat so it monitors data in a remote machine or for multiple machines. This requires a more elaborate setup but offers enough security in order to run thermostat in an untrusted network. It is highly recommended to use this kind of setup for production environments.

1.2.3 Installation Location vs. User Data

Thermostat distinguishes between where it is installed and where it places runtime data. The former is generally referred to as THERMOSTAT_HOME, the latter as USER_THERMOSTAT_HOME. The main difference between THERMOSTAT_HOME and USER_THERMOSTAT_HOME is that the former is assumed to be read-only (or only writable by root), the latter is in a user-writable location. For a thermostat image produced from its sources this means that THERMOSTAT_HOME is the directory where you've put the thermostat sources with /distribution/target/image appended. USER_THERMOSTAT_HOME in general is ~/.thermostat.


The following sections assume that the thermostat boot script is in your PATH. For a thermostat image, which was built from source this script is located in THERMOSTAT_HOME/bin.

1.2.4 Sanity Test

To ensure that Thermostat is built/installed in a sane way, make sure that the thermostat boot script is in your PATH: 

$ which thermostat

If this comes back with an error, make sure to add THERMOSTAT_HOME/bin to your PATH variable (assuming bash). Alternatively, use the absolute path to the script. Now that we've verified that the script is available run: 

$ thermostat --version

If thermostat is set up correctly, it will print a version: 

Thermostat version 1.0.0

An error indicates a problem with the installation. See how to debug problems for suggestions on how to fix this.

1.3 Development: Local Thermostat Setup

1.3.1 Storage

The mongodb URL storage endpoint can be started as follows. In the default configuration, this will start a mongodb server binding to 127.0.0.1 port 27518. Defaults can be changed via THERMOSTAT_HOME/etc/db.properties or USER_THERMOSTAT_HOME/etc/db.properties: 

$ thermostat storage --start

The endpoint server can be stopped via: 

$ thermostat storage --stop

If you don't know if you have a thermostat storage process already running or not you can use the following command in order to determine if it's up or down: 

$ thermostat storage --status

The storage layer is really just a wrapper over a database server (mongodb in this case).It will ensure a server is started with the appropriate configuration so other components can use it. You can grep for a mongod process after you've run thermostat storage --start in order to see details about the mongod configuration.

1.3.2 Agent

After the storage layer is up and running, the agent, or data collector, can be started as follows: 

$ thermostat agent

The agent can be stopped by pressing CTRL+C on the terminal where the agent was started. Alternatively a SIGTERM can be sent to the agent process in order to stop it. When the agent is stopped, no more data will be collected. By default, agents use a mongodb connection URL of mongodb://127.0.0.1:27518. This can be changed by either using the -d option or by changing configuration in THERMOSTAT_HOME/etc/agent.properties or USER_THERMOSTAT_HOME/etc/agent.properties.

The thermostat agent starts a networked server. This is used when a user wants to perform certain actions against a JVM (for example, trigger a full Garbage Collection on a target JVM). By default, thermostat agents bind to port 12001.

1.3.3 Swing Client

The built-in Swing GUI client can be started via: 

$ thermostat gui

See the swing client section for more information about using it.

There is also a built-in thermostat shell which can be started as follows: 

$ thermostat shell

See the the shell client section for more information about using the shell.

1.3.4 Service

Thermostat provides an additional command, service, which is a short-cut for starting the storage and agent directly. Although it is considered an internal option, it was decided to leave it available for now since, even if it doesn't offer the flexibility and the number of options of the other commands, it is still very useful to quickly run Thermostat: 

$ thermostat service

NOTE: This command is a convenience and may go away in a future release.

1.4 Production: Thermostat Setup

NOTE: Before proceeding with a production setup of thermostat, we strongly advise you to have a look at the our Thermostat security considerations page.

In contrast to a simple development setup involves a reasonably secure thermostat production setup the following components:

Storage Backend

Currently, the only supported storage backend is mongodb. It can be mongod processes or sharded mongos and mongod processes. The mongodb:// URL should be only exposed to the next component, the web storage cluster.

Web Storage Cluster

This should be the only publicly exposed storage layer. This forces thermostat agents and built-in clients to connect to http:// style URLs for increase security. A web storage cluster can be a load-balanced setup of servlet containers each running the thermostat web application end point.

Agents

Regular thermostat agent processes as described previously.

Clients

Regular thermostat client processes as described previously.

In the following sections we assume a non-clustered setup for simplicity. In this setup, the storage back-end runs on host storage-backend.example.com with IP 192.168.1.1, the web service runs on host webstorage.example.com with IP 192.168.0.2. Thermostat agents and clients can only reach host webstorage.example.com. storage-backend.example.com is separated from agents and client via means of a firewall, for example. In the below example configuration one thermostat agent will run on host agent1.example.com with IP 192.168.0.100.

1.4.1 Backing Storage Setup

This section describes configuration of the backing storage node, namely storage-backend.example.com above. In order to be able to use the mongodb storage backend in a distributed environment, it needs to be configured to bind to a proper IP address of the host on which storage is running on. Moreover authentication needs to be properly set up for it to work. Here is how this can be done.

First we start mongodb storage: 

$ thermostat storage --start

Next, we add an administrative user, adminUser in mongodb. 

$ mongo 127.0.0.1:27518
MongoDB shell version: 2.0.6
connecting to: 127.0.0.1:27518/test
> use admin
switched to db admin
> db.addUser("adminUser", "adminPassword")
{ "n" : 0, "connectionId" : 3, "err" : null, "ok" : 1 }
{
  user" : "adminUser",
  "readOnly" : false,
  "pwd" : "b9453af5a2b4eb85e098a70c2fbf38b4",
  "_id" : ObjectId("504f019876aefbdbe7b34983")
}

Then we authenticate as this administrative user and add a regular user to the thermostat DB. We'll later use this user for the thermostat DB for connections from the web storage endpoint node. 

> db.auth("adminUser", "adminPassword")
1 
> use thermostat
switched to db thermostat
> db.addUser("thermostat", "thermostat")
> exit
bye

Finally we stop the mongodb storage server: 

$ thermostat storage --stop

Next, we configure storage settings to bind to our host's IP address. In order to do so change the BIND parameter in THERMOSTAT_HOME/etc/db.properties or USER_THERMOSTAT_HOME/etc/db.properties. The bind port which the thermostat storage command will use can be configured in a similar fashion. In our example, we change the BIND parameter to: BIND=192.168.1.1 and leave the rest as is.

Finally, on host storage-backend.example.com, we execute: 

$ thermostat storage --start

At this point, we have the backing storage node configured.

1.4.2 Web Storage Cluster

Now with backing storage up and running on host storage-backend.example.com, we are ready to configure the Web layer for increased security. In a nutshell, this is a regular web application to be deployed in a servlet container of your choosing. At this point, we know the thermostat webapp works with Tomcat 6, Tomcat 7, Jetty 8, Jetty 9, JBoss AS 7 and Wildfly 8.

NOTE: The thermostat webapp comes with a built-in JAAS login module. This will likely require some configuration for your servlet container in order to know which login module class to use for the thermostat web storage application. It usually requires adding some Java option to the servlet container's boot script such as JAVA_OPTS="-Djava.security.auth.login.config=$THERMOSTAT_HOME/etc/thermostat_jaas.conf".

Since deploying a web application archive is fairly servlet container specific we fix the servlet container we'll use for host webstorage.example.com to Apache Tomcat 7. The binary distribution can be downloaded from their website. Also, we only describe a non-clustered setup here for simplicity. In a real production environment you'd want to have this clustered in some way or another. However, that's beyond the scope of this guide. We call the directory where you've extracted the binary distribution of Apache Tomcat CATALINA_HOME. The structure looks similar to this: 

$ ls -1 $CATALINA_HOME
bin
conf
lib
LICENSE
logs
NOTICE
RELEASE-NOTES
RUNNING.txt
temp
webapps
work

Next, we'll need to get the thermostat web application archive. For a buit-from-source thermostat distribution the web archive is in THERMOSTAT_HOME/web/war/target/thermostat-web-war-<VERSION>.war. This archive can be copied to CATALINA_HOME/webapps/thermostat.war. Finally, we can start tomcat as follows: 

$ cd $CATALINA_HOME
$ export JAVA_OPTS="-Djava.security.auth.login.config=$(echo $THERMOSTAT_HOME)/etc/thermostat_jaas.conf"
$ ./bin/startup.sh

This start a tomcat instance on localhost, port 8080. Tomcat configuration will have to get adjusted in order to bind to webstorage.example.com instead. Please refer to the Apache Tomcat configuration guide in order to do this. It is also advised that only a SSL enabled connector gets exposed to thermostat agents and clients. Please see the Thermostat security considerations page for pointers as to how to do this. If all went OK, then the thermostat web storage endpoint is accessible via the following URL: 

https://webstorage.example.com:8443/thermostat/storage

Note that accessing the thermostat context will require authentication. Users are configured via THERMOSTAT_HOME/etc/thermostat-users.properties file by default. Assigning roles to users can be done via file THERMOSTAT_HOME/etc/thermostat-roles.properties. Which files thermostat will use for defined users and roles can also be changed by passing options to the default ThermostatJAASDelegate login module, namely PropertiesUsernameRolesLoginModule. Example: 

ThermostatJAASDelegate {
  com.redhat.thermostat.web.server.auth.spi.PropertiesUsernameRolesLoginModule required debug="true"
  users.properties="/absolute/path/to/thermostat-users.properties"
  roles.properties="/absolute/path/to/thermostat-roles.properties" ;
};

The above example also turns on debugging, for more verbose logging. That is useful in order to debug login problems. A minimal user setup in order for you to test the thermostat context looks as follows: 

# Example file for thermostat-users.properties
# Defines a user called "context-tester" with password "tester"
context-tester=tester

# Example file for thermostat-roles.properties
# Assigns user "context-tester" role "thermostat-realm" which is used
# as a minimal requisite in order to access the realm.
context-tester=thermostat-realm

For more information about roles configuration see our Thermostat security considerations page.

1.4.3 Agent

Since thermostat agents are end points for command-channel requests, we need for them to bind to a remotely accessible IP address. In our example config this would be 192.168.0.100. In order to do so, modify THERMOSTAT_HOME/etc/agent.properties or USER_THERMOSTAT_HOME/etc/agent.properties and set CONFIG_LISTEN_ADDRESS to 192.168.0.100:12001. Furthermore, we want the agent to connect to our web cluster, webstorage.example.com. This can be achieved, but setting the DB_URL configuration property to https://webstorage.example.com:8443/thermostat/storage. Finally, we need to configure credentials with which the agent user should connect to webstorage with. We create a file called agent.auth, namely USER_THERMOSTAT_HOME/etc/agent.auth with the following contents: 

username=agent-tester
password=tester

This will make the thermostat agent use username "agent-tester" with password "tester". Note that only "thermostat agent" will use this "agent.auth" file in USER_THERMOSTAT_HOME. It has be designed that way in order for it to be not interactive. If this file does not exist, it will try to connect to the storage end point using the empty string for username and password. In order for this "agent-tester" agent to be able to connect to the storage endpoint one would have to have this user configured on the webstorage.example.com node. Moreover, appropriate thermostat agent roles would have to be assigned to this agent user.

After these configuration steps the agent can be started on host agent1.example.com as usual with: 

$ thermostat agent

1.4.4 Clients

Similar to agent users, ALL client users would have to be set up on the webstorage.example.com node. Otherwise, they'll fail to authenticate and/or authorize.

The built-in swing client can be launched as usual with: 

$ thermostat gui

This will likely report that it cannot connect to the data collector (see screenshot) as it will attempt to connect to a mongodb:// URL on the local host on a fresh installation by default. When this happens, a configuration dialog for changing the client preferences can be brought up. Click on "Configure" to do so:

Image:thermostat_storage_config.png

In the configuration dialog enter storage URL https://webstorage.example.com:8443/thermostat/storage and an appropriate username/password combination which has been configured of the webstorage node (make sure to check "Save entitlements"):

Image:thermostat_storage_config_dialog.png

Press "OK" and you should see the familiar thermostat swing UI with information collected on host agent1.example.com.

Please see the swing client section for more information on how to use the GUI.

As an alternative to the swing client the the interactive thermostat shell or regular thermostat commands can connect to the webstorage node. Here are a few examples: 

$ thermostat shell
Thermostat version 1.0.0
Thermostat > list-vms
Could not connect to: mongodb://127.0.0.1:27518
Thermostat > connect -d https://webstorage.example.com:8443/thermostat/storage
Thermostat > list-vms
HOST_ID                              HOST                VM_ID STATUS  VM_NAME
44b67f40-e89e-4fbe-bf72-845a4074f81e agent1.example.com  1716  RUNNING com.redhat.thermostat.main.Thermostat

Please see the CLI/shell client section for more information on how to use the shell and CLI.

1.5 Clients

1.5.1 Swing Client

The Swing Client is the most advanced and complex client, as well as rapidly evolving, for this reason a full document has been dedicated to it, detailing various aspect of the user interface and the functions of each plugin. Please refer to the Swing Client v 1.0 User Guide for details.

1.5.2 CLI/Shell Client

If you prefer to use the command line (or are on a headless server) the command line/shell client will come in very handy.

To start the shell client run: 

$ thermostat shell

This will start the shell, print the version and then display a prompt, waiting for you to enter a command. 

$ thermostat shell
Thermostat version 1.0.0
Thermostat >

Once in the thermostat shell, you can type any command here and the shell will execute it and display the result. For getting started, the help command is most useful. it displays a list of all commands that are currently available and what they do: 

Thermostat > help
list of commands:

 help                   show help for a given command or help overview
 clean-data             Drop all data related to all of the specified agents
 connect                persistently connect to storage
 disconnect             disconnect from the currently used storage
 dump-heap              trigger a heap dump on the VM
 find-objects           finds objects in a heapdump
 find-root              finds the shortest path from an object to a GC root
 list-heap-dumps        list all heap dumps
 list-vms               lists all currently monitored VMs
 object-info            prints information about an object in a heap dump
 ping                   using the Command Channel, send a ping to a running agent
 save-heap-dump-to-file saves a heap dump to a local file
 show-heap-histogram    show the heap histogram
 validate               validates a thermostat plug-in XML file against the schema
 vm-info                shows basic information about a VM
 vm-stat                show various statistics about a VM
Thermostat >

The help command can also display detailed information about a specific command: 

Thermostat> help list-vms
usage: thermostat list-vms [-d <url>] [-l <level>]
                 lists all currently monitored VMs

thermostat list-vms
  -d,--dbUrl <dbUrl>          connect to the given URL
  -l,--logLevel <logLevel>    sets the log level for this invocation. Possible
                              values for <logLevel> in decreasing severity are:
                              SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST
                              and OFF

This shows that the list-vms command can display all the VMs that the storage knows about.

To invoke a command, just enter it on the prompt: 

Thermostat > list-vms
HOST_ID                              HOST   VM_ID                                VM_PID STATUS  VM_NAME
013b9df4-b831-4e71-8449-593e83c71626 trotts cc89fc51-ecc8-4642-ae5a-836e11792e78 23551  RUNNING com.redhat.thermostat.main.Thermostat
013b9df4-b831-4e71-8449-593e83c71626 trotts ac076cfc-9378-4246-8390-94342ac6691f 23673  EXITED  com.redhat.thermostat.main.Thermostat
013b9df4-b831-4e71-8449-593e83c71626 trotts 00f70def-46ae-42a3-82f5-217e5629de29 23851  RUNNING com.redhat.thermostat.main.Thermostat

Some commands will ask for a password: 

Please enter username for storage at mongodb://127.0.0.1:27518:
Please enter password for storage at mongodb://127.0.0.1:27518:

If you didn't set one up, the default username and password are "" (empty, blank). Just hit Enter.

1.5.2.1 Commands

Here is a list of some common commands by topic.

  • connection commands
    • connect
    • disconnect
  • VM commands
    • list-vms
    • vm-info
    • vm-stat
  • VM Heap Commands
    • dump-heap
    • list-heap-dumps
    • find-objects
    • find-roots
    • object-info
    • save-heap-dump-to-file
    • show-heap-histogram

Use help command to find out more about any command.

1.6 How To Debug Problems

Like all complex software, problems can occur when you are trying to use Thermostat. This section contains a few tips on identifying and resolving problems. If you think a problem is a bug in Thermostat, please file a bug in our bug tracker.

A general strategy is to simplify things and try alternates until you can 'see' the problem. If you are having problems with the GUI, for example, check if the command line shell is working. If service is not working, check and see if storage by itself is.

1.6.1 Verbose Logging

You can configure the amount, verbosity and other settings of Thermostat logging using a configuration file located at $THERMOSTAT_HOME/logging.properties or $USER_THERMOSTAT_HOME/logging.properties (the latter overrides the former if both exist). If you are using the basic logging.properties template as shipped with Thermostat and use (uncomment) the FileHandler configured there logs will go to $USER_THERMOSTAT_HOME/logs/thermostat.log.

A default logging.properties is located at $THERMOSTAT_HOME/logging.properties, which you can copy over to $USER_THERMOSTAT_HOME/etc/logging.properties.

Probably the most intersting set of lines looks something like this: 

###################################################################
# Thermostat log level config. Default log level is INFO. 
###################################################################
# Possible levels are in descending order:
#com.redhat.thermostat.level=SEVERE
com.redhat.thermostat.level=WARNING
#com.redhat.thermostat.level=INFO
#com.redhat.thermostat.level=CONFIG
#com.redhat.thermostat.level=FINE
#com.redhat.thermostat.level=FINER
#com.redhat.thermostat.level=FINEST
#com.redhat.thermostat.level=OFF

It may be slightly different in your build. All the lines beginning with # are comments. This is a list in least-verbose to most-verbose order (with the exception of OFF!). In this example, only items classified as WARNING are printed. If you want to see more information, comment out that line and uncomment a line such as one with FINE or FINER. Watch out: a level such as FINEST may produce too much output.

1.6.2 Communication between agent and client (Command Channel)

The ping command can be used to check if direct commnication between the agent and the client is working. This requires that agent, client and storage are all running.

First, find the host-id (aka agent-id) of a running agent: 

$ thermostat shell
Thermostat > list-vms

Use vm-info to check that it's really an agent (the 'Command Line' will include 'agent' or 'service').

Then ping the host using the host-id: 

Thermostat > ping host-id-goes-here

If this works, you should see something that looks like: 

Response received from: /127.0.0.1:12000

1.6.3 Other Issues?

Be sure to check our general Frequently Asked Questions (FAQ) page.

1.7 Configuration Reference

This section describes some ofthe common confiugration files used by thermostat.

Thermostat uses two places to search for (most) configuration files: $THERMOSTAT_HOME (where you installed/built thermostat. Possibly /usr/share/thermostat.) and $USER_THERMOSTAT_HOME (~/.thermostat). $THERMOSTAT_HOME is used for system-wide configuration files. These are not expecited to be modified. $USER_THERMOSTAT_HOME stores per-user settings. Sometimes a file is present in both locations and the user-specific file override the system-wide file.

  • $THERMOSTAT_HOME/etc
    • agent.properties specifies settings for use by the agent
    • agent_proxy_jaas.conf contains thermostat internal data. The contents of this file MUST not be modified.
    • commands contains thermostat internal files. The contents of this directory MUST not be modified.
    • db.properties contains settings to use for the storage. This includes what ip address and port the storage should listen at. It also allows enabling encrypting and authentication.
    • logging.properties configures how much thermostat should log and whether information logged should go to standard out or to some log file.
    • osgi-export.properties contains thermostat internal data. The contents of this file MUST not be modified.
    • thermostat_jaas.conf contains JAAS configuration which the thermostat web storage webapp requires. ThermostatJAASLogin must not be changed. ThermostatJAASDelegate may point to a JAAS login module class to use for authenticating users and for fetching roles associated to users.
    • thermostat-roles.properties specifies which users have which roles when running with authentication and security.
    • thermostat-users.properties specifies users (in the form of user names and passwords)
  • $USER_THERMOSTAT_HOME/etc
    • agent.auth contains information used to authenticate an agent
    • agent.properties specifies settings for use by the agent
    • db.properties contains settings to use for the storage. This includes what ip address and port the storage should listen at. It also allows enabling encrypting and authentication.
    • logging.properties configures how much thermostat should log and whether information logged should go to standard out or to some log file.
    • ssl.properties configures which keystores to use, and whether ssl should be enabled.
    • client.properties stores client configuration, inlcuding url for connection to storage and authentication parameters (but not the password).

1.8 Glossary

agent id 
Uniquely identifies an agent (or a host)
JVM 
The Java Virtual Machine. Thermostat currenly only supports OpenJDK/IcedTea.
THERMOSTAT_HOME
This is the location that corresponds to where thermostat was installed. If you built it from source, it is probably /path/to/thermostat/distribution/target/image.
USER_THERMOSTAT_HOME
This is the location where thermostat writes data to. It may also be used to add user-installed plugins and override system wide thermostat configuration. It is typically set to ~/.thermostat for a default install.
vm id 
uniquely identifies a JVM
Retrieved from "https://icedtea.classpath.org/wiki/Thermostat/UserGuideV1.0"
Personal tools