Getting Started With Farrago

This document provides instructions on how to build and run Farrago and its dependencies (including Fennel).

Supported Platforms

So far, the Fennel C++ library only builds and runs on Linux and Windows. This means that if you want to get a fully functional Farrago build, you have to be running one of these operating systems. However, the Java portion is likely to run on other platforms where JDK 1.5 is available. You can use it to test parsing, validation, optimization, and the metadata aspects of DDL, as well as queries against foreign data accessible via Java (a sample HSQLDB source is provided for testing JDBC, and the MDR HOWTO explains MDR/JMI usage).

So, if you're working on a task that doesn't require Fennel, you can do the development anywhere. If you need Fennel, then you have a few options:

Important Windows Note

If you are using Windows, please read and follow the Cygwin setup instructions before going any further.

Get the Source Code

There are two ways to get the Farrago source code; either download a full Eigenbase source release or pull the latest code from the Eigenbase Perforce server. (If you just want to take a look-see through the codebase, you can use p4web.)

If you have downloaded a full Eigenbase source release, you can skip the rest of this section about Perforce and continue on with Initial Build; note that in this case we'll refer to the top-level directory created when you unpacked the release archive as dev in this and other developer docs.

If you're new to Perforce, you might want to read the official Perforce Quick Start. If you don't already have a p4 client program, download the appropriate binary from Perforce (you only need the client, not the server).

There are a number of ways to configure Perforce depending on the platform (on Windows through the GUI, on Unix through environment variables or a .p4env file). The essential information you need to provide is:

In order to make changes, you'll need to send your username to the Farrago project admin and request write access. If you don't have a Sourceforge account yet, go get one. For read-only clients (like if you just want a snapshot of the latest code), use the predefined username "guest" (the server license has a user limit, but there's no limit on the number of clients, so feel free to create multiple workspaces on as many machines as needed).

Once you've got Perforce set up, run p4 client (or edit your default clientspec from P4WIN). Map your view to match this example:


View:
        //open/dev/thirdparty/... //YOURCLIENT/dev/thirdparty/...
        //open/dev/fennel/... //YOURCLIENT/dev/fennel/...
        //open/dev/farrago/... //YOURCLIENT/dev/farrago/...

We'll refer to the YOURCLIENT/dev local filesystem directory as dev in this and other developer docs.

While editing your client, you should also set the LineEnd attribute to share, especially if you are developing on Windows or using Samba. This prevents DOS CRLF's from getting checked in.

Finally, run p4 sync to get all the source files and thirdparty distributions into your client workspace. This may take a while because some of the thirdparty distributions are large.

Initial Build

Before proceeding, make sure you have JDK 1.5 or higher installed, with JAVA_HOME set to the JDK installation root.

For the first build in a new sandbox, there's a lot of one-time setup that needs to be done, including building some of the third-party components from source. This process is automated by the script dev/farrago/initBuild.sh. Change to the dev/farrago directory and run the initial build as follows:


initBuild.sh --with-fennel

  or

initBuild.sh --without-fennel

If you're building without Fennel, this should be fairly quick (under ten minutes on most machines); with Fennel it will take quite a bit longer. If all goes well, you'll be left with a complete Farrago build for your platform. Otherwise, you'll need to dig into the script and its output to see what's going wrong. If you can't figure it out, ask for help on the farrago-developers mailing list. You should probably include the last 100 lines of build output including the error.

Fennel and Autotools

If you are building with Fennel, the initBuild script will check to see if you have a autotools versions (autoconf, automake, and libtool) which are known to work; if you don't, the script will fail before starting the C++ build. If this occurs, you can either follow the directions in the script error message to build and install the autotools from thirdparty, or update your machine using the mechanism provided by your OS distribution (Linux or Cygwin).

If you need help with problems in the C++ portion of the build, use the fennel-developers mailing list.

Environment Variables

The initial build script takes care of setting some necessary environment variables internally, but once it completes they are left unset. For subsequent runs and builds of Farrago, you will need to make sure these are set. Scripts are provided for this purpose. You can run them automatically from a login script such as .bash_profile, or run them manually before working with Farrago:

source dev/fennel/fennelenv.sh dev/fennel
source dev/farrago/farragoenv.sh dev/thirdparty

As usual, you only need the Fennel script if Fennel support is enabled. Don't execute these scripts directly; you must use . or source in order for the environment variable settings to take effect for subsequent commands.

Incremental Builds

Once you already have a working build, updating to the latest code (assuming you're using the Perforce server) requires following these steps:
  1. p4 sync. You'll probably want to examine the output to see what has changed (or you can ignore it and blindly carry out the rest of the steps unconditionally).
  2. If anything under dev/thirdparty/... has changed, cd to that directory and run make all. New thirdparty components and upgrades to new versions are fairly rare. Upgrades to C++ components usually require you to repeat the original initBuild.sh process.
  3. If anything under dev/fennel/... has changed (and you're building with Fennel support), cd to that directory and run make. If it has been a long time since your last sync, you may get errors, in which case you should run make clean and then try again. If you still get errors, see below on forcing a full rebuild.
  4. cd to dev/farrago and run ant test to make sure you'll be working from a good build (WARNING: this will rebuild your catalog automatically, destroying any data in your Farrago database). To avoid running the tests, run ant createCatalog instead.
If none of the C++ components under thirdparty have been upgraded, and you'd like to force a rebuild of everything else, skip the steps after the sync and instead run

cd dev/farrago
initBuild.sh --with-fennel --skip-fennel-thirdparty-build

This will recapitulate the original full build, but skipping the time-consuming thirdparty C++ components.

Catalog Dump

At any time, you can get a dump of the catalog by running ant exportCatalog; the output will be an XMI file in dev/farrago/catalog/xmi/FarragoCatalogExport.xmi. This can be useful for debugging the effects of DDL execution.

Catalog Backup/Restore

The ant createCatalog task (invoked automatically as part of ant test) creates a backup of the initial catalog (and database if Fennel is enabled). If your catalog gets corrupted, you can recover with ant restoreCatalog. This is a lot faster than ant createCatalog (which is another way to get back to a good state).

Interactive SQL

You can execute interactive SQL from a command-line via the open-source JDBC sqlline project. Just run the sqllineEngine script from dev/farrago. After connecting, try select * from sales.emps; (but if you disabled Fennel, don't be disappointed when the result set comes back empty).

If you prefer using a GUI, try the dbmanEngine script instead.

Tracing

See the wiki docs for instructions on tracing Farrago.

Unit Testing

See the wiki docs for information on Farrago unit tests.

Debugging

The developer FAQ contains some useful information on debugging, including how to set up Eclipse.

A standalone debugger (JSwat) is also provided in the thirdparty distribution. Run ant jswat.sqlline or ant jswat.isql to use this debugger for execution of interactive SQL commands. For sqlline, input/output is performed through the JSwat Output pane at the bottom left of the screen. The junitDebug script can also be used to debug one unit test at a time. If you want to use another debugger, the jswatDebug ant macro definition in dev/farrago/buildMacros.xml might be helpful as a template for passing the right startup information. Also, as part of a successful build, the full classpath is always dumped to the file dev/farrago/classpath.gen.

Client/Server Operation

So far, all of the execution modes discussed have been embedded engine mode, where the Farrago engine is loaded directly into a single process such as sqlline. This is the most useful for debugging, but scripts are also available to test client/server mode. Run the farragoServer script to start the server in a shell (after startup, it waits listening for connections; enter the !quit command in that shell to shut it down). From another shell, run sqllineClient or dbmanClient to connect and execute commands.

Coding

Please read the coding conventions documentation. Before starting development projects, consider using private branching.

Feedback

If you have trouble with any of this, or have other questions, use the farrago-developers mailing list.