README

###############################################################################
# About hshetl
###############################################################################

hshetl is a tool and library for extract transform load data between different
data sources.

Currently supported sources are
 - files
 - ldap
 - oracle
 - mysql
 - sqlite3
 - postgresql

###############################################################################
# Install on Debian
###############################################################################

Install the tool by using the setup.py script. It resolves all dependencies.
Currently the setup was only tested on debian based linux distributions.

You will need to following packages for the building process:
 - python-dev
 - python-setuptools
 - libmysqlclient-dev
 - libldap2-dev
 - libsasl2-dev
 - libssl-dev
 - postgresql-server-dev-9.1
 - mercurial

sudo apt-get install python-dev python-setuptools libmysqlclient-dev \
libldap2-dev libsasl2-dev libssl-dev mercurial

We need need a higher version of distribute... Execute on Shell:
pip install -U distribute

The environment variable ORACLE_HOME and LD_LIBRARY_PATH must point to your instantclient.
export ORACLE_HOME=/path/to/instant_client_11_2
export LD_LIBRARY_PATH=/path/to/instant_client_11_2

###############################################################################
# Virtualenv
###############################################################################

For testing purpose it is recommended to use python virtualenv. It is available
in the package:
 - python-virtualenv

# create a virtualenv
virtualenv --no-site-packages path/to/your/virtual/env
# activate a virtualenv
source path/to/your/virtual/env/bin/activate
# deactivate a virtualenv
deactivate

###############################################################################
# Tests
###############################################################################
We used nosetests to run the included unittests. Install it and run nosetests
from inside the package directory ./hshetl

If you want to run the tests from within your IDE, configure it to use nose
as the test runner. We have tested this with Eclipse and PyDEV IDE.

###############################################################################
# Documentation
###############################################################################

The documentation is rendered using sphinx.

sudo apt-get install python-sphinx

We also rely on graphviz for inheritance diagrams.

sudo apt-get install graphviz

And on sphinx-argpase

sudo pip install sphinx-argparse

Render the documentation:

cd docs
make html

if you have changes that will reflect the diagramms use:
sphinx-build -b html -E -D graphviz_dot=/usr/bin/dot source build/html

###############################################################################
# Usage
###############################################################################

usage: hshetl [-h] [-d WORKING_DIRECTORY] [-v] [--init] [config_file]

positional arguments:
  config_file           The path to the configuration file. An absolute path
                        is taken as it is. Relative paths are relative to the
                        config directory A relative path starting with . is
                        relative to your working directory (default:
                        config.yml)

optional arguments:
  -h, --help            show this help message and exit
  -d WORKING_DIRECTORY, --directory WORKING_DIRECTORY
                        The working directory where config file and all
                        related files are expected. By default this is the
                        folder you were in when executing hshetl. (default:
                        /home/ahrensde/Local/git/hshetl)
  -v, --verbose         Increase verbosity (default: False)
  --init                Make your current directory a new working directory
                        with sample content. (default: False)

The tool completely depends on a YAML formatted configuration file. Above
you can see how to specify the place of the config file.

If you install the tool on your computer a sample configuration file is
placed inside /home/username/.hshetl/config.yml.sample