Skip to content
Snippets Groups Projects
Commit ba1d1314 authored by Tim Fechner's avatar Tim Fechner
Browse files

[TASK] Updated install instructions.

Most information of the README file moved to the sphinx documentation.
parent fb94fbaf
Branches
No related tags found
No related merge requests found
###############################################################################
# About hshetl
###############################################################################
===[ About ]====================================================================
hshetl is a tool and library for extract transform load data between different
data sources.
hshetl is a tool and library for extract transform load data between
different data sources.
Currently supported sources are
- files
......@@ -13,107 +11,19 @@ Currently supported sources are
- sqlite3
- postgresql
###############################################################################
# Install on Debian
###############################################################################
For a complete description of hshetl, visit:
http://etl.it.hs-hannover.de/
Install the tool by using the setup.py script. It resolves all dependencies.
Currently the setup was only tested on debian based linux distributions.
===[ Installation ]=============================================================
You will need to following packages for the building process:
- python-dev
- python-setuptools
- libmysqlclient-dev
- libldap2-dev
- libsasl2-dev
- libssl-dev
- postgresql-server-dev-9.1
- mercurial
'sudo python setup.py install'
sudo apt-get install python-dev python-setuptools libmysqlclient-dev \
libldap2-dev libsasl2-dev libssl-dev mercurial
For the complete installation guide, visit:
http://etl.it.hs-hannover.de/install.html
We need need a higher version of distribute... Execute on Shell:
pip install -U distribute
===[ Usage ]====================================================================
The environment variable ORACLE_HOME and LD_LIBRARY_PATH must point to your instantclient.
export ORACLE_HOME=/path/to/instant_client_11_2
export LD_LIBRARY_PATH=/path/to/instant_client_11_2
'hshetl -h'
###############################################################################
# Virtualenv
###############################################################################
For testing purpose it is recommended to use python virtualenv. It is available
in the package:
- python-virtualenv
# create a virtualenv
virtualenv --no-site-packages path/to/your/virtual/env
# activate a virtualenv
source path/to/your/virtual/env/bin/activate
# deactivate a virtualenv
deactivate
###############################################################################
# Tests
###############################################################################
We used nosetests to run the included unittests. Install it and run nosetests
from inside the package directory ./hshetl
If you want to run the tests from within your IDE, configure it to use nose
as the test runner. We have tested this with Eclipse and PyDEV IDE.
###############################################################################
# Documentation
###############################################################################
The documentation is rendered using sphinx.
sudo apt-get install python-sphinx
We also rely on graphviz for inheritance diagrams.
sudo apt-get install graphviz
And on sphinx-argpase
sudo pip install sphinx-argparse
Render the documentation:
cd docs
make html
if you have changes that will reflect the diagramms use:
sphinx-build -b html -E -D graphviz_dot=/usr/bin/dot source build/html
###############################################################################
# Usage
###############################################################################
usage: hshetl [-h] [-d WORKING_DIRECTORY] [-v] [--init] [config_file]
positional arguments:
config_file The path to the configuration file. An absolute path
is taken as it is. Relative paths are relative to the
config directory A relative path starting with . is
relative to your working directory (default:
config.yml)
optional arguments:
-h, --help show this help message and exit
-d WORKING_DIRECTORY, --directory WORKING_DIRECTORY
The working directory where config file and all
related files are expected. By default this is the
folder you were in when executing hshetl. (default:
/home/ahrensde/Local/git/hshetl)
-v, --verbose Increase verbosity (default: False)
--init Make your current directory a new working directory
with sample content. (default: False)
The tool completely depends on a YAML formatted configuration file. Above
you can see how to specify the place of the config file.
If you install the tool on your computer a sample configuration file is
placed inside /home/username/.hshetl/config.yml.sample
\ No newline at end of file
For the complete introduction for hshetl, visit:
http://etl.it.hs-hannover.de/commandlineintro.html
development
===========
This development guide is for debian based linux distributions.
For development purpose some python packages are **required**:
- mock
- fakeldap
- pyfakefs
Also some debian packages are used:
- python-nose
- python-virtualenv
.. code-block:: bash
# Get debian packages
sudo apt-get install python-nose python-virtualenv
# Get python packages
sudo pip install mock fakeldap pyfakefs
use virtualenv
--------------
For testing purpose it is recommended to use python virtualenv. It is available in the package python-virtualenv
Create a virtualenv:
.. code-block:: bash
# Create a virtualenv
virtualenv --no-site-packages path/to/your/virtual/env
# Activate a virtualenv:
source path/to/your/virtual/env/bin/activate
# Deactivate a virtualenv:
deactivate
run tests
---------
We used nosetests to run the included unittests.
Run nosetests from inside the package directory ./hshetl
If you want to run the tests from within your IDE, configure it to use nose as the test runner.
We have tested this with Eclipse and PyDEV IDE.
build the documentation
-----------------------
You need Sphinx to generate a Sphinx docu:
.. code-block:: bash
# Get debian package
sudo apt-get install python-sphinx
# get python package
sudo pip install sphinx-argparse
To generate html documentation files, go to ./hshetl/docs and type
.. code-block:: bash
# Generate swings documentation into './hshetl/docs/build/html/' directory
sudo make html
\ No newline at end of file
......@@ -11,6 +11,7 @@ hshetl documentation
.. toctree::
:maxdepth: 3
install
commandlineintro
shellintro
connectors
......@@ -19,4 +20,5 @@ hshetl documentation
extractors
transformers
loaders
develop
diagramms
installation
============
hshetl must be installed to work properly.
.. note::
Currently the setup was only tested on debian based linux distributions.
install on Debian
-----------------
Install hshetl by using the setup.py script as shown below. It resolves all dependencies.
You will need to following packages for the building process:
- python-setuptools
- libssl-dev
- postgresql-server-dev-9.1
- python-dev
- libmysqlclient-dev
- libldap2-dev
- libsasl2-dev
**You can simply run these commands to install the needed packages and hshetl:**
.. code-block:: bash
# Get debian packages
sudo apt-get install python-setuptools libssl-dev postgresql-server-dev-9.1 python-dev libmysqlclient-dev libldap2-dev libsasl2-dev
# Install hshetl
sudo python2 setup.py install
.. note::
If you want to use oracle database stuff, you must install the python package 'cx_Oracle'!
See http://cx-oracle.sourceforge.net/
.. note::
The environment variable ORACLE_HOME and LD_LIBRARY_PATH must point to your instantclient.
.. code-block:: bash
export ORACLE_HOME=/path/to/instant_client_11_2
export LD_LIBRARY_PATH=/path/to/instant_client_11_2
install on Windows
------------------
.. warning::
Not tested yet ...
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment