From ba1d1314bac9b5bf1938708053500ba8be504b32 Mon Sep 17 00:00:00 2001 From: Tim Fechner <tim.fechner@hs-hannover.de> Date: Tue, 11 Mar 2014 16:22:39 +0100 Subject: [PATCH] [TASK] Updated install instructions. Most information of the README file moved to the sphinx documentation. --- README | 116 +++++----------------------------------- docs/source/develop.rst | 74 +++++++++++++++++++++++++ docs/source/index.rst | 2 + docs/source/install.rst | 56 +++++++++++++++++++ 4 files changed, 145 insertions(+), 103 deletions(-) create mode 100644 docs/source/develop.rst create mode 100644 docs/source/install.rst diff --git a/README b/README index c812204..64397bf 100644 --- a/README +++ b/README @@ -1,9 +1,7 @@ -############################################################################### -# About hshetl -############################################################################### +===[ About ]==================================================================== -hshetl is a tool and library for extract transform load data between different -data sources. +hshetl is a tool and library for extract transform load data between +different data sources. Currently supported sources are - files @@ -13,107 +11,19 @@ Currently supported sources are - sqlite3 - postgresql -############################################################################### -# Install on Debian -############################################################################### +For a complete description of hshetl, visit: +http://etl.it.hs-hannover.de/ -Install the tool by using the setup.py script. It resolves all dependencies. -Currently the setup was only tested on debian based linux distributions. +===[ Installation ]============================================================= -You will need to following packages for the building process: - - python-dev - - python-setuptools - - libmysqlclient-dev - - libldap2-dev - - libsasl2-dev - - libssl-dev - - postgresql-server-dev-9.1 - - mercurial +'sudo python setup.py install' -sudo apt-get install python-dev python-setuptools libmysqlclient-dev \ -libldap2-dev libsasl2-dev libssl-dev mercurial +For the complete installation guide, visit: +http://etl.it.hs-hannover.de/install.html -We need need a higher version of distribute... Execute on Shell: -pip install -U distribute +===[ Usage ]==================================================================== -The environment variable ORACLE_HOME and LD_LIBRARY_PATH must point to your instantclient. -export ORACLE_HOME=/path/to/instant_client_11_2 -export LD_LIBRARY_PATH=/path/to/instant_client_11_2 +'hshetl -h' -############################################################################### -# Virtualenv -############################################################################### - -For testing purpose it is recommended to use python virtualenv. It is available -in the package: - - python-virtualenv - -# create a virtualenv -virtualenv --no-site-packages path/to/your/virtual/env -# activate a virtualenv -source path/to/your/virtual/env/bin/activate -# deactivate a virtualenv -deactivate - -############################################################################### -# Tests -############################################################################### -We used nosetests to run the included unittests. Install it and run nosetests -from inside the package directory ./hshetl - -If you want to run the tests from within your IDE, configure it to use nose -as the test runner. We have tested this with Eclipse and PyDEV IDE. - -############################################################################### -# Documentation -############################################################################### - -The documentation is rendered using sphinx. - -sudo apt-get install python-sphinx - -We also rely on graphviz for inheritance diagrams. - -sudo apt-get install graphviz - -And on sphinx-argpase - -sudo pip install sphinx-argparse - -Render the documentation: - -cd docs -make html - -if you have changes that will reflect the diagramms use: -sphinx-build -b html -E -D graphviz_dot=/usr/bin/dot source build/html - -############################################################################### -# Usage -############################################################################### - -usage: hshetl [-h] [-d WORKING_DIRECTORY] [-v] [--init] [config_file] - -positional arguments: - config_file The path to the configuration file. An absolute path - is taken as it is. Relative paths are relative to the - config directory A relative path starting with . is - relative to your working directory (default: - config.yml) - -optional arguments: - -h, --help show this help message and exit - -d WORKING_DIRECTORY, --directory WORKING_DIRECTORY - The working directory where config file and all - related files are expected. By default this is the - folder you were in when executing hshetl. (default: - /home/ahrensde/Local/git/hshetl) - -v, --verbose Increase verbosity (default: False) - --init Make your current directory a new working directory - with sample content. (default: False) - -The tool completely depends on a YAML formatted configuration file. Above -you can see how to specify the place of the config file. - -If you install the tool on your computer a sample configuration file is -placed inside /home/username/.hshetl/config.yml.sample \ No newline at end of file +For the complete introduction for hshetl, visit: +http://etl.it.hs-hannover.de/commandlineintro.html diff --git a/docs/source/develop.rst b/docs/source/develop.rst new file mode 100644 index 0000000..3aab3de --- /dev/null +++ b/docs/source/develop.rst @@ -0,0 +1,74 @@ +development +=========== + +This development guide is for debian based linux distributions. + +For development purpose some python packages are **required**: + +- mock +- fakeldap +- pyfakefs + +Also some debian packages are used: + +- python-nose +- python-virtualenv + + +.. code-block:: bash + + # Get debian packages + sudo apt-get install python-nose python-virtualenv + + # Get python packages + sudo pip install mock fakeldap pyfakefs + + +use virtualenv +-------------- + +For testing purpose it is recommended to use python virtualenv. It is available in the package python-virtualenv + +Create a virtualenv: + +.. code-block:: bash + + # Create a virtualenv + virtualenv --no-site-packages path/to/your/virtual/env + + # Activate a virtualenv: + source path/to/your/virtual/env/bin/activate + + # Deactivate a virtualenv: + deactivate + + +run tests +--------- + +We used nosetests to run the included unittests. +Run nosetests from inside the package directory ./hshetl + +If you want to run the tests from within your IDE, configure it to use nose as the test runner. +We have tested this with Eclipse and PyDEV IDE. + + +build the documentation +----------------------- + +You need Sphinx to generate a Sphinx docu: + +.. code-block:: bash + + # Get debian package + sudo apt-get install python-sphinx + + # get python package + sudo pip install sphinx-argparse + +To generate html documentation files, go to ./hshetl/docs and type + +.. code-block:: bash + + # Generate swings documentation into './hshetl/docs/build/html/' directory + sudo make html \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 28ba266..c2845da 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -11,6 +11,7 @@ hshetl documentation .. toctree:: :maxdepth: 3 + install commandlineintro shellintro connectors @@ -19,4 +20,5 @@ hshetl documentation extractors transformers loaders + develop diagramms diff --git a/docs/source/install.rst b/docs/source/install.rst new file mode 100644 index 0000000..3330168 --- /dev/null +++ b/docs/source/install.rst @@ -0,0 +1,56 @@ +installation +============ + +hshetl must be installed to work properly. + +.. note:: + + Currently the setup was only tested on debian based linux distributions. + + +install on Debian +----------------- + +Install hshetl by using the setup.py script as shown below. It resolves all dependencies. + +You will need to following packages for the building process: + +- python-setuptools +- libssl-dev +- postgresql-server-dev-9.1 +- python-dev +- libmysqlclient-dev +- libldap2-dev +- libsasl2-dev + +**You can simply run these commands to install the needed packages and hshetl:** + +.. code-block:: bash + + # Get debian packages + sudo apt-get install python-setuptools libssl-dev postgresql-server-dev-9.1 python-dev libmysqlclient-dev libldap2-dev libsasl2-dev + + # Install hshetl + sudo python2 setup.py install + +.. note:: + If you want to use oracle database stuff, you must install the python package 'cx_Oracle'! + +See http://cx-oracle.sourceforge.net/ + +.. note:: + + The environment variable ORACLE_HOME and LD_LIBRARY_PATH must point to your instantclient. + +.. code-block:: bash + + export ORACLE_HOME=/path/to/instant_client_11_2 + export LD_LIBRARY_PATH=/path/to/instant_client_11_2 + + +install on Windows +------------------ + +.. warning:: + + Not tested yet ... -- GitLab