<pre>

	1.	Introduction.

DBPUMP was intended to be a new reincarnation of existing DBDUMP utility.
Last version of DBDUMP is complete in terms of plain RDBMS-independent operations, and the only
thing to be improved for generic-purpose operations is the interface. Meanwhile Virtuoso,
providing much more various functionalities than plain RDBMS, has some uncommon data types and
schema features. Currently they are not supported in DBDUMP, but should.
And what is more important DBPUMP is aimed to be not only the specialized 'DBDUMP-for-Virtuoso' 
but rather a backup/restore utility with support for all Virtuoso features.


	2.	VSP - pages hierarchy

DBDUMP is a single command-line program with ability to be run via CGI and in this case 
to generate HTML content 'on-the-fly'. In DBPUMP we have the situation with separation of interface
content and sensible work. DBDUMP consists of a number of independent components each of which can be 
called by name and VSP pages can be considered just as a visually convenient method to call this 
components. Each page consists of three parts - header, body and footer. In header we can see the title 
of current page and the set of buttons - jumps to other pages. Buttons can have different colors what 
means their status: blue-current page, red-forbidden from current state, green- available. Page body 
contains the current contest and footer currently is empty with exception for possibility to turn on 
a debug printing.

The root page for all DBPUMP VSP-tree is 'Intro' (dbdump.vsp). Its just contains a general text and 
zero of useful information. In this page all links are permitted. This page uses the only DBPUMP 
component - 'retrieve_oper_pars' to deserialize the current context. Here there should be mentioned 
what all DBPUMP vsp-pages use this component explicitly or implicitly and we will not refer to this 
component later.

The next page is 'Datasource'(select_datasource.vsp). Here you can choice parameters for ODBC connection - 
default qualifier, username, password and datasource name. The datasource choice list is provided and 
refers to 'select_datasource' DBPUMP component but in current state this works only at WIN32 with 
native ODBC driver manager. This information will be used in two ways: first of all for any future 
ODBC connection and secondly as arguments for internal isql call. In this case to achieve compatibility 
when defining names for new datasources you should use special (isql-compatible) datasource names like 
'localhost:1111'.

'Options' (select_options.vsp) page represents all options and execution parameters which can be found 
in DBPUMP vsp pages. Initially here there was all the bulk of DBDUMP options but due to above mentioned 
DBDUMP specialization they was mostly obsolete.

'S-Dump' (dump_schema.vsp) page lets you to save the current database schema. This page uses 'choice_rdir'
component to obtain server filesystem subtree and 'get_schema_comment' instead of 'retrieve_oper_pars'
which is in fact just wrapper around it. So you can choice necessary folder to save schema from list or 
write it manually. If you defined non-existing path manually and 'allow_make_path' switch in 'Options'
is set to on, new path will be created and used. Server folders which already have saved schemas have 
corresponding suffices in brackets. You can change current server folder by double-click on necessary 
one and in so case if schema dump exists in this place you will see old comment. Also you can define 
comment to be stored in this schema. Later you can use it for orientation in a set of similar dumps.
Schema dump contains:
	- comment
	- creation of all users
	- creation of new qualifiers
	- drops of all tables, the order of tables drops corresponds to the order of 
	  tables inheritance via foreign keys and 'under' links.
	- creation clauses for all tables in partially back (in dependent sets) order 
	  in comparison with drop section, all indexes based on this table also are mentioned here
	- all triggers for above mentioned tables
	- views section, and normal and xml views are created here
	- stored procedures 
	

'S-Restore' (restore_schema.vsp) page is set aside for restoring information from previously made dumps. 
This page has very similar with previous explained page external view and the same components but just 
inverse action. It never connects to ODBC datasource directly but uses 'isql' call to run saved schema 
which is just plain PL-SQL text.

'T-Dump' (dump_table.vsp) - using this page you can make dumps of selected tables. And its time to 
explain the internal details of such dump. The thing which in context of DBPUMP is called 'Current Folder' 
in fact means the root directory of tables dump. In this root folder there will be saved the schema 
(if needed) and placed temporary files for 'isql'. Each table dump will be placed in subfolder of this 
root. The name of such subfolder is not the same with table name because of table name can be 
too long and/or contain invalid from the file system's point of view symbols. And such name is 
being calculated as MD5 value of raw table name with suffix '.table'. In this subfolder there 
can be files of 3 kinds:
	- blob.xxxx - files with outlined blobs, file size is in general limited by 
	  'split_by' option in 'Options' page. 
	  In general here means what if we have  blob greater then 'split_by' parameter, 
	  current blob file will be closed and next will be written without size restriction.
	  xxxx here means 0000 for first file, 0001 for second, etc...
	- data.xxxx - files with table data. Here and later we use the same 
	  rules with files lengths. Files have the next structure:
		- magic: "--DBPUMP--\n" written through 'PrpcWriteObject', 
		  we should note here what all values in this file are written via this mechanism.
		- raw table name e.g.: 'Demo.demo.Order_Details'
		- comment defined by user in 'T-Dump' page
		- serialization of DBPUMP's internal state at the moment of dump 
		  (we will talk about the mechanism of permanent state transmission few later).
		- the beginning of sql insert statement up to the data values, this will be used 
		  for data restore. Certainly this can be easily calculated through column info 
		  but in such form it is very useful to control data serialization and in general 
		  you can consider it as additional reserve point.
		- reserve
		- number of 'before tables' - tables which should be 
		  created before the current one (and, obviously, drooped after it)
		- above described number of strings with tables names
		- number of 'after tables' - which are antipodes of 'before tables'
		- above described number of strings with tables names
		- number of schema clauses which are:  drops, table & 
		  indices creations, grants and triggers
		- above described number of strings with sql statements
		- number of columns
		- for each column we have:
			- column type as returned by ODBC 'SQLColumns'
			- column bind type - as was used in ODBC 'SQLBindCol' for main loop's 'select'.
			- is that blob flag
			- column name 
			- reserve
		- the rest of file is filled with data row by row, column by column in the same 
		  order as filled above. The number of 'PrpcWriteObject' call per column depends 
		  only on column type, usually this is 1 call as for integer or varchar up to 7 
		  calls in case of TIMESTAMP. The most complex case is the blob type 
		  (SQL_LONGVARCHAR, SQL_LONGVARBINARY, SQL_WLONGVARCHAR). If blob length is less then 
		  some threshold (currently 64 bytes) it will be written as varchar 
		  with pre-written integer zero, in other case the blob's record consists of  
		  three items: integer length in bytes, starting position of blob in file, 
		  the name of blob file.
	- text.xxxx - files for dump in plain text mode, this is very similar to what you get at 
	  'T-Restore' procedure with the following exceptions:
		- there will not be possible to restore this data by no means except 
		  manual 'isql' call
		- even this will not work if your table has foreign key(s) and/or super-table
		- each table slice will have stored procedure with table creation, but only in 
		  first one there will be explicit call of this procedure
	- isql.code - temporary log file with all sql statements concerning this table
Now let's glance at 'T-Dump' page itself. In the upper area you can see two list-box for 
tables selection. In left one there are available tables, in left - selected for dump.
With buttons '<', '<<', '>>' and '>' you can manipulate by them. There should be noticed what 
the number of really dumped tables can be greater then the number of selected tables. 
The reason of that exists in dependencies with other tables via foreign keys and super-tables.
Below table filters and comments are placed. Filter works on the list of available tables, 
to activate them you should enter some value in one of them and press 'Refresh' button below.
Comment will be written in every table dump and if schema was dumped in this folder, its comment 
will be read. 'Split_by' parameter have been mentioned above and almost always means maximal 
dump-file slice length. SQL/Binary switch lets you to select method of data dump. 
In the bottom area of screen there are placed already familiar remote path list-box in this case 
advanced by the content of current folder i.e. here you also can see names of already dumped 
in this folder tables. This page uses following DBPUMP components: 
	- 'choice_tables' to retrieve selected tables
	- 'retrieve_tables' to obtain available via datasource tables
	- 'choice_rpath' to get remote filesystem sub-tree
	- 'choice_rdir' to retrieve dumped tables in current folder
	- 'get_schema_comment_and_pars_retrieve' to not only deserialize 
	  context but also to get the schema comment in current folder (if exists).

'T-Restore' (restore_tables.vsp) - using this page you can restore previously saved state of 
selected tables. This page consists of the same as in 'T-Dump' 'remote directory' control, 
analogous comment textarea, username&password to be passed to 'isql', 'Name Filter' which can be 
activated by refresh button, 'Qualifier' field to set default qualifier for 'isql' connection and 
'Print To Screen' flag to redirect isql output to http client (but this can exhaust your virtual 
memory). This page uses following DBPUMP components: 
	- 'choice_rpath' for the same reasons as in previous page
	- 'choice_rdir' ...
	- 'get_schema_comment_and_pars_retrieve' ...
The algorithm of this page's action is relatively complex. For every requested table we should 
recursively get lists of dependent tables. As result of this procedure we will get new (probably 
greater) ordered list of requested tables. Having this list we can convert tables to plain isql 
output through 4 passes:
	- drops in back order
	- creations in normal order
	- rows of data in normal order
	- triggers creations in any order
The result of conversion piece by piece will be sent to isql and common error output will be 
returned to http client. Here we should outline two moments of data transmission via 'isql'.
	- blobs: as we remember, short strings was inlined in 'data.xxxx'-file and should be 
	  printed as normal strings. For long blobs the situation is more complex: in output 
	  will be printed bif-function 'blob_to_string_output' with parameters: file name
	  (full name in this case due to multi-layer process of restoring), offset in file 
	  and blob length in bytes.
	- wide strings: there is no correct way to represent general unicode string and so we 
	  will convert unicode string to UTF-8 form and convert them back by bif-function 
	  'charset_recode' with parameters utf-8 sting, 'UTF-8'  and '_WIDE_'.

'Debug' (debug.vsp) - this page is useful (and this is reflected in its name) to switch 
'debug_in_footer' flag which allows to print the current context in footer of every page.


	3.	DBPUMP's context

This context consists of the set of named and internally registered character values. 
Internally each component can get a value of every such value and use its value by any way. 
There exists the only parameter with pre-defined name 'all_together_now' and trying to obtain its 
value you'll get the serialization of all other parameters. Symmetrically, an attempt to set 
value of this parameter will yield to deserialization of previously saved state. Consequently 
every VSP-page should have hidden input element with such name. Because of every VSP page in 
first steps runs 'retrieve_oper_pars' (or its relative) which returns deserialization of all 
parameters including that (without recursion, sure). So if we have synonymous hidden field,
at html form submission its value will be posted and values of all (not only existing in 
current form) parameters will be restored. So if html form has input elements with registered 
in DBPUMP names, values of this parameters will be also accepted by DBPUMP and serialized in 
the next iteration of 'all_together_now'.
	

	4.	DBPUMP's command line
DBPUMP has different variants of execution:
	- normal. Command line syntax is: 
	  dbpump component-name [ component-arg [ output-file ] ] [par1=value1]...[parN=valueN]
	- via CGI, all the same but by urlified  way i.e. environment variable QUERY_STRING should 
	  contain component-name&component-arg&output-file-name&oper-par1-name=oper-par1-value...
	- via file: dbpump @file-name
	  and in this file we should store all arguments in the first(normal) form
	  'component-name' is one of the set of internally registered components names, 
	  if this name is unknown result will be void.
Some components have arguments, some not. Extra argument can not break something  and if 
component doesn't require argument but you are interested in file output, you can freely pass as 
argument whatever you want. All parameters have names consisting only from printable chars but 
in value of parameter there can be placed any string. Due to this, values of parameters should be 
passed in urlified form (i.e. ' '='+', printable chars as is, other in form '%xx').

	5.	DBPUMP's components
DBPUMP doesn't have own behavior, all what it can is to construct operational parameters pool, 
run required component, pass to it argument and print out component's messages.
In fact DBPUMP is a set of independent mini-programs under one roof. Such components are:
	- 'retrieve_oper_pars' - we have already talked about it. 
	  This component just prints all not-empty parameters in form:
	  par1-name=par1-urlified-value&par2-name=par2-urlified-value&...
	  Used operational parameters: all or required 
	  Argument: '*' for all or list of parameters with delimiter '@'

	- 'select_datasources' - when on WIN32 DBPUMP was not linked with native odbc32.lib 
	  this returns a list of available datasources in form:
	  dsn1-name=dsn1-description&dsn2-name=dsn2-description&...
	  In other cases this returns nothing.
	  Used operational parameters: none 
	  Argument: none

	- 'retrieve_tables' - this returns list of available tables in similar form:
	  table1-name=table1-name&table2-name=table2-name&...
	  Used operational parameters:
		- 'datasource'
		- 'user'
		- 'password'
		- 'qualifier'
	  Argument: none

	- 'choice_tables' - this will give you the list of tables selected for dump 
	  in the same form as previous component.
	  Used operational parameters:
		- 'choice_sav' - summarization of all 'choice_tables' from listbox with such name
	  Argument: none

	- 'choice_rpath' - this component will give you the sub-tree of server filesystem. 
	  This component will recursively fall into each subfolder (including symlinks, 
	  this generates the problem of cycling, this situation yields to zero output after 
	  maximal depth will be achieved). 
	  The root folder of this sub-tree will be placed in  'ServerRoot' parameter of 
	  section [HTTPServer] of Virtuoso config file. 
	  Each subfolder will occupy one pair of output:
	  real_path1=indented-name1&real_path2=indented-name2&...
	  Used operational parameters: none 
	  Argument: none

	- 'choice_rdir' - returns the content of current folder. To do this it should descend 
	  in all subfolders and try to find file 'data.0000'.
	  If it exists, the header should be read and table name found. The result will be in form:
	  table1-name=table1-name&table2-name=table2-name&...
	  Used operational parameters:
		- 'rpath' - selected row in corresponding listbox
	  Argument: none

	- 'dump_tables' this component will try to do the above described 
	  procedure of tables dump, the error and info output of which 
	  will be put into operational parameter 'result' and then all 
	  parameters will be output as in 'retrieve_oper_pars'.
	  This component is called from 'dump_tables_itself.vsp'
	  Used operational parameters:
		- 'datasource'
		- 'user'
		- 'password'
		- 'comment'
		- 'dump_name' remote path, similar with 'rpath' but can be entered by hands
		- 'choice_sav' - summarization of all 'choice_tables' from listbox with such name
	  Argument: '*' for all or list of parameters with delimiter '@'

	- 'dump_schema' - the same but the procedure is schema dump.
	  This component is called from 'dump_schema_itself.vsp'
	  Used operational parameters:
		- 'datasource'
		- 'user'
		- 'password'
		- 'comment'
		- 'dump_name' remote path, similar with 'rpath' but can be entered by hands
	  Argument: '*' for all or list of parameters with delimiter '@'

	- 'restore_tables' - the same but the procedure is tables restore.
	  This component is called from 'restore_tables_itself.vsp'
	  Used operational parameters:
		- 'datasource'
		- 'user'
		- 'password'
		- 'dump_name' remote path, similar with 'rpath' but can be entered by hands
		- 'choice_sav' - summarization of all 'choice_tables' from listbox with such name
	  Argument: '*' for all or list of parameters with delimiter '@'

	- 'restore_schema' - the same but the procedure is schema restore.
	  This component is called from 'restore_schema_itself.vsp'
	  Used operational parameters:
		- 'datasource'
		- 'user'
		- 'password'
		- 'dump_name' remote path, similar with 'rpath' but can be entered by hands
	  Argument: '*' for all or list of parameters with delimiter '@'

	- 'get_schema_comment' - this will try to find file 'schema.sql' in current folder and if 
	  such file exists the comment will be extracted,
	  placed into 'comment' operational parameter and then all 
	  parameters will be output as in 'retrieve_oper_pars'.
	  Used operational parameters: all or required 
	  Argument: '*' for all or list of parameters with delimiter '@'
	

	6.	DBPUMP's todo.

As we can see, DBPUMP represents very simple, flexible  and scalable paradigm of behavior. 
It seems to be useful to separate DBPUMP's framework for work with operational parameters 
and components loading from one side and components themselves from other one. By so means 
we will need single description of all components with indications to necessary shared libraries. 
In this case main program in the moment of arguments parsing can load additionally necessary 
library and call required procedure. And we will have an open environment for dbpump-like 
utilities(packages) with single style, administering and behavior.

















</pre>
