On Mon, 28 Jan 2013, Ron Pinkas wrote:
Hi,
Sorry for late respone.
>> Please only remember that I was not alone in Harbour modifications in
>> last years. Viktor made really great job in general code and used types
>> cleanup. Mindaugas added important extensions to compiler and HVM/RTL
>> code, Pritapl is extensively working on set of addons like HBIDE, HBQT,
>> HBXBPP. Also many other people worked on Harbour.
> Thanks for your modesty, but I am convinced that none of this would
> ever take place, unless you first revived Harbour, single handedly,
> as you did, and even after, those IMO, are minor, and superficial
> contributions, compared to your work.
Thanks for nice words but it's not really true. I think that
people using HBMK2 found Viktor modifications as the most
important ones because they allow him to easy move to Harbour
and without any doubts such basic functionality is the most
important for them so for sure it was extremely important for
the whole project.
Anyhow I'm most interested in the future of both projects
so lets leave this discussion. I feel that I can hurt someone
continuing it, i.e it was not my intention to discard Andi
work. I only wanted to point that some not working ports may
create false imaginations about real functionality of original
code and competitions of its authors. I hope that Andi seriously
rethink his decision because I'm finding him as the very important
person in the future modifications.
>> In Harobur there is text filewhere I tried to discribe main differences
>> between both projects:
>>
http://http://harbour-project.svn.sourc ... t?view=log
>> This may show you what should be done. I'll try to update this file
>> and add some information about other differences.
>> As you can seen not too much. I think that in less then month all
>> modifications can be finished.
> Actually this is not shocking for me, and I was debating it my self.
> I am not sure what would be easier, as it mostly depends on the
> DRIVE of the specific individual[s] interested in caring this task.
> Sadly, I am not in a position to volunteer for this Job, for many
> personal reasons, so I can not suggest what would be easier. I trust
> that the person willing to take on this Job, will have his
> preference.
The most important differences between both project are well known.
Anyhow it's also something that it was not very popular in news list
but it's very important for general language description in the future:
small incompatibilities to Clipper and anomalies in implementation
(also in Clipper) which where cleaned or at least documented in
Harbour. We do not have to follow exactly some decisions anyhow
it's much easier to operate on code where some problems are resolved
even to add again some extensions because programmer can see some
interactions which were unknown for him in initial implementation.
It also allows to eliminate duplicated code or code which is to
danger and may break HVM code.
Good example is OOP implementation. Many of __cls*() and __obj*()
functions are repeated or very similar and I can onlu guess that
no one exactly knows what each of them and what are the differences
without careful xHarbour core code analyzing. Here the situation
in Harbour is also bad. I left many functions just for backward
compatibility but without any doubts some of them should be removed
and rest replaced by new small set of functions with precisely
defined actions. We should also agree some extensions so they can
be implemented in all places. If we do not understand something
in current implementation then we should remove it - I'm serious
there are things which never worked or stop to work correctly long
time ago and now it's not possible to guess why we have them and
what they should exactly do. In last week I tried to document
xHarbour HVM OOP functions and compare the to Harbour. Also with
other functionality and it's really gray area. I.e. why xHarbour
has PUBLISHED scope. Looks that it's the same as EXPORT + persistent
flag or at least it duplicates such functionality but with separate
set of functions (also duplicated). For serialization code new
duplicated functions were added which tries to merge both functionality.
To make it more complicated we have yet another serialization method
by HBPRESISTENT class which is merged with other serializations with
many side effects, etc.
>> The windows only extensions and windows API wrappers have WIN_ and
>> WAPI_ prefixes in Harbour and are part of HBWIN library. I think it's
>> good solution because it well separates non portable MS-Windows only
>> code what helps users to create portable programs and isolate local
>> to system extensions.
> Agree, except, I would in general prefer resolution by means of
> namespace support. IMO, NameSpaces are critical feature of any
> modern compiler, and long ago we should have implemented and
> standardized namespace usage to have code like this:
> USING NAMESPACE Clipper
> To force usage of STRICT Clipper compatible RTL Functions, vs.
> USING NAMESPACE Harbour
> To prefer EXTENDED Clipper RTL. or of course manual overrriding:
> Clipper.SubStr(...) vs. Harbour.SubStr(), etc.
> As well as:
> USING NAMESPACE Windows
> I hope you agree that it is more flexible, and elegant, then using prefixes.
I agree that namespace is very important. Anyhow it is much more wider
problem which has many interactions with compiler and with HVM at runtime
when dynamic libraries are loaded and unloaded during application life.
In case of [x]Harbour it also interacts with macrocompiler so we need
dynamic code execution context. Such context should resolve also other
things like limited functionality (sandboxes) for foreign code. Finally
it should also include support for named parameters with default
values what seems to be very important in current days, i.e. I have
dynamic JSON RPC (1.0 and 2.0) server, client and peer connection library
for Harbour so I can exchange data with most of other languages but without
named parameters in core Harbour code it has reduce functionality to
position parameters or dedicated user functions which can decode default
parameter values from partial parameters in hash array.
A lot of things have to be changed to fully implement everything and
I'm not sure it can be done in reasonable time by volunteers.
>> The namespace support is sth what you will have to make yourselves.
>> I'm not familiar with this code.
> It has been many years since I wrote it, and I know it is not 100%
> elegant, due to reliance on PP code, and having to use multiple
> compilation when cyclic reliance between 2 sources exists, but I
> believe it to be very complete and stable, and it offers great
> flexibility and full namespace support, to both C code, and PRG
> code, as well as Macro executed code. I would strongly appreciate
> your review \xharbour\doc\namespaces.txt for the full set of
> functionality it offers, in terms of a Namespace MODEL, then we
> could discuss the code, which I am really much less interested in.
OK, I'll try to look at it in spare time.
>> In Harbour compiler can be used at runtime. It means that
>> program which works like xBaseScript can be implemented in just few
>> lines and .prg files are executed with full speed just like
>> after -gc2 compilation.
> Interesting - what about the copyright exception which was NOT given
> to the Compiler sources?
Still exists so such programs have to be released on GPL license.
In fact in Harbour the folowing files do not contain Harobur excption:
cmdcheck.c
genc.c
genhrb.c
harbour.y
hbfunchk.c
hbgenerr.c
hbident.c
hbmain.c
A lof of code inside is written by me but not all so I still have on
my TODO list to rewrite them. It can be done quite easy starting
from macrocompiler grammar rules and creating new flow control for
compiler mode. It should resolve many bad things which are forced
by current grammar rules defined in harbour.y
Anyhow it's not critical for me and I do not find anything wrong
that HBRUN in Harbour is on GPL license.
>> I even create hbscript which can be used as active script in
>> MS-Windows. I haven't committed it yet. It covers similar functionality
> >from xHarbour.com - I tested it with xHarbour examples and they
>> worked well.
>
> Very good
I'll try to commit it soon so you can see it in action.
It was written by me and Mindaugas.
It's pure C code though it defines C++ objects for COM/OLE
interface. If you agree then I would like to attach one of
your html example file from xBaseScript just for comparison.
I'm not MS-Windows programmer so this code to be alive needs
support from Windows users.
>> The real problem can appear at runtime and is caused by binary
>> compatibility with older code.
>> HB_SERIALIZE() gives incompatible results. It means that
>> HB_DESERIALIZE() from Harbour cannot decode data encoded by
>> current xHarbour HB_SERIALIZE().
>> HB_SERIALIZE() is used by SQLRD so this problem is not such trivial
>> because it may block migrating to new xHarbour version.
>> Probably this can be resolved adding support for signatures used
>> by xHarbour in serialization code. I haven't though about it
>> before. I'll look at it closer and if it's possible I'll commit
>> such patch to Harbour.
> Saw you already committed.
Yes and I added also support for decoding data with cross refrences,
objects and HBPERSITENT objects. I also recognizes correctly serialized
codeblocks skipping their body (decoded as NIL).
It means that now everything except codeblocks serialized directly
or indirectly by xHarbour version of HB_SERIALIZE() is correctly
decoded by Harbour. When I was checking xHarbour serializtion code
I've found some problems which definitely have to be fixed.
I'll commit critical fixes soon.
>> hb_valToExp() gives different results then ValToPrg() but in fact
>> Harbour creates real expressions which can be macrocompiled so it's
>> rather like a fix for current xHarbour behavior.
> Mmm, are you not aware of xHarbour's ValToPrgExp()?
I have to missed it or I exploited some problems with it in the past.
When I touched HB_SERIALIZE() I also checked other serialization code
with some OOP functionality in Harbour and xHarbour. As result I
created new general functions __obj{Set,Get,Restore}IVars() which
can be used as general serialization function regardless of object
definition. They can serialize whole object instance area also
overloaded by descendant classes which needs super casting or
have nonvirtual hidden messages. These functions can be used to
replace all other __obj*() and __cls*() functions which tries to
operate on object instance variables. I also updated hb_valToExp()
to use them so now the serialization is more similar to xHarbour
though it does not use PRIVATE variables so deserialization is
reentrant safe.
In the future I will want to document how programmer can control
which object items should be serialized automatically.
I plan to add support for VAR ... NOSAVE like in xBase++ to
explicitly disable serialization for given variables. For
backward compatibility if objects has variables with PERSISTENT
attribute then only this variables will be saved. It should cover
HBPERSITENT class functionality. Pointer items and codeblock
will be eliminated. Exported persitent variables will give
functional replacement for published scope.
Looking at different serialization code implemented in xHarbour
I tried to document some most important things I found. It should
help in creating one centralized and well documented serialization
code:
1. HB_SERIALIZE()
- it does not serialize timestamp values "T"
- it wrongly serialize LONGLONG values and then cannot desrialize
data containing LONGLONG values
- it does not serialize pointer items "P"
- for objects it uses:
HBPersistent:SaveToText()
or
__ClsGetPropertiesAndValues( oObj )
- stores CODEBLOCKs as "B" + HB_SERIALIZE( HB_SaveBlock( <bCode> ) )
2. ValToPrg() / ValToPrgExp()
- it does not correctly serialize timestamp values "T"
- potential problems with strings containig chr(0) or "\" characters
StringToLiteral() does not work correctly.
- it does not serialize memo strings "M"
- it converts pointer items to numbers "P"->"N"
- no support for cross and cyclic references in hashes
- deserialization by macrocompiler is not reentrant safe so if
execution of macrocompiled data activates some code which may
internally use deserialization of other ValToPrg() values then
it breaks upper level data because it operates on common private
variable. It is not unusual situation, i.e. it cannot be used
with code where objects retrieve their initial state from
serialized data inside constructors.
- for objects it uses:
__objGetValueDiff( oObj )
effectively it works like __ClsGetPropertiesAndValues( oObj ) in
HBPersistent():SaveToText() but saves also EXPORTED variables and
does not saves private and hidden vars declared with PERSISTENT
attribute.
Unfortunately it wrongly saves CLASSVARs with some unexpected
values extracted from object instance area - this is serious
bug because it badly interacts with deserialization code.
3. HBPersistent():SaveToText()
- it does not correctly serialize hashes (generates broken output)
because it wrongly uses ValToPrg() instead of ValToPrgExp() for
types which are not directly supported.
- no support for cross and cyclic references
- to serialize objects which are HBPersistent class descendant
(so also Self) it uses:
__ClsGetPropertiesAndValues( oObj )
but reduces the properties list to the ones which have different
values different then in object created by:
__ClsInst( Self:ClassH )
- other objects in instance variables are not serialized.
4. Functions to clone complex items at runtime.
They are not strictly serialization functions but share some similar
problems like cross references and creating next object instances so
they have to be synced with serialization code. Now xHarbour has:
AClone(), HClone(), __objClone()
- HClone() in xHarbour does not clone anything, it's simple HCOPY()
so it has to be fixed and bound with AClone() code so they cab use
common references list.
- AClone() - clones only arrays but doesn't clone hashes and does not
make any deeper check for hash contents so items like:
a := { { "V" => NIL } }
a[1,"V"] := a
are not correctly cloned. It means that both functions have to use
common internal clone engine like in Harbour. So far in xHarbour
only HB_SERIALIZE() tries to follow internal references in arrays,
object and hashes.
- __objClone() - it does not clone nested objects - only instance
area is cloned. In xHarbour there is support for class method
declared with CONSTRUCTOR attribute. Such methods are executed with
HB_OO_MCLSCTOR_INSTANCE parameter when new object is created and
with HB_OO_MCLSCTOR_CLONE parameter inside __objClone() function.
It means that theoretically it's possible to implement some custom
initialization for cloned objects and deeper cloning for object
items. Unfortunately it's not bound with internal list of cross
references so user cannot implement custom deep cloning for objects
with internal cross references. Also AClone() is not bound with
it and ignores object with defined clone/copy/init constructor.
This have to be resolved yet also in Harbour. We should agree
how to define copy/init constructors or which messages send to
new objects when they are cloned or deserialized - now
deserialization code completely ignores copy constructors, in
opposite Xbase++ sends :notifyLoaded() message to deserialized
objects so programmer can overload it and make some necessary
initialization.
We should also add support for nested cloning so user can access
cloned item list and redirect it to common clone engine or at
least define that objects support deep cloning so we can use
standard array cloning method for such objects.
I also checked some helper functions used in serialization/clone
operations and here are some notes:
- HB_SaveBlock( <bCode> ) does not serialize detached locals so it's not
necessary to detect cyclic references between them and other data.
Anyhow general codeblock serialization may support it and in such
case it's necessary to bind it with common list of references.
- StringToLiteral() does not work correctly, use HB_STRTOEXP() instead
which gives correct results also for binary strings and it's much
faster.
BTW I'm the author of this function not David
- __objGetValueDiff( oObj )
It uses __clsGetIVarNamesAndValues( Self, HB_OO_CLSTP_EXPORTED + ;
HB_OO_CLSTP_PUBLISHED )
but reduces the properties list to the ones different then in
__clsGetIVarNamesAndValues( __ClsInst( Self:ClassH ), ;
HB_OO_CLSTP_EXPORTED + ;
HB_OO_CLSTP_PUBLISHED )
This reduction (just like in HBPersistent():SaveToText()) does not
work correctly for complex items because they are cloned during
object initialization and deep comparison is not implemented so
it never reduce arrays used in INIT clause in class definition.
- __ClsGetPropertiesAndValues( oObj )
It looks for HB_OO_CLSTP_PUBLISHED or PERSITENT messages which
have non zero uiData index.
It means that it gives wrong results for
CLASS VAR
declared as PUBLISHED and
METHOD/ACCESS/ASSIGN ... INLINE
declared as PERSISTENT or PROPERTY
because they use uiData internally as index to diffeent structures,
this is serious bug.
- __clsGetIVarNamesAndValues( oObj, nScope )
It looks for variables with given scope (or all if scope is 0)
checking message function hb___msgGet*Data().
It gives wrong results for
CLASS VAR
this is serious bug,
fortunately ignores shared vars due to 0 in uiData PMETHOD member.
I also found that either Harbour and xHarbour cannot serialize correctly
supercast objects. In fact the results are unpredictable because it
creates two objects of given class but the first one has only one
element instance are and in this element it has real object. Definitely
it has to be fixed in both compilers. I already added it to my TOFIX list.
Enough, lot about serialization code but binary compatibility with
existing data stored somewhere is usually the most important thing
which can block future development and extensions. This is the
reason why I never liked to add extensions to RDD if they are not
precisely defined and can create anomalies (i.e. SIx3 encryption in
memos). RDDs is yet another thing anyhow here the differences are
much smaller.
In general RDD is subject for separate discussion but here situation
is much simpler. xHarbour has few copies of DBFCDX called BMDBFCDX,
DBFMDX, REDBFCDX, REDBFFPT, DBFNET, etc. All such RDDs can be safely
removed because they do not give any new functionality in comparison
to pure DBFCDX in Harbour core code and BM* RDDs in contrib. There
are some differences in DBF headers but as I can see most of them
you have to remove when people begins to report serious compatibility
problems with Clipper and other xbase compatible languages.
>> I believe that you can quite fast catch all such differences and
>> agree what to do with them.
> Yes, tough I suspect there are many more hidden, such as extension
> to Clipper RTL functions, that in Harbour were either not
> implemented, or implemented as new HB_<SomeFunc>().
Yes but most of them are documented in XHB library so it should
be easy to update the code.
Enough. Now I really have to return to my own work.
best regards,
Przemek
Enviado: 04 Jun 2008 22:31