XmlGen.py - XML I/O map generator
Top generator for XML I/O maps. The header comment is the most authoritative reference for the structure of XML I/O maps and XML I/O style, tags, etc. generally.
Size 28.4 kB - File type text/python-sourceFile contents
"""
======================COPYRIGHT/LICENSE START==========================
XmlGen.py: Code generation for CCPN framework
Copyright (C) 2005 Wayne Boucher (CCPN Project)
=======================================================================
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public
License as published by the Free Software Foundation; either
version 2 of the License, or (at your option) any later version.
A copy of this license can be found in ../license/GPL.license
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
======================COPYRIGHT/LICENSE END============================
for further information, please contact :
- CCPN website (http://www.ccpn.ac.uk/)
- email: ccpn@bioc.cam.ac.uk
=======================================================================
If you are using this software for academic purposes, we suggest
quoting the following references:
===========================REFERENCE START=============================
R. Fogh, J. Ionides, E. Ulrich, W. Boucher, W. Vranken, J.P. Linge, M.
Habeck, W. Rieping, T.N. Bhat, J. Westbrook, K. Henrick, G. Gilliland,
H. Berman, J. Thornton, M. Nilges, J. Markley and E. Laue (2002). The
CCPN project: An interim report on a data model for the NMR community
(Progress report). Nature Struct. Biol. 9, 416-418.
Rasmus H. Fogh, Wayne Boucher, Wim F. Vranken, Anne
Pajon, Tim J. Stevens, T.N. Bhat, John Westbrook, John M.C. Ionides and
Ernest D. Laue (2005). A framework for scientific data modeling and automated
software development. Bioinformatics 21, 1678-1684.
===========================REFERENCE END===============================
Top level classes for generation of I/O maps.
Subclasses in other files will then complete the I/O maps and write them, or
derived products like XML schemas, to disk. Note that the maping dictionaries
generated in these classes are identical in structure to those written to disk
(in Python) and could be used directly for XML I/O once language specific
extensions were added
Structure of I/O maps
A top level dictiooary (globalMap) contains:
package specific mapping dictionaries for every package
(key is package.shortName)
A dictionary 'loadMaps' of XML tag : element I/O map
This has maps for all legal XML tags.
Note that all XML tags are unique across the entire model.
They are of the following forms:
- PPPP.Cccccc for AbstractTypes, where the prefix PPPP is the
package.shortname and Cccccc is the AbstractType.name
- PPPP.Cccccc.aaaa for ClassElemnts, where PPPP and Cccccc are as above,
and aaaa is the ClassElement.name
- PPPP.exo-Cccccc for exolinks. The tqg descrines an out-of-package link *to*
Class Cccccc in package with shortName PPPP.
A dictionary 'mapsByGuid' of model element guid : element I/O map
which is used for model comparison and for crossreferences between I/O maps.
These maps include all abstract types and all attributes or links,
but not the exolinks descriptors
The maps form different packages refer to each others contents,
importing package to imported package, and Implementation package to all others.
All dictionaries contain enough information to allow you to write to file
a reference to teh mapping dictionary given teh globalMap
Every package pecific mapping contains:
- 'globalRelease' - the global release version
- '.prefix' - the package prefix (== package.shortName)
- '.name':'mapping' (fixed value) - describes the kind of dictionary
for use in subsequent generation
- 'abstractTypes' - the AbstractTypes dictionary
- 'exolinks' - the dictionary of out-of-package link descriptors
the abstractTypes dictionary contains
- '.prefix' - the package prefix (== package.shortName)
- '.name':'abstractTypes' (fixed value) - describes the kind of dictionary
It further contains an I/O map for every AbstractType
(Class (also abstract), DataObjType (also abstract) or
DataType (not abstract) ). The dictionary key is the
AbstractType.name.
the exolinks dictionary contains
- '.prefix' - the package prefix (== package.shortName)
- '.name':'exolinks' (fixed value) - describes the kind of dictionary
It further contains an I/O map for out-of-package links to every
non-abstract class in the package. The dictionary key is the Class.name.
All I/O maps contain the following:
- 'tag': The XML tag to use when writing. May differ from the corresponding
key in the 'loadMaps' which is used for reading.
- 'guid': The guid for the corersponding model element
- 'type' the element type, which is one of the following:
'class', 'cplx', 'simple', 'exo', 'attr', 'link', 'child', 'coll'
Further content depends on the type - some are a bit arcane, they are chosen
to make XML I/O fast, not to make the description simple:
- type 'cplx' - for DataObjTypes (and 'class' - for Classes) further has
'eType' : 'cplx' (fixed value)
'class' : The actual API class corresponding to the DataObjType (or Class)
Also serves as a constructor.
'content' : a dictionary of ClassElement.name : I/O map for classElement
'headerAttrs' : A list of attribute names for attributes that
are written as XML attributes in the Class XML element.
'simpleAttrs' : A list of ClassElement names for attributes with DataType
valueType and intrapackage links that are not included in
'headerAttrs'
'cplxAttrs' : A list of ClassElement names for child links, attributes of
DataObjType valueType, and out-of-package links
For use in version conversion, two more elements are optional:
'cnvrt': a function that takes an object as a parameter. It is called on
the relevant object(s) after reading and link dererferencing, but
before validity checking. The function converts from the then
(possibly intermediate) state of the object to the final state.
'priority' : An integer that determines the order (lowest value first)
in which the cnvrt functions are called. Intended to ensure
that later conversions can rely on earlier conversions to be
finished.
- type 'class' - has all the tags used for type 'cplx', plus the following
'optLinks': A list of role names for intrapackage roles where
role.hicard == 1 and role.otherRole.hicard != 1.
These may be treated as headerAttrs, simpleAttrs, or skipped,
depending on the ettings of the save function.
'fromParent': equal to clazz.parentRole.otherRole.name
'objkey' : Optional. The name of the single simple attribute that forms
the local key of the class. Where appropriate it allows
faster object creation
type 'simple' - for DataTypes, further has toStr, cnvrt,
'toStr' : a function to convert a value to a string,
or the string 'text' for string types
'cnvrt' : a function to convert a value from a string,
or the string 'text' for string types
type 'exo' - for exolinks
'eType' : 'cplx' (fixed value)
'class' : The actual API class corresponding to the Class
'keyMaps': a list of I/O maps that describes the fullKey of the object -
used for writing.
For key elements of DataType type the I/O map of the dataType is
used. For key elements of DataObjType type, the mapping of the
relevant package is used. For key elements of type Class the
exolinks directory of the relevant package is used.
For use in version conversion:
'proc': Optional . fixed value 'delay'. In the normal case each
exolink is dereferenced as soon as it has been read.
For proc == 'delay' processing is suspended to allow
version compatibility code for the attribtue containing it to
do its own processing
Finally there are the types for ClassElements - attributes and links
all maps have tag, type and guid
Further (square brackets may be needed for schemas, not used in I/O
type 'attr' - attributes of DataType valueType
'hicard' : attr.hicard
'locard' : attr.locard. Not needed for I/O but may be so for XML schemas.
'name' : attr.name. Used for writing XML attributes and for setting the
result when reading
'data' : The I/O map dictionary of the valueType. Must be type 'simple'
'proc' : Optional. 'direct' (fixed value). If proc=='direct' the attribute
is set into the object bypassing the API. Can only happen when
hicard == 1 and there are no constraints.
'default' : attribute default value.
'implSkip' : Optional. True (fixed value). If set, do not write this attribute
if the package being written is Implementation. Used for writing
TopObjects in Implementaion package.
'eType' : Optional. 'cplx' (fixed value). If set attribute is read and
written with an XMl element contianing other elements. Used (e.g.)
for DataTypes that may contain whitespace (like String).
'isDerived': attr.isDerived. Not needed for I/O but may be so for XML schemas.
Optional. Only set if true.
type 'link' - intrapackage links
'hicard' : link.hicard
'name' : link.name. Used for writing XML attributes and for setting the
result when reading
'implSkip' : Optional. True (fixed value). If set, do not write this link
if the package being written is Implementation. Used for writing
TopObjects in Implementaion package.
'locard' : link.locard. Not needed for I/O but may be so for XML schemas.
'isDerived': link.isDerived. Not needed for I/O but may be so for XML schemas.
Optional. Only set if true.
type 'child' - child links
'hicard' : link.hicard
'eType' : 'child' (fixed value).
'content' : The abstractDataTypes dictionary for the package containing the
child class.
'implSkip' : Optional. True (fixed value). If set, do not write this link
if the package being written is Implementation. Used for writing
TopObjects in Implementaion package.
'locard' : link.locard. Not needed for I/O but may be so for XML schemas.
'proc' : Optional. 'loadDelayed' (fixed value). If set, the end of this
link will trigger a call to loadDelayedData.
Needed for the special case of reading Implementation.xml, where
certain objects must be set and linked before the rest of the
data can be processed.
type 'coll' - Attributes of DataObjType type or our-of-package links.
'hicard' : attr.hicard
'locard' : attr.locard. Not needed for I/O but may be so for XML schemas.
'name' : attr.name. Used for setting the result when reading
'eType' : 'cplx' (fixed value).
'content' : The abstractDataTypes dictionary for the package containing the
DataObjType (for DataObjType attributes) or the exolinks
dictionary for the package containing the valueType
(for exolinks).
'implSkip' : Optional. True (fixed value). If set, do not write this link
if the package being written is Implementation. Used for writing
TopObjects in Implementaion package.
'isDerived': attr.isDerived. Not needed for I/O but may be so for XML schemas.
Optional. Only set if true.
'proc' : Optional. 'direct' (fixed value). If proc=='direct' the attribute
is set into the object bypassing the API. Can only happen when
hicard == 1 and there are no constraints.
"""
from memops.metamodel import MetaModel
MemopsError = MetaModel.MemopsError
from memops.metamodel import ImpConstants
import memops.general.Constants as genConstants
from memops.scripts_v2.core.ModelTraverse import ModelTraverse
headerTextTypeNames = ['Token','Word']
mandatoryAttributes = ()
repositoryTag = '$Name: $'
repositoryId = '$Id: XmlGen.py,v 1.12 2007/05/04 18:40:13 rhfogh Exp $'
# Requires other writers also to be implemented in subclass
class XmlGen(ModelTraverse):
###########################################################################
###########################################################################
def __init__(self):
# init handling
super(XmlGen, self).__init__()
pp = self.modelPortal.topPackage
self.implPackage = pp.metaObjFromQualName(self.implPackageName)
#self.baseClass = self.implPackage.getElement(ImpConstants.baseClassName)
self.stringType = self.implPackage.getElement(genConstants.string_code)
self.dataObject = self.implPackage.getElement(ImpConstants.dataObjClassName)
self.topObject = self.implPackage.getElement(ImpConstants.topObjClassName)
self.dataRoot = self.implPackage.getElement(ImpConstants.dataRootName)
self.dictType = self.implPackage.getElement('StringKeyDict')
for tag in mandatoryAttributes:
if not hasattr(self, tag):
raise MemopsError(" XmlGen lacks mandatory %s attribute" % tag)
# has to be done this way to allow for different initialisation orders
if not hasattr(self, 'modelFlavours'):
self.modelFlavours = {}
###########################################################################
###########################################################################
# overrides ModelTraverse
def initLeafPackage(self, package):
""" Set up dictionaries and write topObject maps
"""
self.prefix = package.shortName
if package is self.implPackage:
self.globalMap = dd = {}
dd['mapsByGuid'] = {}
dd['loadMaps'] = {}
for pp in self.modelPortal.leafPackagesByImport():
prefix = pp.shortName
self.globalMap[prefix] = mapping = {}
# set top level map
mapping['abstractTypes'] = abstractTypes = {'.prefix':prefix,
'.name':'abstractTypes'}
mapping['exolinks'] = exolinks = {'.prefix':prefix, '.name':'exolinks'}
mapping['guid'] = pp.guid
mapping['.prefix'] = prefix
mapping['.name'] = 'mapping'
mapping['globalRelease'] = genConstants.currentModelVersion
###########################################################################
###########################################################################
# overrides ModelTraverse
def processLeafPackage(self, package):
""" make data dictionaries for package containing actual code
"""
# data types
# Note that inheritance order means that
# supertypes are processed before subtypes
for xx in self.modelPortal.dataTypesByInheritance(package):
if not xx.isAbstract:
self.processDataType(xx)
# dataObjTypes
for xx in self.modelPortal.dataObjTypesByInheritance(package):
self.processDataObjType(xx)
self.endComplexDataType(xx)
# classes
for xx in self.modelPortal.classesByInheritance(package):
self.processClass(xx)
self.endComplexDataType(xx)
# exolink maps
if package is not self.implPackage:
for xx in self.modelPortal.classesByInheritance(package):
if not xx.isAbstract:
self.processExoLink(xx)
###########################################################################
###########################################################################
# overrides ModelTraverse
def endLeafPackage(self, package):
""" processing actions for end of LeafPackage
"""
del self.prefix
###########################################################################
###########################################################################
# overrides ModelTraverse
def processDataType(self, xx):
""" toStr, cnvrt, and baseType postponed, being language dependent
"""
guid = xx.guid
tag = self.xmlTag(xx)
result = {'tag':tag, 'type':'simple', 'guid':guid,}
self.globalMap[self.prefix]['abstractTypes'][xx.name] = result
self.globalMap['mapsByGuid'][guid] = result
self.globalMap['loadMaps'][tag] = result
###########################################################################
###########################################################################
# overrides ModelTraverse
def processDataObjType(self, xx):
guid = xx.guid
result = {'type':'cplx', 'eType':'cplx', 'guid':guid, 'content':{},
'headerAttrs':[], 'simpleAttrs':[], 'cplxAttrs':[],}
self.globalMap[self.prefix]['abstractTypes'][xx.name] = result
self.globalMap['mapsByGuid'][guid] = result
if not xx.isAbstract:
tag = self.xmlTag(xx)
result['tag'] = tag
self.globalMap['loadMaps'][tag] = result
###########################################################################
###########################################################################
# overrides ModelTraverse
def endComplexDataType(self, xx):
self.processClassAttrs(xx)
if isinstance(xx, MetaModel.MetaClass):
self.processClassRoles(xx)
if self.dataRoot in xx.getAllSupertypes():
classMap = self.globalMap[self.prefix]['abstractTypes'][xx.name]
cplxAttrs = (classMap['cplxAttrs'])
# these roles must be stored and loaded before anything else
ic = ImpConstants
cplxAttrs.remove(ic.packageLocatorRole)
cplxAttrs.remove(ic.repositoryRole)
cplxAttrs[:0] = [ic.repositoryRole, ic.packageLocatorRole]
###########################################################################
###########################################################################
# overrides ModelTraverse
def processClass(self, xx):
guid = xx.guid
result = {'type':'class', 'eType':'cplx', 'guid':guid, 'content':{},
'headerAttrs':[], 'simpleAttrs':[], 'optLinks':[], 'cplxAttrs':[],
}
pr = xx.parentRole
if pr is not None:
result['fromParent'] = pr.otherRole.name
ll = xx.keyNames
if len(ll) == 1:
tag = ll[0]
if xx.getElement(tag).hicard == 1:
result['objkey'] = tag
self.globalMap[self.prefix]['abstractTypes'][xx.name] = result
self.globalMap['mapsByGuid'][guid] = result
if not xx.isAbstract:
tag = self.xmlTag(xx)
result['tag'] = tag
self.globalMap['loadMaps'][tag] = result
###########################################################################
###########################################################################
def processClassAttrs(self, clazz):
""" process atttributes for class *or* DataObjType
"""
mapping = self.globalMap[self.prefix]
classMap = mapping['abstractTypes'][clazz.name]
classContent = classMap['content']
for elem in clazz.getAllAttributes():
elemMap = self.processAttribute(elem, clazz)
if elemMap is not None:
# connect up
classContent[elemMap['name']] = elemMap
typ = elemMap['type']
# add to apprropriate list for writing order
if not elemMap.get('isDerived'):
if typ == 'coll':
classMap['cplxAttrs'].append(elemMap['name'])
else:
# typ == 'attr'
if elem.hicard == 1:
classMap['headerAttrs'].append(elemMap['name'])
else:
classMap['simpleAttrs'].append(elemMap['name'])
###########################################################################
###########################################################################
def processClassRoles(self, clazz):
""" process atttributes for class *or* DataObjType
"""
mapping = self.globalMap[clazz.container.shortName]
classMap = mapping['abstractTypes'][clazz.name]
classContent = classMap['content']
children = []
for elem in clazz.getAllRoles():
elemMap = self.processRole(elem, clazz)
if elemMap is not None:
# connect up
classContent[elemMap['name']] = elemMap
typ = elemMap['type']
if typ == 'link':
# intrapackage link
otherRole = elem.otherRole
if elem.hicard == 1 and otherRole and otherRole.hicard != 1:
classMap['optLinks'].append(elemMap['name'])
else:
classMap['simpleAttrs'].append(elemMap['name'])
elif typ == 'child':
# intrapackage child link
children.append(elemMap['name'])
else:
# exolink
classMap['cplxAttrs'].append(elemMap['name'])
classMap['cplxAttrs'].extend(children)
###########################################################################
###########################################################################
# overrides ModelTraverse
def processAttribute(self, elem, inClass):
result = {}
if elem.isImplementation or elem.isAbstract:
return None
# derived elements
if elem.isDerived:
if elem.changeability == ImpConstants.frozen:
return None
else:
result['isDerived'] = True
if elem.container is inClass:
# make new map
hicard = elem.hicard
valueType = elem.valueType
vtPrefix = valueType.container.shortName
guid = elem.guid
name = elem.name
tag = self.xmlTag(inClass, elemName=name)
result['name'] = name
result['tag'] = tag
result['guid'] = guid
result['hicard'] = hicard
result['locard'] = elem.locard
# set type
if isinstance(valueType, MetaModel.MetaDataType):
result['type'] = 'attr'
if (self.stringType in valueType.getAllSupertypes()
and elem.name not in headerTextTypeNames):
result['eType'] = 'cplx'
result['data'] = self.globalMap[vtPrefix]['abstractTypes'][valueType.name]
else:
# complex data type
result['type'] = 'coll'
result['eType'] = 'cplx'
result['content'] = self.globalMap[vtPrefix]['abstractTypes']
# default value
default = elem.defaultValue
if default:
if hicard == 1:
result['default'] = default[0]
elif elem.isUnique:
result['default'] = set(default)
else:
result['default'] = default
# processing tag
if hicard == 1 and not elem.isDerived:
if isinstance(valueType, MetaModel.MetaDataType):
if (not valueType.getAllConstraints() and valueType.length is None
and (valueType.isOpen or not valueType.enumeration)):
# primitive, unconstrained type. set directly into memory
result['proc'] = 'direct'
# for topObject shells in Impl package only guid and keys are stored.
if inClass is self.dataObject:
# Note that dataObject has no subclasses in Impl but TopObject
# inherits from it
result['implSkip'] = True
if self.topObject in inClass.getAllSupertypes():
if name != 'guid' and name not in inClass.keyNames:
result['implSkip'] = True
# connect up
self.globalMap['mapsByGuid'][guid] = result
self.globalMap['loadMaps'][tag] = result
else:
# map exists - get it
result = self.globalMap['mapsByGuid'][elem.guid]
#
return result
###########################################################################
###########################################################################
# overrides ModelTraverse
def processRole(self, elem, inClass):
result = {}
hicard = elem.hicard
valueType = elem.valueType
vtPrefix = valueType.container.shortName
package = elem.container.container
if elem.isImplementation or elem.isAbstract:
return None
# derived elements
if elem.isDerived:
if elem.changeability == ImpConstants.frozen:
return None
else:
result['isDerived'] = True
# processing tag
hierarchy = elem.hierarchy
if hierarchy == ImpConstants.parent_hierarchy:
# parent link
return None
elif hierarchy == ImpConstants.child_hierarchy:
# intra- or inter-package childlink
result['type'] = 'child'
result['eType'] = 'child'
result['content'] = self.globalMap[vtPrefix]['abstractTypes']
if (inClass is self.dataRoot
and elem.name == ImpConstants.packageLocatorRole):
result['proc'] = 'loadDelayed'
elif valueType.container is package:
if hierarchy != ImpConstants.child_hierarchy:
# intra-package crosslink
result['type'] = 'link'
elif (valueType.container in package.accessedPackages
and elem.container is not self.dataObject):
# wrong way exolink
# The exception is DataObject.access
return None
elif valueType is self.dataObject:
# AccessObject.dataObjects
return None
else:
# normal exolink
result['type'] = 'coll'
result['eType'] = 'cplx'
result['content'] = self.globalMap[vtPrefix]['exolinks']
if elem.container is inClass:
# make new map
guid = elem.guid
name= elem.name
tag = self.xmlTag(inClass, elemName=name)
result['name'] = name
result['tag'] = tag
result['guid'] = guid
result['hicard'] = hicard
result['locard'] = elem.locard
# connect up
self.globalMap['mapsByGuid'][guid] = result
self.globalMap['loadMaps'][tag] = result
kNames = inClass.keyNames
if ((name not in kNames and self.topObject in inClass.getAllSupertypes())
or inClass is self.dataObject):
# for topObject shells in Impl package only guid and keys are stored.
# Note that dataObject has no subclasses in Impl but TopObject
# inhjerits from it
result['implSkip'] = True
else:
# map exists - get it and ignore work done so far
result = self.globalMap['mapsByGuid'][elem.guid]
#
return result
###########################################################################
###########################################################################
def processExoLink(self, clazz):
"""
"""
globalMap = self.globalMap
tag = self.xmlTag(clazz, var='exo')
guid = clazz.guid
keyMaps = []
result = {'tag':tag,
'name':clazz.name,
'type':'exo',
'eType':'cplx',
'guid':guid,
'keyMaps':keyMaps}
# get parent list
# set up. First parent list
parents = [clazz]
for pp in parents:
pr = pp.parentRole
if pr is None:
break
else:
parents.append(pr.valueType)
# remove MemopsRoot
parents.pop()
topObject = parents.pop()
keyElements = [topObject.getElement('guid')]
parents.reverse()
for pp in parents:
for ss in pp.keyNames:
elem = pp.getElement(ss)
keyElements.append(elem)
for elem in keyElements:
vt = elem.valueType
packageMap = globalMap[vt.container.shortName]
if isinstance(vt, MetaModel.MetaClass):
# exolink
elemMap = packageMap['exolinks']
elif isinstance(vt, MetaModel.MetaDataObjType):
# MetaDataObjType
elemMap = packageMap['abstractTypes']
else:
# simple type
elemMap = packageMap['abstractTypes'][vt.name]
keyMaps.extend(elemMap for dummy in range(elem.hicard))
self.globalMap[self.prefix]['exolinks'][clazz.name] = result
self.globalMap['loadMaps'][tag] = result
#
return result
###########################################################################
###########################################################################
def xmlTag(self, clazz, elemName=None, var=None):
""" make xml tag for element corresponding to MetaObject
"""
prefix = clazz.container.shortName
className = clazz.name
if var == 'exo':
className = 'exo-%s' % className
if elemName is None:
return '%s.%s' % (prefix, className)
else:
return '%s.%s.%s' % (prefix, className, elemName)
###########################################################################