Redland Storages Limits (was Re: [redland-dev] Messing up bdb)

[Dave Beckett]

On Thu, 2005-06-16 at 09:19 +0200, Danny Ayers wrote:
> > On 6/15/05, Dave Beckett <dave.beckett at bristol.ac.uk> wrote:
> > 
> > > Yes, redland uses a 2-byte field to store the length of literals
> > > when it encodes and stores them. Suggestions what to do when it's too
> > > long  are welcome - probably at the least it should warn.
> 
> What are the limits on URI strings? Look like I'm getting corruption there too.

2-byte length also, so 0-655535 bytes

> (255 would seem a good choice, except that might cause problems with
> SPARQL queries...)

No, 255 is way too short.  Redland does limit RDF literal
language tags to 255 bytes when stored but I'm not worried about that.

Let me summarise:

Redland has no limits(*) to the size of data in general, only when
stored in some triple stores.

The "hashes" and "memory" storages have these limits from using
the encoding defined in librdf_node_encode() and librdf_node_decode():
  RDF literal
    literal string: 65535 bytes 
      This is UTF-8 encoding of the literal, so it may be
      shorter in terms of Unicode codepoints.
    language string: 255 bytes
      ASCII - see the language RFCs for why.
    datatype URI string: 65535 bytes
      UTF-8
  
  RDF URI
    URI string: 65535 bytes
      UTF-8

  blank node
    identifier string: 65535 bytes
      UTF-8 I think. It's an internal identifier - you shouldn't care!

The "mysql", "sqlite" and "tstore" storages are dependent on the
underlying database limits:

  MySQL uses a LONGTEXT for literal strings and a TEXT for language
  strings, URI strings and blank node identifiers strings.  I assume
  they are all Unicode.

  SQLite uses TEXT for all strings.

  3Store uses MySQL underneath, I've not looked up it's schema recently.

The remaining storages "file" and "dir" have the limits of the "memory"
storage.

Dave

(*)Practical limits, unless you are worried about 4G length strings