[redland-dev] Messing up bdb
Dave Beckett
dave.beckett at bristol.ac.uk
Wed Jun 15 22:05:52 BST 2005
On Wed, 15 Jun 2005, Danny Ayers wrote:
> Dave, I think I might have found what was causing my db foulups
> earlier, looks like an overflow (or break in the middle of a stream).
> Symptom was -
>
> Error:Illegal node encoding 'u' seen
>
> I checked the db42 install, reinstalled Redland, ran the crawl again.
> Same error, except this time it was encoding 'a'. It appeared in the
> same bunch of triples as before, from:
> http://www.greenrobe.com/node/feed
> XSLT'd with:
> http://purl.org/net/syndication/subscribe/feed-rss1.0.xsl
>
> The last bit of that feed is a <description> about 110,000 chars long,
> which would have been added as a literal, or at least attempted. Time
> I started trimming ;-)
Yes, redland uses a 2-byte field to store the length of literals
when it encodes and stores them. Suggestions what to do when it's too
long are welcome - probably at the least it should warn. I can add
support for literals with 4-byte lengths if people want them (with a
new encoding type).
Note though, that the BDB indexing means that you copy literals
for every time 'o' (triple object) is mentioned in an index, so
that means 300K bytes+.
Maybe the SQLite store would work a little better as it normalises a
little, but it's still new so take care.
Dave
More information about the redland-dev
mailing list