org.sindice.siren.analysis

Class JsonTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.sindice.siren.analysis.JsonTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class JsonTokenizer
extends org.apache.lucene.analysis.Tokenizer

A tokenizer for data following the JSON syntax.

The tokenizer parses JSON data and generates a token for each field name and each value. The tokenizer attaches a node label and a datatype to each token.

Regarding datatype, the convention is the following:

• If a field name is parsed, the datatype JSONDatatype.JSON_FIELD is assigned;
• If a value string is parsed, the datatype XSDDatatype.XSD_STRING is assigned;
• If a boolean value is parsed, the datatype XSDDatatype.XSD_BOOLEAN is assigned;
• If a numerical value is parsed, the datatype XSDDatatype.XSD_LONG is assigned;
• If a numerical value with a fraction is parsed, the datatype XSDDatatype.XSD_DOUBLE is assigned;

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATATYPE_LABEL Datatype JSON schema: field for the datatype label
static String	DATATYPE_VALUES Datatype JSON schema: field for the datatype value
static int	FALSE
static int	LITERAL
static int	NULL Token Definition
static int	NUMBER
protected static String[]	TOKEN_TYPES
static int	TRUE

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Constructor Summary

Constructors

Constructor and Description
JsonTokenizer(Reader input)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
static String[]	getTokenTypes()
boolean	incrementToken()
void	reset()

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset, setReader

Methods inherited from class org.apache.lucene.analysis.TokenStream

end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

NULL

public static final int NULL

Token Definition

See Also:

Constant Field Values

TRUE

public static final int TRUE

See Also:

Constant Field Values

FALSE

public static final int FALSE

See Also:

Constant Field Values

NUMBER

public static final int NUMBER

See Also:

Constant Field Values

LITERAL

public static final int LITERAL

See Also:

Constant Field Values

DATATYPE_LABEL

public static final String DATATYPE_LABEL

Datatype JSON schema: field for the datatype label

See Also:

Constant Field Values

DATATYPE_VALUES

public static final String DATATYPE_VALUES

Datatype JSON schema: field for the datatype value

See Also:

Constant Field Values

TOKEN_TYPES

protected static String[] TOKEN_TYPES

Constructor Detail

JsonTokenizer

public JsonTokenizer(Reader input)

Method Detail

getTokenTypes

public static String[] getTokenTypes()

incrementToken

public final boolean incrementToken()
                             throws IOException

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

reset

public void reset()
           throws IOException

Overrides:

reset in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.lucene.analysis.Tokenizer

Throws:

IOException