TokenizationResult

TokenizationResult schema from Apache OpenNLP

Machine LearningNatural Language ProcessingNLPText ProcessingApacheOpen SourceJava

Properties

Name	Type	Description
tokens	array	Extracted tokens
spans	array
probabilities	array	Confidence for each token boundary

View JSON Schema on GitHub

JSON Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://raw.githubusercontent.com/api-evangelist/apache-opennlp/refs/heads/main/json-schema/apache-opennlp-tokenization-result-schema.json",
  "title": "TokenizationResult",
  "description": "TokenizationResult schema from Apache OpenNLP",
  "type": "object",
  "properties": {
    "tokens": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Extracted tokens",
      "example": [
        "Pierre",
        "Vinken",
        ",",
        "61",
        "years",
        "old"
      ]
    },
    "spans": {
      "type": "array",
      "items": {
        "$ref": "#/components/schemas/Span"
      }
    },
    "probabilities": {
      "type": "array",
      "items": {
        "type": "number"
      },
      "description": "Confidence for each token boundary"
    }
  }
}