Skip to main content

JSON Schemas

Incubating

Important Capabilities

CapabilityStatusNotes
DescriptionsExtracts descriptions at top level and field level
Detect Deleted EntitiesWith stateful ingestion enabled, will remove entities from DataHub if they are no longer present in the source
Extract OwnershipDoes not currently support extracting ownership
Extract TagsDoes not currently support extracting tags
Platform InstanceSupports platform instance via config
Schema MetadataExtracts schemas, following references

This source extracts metadata from a single JSON Schema or multiple JSON Schemas rooted at a particular path. It performs reference resolution based on the $ref keyword.

Metadata mapping:

  • Schemas are mapped to Datasets with sub-type Schema
  • The name of the Schema (Dataset) is inferred from the $id property and if that is missing, the file name.
  • Browse paths are minted based on the path

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[json-schema]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

pipeline_name: json_schema_ingestion
source:
type: json-schema
config:
path: <path_to_json_file_or_directory or url> # e.g. https://json.schemastore.org/petstore-v1.0.json
platform: <choose a platform that you want schemas to live under> # e.g. schemaregistry
# platform_instance: <add a platform_instance if there are multiple schema repositories>
stateful_ingestion:
enabled: true # recommended to have this turned on

# sink configs if needed

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
path 
One of string(file-path), string(directory-path), string(uri)
Set this to a single file-path or a directory-path (for recursive traversal) or a remote url. e.g. https://json.schemastore.org/petstore-v1.0.json
platform 
string
Set this to a platform that you want all schemas to live under. e.g. schemaregistry / schemarepo etc.
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
use_id_as_base_uri
boolean
When enabled, uses the $id field in the json schema as the base uri for following references.
Default: False
env
string
The environment that all assets produced by this connector belong to
Default: PROD
uri_replace_pattern
URIReplacePattern
Use this if URI-s need to be modified during reference resolution. Simple string match - replace capabilities are supported.
uri_replace_pattern.match 
string
Pattern to match on uri-s as part of reference resolution. See replace field
uri_replace_pattern.replace 
string
Pattern to replace with as part of reference resolution. See match field
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
The type of the ingestion state provider registered with datahub.
Default: False
stateful_ingestion.ignore_new_state
boolean
If set to True, ignores the current checkpoint state.
Default: False
stateful_ingestion.ignore_old_state
boolean
If set to True, ignores the previous checkpoint state.
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Configuration Notes

  • You must provide a platform field. Most organizations have custom project names for their schema repositories, so you can pick whatever name makes sense. For example, you might want to call your schema platform schemaregistry. After picking a custom platform, you can use the put platform command to register your custom platform into DataHub.

Code Coordinates

  • Class Name: datahub.ingestion.source.schema.json_schema.JsonSchemaSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for JSON Schemas, feel free to ping us on our Slack.