Kafka Replication to Partition

BATCH_IMPORT

All Entry Source that publishes batch data – via a flat file, either CSV, TSC etc, shall publish messages to this topic. Each message shall correspond to one single batch of records.

Message Attributes
  • Entry Source ID
  • Journey ID
  • Data
    • Timestamp – time stamp when the message was added to the topic
    • Entry Source Type
    • FQN
Object Type
  • entrySourceID:
  • journeyID:
  • entrySourceType: enum
  • data: FQN of the file containing the data
  • timestamp:

STREAMING_IMPORT

All Entry Source that publishes streaming data – via a series of individual records, shall publish messages to this topic. Each message shall correspond to one single record.

Message Attributes
  • Entry Source ID (if available)
  • Data (in JSON format)
  • Metadata
    • Timestamp – time stamp when the message was added to the topic
    • Entry Source Type
    • Other Metadata
Object Type
  • entrySourceID: (this is optional)
  • entrySourceType: enum
  • data: (JSON of single record)
  • timestamp:
  • metaData: (JSON of other data)

DATA_MAP

This topic will have messages once the incoming data is cleaned and needs to be mapped to a Data Definition.

Message Attributes
  • Entry Source ID
  • Journey ID
  • Data
    • Timestamp – time stamp when the message was entered in the topic
    • Data, in JSON
    • Number of records, if in batch
Object Type
  • entrySourceID:
  • journeyID
  • entrySourceType: enum
  • timeStamp
  • data: <JSON string>
  • numRecords:

RAW_DATA_FETCH

This topic will have messages to fetch unmapped data, from the DATA ARCHIVE table.

Message Attributes
  • Entry Source ID
  • Timestamp – time stamp when the message was entered in the topic
  • State – state of data that need to be fetched, RAW or CLEANED
  • Number of records – Number of records to be fetched. If not specified, then all records
Object Type
  • entrySourceID:
  • timeStamp
  • state: enum
  • numberRecords:

END_JOURNEY

This topic shall hold all messages for the various running Journeys for which some of the audiences have reached a logical end of the defined Journey.

JOURNEY_EVENTS

This topic shall hold all the messages for the various events during the lifetime of a running Journey instance.

Message Attributes
  • ID – this shall be a combination of the Journey ID and an optional Node ID
  • Data
    • State – the state of the Journey
    • Timestamp
    • Metadata
Object Type
  • ID: journeyID
  • state: enum:
  • timeStamp
  • data:
PUBLISHED
  • ID – Journey ID
  • Data
    • State – PUBLISHED
    • Timestamp - time stamp when the message was entered in the topic
    • Metadata – FQN for JSON file having the complete Journey structure
Object Type
  • ID: journeyID
  • state: enum:PUBLISHED
  • timeStamp
  • data: JSON for Journey structure (this can be the serialized form of the JSON)
PAUSE
  • ID – Journey ID
  • Data
    • Timestamp - time stamp when the message was entered in the topic
    • State – PAUSE
Object Type
  • ID: journeyID
  • state: enum:PAUSE
  • timestamp
  • data: <empty>
JOURNEY_PAUSED
  • ID – Journey ID
  • Data
    • Timestamp - time stamp when the message was entered in the topic
    • State – PAUSED
Object Type
  • ID: journeyID
  • state: enum:PAUSED
  • timestamp
  • data: <empty>
PARSED
  • ID – Journey ID
  • Data
    • Timestamp - time stamp when the message was entered in the topic
    • State – PARSED
Object Type
  • ID: journeyID
  • state: enum:PARSED
  • timestamp
  • data: <empty>
MAPPED
  • ID – Journey ID
  • Data
  • Timestamp - time stamp when the message was entered in the topic
  • State – PARSED
Object Type
  • ID: journeyID
  • state: enum:MAPPED
  • timestamp
  • data: <empty>
FETCHED
  • ID – Journey ID
  • Data
    • Timestamp - time stamp when the message was entered in the topic
    • List of record IDs
Object Type
  • ID: journeyID#nodeID
  • state: enum:FETCHED
  • timestamp
  • numberRecords:
  • listRecords:
DELAY_EXECUTED
  • ID – this ID shall be the Journey ID and the node ID for the following node
  • Data
    • State – DELAY_EXECUTED
    • Timestamp - time stamp when the message was entered in the topic
    • List of record IDs
Object Type
  • ID: journeyID#nodeID
  • state: enum:DELAY_EXECUTED
  • timestamp
  • numberRecords:
  • listRecords:
EMAIL_SENT
  • ID – this ID shall be the Journey ID and the node ID for the following node
  • Data
    • State – EMAIL_SENT
    • Timestamp - time stamp when the message was entered in the topic
    • List of record IDs
Object Type
  • ID: journeyID#nodeID
  • state: enum:EMAIL_SENT
  • timestamp
  • numberRecords:
  • listRecords:

JOURNEY_PARSE

This topic shall hold all messages for journeys to be validated and parsed.

Message Attributes
  • ID – Journey ID
  • Data
    • Timestamp - time stamp when the message was entered in the topic
    • State – PARSE
    • Data
Object Type
  • ID: journeyID
  • state: enum:PARSE
  • timestamp
  • data: <empty>

JOURNEY_CACHE

This topic shall hold all messages for Journey Data that needs to be added to the cache.

Message Attributes
  • Journey ID
  • Entry Source ID
  • Data
    • Timestamp - time stamp when the message was entered in the topic
    • `Data
Object Type
  • ID: journeyID
  • state: enum:PARSE
  • timestamp
  • data: <empty>

ASYNC_DATABASE_INSERTS

This topic shall hold all messages that are to be inserted into some database table asynchronously.

Message Attributes
  • Entity
  • Name
  • Message
  • Timestamp

JOURNEY_MAP

This topic shall hold all messages for the Journey Map Service. This data shall consist of all the details of each Entry Source associated with the Journey.

Message Attributes
  • Journey ID
  • Data
    • List of Entry Sources for the Journey, which includes ID, type and FQN

JOURNEY_DATA

This topic shall hold all messages that are to be pushed to database tables for log or state information. These messages do not impact execution of the Journey.

Message Attributes
  • Service ID
  • Data
    • Query
    • Timestamp

EMAIL_SEND

This topic shall hold all the messages from the running instances of journeys that need emails to be sent out.

Message Attributes
  • ID – this shall be a combination of the Journey ID and Node ID
  • Audience IDs – the list of audience IDs that are to be sent the email

OUTGOING_EMAIL

This topic shall hold all the messages from the Journey application that need to be sent out as emails. These will either be acted upon by a suitable Adaptor Service or the Universal Connector.

Message Attributes
  • sourceAppId: name of application - Journey
  • sourceInstanceId – Touch Point ID
  • Audiences – List of audience data

EMAIL_RESPONSES

This topic shall hold all the messages received from the delivery engine via configured connector as audience response events to emails sent.

Message Attributes
  • Touch Point ID
  • Audience ID
  • Event ID
  • Timestamp

OUTGOING_SMS

This topic shall hold all the messages from the Journey application that need to be sent out as SMS’s. These will either be acted upon by a suitable Adaptor Service or the Universal Connector.

Message Attributes
  • sourceAppId: name of application - Journey
  • sourceInstanceId – Touch Point ID
  • Audiences – List of audience data

SMS_RESPONSES

This topic shall hold all the messages received from the delivery engine via configured connector as audience response events to SMS’s sent.

Message Attributes
  • Touch Point ID
  • Audience ID
  • Event ID
  • Timestamp

DELAY

This topic shall hold all the messages from the running instances of journeys that need to add a delay before the execution of the following touch point.

Message Attributes
  • ID – this shall be a combination of the Journey ID and Node ID
  • Configuration – this shall be the configuration data for the Delay Touch Point
  • Structure – this shall be the JSON structure extracted from the Journey JSON

DECISION_SPLIT

This topic shall hold all the messages from the running instances of journeys that need to split incoming data based on some filter.

Message Attributes
  • ID – this shall be a combination of the Journey ID and Node ID
  • Configuration – this shall be the configuration data for the Decision Split Touch Point
  • Structure – this shall be the JSON structure extracted from the Journey JSON

JOURNEY_PAUSE

This topic shall hold messages from the Orchestration Service to pause a specific Journey.

Message Attributes
  • ID – the ID of the Journey which must be Paused

DATA_PAUSED

This topic shall hold all the messages from the running instances of Journeys that have been Paused.

Message Attributes
  • Name – name of the Source topic
  • ID – the Journey ID
  • Message – the original message

JOURNEY_RESUME

This topic shall hold messages from the Orchestration Service to resume a specific Journey.

Message Attributes
  • ID – the ID of the Journey which must be Resumed

DATA_RESUMED

This topic shall hold all the messages that were in the Paused state but now need to be resumed by the individual Services.

Message Attributes
  • Name – name of the Source topic
  • ID – the Journey ID
  • Message – the original message

JOURNEY_ENGINE_ERRORS

This topic shall hold all the messages from the engine which are errors that took place and that could be displayed on the Web on an administrative console when required.

Message Attributes
  • ID – Some ID that identifies the entity that caused the error – Journey ID, Audience ID
  • State – State of the error – MEDIUM / HIGH / CRITICAL
  • Category – Name of the Service that detected the error
  • Text – The actual error text

JOURNEY_GOAL_VERIFICATION

This topic shall hold all messages for the Journey Goal Verification Service which needs to verify if the goal for the Journey has been achieved.

Message Attributes
  • ID – Journey ID
  • EVENT_TYPE – EMAIL RESPONSE, SMS_REPOSNE, PUSH_RESPONSE
  • Data
  • Timestamp

JOURNEY_GOAL_ACHIEVED

This topic shall hold messages for the Service that handles notification for when a Journey has achieved the set goal.

Message Attributes
  • ID – Journey ID
  • EVENT_TYPE – TIME, COUNT
  • Timestamp

JOURNEY_ENGINE_MONITORING

This topic shall be used by the Engine to send monitoring data across to the web.

Message Attributes
  • Name – Name of Service is message refers to a Service or then ENGINE indicating that the message is for Journey Engine as a whole
  • State – STARTING, GOING DOWN, NOT RESPONDING
  • Thread ID
  • Timestamp
  • IP Address

States, Types & Status

  1. ENTRY SOURCE TYPES
    • Flat File
    • REST
  2. DATA PIPELINE STATUS
    • Imported
    • Cleaned
    • Mapped
  3. LOG TYPES
    • File
  4. LOG SUB TYPES
  5. DATA PIPELINE STATES
  6. JOURNEY STATES
    • PUBLISHED - when the Journey is pushed into the Published state from the front end
    • PARSED – when the Journey is parsed successfully
    • RUNNING – when the Journey has mapped data and has begun execution
    • PAUSED – when the Journey has been Paused, either explicitly or implicitly. The current Run of the Journey shall be completed
    • STOPPRED – when the Journey has been Stopped, either explicitly or implicitly. The current Run of the Journey shall be terminated
    • COMPLETED – when the Journey has completed its logical execution
    • GOAL ACHIEVED – when the Journey has achieved its defined goal
  7. AUDIENCE TABLE STATES
    • NEW – these are new records for a given Journey ID, that have not been processed
    • PROCESSED – these are records for a given Journey ID, that have been processed
    • COMPLETED – these audience for a given Journey have completed the Journey – the logical end
    • GOAL ACHEIVED – the Journey associated with these audience has achieved its goal