Kafka Replication to Partition
BATCH_IMPORT
All Entry Source that publishes batch data – via a flat file, either CSV, TSC etc, shall publish messages to this topic. Each message shall correspond to one single batch of records.
- Entry Source ID
- Journey ID
- Data
- Timestamp – time stamp when the message was added to the topic
- Entry Source Type
- FQN
- entrySourceID:
- journeyID:
- entrySourceType: enum
- data: FQN of the file containing the data
- timestamp:
STREAMING_IMPORT
All Entry Source that publishes streaming data – via a series of individual records, shall publish messages to this topic. Each message shall correspond to one single record.
- Entry Source ID (if available)
- Data (in JSON format)
- Metadata
- Timestamp – time stamp when the message was added to the topic
- Entry Source Type
- Other Metadata
- entrySourceID: (this is optional)
- entrySourceType: enum
- data: (JSON of single record)
- timestamp:
- metaData: (JSON of other data)
DATA_MAP
This topic will have messages once the incoming data is cleaned and needs to be mapped to a Data Definition.
- Entry Source ID
- Journey ID
- Data
- Timestamp – time stamp when the message was entered in the topic
- Data, in JSON
- Number of records, if in batch
- entrySourceID:
- journeyID
- entrySourceType: enum
- timeStamp
- data: <JSON string>
- numRecords:
RAW_DATA_FETCH
This topic will have messages to fetch unmapped data, from the DATA ARCHIVE table.
- Entry Source ID
- Timestamp – time stamp when the message was entered in the topic
- State – state of data that need to be fetched, RAW or CLEANED
- Number of records – Number of records to be fetched. If not specified, then all records
- entrySourceID:
- timeStamp
- state: enum
- numberRecords:
END_JOURNEY
This topic shall hold all messages for the various running Journeys for which some of the audiences have reached a logical end of the defined Journey.
JOURNEY_EVENTS
This topic shall hold all the messages for the various events during the lifetime of a running Journey instance.
- ID – this shall be a combination of the Journey ID and an optional Node ID
- Data
- State – the state of the Journey
- Timestamp
- Metadata
- ID: journeyID
- state: enum:
- timeStamp
- data:
- ID – Journey ID
- Data
- State – PUBLISHED
- Timestamp - time stamp when the message was entered in the topic
- Metadata – FQN for JSON file having the complete Journey structure
- ID: journeyID
- state: enum:PUBLISHED
- timeStamp
- data: JSON for Journey structure (this can be the serialized form of the JSON)
- ID – Journey ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- State – PAUSE
- ID: journeyID
- state: enum:PAUSE
- timestamp
- data: <empty>
- ID – Journey ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- State – PAUSED
- ID: journeyID
- state: enum:PAUSED
- timestamp
- data: <empty>
- ID – Journey ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- State – PARSED
- ID: journeyID
- state: enum:PARSED
- timestamp
- data: <empty>
- ID – Journey ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- State – PARSED
- ID: journeyID
- state: enum:MAPPED
- timestamp
- data: <empty>
- ID – Journey ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- List of record IDs
- ID: journeyID#nodeID
- state: enum:FETCHED
- timestamp
- numberRecords:
- listRecords:
- ID – this ID shall be the Journey ID and the node ID for the following node
- Data
- State – DELAY_EXECUTED
- Timestamp - time stamp when the message was entered in the topic
- List of record IDs
- ID: journeyID#nodeID
- state: enum:DELAY_EXECUTED
- timestamp
- numberRecords:
- listRecords:
- ID – this ID shall be the Journey ID and the node ID for the following node
- Data
- State – EMAIL_SENT
- Timestamp - time stamp when the message was entered in the topic
- List of record IDs
- ID: journeyID#nodeID
- state: enum:EMAIL_SENT
- timestamp
- numberRecords:
- listRecords:
JOURNEY_PARSE
This topic shall hold all messages for journeys to be validated and parsed.
- ID – Journey ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- State – PARSE
- Data
- ID: journeyID
- state: enum:PARSE
- timestamp
- data: <empty>
JOURNEY_CACHE
This topic shall hold all messages for Journey Data that needs to be added to the cache.
- Journey ID
- Entry Source ID
- Data
- Timestamp - time stamp when the message was entered in the topic
- `Data
- ID: journeyID
- state: enum:PARSE
- timestamp
- data: <empty>
ASYNC_DATABASE_INSERTS
This topic shall hold all messages that are to be inserted into some database table asynchronously.
- Entity
- Name
- Message
- Timestamp
JOURNEY_MAP
This topic shall hold all messages for the Journey Map Service. This data shall consist of all the details of each Entry Source associated with the Journey.
- Journey ID
- Data
- List of Entry Sources for the Journey, which includes ID, type and FQN
JOURNEY_DATA
This topic shall hold all messages that are to be pushed to database tables for log or state information. These messages do not impact execution of the Journey.
- Service ID
- Data
- Query
- Timestamp
EMAIL_SEND
This topic shall hold all the messages from the running instances of journeys that need emails to be sent out.
- ID – this shall be a combination of the Journey ID and Node ID
- Audience IDs – the list of audience IDs that are to be sent the email
OUTGOING_EMAIL
This topic shall hold all the messages from the Journey application that need to be sent out as emails. These will either be acted upon by a suitable Adaptor Service or the Universal Connector.
- sourceAppId: name of application - Journey
- sourceInstanceId – Touch Point ID
- Audiences – List of audience data
EMAIL_RESPONSES
This topic shall hold all the messages received from the delivery engine via configured connector as audience response events to emails sent.
- Touch Point ID
- Audience ID
- Event ID
- Timestamp
OUTGOING_SMS
This topic shall hold all the messages from the Journey application that need to be sent out as SMS’s. These will either be acted upon by a suitable Adaptor Service or the Universal Connector.
- sourceAppId: name of application - Journey
- sourceInstanceId – Touch Point ID
- Audiences – List of audience data
SMS_RESPONSES
This topic shall hold all the messages received from the delivery engine via configured connector as audience response events to SMS’s sent.
- Touch Point ID
- Audience ID
- Event ID
- Timestamp
DELAY
This topic shall hold all the messages from the running instances of journeys that need to add a delay before the execution of the following touch point.
- ID – this shall be a combination of the Journey ID and Node ID
- Configuration – this shall be the configuration data for the Delay Touch Point
- Structure – this shall be the JSON structure extracted from the Journey JSON
DECISION_SPLIT
This topic shall hold all the messages from the running instances of journeys that need to split incoming data based on some filter.
- ID – this shall be a combination of the Journey ID and Node ID
- Configuration – this shall be the configuration data for the Decision Split Touch Point
- Structure – this shall be the JSON structure extracted from the Journey JSON
JOURNEY_PAUSE
This topic shall hold messages from the Orchestration Service to pause a specific Journey.
- ID – the ID of the Journey which must be Paused
DATA_PAUSED
This topic shall hold all the messages from the running instances of Journeys that have been Paused.
- Name – name of the Source topic
- ID – the Journey ID
- Message – the original message
JOURNEY_RESUME
This topic shall hold messages from the Orchestration Service to resume a specific Journey.
- ID – the ID of the Journey which must be Resumed
DATA_RESUMED
This topic shall hold all the messages that were in the Paused state but now need to be resumed by the individual Services.
- Name – name of the Source topic
- ID – the Journey ID
- Message – the original message
JOURNEY_ENGINE_ERRORS
This topic shall hold all the messages from the engine which are errors that took place and that could be displayed on the Web on an administrative console when required.
- ID – Some ID that identifies the entity that caused the error – Journey ID, Audience ID
- State – State of the error – MEDIUM / HIGH / CRITICAL
- Category – Name of the Service that detected the error
- Text – The actual error text
JOURNEY_GOAL_VERIFICATION
This topic shall hold all messages for the Journey Goal Verification Service which needs to verify if the goal for the Journey has been achieved.
- ID – Journey ID
- EVENT_TYPE – EMAIL RESPONSE, SMS_REPOSNE, PUSH_RESPONSE
- Data
- Timestamp
JOURNEY_GOAL_ACHIEVED
This topic shall hold messages for the Service that handles notification for when a Journey has achieved the set goal.
- ID – Journey ID
- EVENT_TYPE – TIME, COUNT
- Timestamp
JOURNEY_ENGINE_MONITORING
This topic shall be used by the Engine to send monitoring data across to the web.
- Name – Name of Service is message refers to a Service or then ENGINE indicating that the message is for Journey Engine as a whole
- State – STARTING, GOING DOWN, NOT RESPONDING
- Thread ID
- Timestamp
- IP Address
States, Types & Status
- ENTRY SOURCE TYPES
- Flat File
- REST
- DATA PIPELINE STATUS
- Imported
- Cleaned
- Mapped
- LOG TYPES
- File
- LOG SUB TYPES
- DATA PIPELINE STATES
- JOURNEY STATES
- PUBLISHED - when the Journey is pushed into the Published state from the front end
- PARSED – when the Journey is parsed successfully
- RUNNING – when the Journey has mapped data and has begun execution
- PAUSED – when the Journey has been Paused, either explicitly or implicitly. The current Run of the Journey shall be completed
- STOPPRED – when the Journey has been Stopped, either explicitly or implicitly. The current Run of the Journey shall be terminated
- COMPLETED – when the Journey has completed its logical execution
- GOAL ACHIEVED – when the Journey has achieved its defined goal
- AUDIENCE TABLE STATES
- NEW – these are new records for a given Journey ID, that have not been processed
- PROCESSED – these are records for a given Journey ID, that have been processed
- COMPLETED – these audience for a given Journey have completed the Journey – the logical end
- GOAL ACHEIVED – the Journey associated with these audience has achieved its goal