Sling
Slingdata.ioBlogGithubHelp!
  • Introduction
  • Sling CLI
    • Installation
    • Environment
    • Running Sling
    • Global Variables
    • CLI Pro
  • Sling Platform
    • Sling Platform
      • Architecture
      • Agents
      • Connections
      • Editor
      • API
      • Deploy from CLI
  • Concepts
    • Replications
      • Structure
      • Modes
      • Source Options
      • Target Options
      • Columns
      • Transforms
      • Runtime Variables
      • Tags & Wildcards
    • Hooks / Steps
      • Check
      • Command
      • Copy
      • Delete
      • Group
      • Http
      • Inspect
      • List
      • Log
      • Query
      • Replication
      • Store
      • Read
      • Write
    • Pipelines
    • Data Quality
      • Constraints
  • Examples
    • File to Database
      • Custom SQL
      • Incremental
    • Database to Database
      • Custom SQL
      • Incremental
      • Backfill
    • Database to File
      • Incremental
    • Sling + Python 🚀
  • Connections
    • Database Connections
      • Athena
      • BigTable
      • BigQuery
      • Cloudflare D1
      • Clickhouse
      • DuckDB
      • DuckLake
      • MotherDuck
      • MariaDB
      • MongoDB
      • Elasticsearch
      • MySQL
      • Oracle
      • Postgres
      • Prometheus
      • Proton
      • Redshift
      • StarRocks
      • SQLite
      • SQL Server
      • Snowflake
      • Trino
    • Storage Connections
      • AWS S3
      • Azure Storage
      • Backblaze B2
      • Cloudflare R2
      • DigitalOcean Spaces
      • FTP
      • Google Drive
      • Google Storage
      • Local Storage
      • Min.IO
      • SFTP
      • Wasabi
Powered by GitBook
On this page
  • Configuration
  • Properties
  • Output
  • Examples
  • Print out Runtime State
  • Conditional Warning Log
  • Debug Information
  • Performance Metrics Logging
  • Error Context Logging
  • Audit Trail Logging
  • Data Quality Logging
  • 8. Environment-Specific Logging (Pre-Hook)
  • 9. Resource Usage Logging (Post-Hook)
  • 10. Milestone Logging (Post-Hook)
  1. Concepts
  2. Hooks / Steps

Log

Log hooks allow you to output custom messages during the replication process. This is useful for debugging, monitoring, and creating audit trails of your data pipeline operations.

Configuration

- type: log
  message: "Custom log message"  # Required: The message to log
  level: info                    # Optional: Log level (info/warn/debug)
  on_failure: abort              # Optional: abort/warn/quiet/skip

Properties

Property
Required
Description

message

Yes

The message to log

level

No

Log level (info/warn/debug). Defaults to info

on_failure

No

What to do if logging fails (abort/warn/quiet/skip)

Output

When the log hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
level: "info"    # The log level used
message: "Custom log message"  # The message that was logged

You can access these values in subsequent hooks using the following syntax (jmespath):

  • {state.hook_id.status} - Status of the hook execution

  • {state.hook_id.level} - The log level used

  • {state.hook_id.message} - The message that was logged

Examples

Print out Runtime State

Log the basic status of a stream after completion:

hooks:
  post:
    - type: log
      message: "runtime_state => {runtime_state}"

    - type: log
      message: "Stream {run.stream.name} completed with status {run.status}. Processed {run.total_rows} rows."

Conditional Warning Log

Log a warning when row count is below threshold:

hooks:
  post:
    - type: log
      if: "run.total_rows < 1000"
      message: "⚠️ Warning: Low record count for {run.stream.name}. Only processed {run.total_rows} rows."
      level: warn

Debug Information

Log detailed information before starting the replication:

hooks:
  pre:
    - type: log
      message: |
        Starting replication for:
        Stream: {run.stream.name}
        Source Schema: {run.stream.schema}
        Target Table: {run.object.full_name}
        Environment: {target.environment}
        Timestamp: {timestamp.datetime}
      level: debug

Performance Metrics Logging

Log performance metrics after successful completion:

hooks:
  post:
    - type: log
      if: run.status == "success"
      message: |
        Performance Metrics for {run.stream.name}:
        - Total Rows: {run.total_rows}
        - Total Bytes: {run.total_bytes}
        - Duration: {run.duration} seconds
        - Processing Rate: {run.total_rows / (run.duration)} rows/second
      level: info

Error Context Logging

Log detailed context when errors occur:

hooks:
  post:
    - type: log
      if: run.status == "error"
      message: |
        ❌ Error in stream {run.stream.name}:
        - Source: {source.type}
        - Target: {target.type}
        - Object: {run.object.full_name}
        - Environment: {target.environment}
        - Start Time: {run.start_time}
        - End Time: {run.end_time}
        Please check the logs for more details.
      level: warn

Audit Trail Logging

Create a detailed audit trail of replication activities:

hooks:
  pre:
    - type: log
      message: |
        🚀 Starting replication task:
        - Stream: {run.stream.name}
        - Mode: {run.mode}
        - Environment: {target.environment}
        - User: {source.user}
        - Start Time: {timestamp.datetime}
      level: info
  post:
    - type: log
      message: |
        📋 Replication task completed:
        - Stream: {run.stream.name}
        - Status: {run.status}
        - Rows Processed: {run.total_rows}
        - Duration: {run.end_time - run.start_time} seconds
        - End Time: {timestamp.datetime}
      level: info

Data Quality Logging

Log data quality metrics after processing:

hooks:
  post:
    - type: log
      message: |
        Data Quality Report for {run.stream.name}:
        - Total Records: {run.total_rows}
        - Null Rate: {state.quality_check.result.null_rate}%
        - Duplicate Rate: {state.quality_check.result.duplicate_rate}%
        - Invalid Format Rate: {state.quality_check.result.invalid_rate}%
      level: info

8. Environment-Specific Logging (Pre-Hook)

Adjust log verbosity based on environment:

hooks:
  pre:
    - type: log
      if: "target.environment == 'production'"
      message: "⚠️ Running production replication for {run.stream.name}"
      level: warn
    - type: log
      if: "target.environment != 'production'"
      message: "Starting {target.environment} replication for {run.stream.name}"
      level: debug

9. Resource Usage Logging (Post-Hook)

Log resource usage statistics:

hooks:
  post:
    - type: log
      message: |
        Resource Usage for {run.stream.name}:
        - Memory Used: {state.resource_check.result.memory_used}
        - CPU Usage: {state.resource_check.result.cpu_usage}%
        - Disk IO: {state.resource_check.result.disk_io_bytes} bytes
      level: debug

10. Milestone Logging (Post-Hook)

Log important milestones during processing:

hooks:
  post:
    - type: log
      if: "run.total_rows >= 1000000"
      message: "🏆 Milestone: Processed over 1 million records in {run.stream.name}"
      level: info
    - type: log
      if: "run.total_bytes >= 1073741824"  # 1GB
      message: "🏆 Milestone: Processed over 1GB of data in {run.stream.name}"
      level: info
PreviousListNextQuery

Last updated 3 months ago