Write hooks allow you to write content to files in any file-based storage connection. This is particularly useful for creating reports, saving processed data, generating configuration files, or writing logs.
Configuration
- type: write
to: "connection/path/to/file.txt" # Required: Destination Location
content: "text content to write" # Required: Content to write
on_failure: abort # Optional: abort/warn/quiet/skip
id: my_id # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
Properties
Property
Required
Description
to
Yes
content
Yes
The content to write to the file. Supports variable substitution. Can also use file://path/to/file to read content from a local file.
on_failure
No
What to do if the write fails (abort/warn/quiet/skip)
Output
When the write hook executes successfully, it returns the following output that can be accessed in subsequent hooks:
status: success # Status of the hook execution
target_url: "s3://bucket/path/to/file.txt" # The normalized URI of the target file
bytes_written: 1024 # Number of bytes written
You can access these values in subsequent hooks using the following syntax (jmespath):
{state.hook_id.status} - Status of the hook execution
{state.hook_id.target_url} - The normalized URI of the target file
{state.hook_id.bytes_written} - Number of bytes written
Examples
Generate Data Processing Report
Create a summary report after data processing:
hooks:
post:
- type: write
to: "s3/reports/processing_summary_{timestamp.YYYY-MM-DD}.txt"
content: |
Data Processing Summary
======================
Stream: {run.stream.name}
Start Time: {run.start_time}
End Time: {run.end_time}
Total Rows: {run.total_rows}
Status: {run.status}
Target: {target.connection}/{target.object}
Total Bytes Written: {run.total_bytes}
Generated on: {timestamp.YYYY-MM-DD HH:mm:ss}
