Articles about data engineering | UserComp.com

ClickHouse Deduplication using Idempotency id Column

Learn how to deduplicate data in ClickHouse using an idempotency id column. Utilize the unique field to ingest rows and prevent duplicate data, making your ClickHouse database more efficient and organized.

Managing Rapidly Growing Apache Iceberg Metadata on AWS

As a developer with a decade of experience, I've recently started working in Data Engineering and set up a few Iceberg tables on AWS Glue and S3. However, I'm facing issues with the exponential growth of Apache Iceberg metadata. In this article, we'll discuss how to manage rapidly growing Iceberg metadata on AWS.