Data Ingestion In Plain Java

Building Data Ingestion Logic using Plain Java

Originally Published in https://asyncq.com/

Introduction

• In this article, we will build simple data ingestion design that will be using simple Java SDK
• We will Read data from remote sources and write to Database.

Use Case

• Our use cases is to ingest News Articles from RSS Sources.
• We will build RSS Client that will ingest data and then write to RDBMS.

Source

• As i mentioned our source is RSS Feed. We will be ingesting news article from Google cloud tech blogs.
• If you want to see detailed blog about building RSS clients please visit here
• In the scope of this blog we will use dependency called Rome, which will help us connecting to RSS feed.
• Below is complete logic which ingests data from google cloud blog and map it to NewsArticle POJO

NewsArticle POJO

• NewsArticle basically consists of properties that we are…